From Black-Box to Glass-Box: Explainable AI for Applied Intelligent Software Systems Across the Software Development Life Cycle

Abstract

Artificial intelligence (AI), particularly large language models (LLMs), is increasingly embedded across the Software Development Life Cycle (SDLC), yet the opacity of these systems limits trust, verification, and certification in software engineering (SE), especially in safety- and regulation-critical domains. This paper presents a systematic mapping study (SMS) of explainable AI (XAI) in SE, spanning classical machine learning, deep learning, and LLM-based tools across all SDLC phases. Using an SDLC- and persona-aware taxonomy, we synthesize evidence on explanation techniques, evaluation practices, and deployment contexts. Our analysis reveals a pronounced imbalance: XAI research focuses on implementation and testing, while requirements, design, maintenance, and operations remain underexplored. Faithfulness validation and human-centred evaluation are limited, and LLM-based explanations introduce additional risks related to over-trust and pipeline opacity. We conclude with a research agenda centered on faithfulness-by-design, persona-aware explanations, and certification-ready XAI pipelines, outlining a path from black-box to glass-box SE in high-stakes intelligent systems.