For most of the history of machine learning, explainability was a research topic: interesting, occasionally useful, but not required for deployment. That era is over.

The EU AI Act, in force since 2024 with staggered compliance deadlines through 2027, mandates transparency and human oversight for high-risk AI systems. GDPR Article 22 requires that automated decisions with significant impacts be explainable to the affected individual. The FCA in the UK and the EBA across the EU have published guidance requiring explainable credit models. FDA guidance for AI-based medical devices requires documented model performance and decision logic.

Explainability is not a feature. It is the price of deployment in regulated environments.

The regulatory landscape for AI explainability converged faster than most organisations expected. Three frameworks are driving adoption:

EU AI Act (2024–2027 rollout) High-risk AI systems - defined by the regulation to include AI used in hiring, credit, insurance, healthcare, education, and law enforcement - must be transparent, provide human oversight capability, and maintain detailed documentation of training data, model performance, and decision logic. Providers of high-risk AI must make available "sufficiently detailed" explanations for individual decisions.

GDPR Article 22 Automated decision-making that produces "legal or similarly significant" effects on individuals requires the controller to provide "meaningful information about the logic involved." The standard interpretation: for credit scoring, hiring tools, and insurance pricing, the individual has the right to an explanation of why they received the outcome they did. A black-box model cannot satisfy this requirement.

Financial sector guidance The EBA (European Banking Authority) guidance on internal governance explicitly addresses model risk management, requiring that credit models be interpretable and that their key drivers be explainable to non-technical stakeholders. The FCA has issued similar expectations for UK firms.

The practical implication: if you deploy a tree-based ensemble or a neural network to make decisions in any of these domains without an explainability layer, you are deploying a system that your compliance team cannot approve, your affected individuals can challenge, and your regulators can sanction.

SHAP (SHapley Additive exPlanations) is the dominant method for explaining ML model predictions in production. It is grounded in cooperative game theory - specifically, the Shapley value, which provides a mathematically principled way to attribute outcomes to individual players in a cooperative game.

Applied to ML: each prediction is the "game," each feature is a "player," and the SHAP value for each feature is its fair contribution to the final prediction.

The key properties that make SHAP the right choice for regulatory contexts:

Consistency: If a model relies more on a feature than another model, the SHAP values reflect that - they are monotone with respect to feature importance.

Local accuracy: The SHAP values for a prediction sum to the difference between the prediction and the expected prediction. The explanation is mathematically exact, not an approximation.

Model-agnosticity: SHAP works with gradient boosting, random forests, neural networks, and linear models. The same explanation framework applies regardless of model architecture.

Explainability operates at two levels, each serving a different purpose:

Global explanations characterise model behaviour across the full dataset: which features matter most in aggregate? A global SHAP summary plot shows the mean absolute SHAP value for each feature, ranked from most to least influential. This tells you what the model has learned - which signals drive predictions across all cases.

Global explanations serve:

• Model validation: do the top features make domain sense?
• Regulatory documentation: which variables drive decisions and how?
• Feature selection: which inputs are not contributing and can be removed?

Local explanations characterise a single prediction: why did the model produce this specific output for this specific input? A SHAP waterfall plot shows the contribution of each feature to a single prediction - starting from the baseline and working through each feature's contribution until reaching the final score.

Local explanations serve:

• Individual rights (GDPR Article 22): "why was my loan declined?"
• Operational decisions: "what drove this fraud flag?"
• Appeals and disputes: "which factors would I need to change to get a different outcome?"

An explanation without a record of what decision it was attached to is not auditable. Regulatory compliance requires that explanations be stored alongside the decisions they explain - permanently, tamper-evidently, and in a format that can be queried by auditors.

The complete audit record for an AI decision:

When an individual exercises their Article 22 right to explanation, the compliance team can retrieve this record and provide the human_readable explanation along with the specific feature values - without touching the production system or re-running the model.

Not all models are equally explainable:

Gradient boosting (XGBoost, LightGBM, CatBoost): Highly explainable with TreeSHAP. Fast, exact computation. The best choice for tabular data where explainability is required. Aicuflow's default for classification and regression tasks.

Random forests: Also supported by TreeSHAP. Slightly less accurate than gradient boosting in most benchmarks but widely trusted in regulated industries due to their interpretability reputation.

Neural networks: Explainable with KernelSHAP or GradientSHAP, but the computational cost is higher and the explanations are approximate. For high-stakes regulated decisions, tree-based models are generally preferred.

Linear models: Inherently interpretable - the coefficients are the feature importances. SHAP can still be used for consistency, but the model itself is already explanatory.

LLMs / RAG systems: Explaining why a language model produced a specific response is an open research problem. Faithfulness metrics (does the response stay within the retrieved context?) are more tractable than attribution to specific features. For regulatory purposes, the explainability layer for LLM-based systems is primarily the audit of what content was retrieved.

Explainability cannot be added after the model is deployed - it must be designed in from the start. The production pattern:

1. Choose an explainable model type where regulatory requirements apply. Gradient boosting with TreeSHAP for tabular data is the most practical choice.

2. Train the explainer alongside the model. In Aicuflow, SHAP is computed automatically after every training run - you do not need to configure it separately.

3. Include explanations in every inference response. The deployed API returns predictions and SHAP contributions in the same response. Every downstream system receives both.

4. Log every decision with its explanation. The audit log record includes the full SHAP breakdown, the model version, and the input hash. Append-only storage, exportable to your SIEM.

5. Build the explanation UI for business users. Decision-makers need to see explanations in natural language, not JSON. Aicuflow generates both the raw SHAP values and a natural-language summary of the top contributors.