Explainable AI for Regulated Industries: Making Black-Box Models Compliant
When a Saudi bank deploys an AI-powered loan approval system and a regulator from the Saudi Central Bank (SAMA) asks why a specific applicant was rejected, the question is deceptively simple. The data science team can point to a model that performs well in aggregate, but the individual decision — the one that affected a real person's financial life — may be buried inside millions of neural network parameters that no one can meaningfully interpret. That gap between performance and explanation is where regulated industries in the Kingdom now find themselves.
The tension is not incidental. The most powerful AI architectures — deep neural networks, large ensemble models, transformer-based systems — tend to be the least transparent about their reasoning. Simpler models sacrifice some predictive power in exchange for interpretability. For most commercial applications, this trade-off is a design preference. For organizations operating under Saudi Arabia's regulatory frameworks, it is increasingly a compliance matter.
SAMA's AI guidelines place algorithmic transparency at the center of responsible deployment in financial services. The Saudi Data and AI Authority (SDAIA) calls for interpretable decision-making across high-impact domains. The National Cybersecurity Authority (NCA) requires auditability for AI systems embedded in critical infrastructure. The Personal Data Protection Law (PDPL) grants individuals meaningful rights with respect to automated decisions that affect them. Together, these instruments form a governance landscape that cannot be satisfied by pointing to a model's overall accuracy alone. When the model decides who receives a loan, who receives a diagnosis, or who is flagged for a security review, the reasoning behind that decision carries legal, ethical, and operational weight.
This lab note examines how organizations in KSA's finance and healthcare sectors can approach explainability not as a retrofit or a compliance checkbox, but as an architectural and governance discipline that shapes AI systems from the beginning.
The Regulatory Case for Explainability
Saudi Arabia's approach to AI governance did not emerge in isolation. SDAIA's frameworks draw on the international consensus that high-risk AI applications — those affecting credit, employment, healthcare, and public safety — require transparency as a precondition for legitimate deployment. The EU's AI Act, which classifies credit scoring and medical diagnostics as high-risk uses mandating appropriate explainability, reflects the same principle that SAMA and SDAIA have embedded into KSA-specific guidance.
What distinguishes the Saudi context is how explainability requirements connect to Vision 2030's governance principles. The push for financial inclusion, digital health expansion, and smart infrastructure all depend on public trust in the AI systems that will underpin those services. Without the ability to explain decisions, that trust is difficult to establish and nearly impossible to defend when things go wrong.
The PDPL, administered by SDAIA, is particularly significant for automated decision-making. It creates obligations around transparency that go beyond simply disclosing that an AI system exists. Individuals subject to consequential automated decisions have rights that require organizations to be able to surface meaningful, human-readable reasoning — not statistical abstractions. For financial institutions, this intersects with SAMA's requirements for fair and transparent treatment of customers. For healthcare providers, it interacts with the Saudi Food and Drug Authority's (SFDA) evolving standards for AI-based clinical decision support tools.
The NCA's Essential Cybersecurity Controls add another dimension: AI systems embedded in critical infrastructure must be auditable. This means not only logging inputs and outputs, but maintaining the ability to reconstruct and examine the reasoning behind specific decisions after the fact. In sectors like healthcare and energy-adjacent finance, where AI outcomes can have physical and human consequences, this auditability requirement is among the most technically demanding aspects of compliance.
The Explainability Spectrum: From Glass Box to Black Box
Explainability is not a binary property. It exists on a spectrum, and understanding where different model architectures fall on that spectrum is foundational to making sensible design decisions.
At one end sit what practitioners call glass box models — linear regression, logistic regression, decision trees, generalized additive models. These architectures offer complete transparency. Every parameter has a clear semantic meaning, and any decision can be traced through the model's logic step by step. For use cases where the underlying relationships are not extraordinarily complex and the input features are well-understood, glass box models can deliver both regulatory compliance and sufficient predictive performance. A credit scoring system built on logistic regression, for instance, can explain exactly which factors — payment history, credit utilization, length of credit history — drove a particular score, and by how much each contributed.
Moving along the spectrum, interpretable models occupy a middle ground. Rule-based systems, decision lists, monotonic gradient boosting machines, and similar architectures can handle more complex patterns while maintaining human-readable logic. The monotonicity constraint is worth highlighting in the KSA financial context: for credit models, it is both a regulatory expectation and a fairness guarantee that increasing a customer's income should never decrease their creditworthiness. Interpretable models can enforce such constraints directly, satisfying both explainability requirements and basic intuitions about how credit decisions should work.
At the far end of the spectrum are black box models — deep neural networks, large random forests, complex ensembles — that extract their predictive power from non-linear interactions too intricate for humans to trace directly. These architectures often outperform their interpretable counterparts on complex tasks, particularly when input data is unstructured (medical images, free text, sensor streams). For many regulated use cases, this performance may be necessary. A deep learning system for diagnostic imaging in a Saudi hospital may genuinely detect patterns that a simpler model cannot.
Post-hoc explainability techniques bridge the gap between black box performance and accountability requirements. LIME (Local Interpretable Model-agnostic Explanations) and SHAP (SHapley Additive Explanations) generate explanations for individual predictions by examining how changes to input features affect model outputs. Counterfactual explanations answer a different but often more useful question: what would need to change for this decision to have been different? For a loan applicant who was declined, a counterfactual might indicate that a modest reduction in existing debt obligations, combined with a longer credit history, would have changed the outcome — information that is both explainable and actionable.
Hybrid architectures represent a fourth approach: systems where an interpretable core model handles the majority of cases while a more complex component addresses edge cases or processes specific sub-tasks. The interpretable layer carries most of the explanatory burden and satisfies regulatory requirements for routine decisions, while the complex component is constrained to contexts where its outputs feed into — but do not fully determine — the interpretable layer's logic. This design pattern has found traction in Saudi fintech applications where the distribution of cases is heavy-tailed: most decisions are relatively straightforward, but a meaningful minority require the pattern-recognition capabilities of more powerful models.
Designing for Explainability in Finance
Saudi financial institutions face a specific constellation of explainability demands. SAMA's guidelines apply across retail banking, insurance, investment management, and fintech. The PDPL applies to any automated processing of personal data that produces decisions affecting individuals. And the Kingdom's consumer protection expectations — embedded in SAMA's conduct standards — require that explanations given to customers be not just technically accurate but genuinely comprehensible.
For credit and lending decisions, the core requirement is that an applicant who is declined receives a specific, honest explanation rather than a generic system message. This is not satisfied by listing the model's top features in abstract; it requires translating the model's reasoning into plain language that corresponds to the applicant's actual circumstances. "Your application was declined primarily because of a high ratio of outstanding credit to available credit, and secondarily because of the recency of new account openings" is the kind of explanation that satisfies both regulatory and human requirements. Generating this from a black box model requires careful engineering of the post-hoc explanation layer and deliberate translation of technical outputs into customer-facing language.
For internal risk management and fraud detection, the audience shifts to analysts, auditors, and risk committees. Here, more technical explanations — feature importance rankings, SHAP value breakdowns, confidence intervals — are appropriate. But the same principle applies: the explanation should connect the model's output to inputs that an expert can interrogate and, if necessary, override. An unexplainable fraud alert that analysts cannot evaluate is both a governance problem and an operational one, since it degrades the usefulness of the system and creates liability if the alert is wrong.
Saudi Islamic finance adds further considerations. Shariah compliance in financial products involves principles that cannot simply be optimized away by a machine learning model. AI systems supporting product recommendation or portfolio construction in Islamic finance must be explainable not just to regulators but to Shariah scholars who review product structures. This is an area where interpretable models, or at minimum models with robust post-hoc explanation, are not optional — they are required by the governance structure of the institutions themselves.
Designing for Explainability in Healthcare
Healthcare AI in Saudi Arabia sits at the intersection of SFDA's medical device and clinical software standards, SDAIA's broader AI governance framework, and the PDPL's protections over sensitive health data. The National Health Information Center (NHIC) plays a growing role in standardizing health data governance, which affects how AI systems trained on health records are validated and documented.
Clinical decision support tools present the most common explainability challenge. A system that suggests a diagnosis, recommends a treatment, or flags a patient as high-risk for a particular outcome must provide reasoning that a clinician can evaluate. The physician is ultimately responsible for the decision; the AI is a tool. This means explanations need to be calibrated to clinical expertise — not simplified to the point of being useless, but not so technical that they require a data science background to interpret.
For diagnostic imaging AI, SHAP-based heat maps that highlight the regions of an image that drove a particular prediction are now a standard approach. These visualizations connect the model's output to anatomical features that clinicians can assess independently. A system that flags a chest radiograph for further review, and can highlight the specific regions of interest that triggered the alert, is far more useful — and far more auditable — than one that returns only a probability score.
For risk stratification and predictive analytics applied to patient populations, counterfactual explanations are particularly valuable. Clinicians do not just want to know that a patient is at elevated risk; they want to know which modifiable factors are driving that risk and what interventions might change the trajectory. An interpretable model or a well-designed post-hoc explanation layer can provide exactly that information, turning a prediction into a care planning tool.
The SFDA's standards for AI-based medical software are continuing to develop, and organizations should expect increasing specificity around validation requirements for explanation mechanisms. It is not sufficient to demonstrate that a model performs well on aggregate metrics; regulators will increasingly require evidence that the model's explanations are accurate — that the features the model claims to rely on are genuinely driving its decisions, and that explanations do not mislead clinicians into inappropriate confidence or inappropriate dismissal.
Building the Governance Architecture
Explainable AI is not a technique — it is a governance discipline. The technical methods for generating explanations are well-established and improving rapidly. The organizational infrastructure for using those explanations responsibly is what most regulated Saudi organizations have yet to fully build.
The foundation is a model charter: a structured document that, for each AI system, specifies the regulatory obligations the system must satisfy, the explanation types required, the audiences for those explanations, the validation procedures that demonstrate explanation accuracy, and the thresholds at which decisions must be escalated for human review. This document should be created before a model is built, not after, because it shapes fundamental architecture decisions.
Explanation quality must be validated continuously, not just at deployment. Models drift as the data they encounter in production diverges from their training distribution. When models drift, their explanations can become inaccurate — the features the model claims to rely on may no longer be the features actually driving its decisions. Monitoring explanation consistency over time, comparing explanations across similar cases, and establishing review processes when explanation quality degrades are all components of a mature governance framework.
An explanation registry — a maintained inventory of every AI system in production, its explanation methods, its validation status, and its recent performance metrics — serves as a key artifact for regulatory inspections and internal governance reviews. For organizations operating under NCA audit requirements, this registry is not optional; it is the documentation that demonstrates the organization can account for its AI systems.
The human oversight layer deserves particular attention. Explainability serves a purpose only if the humans receiving explanations are equipped to act on them. A clinician who sees a SHAP heat map but has no training in interpreting it gains little from the explanation. A loan officer who receives a technically accurate but incomprehensible breakdown of model features cannot meaningfully exercise the judgment that regulators expect. Training programs, interface design, and escalation procedures all need to be designed in concert with the explanation system itself.
Arabic-language explanation quality is a dimension that Saudi organizations sometimes underinvest in. Regulators, customers, and clinical staff who primarily work in Arabic require explanations that are not only translated but culturally and contextually appropriate. Technical Arabic terminology in AI and finance is still evolving, and explanations that work well for English-speaking audiences may be confusing or misleading in direct translation. Developing and validating Arabic-language explanation templates, with input from Arabic-speaking domain experts, is a non-trivial but necessary piece of the compliance infrastructure.
From Compliance Requirement to Competitive Asset
Explainability is sometimes framed as a constraint — something that limits model complexity and adds overhead to development and governance processes. That framing misses the strategic dimension. Regulated institutions in Saudi Arabia that invest in genuine explainability infrastructure are building capabilities that serve them well beyond narrow compliance.
Organizations that can explain their AI decisions to regulators can build the kind of relationships with SAMA, SDAIA, and the NCA that accelerate approvals for new AI initiatives. Organizations that can explain decisions to customers build trust that is increasingly difficult to acquire in markets where AI skepticism is growing. Organizations that can explain decisions to their own analysts and risk committees make better use of their AI systems, because the explanations enable human judgment to complement machine intelligence rather than simply defer to it.
The PDPL's automated decision-making provisions, SAMA's transparency requirements, and the NCA's auditability controls are not going away. They are becoming more detailed and more rigorously enforced as regulators develop the technical capacity to assess AI systems directly. The institutions that treat explainability as a foundational design principle — not an afterthought applied to finished models — will be better positioned for every compliance review, every regulatory inquiry, and every moment when an AI decision is called into question by a customer, a clinician, or an auditor.
The question is not whether to explain AI decisions. For regulated industries in the Kingdom, that question has been answered by the regulatory framework. The productive question is how to explain decisions in ways that are accurate, useful, and sustainable at scale — and how to build the organizational capacity to maintain explanation quality as models evolve and regulatory expectations develop.
Published by PeopleSafetyLab — AI safety and governance research for KSA organizations.