Explainability Under Pressure: AI Decisions in Credit, Insurance and Hiring

Published: at 06:50 AMSuggest Changes

The head of credit risk walked into a regulatory review with a clean answer. Every declined application received a letter listing the main reasons, generated automatically from the model’s top contributing features. The regulator did not ask about the letter. It asked for one decision from six months earlier: which model version made the call, on which input values, and what a human had done to it afterwards. The team could produce the letter. They could not reproduce the decision.

That gap is where explainability stops being a data-science deliverable and becomes an operating obligation. The model is one input into a decision; the obligation runs to the person on the other end and to the examiner who will reconstruct it later. Treating explainability as a property of the output — a sentence, a chart, a feature-importance plot — is what produces a team that is confident about its explanations and unable to account for its decisions.

An explanation is a property of the decision

The word “explanation” hides four obligations, and most programmes satisfy one while quietly failing the other three.

Use those four as a pass/fail test. A letter a customer cannot follow fails the first. A reason code drawn from a surrogate model fails the second. A justification written by a complaints analyst after the fact fails the third. A decision no one can rebuild fails the fourth. Enforcement tends to find whichever is weakest, and it is rarely the one the team has been polishing.

Ownership follows from the test. The data-science team can build the machinery that produces faithful, contemporaneous reasons. It cannot decide what the firm is willing to tell a customer, how long it will keep a decision, or who is allowed to reverse one. Those are business and control decisions, and they belong to the accountable executive for the product, with risk and legal setting the floor. Organisations that leave explainability inside the modelling team get elegant interpretability and unaccountable decisions.

Post-hoc rationalisation is a story about the decision

An explanation produced after a black-box decision carries a quiet problem: it is itself an approximation. Methods such as LIME and SHAP fit a simpler, interpretable model around the original one. They earn their place in debugging and analyst workflows. They are not a record of why this applicant was declined, and different methods can disagree on the same input.

The Consumer Financial Protection Bureau has been direct about the credit case. In Circular 2023-03 it states that creditors cannot rely on the checklist of reasons in its sample adverse-action forms where those reasons do not accurately or specifically identify the principal reason for the adverse action, and that using complex algorithmic underwriting does not excuse the obligation. The reasons disclosed must be specific and true to the decision. A plausible letter the model cannot support is a second violation stacked on the first. [CFPB, September 2023]

Post-hoc explanations inherit the model’s mistakes. I have seen a risk team discover that its decline letter was being partly populated by a marketing segmentation field that had leaked into the feature set. The letter was consistent with the model. The model was using something the firm would never have chosen to disclose. A surrogate explanation would have carried the same error, with more confidence and less evidence.

Log the decision, not the model

What has to be retained is not the model card. It is the decision record: a per-decision artefact that lets a reviewer reconstruct what happened without rerunning the model and without guessing.

The reason codes must be generated at the point of decision. A common failure is to log the model output and rebuild the explanation later from feature importances, which shift with every retraining. Two years and three retrainings later, that reconstruction is fiction with a confident tone.

This is a different discipline from model monitoring. Monitoring tells you whether the model is drifting today. The decision record tells you what the firm did to a specific person on a specific date. Teams that invest in the first and skip the second end up able to describe their model’s behaviour in aggregate while being unable to answer a single question about an individual case.

Retention is a governance choice with a legal floor. The National Association of Insurance Commissioners’ model bulletin, adopted in December 2023 and since implemented by a majority of states, expects insurers to be able to produce documentation and explanations for AI-supported decisions on request, and to govern third-party models with the same rigour as their own. [NAIC, December 2023] New York City’s Local Law 144, enforced since July 2023, requires employers using automated employment decision tools to commission an annual independent bias audit, publish a summary, and notify candidates in advance. [New York City, July 2023] The design tension is real: retain enough to reconstruct the decision, hold as little personal data as the law allows. The workable resolution is to keep the decision artefact and its reason codes, and to be disciplined about everything else.

When you cannot see the weights

Most of these decisions are not made by a model you built. A lender buys a scoring engine. An insurer adopts a vendor’s underwriting platform. An employer runs a third-party assessment. The employer can still be liable for a vendor tool that screens out a qualified applicant with a disability, and regulators do not accept “we could not see inside it” as a defence. [EEOC, May 2022]

The response is contractual and architectural, not forensic. Require the explanation to be produced by the vendor’s system at decision time and returned with the decision, so you are never reverse-engineering it later. Insist on per-decision reason codes and counterfactuals you can put in front of a customer. Demand model documentation sufficient to complete your own impact assessment. Secure validation access, an audit right and, for critical systems, source or model escrow with a defined trigger. Write the exit into the contract: the data and artefacts you will need to move the book or rebuild the decision history elsewhere.

If a vendor will not supply those, that is a procurement risk, and it should be priced, escalated and governed as one. The alternative is discovering the limitation during an examination, when the only remaining lever is to stop using the system.

Designing a contestability path

Explainability only earns its keep if a person can act on it. Contestability is the operating half of the same obligation, and it has a shape.

Track contestability the way you track incidents: reversal rate, time to review, disputes per thousand decisions, and the share of decisions whose reasons were generated at inference rather than reconstructed. Those numbers expose a brittle explanation long before a regulator does.

One caution against a common shortcut. A “human in the loop” who rubber-stamps the model’s output is not contestability; it is the same automated decision with a signature attached. Review is only meaningful when the reviewer has the information, the time and the authority to reach a different answer. The ICO and the Alan Turing Institute made the underlying point when they set out their guidance on explaining AI decisions: an explanation has to be meaningful to the person affected and grounded in the context in which the decision operates, not produced to satisfy a template. [ICO and Alan Turing Institute, May 2020]

The test

Ask your team for one real declined decision from six months ago — a customer, a claimant, a candidate — and have them reconstruct it end to end: the model version, the input data, the reason codes, the human touch, the letter that was sent. Give them a day. If they can reproduce the decision, you have a control. If they can only reproduce the story you told about it, you have a model and a complaints script. Enforcement will not ask to see the model. It will ask you to account for the decision.

Sources

Previous Post
When the Agent Goes Down, Who Can Still Do the Work?
Next Post
Shadow AI Is Not Shadow IT