Explainability versus accuracy: choosing where to stand

Three ways to see it

1. 1

   Explainability sits on a sliding scale. At one end are simple linear models and shallow decision trees that are inherently readable. At the other end are deep neural networks and large ensembles that need post-hoc tools like SHAP, LIME, integrated gradients, or counterfactuals to extract meaning. Accuracy generally rises as complexity rises, up to a point. The question is not which is better in general but where you should stand for a specific decision context.

2. 2

   Decision context tells you which side to favor. High stakes adverse decisions like credit denial, insurance claim refusal, or job rejection lean toward inherently interpretable models. Low stakes recommendation tasks like product ranking or article suggestions tolerate opacity. Regulated domains in Pakistan including credit scoring under SBP and insurance under SECP increasingly expect a reason explanation that a customer can understand without a data science vocabulary.

3. 3

   Counterfactual explanations are powerful for adverse decisions. Instead of saying which features pushed the score down, the model says what would have to change for the decision to flip: your declared income would need to be 35 percent higher, or your three-month average balance would need to exceed PKR 50,000. Counterfactuals are concrete, actionable, and easier to explain than feature attributions. Use them where the law and risk policy allow.

The why-tree

Try this with Claude

Practical decision rule. If a model decision can lead to a person losing money, opportunity, or a public benefit, the explanation must be readable by a college-educated adult without ML background, available within the same channel as the decision, and consistent with what the model actually did. If you cannot produce that explanation, you should either simplify the model or add a human reviewer who can.