Ask most enterprise teams why their AI system needs explainability and the answer is a sigh: the regulator, the auditor, the risk committee. Explainability lives in their backlog as a compliance chore — something bolted on at the end, usually as a SHAP chart nobody reads. This framing isn't just joyless; it's commercially wrong.
The adoption effect
Across our deployments, the pattern is stark: decision-support systems that show their reasoning get used, and ones that don't get ignored. Loan officers override an unexplained score far more than an explained one. Planners accept a replenishment recommendation dramatically more often when it cites the drivers — seasonal uplift, competitor stockout, promotion calendar — in their own vocabulary. The model is identical; the interface to its reasoning changes everything. Users don't need to trust the maths. They need to see that the machine noticed what they would have noticed.
Designed in, not bolted on
Real explainability is a product decision made at design time. It shapes model choice — a slightly less accurate model whose reasoning can be surfaced often delivers more realised value than an opaque champion, because realised value is accuracy times adoption. It shapes data design, because explanations must be phrased in features that mean something to the user, not in engineered variables with names like ratio_7d_lag3. And it shapes the interface: the explanation belongs next to the recommendation, in one sentence, with drill-down for the sceptical — not in a PDF appendix.
What it costs to retrofit
Teams that defer this discover the cost curve is brutal. Retrofitting explanation onto an opaque pipeline means re-engineering features, rebuilding interfaces, and re-earning user trust already lost to months of unexplained outputs. The checkbox mentality doesn't even satisfy the regulator well — audit-grade explanations generated after the fact are exactly the kind that fall apart under questioning.
Build the explanation with the model, and you satisfy the compliance requirement as a side effect of building a product people actually use. That's the correct order.