Almost every stalled AI project we're brought in to rescue has the same root cause, and it's not the model. It's that nobody checked whether the data underneath could actually support what leadership wanted to build. Here's the six-point audit we run before scoping any AI engagement.
One: data lineage. Can you trace a given field back to its source system and transformation history? If not, you can't debug a model's bad output, because you don't know if it's a model problem or a garbage-in problem. Two: label quality. If the use case needs supervised learning, are the existing labels consistent, or were they generated by five different people with five different definitions of the target category?
Three: freshness and drift. Is the data that would train or ground a model representative of current reality, or is it eighteen months stale? Four: access and governance. Is sensitive data already classified and access-controlled, or would building the AI system also mean untangling years of ad-hoc permission sprawl first? Five: volume, for the specific use case - a retrieval system needs different scale than a fine-tuning job, and "we have a lot of data" is not the same as "we have enough of the right data."
Six, and most commonly skipped: a ground-truth eval set. Before you build anything, do you have a set of real questions or cases with known-correct answers you can measure against? Without it, you can't tell if a change made the system better or just different. Most of our AI engagements start with two to three weeks of exactly this audit - unglamorous, but it's the difference between a model that works in a demo and one that survives contact with real usage.
