A practical guide for organizations deploying AI — built around one question: what are you actually building on top of?
Zillow's AI home valuation model was technically capable. The team that built it was sophisticated. The data it ran on was real, verified, and historically accurate. And then COVID shifted housing markets in ways the historical data could not have anticipated — local desirability, labor shortages, supply chain effects on renovation costs — and the model kept running. Confidently. At scale.
The result: $421 million in losses in a single quarter. Zillow shut down the entire iBuying division and eliminated 25% of its workforce.
This was not an AI failure in the way we usually tell the story. The model did what it was built to do. The failure was that no one asked the harder question before they built: is the foundation this AI is standing on actually ready to hold the weight of the decisions we're making?
This document is a pre-flight checklist. It does not tell you whether your AI tool is good. It tells you whether the environment you're deploying into is ready to support it. These are five different questions — and almost no organization asks all of them before they start building.
Your AI is only as reliable as the data it runs on. Audit the foundation before you build on it.
The most common reason AI projects fail is not that the AI tool was wrong for the problem. It is that the data feeding the AI was not fit for purpose — incomplete, inconsistent, poorly governed, or accurate for yesterday's conditions rather than today's. This is the Zillow lesson, and it is being repeated across enterprise AI deployments at scale.
Before any AI project begins, audit your data on four dimensions:
Zillow's model had all four problems simultaneously — and no one was assigned to watch for any of them. The model had strong historical completeness, poor currency as COVID shifted market conditions, inconsistent local pricing data across geographies, and no governance process to flag when the data environment was diverging from the model's assumptions. The model kept running. Confidently.
Is there someone in the room who knows what the diagram can't show?
Enterprise Architects — and the function they represent, regardless of title — exist to hold the holistic view of how systems connect to each other, what the data actually supports, and what happens downstream when you build the thing you're about to build. This is not a documentation function. It is a judgment function.
The research on AI failure is consistent: the question of whether systems and data can actually support an AI initiative is rarely asked at the beginning, when the cost of the answer is low. It is almost always encountered mid-build or post-launch, when the cost is not.
| Governance Question | Why it matters | Warning signs |
|---|---|---|
| Is there a designated owner for how this AI deployment connects to existing systems? | Without a named owner, integration decisions accumulate silently — each one reasonable in isolation, none of them reviewed as a system. | "That's the dev team's call" applied to every integration decision |
| Has the downstream impact of this AI's outputs been mapped? | AI outputs become inputs to other systems. An error doesn't stay contained — it propagates. Someone needs to have traced the path before deployment. | No impact map exists; downstream systems are discovered in post-incident review |
| Was architectural review part of the project initiation — not the project completion? | Half of C-suite executives believe significant digital value loss comes from poor estimation of expected benefits (Gartner, 2025). Architecture review at initiation finds this gap when it can still be addressed. | Architecture review is a sign-off step at the end, not a shaping step at the beginning |
| Does the person building the AI know about other AI initiatives targeting the same problem or data? | Duplicate initiatives — two teams building the same AI solution for the same business problem — are more common than any organization wants to admit. The waste is significant; the data integrity risk when both go live is higher. | No cross-team visibility mechanism exists; initiative inventory is not maintained |
| Who holds the authority to say "we are not ready to build this yet"? | Pressure to ship is asymmetric. Slowing down a high-profile AI initiative requires organizational authority and evidence. Both need to exist before the moment arrives. | Nobody has this authority; or the authority exists but has never been exercised |
In the 1990s, when client-server computing arrived, companies reclassified the people who thought about how systems connected to each other as overhead. Their role fragmented into specialized titles — business analyst, programmer/analyst — with the emphasis on programming, not the holistic view. Eighteen to thirty-six months later, companies discovered their systems didn't connect the way they thought. They called it the integration problem. The same pattern is visible in AI deployments today. The architecture conversation is being skipped at the beginning, and it will be discovered at the end.
Know what you're connecting the AI to — before the AI starts connecting to it.
AI agents — systems that route instructions to tools like APIs, databases, internal services, and third-party integrations — are increasingly common in enterprise deployments. The routing logic in most orchestration frameworks works by matching the AI's intent to tool descriptions. This means the quality, accuracy, and reliability of your tools' documentation is now a direct input to your AI's behavior.
An AI agent does not know your API has a known issue. It does not know the Lambda function it's calling only works in the development environment. It does not know there's no rate limit on the endpoint it just discovered. It routes confidently, calls confidently, and fails confidently.
The tool is documented accurately and completely. Tool descriptions in agent orchestration frameworks are not metadata — they are the routing instructions. An incomplete description produces a mis-routed call. A missing description produces no call where one was needed.
The tool is accessible in the environment where the AI is running. Development endpoints, internal services accessible only on a corporate network, and sandbox environments that don't reflect production are common points of failure between demo and deployment.
Rate limits exist and are enforced. AI agents call at machine speed. An API that handles human-rate calls reliably may not handle agent-rate calls at all. If the endpoint the AI is calling doesn't have a rate limit, it needs one before the AI is connected to it.
Write operations are explicitly reviewed. Reading from a system and writing to it are not equivalent risks. An AI agent with write access to a production system, calling without a rate limit, can create data quality problems faster than any human could — and it will not know it's doing so.
Known issues with connected tools are documented and their impact on the AI is characterized. A tool with a known intermittent issue is not a tool the AI can rely on. Document the issue, characterize how it affects AI routing, and determine whether the AI deployment should wait for the issue to be resolved.
You can't capture value from infrastructure your leadership doesn't know exists.
AI is increasingly being used to expose and monetize data and services that organizations have been sitting on for years — through APIs that surface internal capabilities to customers, partners, and new products. The opportunity is real. The gap between what's possible and what's being captured is also real, and it's structural.
The AI layer doesn't create value on its own — it surfaces value that's already latent in your data and services. If leadership doesn't understand what APIs exist and what they can support, AI can't be directed toward the highest-value opportunities. Before building AI on top of your API infrastructure, answer these questions:
If the answer is "sort of" or "the team knows," that's not sufficient. Agent orchestration requires explicit, accurate tool descriptions. An inventory that exists in someone's head is an inventory that produces routing failures at deployment.
The decision about which APIs to prioritize for AI exposure is a business and architecture decision together — it requires someone who understands the technical capability and someone with visibility into business value. If those two people are not in the same conversation, the highest-value opportunities are being left undiscovered.
Read and write permissions are not equivalent. An AI agent with write access to a customer-facing API, calling without appropriate rate limiting and audit logging, is a data integrity risk. Before any AI deployment that involves write operations, the access model needs to be explicitly reviewed — not assumed from existing API configurations.
Five questions every AI project should answer before the first sprint begins.
The research is consistent across Gartner, IBM, Forrester, and independent case study analysis: the failure mode in most enterprise AI projects is not the AI. It's the decision to build before the foundation was ready. The Before-We-Start Brief is a structured conversation — not a document — that happens before scope is set, before resources are committed, and before anyone writes a line of code.
Every question has a good answer and a "not yet" answer. The "not yet" answers don't mean don't build — they mean don't build yet.
| Question | The "ready" answer looks like | The "not yet" answer looks like |
|---|---|---|
| What problem is this AI solving — and does that problem actually require AI? | A specific, scoped problem where AI provides a demonstrable advantage over existing approaches | "We should be using AI" / "leadership wants to see AI here" / solving a process problem with technology rather than fixing the process |
| Is the data this AI needs ready — complete, current, consistent, and governed? | A completed data audit against all four dimensions with a named data owner post-launch | "The data is mostly there" / "we'll clean it up as we go" / no data owner identified for ongoing maintenance |
| Has someone mapped what happens if the AI is wrong — and is that consequence acceptable? | A documented impact assessment with a defined blast radius and a confirmed human review protocol for high-stakes outputs | "We'll monitor it" / no blast radius defined / human review is assumed but not specified |
| Are the tools and systems this AI will connect to documented, tested, and rate-limited? | An explicit integration inventory with confirmed access, rate limits, and write-operation review completed before deployment | "The team knows what it connects to" / rate limits not verified / write access assumed from existing API configs |
| Is there someone whose job it is to hold the holistic view — to know what the diagram can't show? | A named person with authority to raise architectural concerns and the organizational standing to be heard when they do | No designated architect; "everyone is responsible" (which means no one is); architectural review scheduled at completion, not initiation |
Pay attention to the organizations that are pausing right now to fix their data foundations before they build on them. They look slower today. They are going to be 24 months ahead of the organizations that didn't ask these questions — because when those organizations try to do the next thing, they will discover that what they built doesn't connect the way they thought. We have seen this before. We are setting it up again.
Before any AI initiative begins, confirm every item below. Incomplete items are not blockers — they are the work.
Complete before scope is set. Re-run if project scope changes significantly.
Data completeness assessed — gaps are documented and their AI impact is characterized
Data currency confirmed — training data reflects current conditions, not only historical ones
Consistency verified — key concepts are defined the same way across all source systems
Data owner named — someone is accountable for quality after launch, not just at launch
Integration ownership assigned — named person responsible for how the AI connects to existing systems
Downstream impact mapped — the path from AI output to downstream systems is traced and understood
Architectural review completed at initiation, not completion
Cross-team visibility confirmed — no duplicate initiative targeting the same problem or data
"Not ready" authority exists — someone has the standing to pause the project and be heard
All connected tools documented accurately and completely
Tools confirmed accessible in the production environment (not only dev or demo)
Rate limits verified on all connected endpoints
Write operations explicitly reviewed and approved with appropriate safeguards
Known tool issues documented with AI impact characterized
API inventory exists and is current — not just in the team's heads
Leadership and business stakeholders understand the value of APIs being exposed to AI
Write-access APIs reviewed separately from read-access APIs for AI exposure
The problem requires AI — and not just a better process or cleaner data
Blast radius of an AI error is defined and acceptable
Human review protocol is specified — not assumed
Holistic view owner is named — someone who knows what the diagram can't show