Why Most AI Pilots Fail Before They Start: The Discovery Problem

The meeting that decides everything

Three months in, the AI pilot has produced no meaningful output. Not because the technology failed — it hasn't even been properly applied yet. The failure happened in the first two weeks, in the room where the project was defined.

I've watched this pattern enough times that it no longer surprises me, though it still frustrates me. A company decides it wants to run an AI pilot. Leadership is engaged. Budget is allocated. Someone is assigned to lead it. A vendor is selected. And almost immediately, the project is set up to fail — not through bad intent but through a very predictable discovery problem.

The discovery problem is simple: no one in the room has done the unglamorous work of understanding the current process well enough to know what AI is supposed to improve.

What "discovery" usually looks like

Most AI engagements skip real discovery. What passes for discovery is usually one of three things:

**A demo.** The vendor shows the technology doing something impressive. The stakeholders decide they want that. The "what problem does this solve?" question gets answered by the demo itself: "it does this." But "this" is rarely the exact shape of the business problem.

**A use case list.** Someone prepiles a list of things AI could potentially do for the organisation. They're usually not wrong — AI could do most of them. But a list of possibilities is not a problem statement. Picking from a list skips the question: which of these would actually move the business if it worked?

**A pain point survey.** Teams are asked what their biggest frustrations are. This produces a list of symptoms. Symptoms are useful input, but acting on symptoms without diagnosing root causes produces the wrong solution more often than not.

Real discovery is slower and less exciting. It involves sitting with the people who do the work, watching the process rather than just hearing about it, and asking uncomfortable questions about why things work the way they do.

The diagnostic questions that actually matter

When I engage with a new AI project, I ask four questions before we discuss technology at all.

**What does success look like, measured how, by when?** Not "we want this to be more efficient." A number. A reduction in hours, error rate, response time, cost per transaction. If the team can't name a metric, the project has no landing zone. You can't evaluate whether it worked.

**Who is affected by this process, and what do they actually do?** Not the official job description — the actual work. The gaps between the job description and the reality are almost always where the complexity lives, and where AI either adds value or breaks things.

**What happens when this process goes wrong today?** Edge cases reveal what matters. If a process has no consequences when it fails, it's probably not important enough to warrant an AI investment. If the consequences are significant, you need to understand them before you automate.

**What's been tried before?** This tells you whether the problem is well-understood or still being diagnosed. A business that has tried three different manual approaches to a problem understands it differently than one encountering it fresh. The history of attempts is information.

Where the actual pilot should start

This is the part most pilots get wrong: they start with the general problem and try to apply AI broadly. The pilots that work start narrow.

Find the highest-frequency, lowest-stakes, most-clearly-defined sub-task within the larger problem. Run the AI on that. Evaluate it rigorously against the current state. Build the measurement infrastructure before scaling.

In practice, this usually means the first month of an AI pilot produces no user-facing output. The team is documenting the current process, establishing baselines, and running the AI in shadow mode — comparing its output against what the human does, without replacing the human yet.

This feels slow. It is slow. But it produces pilots that actually tell you something, rather than pilots that run for three months and produce a report that says "results were inconclusive."

We're still refining this framework — every engagement teaches us something about where the discovery phase needs to go deeper. But the early results suggest that pilots with genuine discovery phases are producing usable output in month two rather than month four.

The vendor's incentive problem

Here's the uncomfortable part: vendors have a financial incentive to skip discovery.

A vendor who does real discovery might conclude that the client's problem doesn't require the vendor's product. Or that the client isn't ready for the implementation yet. Or that the project as scoped will fail and needs to be rescoped. None of these conclusions produce a signed contract.

A vendor who goes straight to solution demo-to-contract has a faster sales cycle and fewer awkward conversations.

This isn't a criticism of vendors — it's a structural incentive problem. The solution is clients who insist on a proper discovery phase as a condition of engagement, and who are willing to pay for it separately before committing to implementation.

The discovery fee is the most honest thing I charge. It costs less than a failed pilot. It produces a clear answer about whether to proceed, and if yes, what to actually build.

The pattern in the portfolios I've seen work

Companies that run successful AI implementations — not just functional pilots but implementations that actually change business outcomes — share a few characteristics.

They had someone internally who could translate between business operations and technical requirements. Not a developer who learned about business, not a business person who learned to prompt — someone who genuinely understood both.

They started with a problem the business had already been working on, not a problem invented for the AI project. The problem had a history, attempts, and stakeholders who cared whether it was solved.

And they treated the discovery phase as real work, not as the polite preliminary before the real work. The scoping document from discovery became the evaluation criteria for the pilot. Success was defined before the technology was selected.

The businesses that get this right are building competitive advantages that are actually durable. Automation using tools like [AutoChat's WhatsApp API](https://autochat.in) follows the same logic — the businesses getting results aren't the ones who automated first; they're the ones who understood their messaging workflows first, then automated the clear parts.

A question for your next pilot

Before you approve the next AI project budget, ask for the discovery document. Not the vendor's proposal — the internal document that answers: what is the current process, who does it, how is it measured today, and what does measurably better look like?

If that document doesn't exist, that's the first deliverable. Everything else can wait.

*Image suggestion: a flow diagram showing two project paths — "discovery-first" leading to clear success criteria vs. "demo-to-contract" leading to inconclusive results — with decision points marked at each stage.*

Want to implement this for your business?

I help business owners build digital systems that actually work. Let's talk about your specific situation.