An AI pilot usually starts well. A small team picks a promising process, connects a model to a sample of documents, and within a few weeks there is a demo that answers questions or drafts replies convincingly. Leadership is impressed. Then the work slows. Months later the pilot is still a pilot: a separate web page a handful of people use, with no date for going live and no clear reason to stop it either.
When a pilot stalls, the model is rarely the reason. Language models are good enough for a wide range of business tasks today. What is missing is a set of ordinary decisions that nobody made at the start, because the pilot felt like an experiment rather than the first phase of a real system. The good news is that these decisions are cheap to make early and expensive to make late.
What stalling looks like
Stalled pilots share a few symptoms. It helps to name them, because each one points back to a decision that was skipped.
- Nobody can say who decides whether it goes live. The sponsor was the IT team, and the business unit that would use it was consulted, not accountable.
- Everyone agrees the demo was impressive, but nobody can say whether it is better than the current process, because the current process was never measured.
- The pilot ran on an export of sample files. Connecting it to the real document store or ERP raises questions about permissions, retention and security that take months to answer.
- Users have to leave the tools they work in — Outlook, SAP, the service desk — to use it, so they stop after the first week.
- There is money for the build but not for running it: nobody owns monitoring, model updates or the monthly compute bill.
The five decisions
Each of these decisions belongs in the first two weeks of a pilot, before anything is built. None of them is technical in the narrow sense. All of them decide whether the technical work will ever be used.
1. A named business owner
Name one person in the business — not IT, not the innovation office — who owns the outcome. They own the process the AI will change, they are measured on it, and they have the authority to change how their team works. IT and data teams are essential, but they cannot decide that a claims team will now review drafted decisions instead of writing them. If the only sponsor is technical, nobody pulls the pilot into daily work.
The owner also makes the hard calls during the pilot: which edge cases matter, what error rate is acceptable, and when a person must stay in the loop. Without that authority, every question becomes a meeting.
2. A baseline and a measure, agreed before building
Decide what success means in numbers the business already cares about — turnaround time per request, cost per document, first-contact resolution, hours spent re-keying — and measure the current process before the pilot starts. Agree the threshold that justifies going live, and the one that justifies stopping. This sounds bureaucratic. It is the opposite: it turns the go-live meeting from a debate about impressions into a short conversation about a number.
Keep a quality measure alongside the business one. A faster process that is wrong more often is not an improvement, so decide early how answers or actions will be checked, by whom, and on how many real cases.
3. Data access settled early, not in week eight
Most pilots are built on a convenient copy of the data. Production needs the real sources, with the real permissions. Start the conversations with security, legal and the system owners in the first week: which systems the pilot may read, under which service account, whether personal data is involved, where the model runs and what is logged. If personal data is involved, bring the privacy team in at the same point, because hosting and retention choices depend on their answers. These reviews take the time they take. The only choice is whether they run alongside the build or after it.
4. Integration into the tools people already use
An assistant in a separate browser tab competes with habit, and habit usually wins. Decide at the start where the output will appear: a draft reply inside the service-desk ticket, a suggested classification in the ERP workflow, an answer in Microsoft Teams, a summary attached to the case record. This decision shapes the architecture, so it cannot wait until the model works. It also tells you which integrations, APIs and approvals the pilot has to include.
5. A day-two operating model and budget
A model in production is a running system, not a finished project. Someone has to monitor answer quality and usage, rerun evaluations when content or prompts change, test and roll out model updates, handle incidents, and keep compute cost within budget. Decide who does each of these, how often, and what it costs per month before the pilot is approved. If the running cost cannot be funded, it is better to know in week two than after launch.
What a good pilot charter contains
We write these decisions down in a short pilot charter — two or three pages, signed by the business owner and the technical lead. It is not a project plan. It is the agreement that makes a clear go or no-go decision possible at the end.
- 1The problem, in the business owner’s words, and the process it affects.
- 2The named business owner, the technical lead, and who else must agree before go-live.
- 3The baseline: how the process performs today, and how that was measured.
- 4Success and stop thresholds for the business measure and the quality measure.
- 5The data sources in scope, the access route for each, and the reviews required: security, privacy and system owners.
- 6Where the output will appear for users, and which integrations that requires.
- 7Where the model runs and what is logged, in line with the organisation’s data classification.
- 8The evaluation set: real examples in Arabic and English, including difficult ones, and who reviews the results.
- 9The day-two plan: monitoring, evaluation cadence, model updates, incident handling and an estimate of monthly running cost.
- 10The decision date, and what happens on each outcome: scale, extend to answer one specific question, or stop.
Stopping is a result too
A pilot that is stopped on its decision date, against a measure agreed in advance, is the process working. It frees budget and attention for a better candidate, and it leaves behind a baseline, an evaluation set and settled data access that the next use case can reuse. What wastes money is the pilot that is neither scaled nor stopped.
The five decisions do not make a pilot slower. They move the slow parts — reviews, integration, funding — to the start, where they run alongside the build instead of after it. That is usually the difference between a pilot that ends in a demo and one that ends in a system people rely on.
When we run pilots with clients at Aahlan, the charter is the first thing we produce together, before any code. If you have a pilot that has stalled, it is also a good place to restart: write the charter for what already exists, and the missing decisions become obvious.