AI Proof of Concept: Success Criteria and a 90-Day Plan

2026-11-22 · Tommaso Maria Ricci

AI Proof of Concept: Success Criteria and a 90-Day Plan

An AI proof of concept should give you a decision within weeks, and a lot of companies are paying for one without getting that decision. According to S&P Global Market Intelligence, the share of companies abandoning most of their AI initiatives before production rose from 17% to 42% in a year, and the average organization scrapped 46% of its proofs of concept before they got there. The survey covered 1,006 respondents and was run in late 2024.

Part of that number is healthy, because a test that kills a bad idea in six weeks has done its job. The expensive failures are the other kind: the demo everybody liked, with no threshold, no owner and no date on which someone has to say yes or no. Those don't get killed. They get extended.

This guide covers how I would run an AI proof of concept when real money follows the result: what to test, how to write success criteria before anyone builds, how many test cases you need, what the economics look like, what to put in the vendor agreement and how to make the decision at the end. It includes a one-page charter, a readiness scorecard and a 30/60/90 day plan. I write as a founder, so I treat a proof of concept as a purchase of information, and I want to know what information the money buys.

What an AI proof of concept is, and what it is not

An AI proof of concept, usually shortened to AI PoC, is a time-boxed test that answers whether a specific AI approach can perform a specific task, on your data, well enough and cheaply enough to justify the next investment.

Time-boxed means there is an end date. Specific task means "draft a reply to a billing dispute", which is testable, where "improve customer experience" is a wish. Your data means the messy export from your system. Well enough means a number agreed in advance. Cheaply enough means somebody has estimated the cost of running it at real volume.

Four terms get mixed up in steering committees, and the confusion costs money because each one answers a different question.

StageQuestion it answersWho uses itWhat comes out
DemoCan this kind of technology do this kind of thing?Nobody, people watchInterest
Proof of conceptCan it do our task on our data at our threshold?A test team, offline or in shadow modeGo, change or stop
PilotDoes it create value inside the real workflow?A limited group of real usersA measured business effect
ProductionCan we run it every day and answer for it?Everyone in scopeAn operated service

A demo uses examples chosen to work. A proof of concept uses examples chosen to represent reality, including the ugly part of it. A pilot adds people, habits and exceptions. Production adds monitoring, support, security reviews and an invoice that arrives every month.

You can skip a stage when the risk is low. A writing assistant for internal meeting notes rarely deserves a formal PoC: buy ten licenses and measure use. What you shouldn't do is rename a stage. If a demo gets called a pilot, someone will ask for a production budget with no business measurement behind it.

When you don't need one at all

Some questions don't need an experiment:

  • The feature is already included in software you own and the downside of a wrong output is small. Switch it on for a group and measure.
  • The task can be solved with a rule, a template or a database query. Do that.
  • The answer is already decided for strategic reasons. Then plan an implementation and stop pretending there is a test.
  • Nobody will act on a negative result. A test with one acceptable outcome is theater.

If the open question is whether to buy a product or build your own, settle the options first with a build vs buy AI software decision, then use the proof of concept to test the assumption that decision depends on most.

Why most AI proofs of concept never reach production

The research is more consistent than the headlines suggest. A Gartner survey of 644 respondents in the United States, Germany and the United Kingdom, conducted in the fourth quarter of 2023, found that on average only 48% of AI projects made it into production, and that getting from prototype to production took eight months. The most cited barrier, named by 49% of participants, was the difficulty of estimating and demonstrating the value of AI projects.

Read that last figure again. The biggest obstacle came before the model. People couldn't show what the thing was worth.

RAND researchers, who interviewed 65 experienced data scientists and engineers for a 2024 report, note that by some estimates more than 80% of AI projects fail, twice the rate of IT projects without AI. They found five root causes: the problem is misunderstood or miscommunicated, the data is inadequate, the team chases technology over the user's problem, the infrastructure to deploy isn't there, or the task is beyond what AI can do today.

Only the last of those five is something a technical test naturally detects. The other four have to be designed into the PoC on purpose. This is why so many tests pass and so many projects still die.

The financial picture is just as sober. In McKinsey's 2026 State of AI survey, with 1,719 participants in 97 nations, nearly nine in ten respondents report regular use of AI in at least one business function, while 37% attribute any EBIT impact to it, about the same as a year earlier. The high performers, who attribute at least 5% of EBIT to AI and describe the impact as significant, are still about 6%.

The five ways a successful test becomes a dead project

Here is how those root causes look from inside a company.

The question was too wide. "Test generative AI in finance" can't fail and can't succeed. Six weeks later there are four half-built ideas and a slide.

The data was prepared by hand. Someone spent two weeks cleaning 300 documents. The model performed beautifully. In production there are 300 documents every morning and nobody to clean them.

The sponsor was curious and the owner was absent. Innovation funded the test. Operations would have to change its procedure, retrain people and carry the errors. Operations was invited to the final presentation.

Nobody priced the review. The output was right most of the time, so a person had to check all of it. Checking took almost as long as doing.

Security arrived at the end. The test ran in a sandbox with copied data. The production request opened a three month review that could have started on day one.

None of these needs a better model. All of them are visible before the build, if somebody is paid to look.

How to choose the use case for an AI PoC

The first use case is a portfolio decision, and it deserves an hour of discipline. If you don't have a list yet, start from a structured set of enterprise AI use cases and cut it down with five filters.

  1. Volume. The task happens often enough that a small gain per case becomes a visible number. Twenty cases a month will never pay for engineering.
  2. Measurability. You can state today's time, cost, error rate or conversion. If you can't, your first project is measurement.
  3. Data access. The inputs and the correct answers exist, and you are allowed to use them for this purpose.
  4. Tolerable error. A wrong output can be caught or reversed before it hurts a customer, an employee or the accounts.
  5. An owner who wants it. A named manager will change the workflow if the numbers hold.

A use case that fails on the fifth filter should be dropped even if it scores perfectly on the other four. I'm rigid on this point. Technology can compensate for mediocre data more easily than for a manager who doesn't care.

Test the riskiest assumption first

Every use case hides one assumption that, if false, makes everything else irrelevant. Find it and test that, before the interface, the integration and the branding.

For contract review the assumption might be that the model finds every non-standard liability clause in scanned documents. For a support assistant it might be that the knowledge base contains the answers at all. For a sales tool it might be that representatives will read a suggestion during a working day. For an agent that updates records, it might be that the target system exposes a safe way to write.

Write it as a sentence that can be false: "On scanned supplier contracts from the last three years, the system finds at least 95% of automatic renewal clauses." Now you have a proof of concept. Before that sentence you had a topic.

A useful habit is to ask the team to list three assumptions and rank them by two things: how likely each is to be wrong, and how much it costs to find out. The one that is likely wrong and cheap to check goes first, even when it's the least exciting. Sometimes it takes two days and ends the project. That is the best return you'll get all year.

AI proof of concept success criteria: write them before the build

AI proof of concept success criteria have to be signed before the first line of code or the first vendor login. The Australian government's guidance on moving AI from proof of concept to scale defines a successful PoC the same way: a clear problem statement, success criteria, appropriate data management and thorough testing.

I would use four groups of criteria. A proof of concept passes only when all four pass.

1. Quality

Define what a correct output is, who judges it and the minimum rate. Separate error types, because they don't cost the same. A missed renewal clause can cost a year of fees. A false alarm costs a lawyer thirty seconds. One accuracy number hides that difference, so set two thresholds: how many real cases must be found, and how many alarms may be false.

For generated text, "good" needs a rubric. Three or four yes or no questions work better than a score from one to ten: Is every fact supported by the source? Is the required information present? Could it be sent with no edits? Two reviewers should grade the same sample independently at the start. If they disagree often, the rubric is broken, and no model will meet a standard your own experts don't share.

2. Economics

State the maximum cost per completed case and the minimum saving or gain. Completed is the important word. Cost per call looks tiny. Cost per case that ends correctly includes retries, long documents, the human review and the cases that fall back to the old process.

3. Operations

Set limits for response time, availability, integration and exception handling. A draft that arrives in forty seconds is fine for a back office queue and useless during a phone call. Decide what happens when the system is unsure or down.

4. Risk

Name the failures that stop the test regardless of averages: personal data shown to the wrong user, an invented policy sent to a customer, an action taken without authorization. These are gates. They are never averaged with quality.

CriterionWeak versionVersion you can decide on
QualityHigh accuracyAt least 90% of drafts pass all rubric questions on 200 unseen cases
Critical errorMinimal hallucinationsZero unsupported policy statements on 300 adversarial and normal cases
EconomicsSaves timeMedian handling time falls from 9 to 4 minutes, review included
CostAffordableRun cost under $0.80 per completed case at forecast volume
OperationsFast95% of responses in under 10 seconds
AdoptionUsers like itIn the pilot, 7 in 10 eligible cases are handled with the tool by week four

The numbers in the table are examples to show the form. Yours come from your baseline and from what an error costs you.

Set the baseline before you touch the model

Measure the human process on the same cases with the same rubric. This step can surprise everyone. A process described as flawless may turn out to be right 86% of the time, and a task everyone calls nine minutes long may take four. Either discovery would change the project.

A baseline also protects you from the opposite error, which is demanding perfection from software that replaces an imperfect process. The fair comparison is between the new system with its controls and the old system with its controls.

How many test cases does an AI proof of concept need?

More than the twenty that fit in a meeting. The number depends on how precise your decision must be, and the arithmetic is short enough to do in front of the steering committee.

Suppose the system gets 90% of cases right. How much should you trust that 90%? With a standard 95% Wilson confidence interval, the plausible range depends heavily on the sample.

Test casesObserved successPlausible range
5090%78.6% to 95.7%
10090%82.6% to 94.5%
20090%85.1% to 93.4%
50090%87.1% to 92.3%

If your threshold is 85%, a result of 45 out of 50 proves very little: the true rate could be below 80%. At 200 cases the lower end just clears the line. This is the difference between "it worked in testing" and evidence.

Rare, serious errors need even more. When you see zero failures in a test, the upper limit of the true failure rate is roughly three divided by the number of cases. Zero critical errors in 100 cases is still compatible with about 3% failures. In 300 cases, with about 1%. If your process handles 4,000 cases a month, 1% is forty incidents. Decide whether you can live with forty before you celebrate the zero.

How to build the test set

The quality of the sample matters as much as its size.

  • Draw it from history at random, then add a deliberate group of hard cases and report that group separately.
  • Cover the segments that matter: languages, countries, product lines, document formats, the customer type that generates most complaints.
  • Keep a holdout. The team tunes prompts and retrieval on one set and is judged on another it has never seen. Without this, you measure how well the team memorized the exam.
  • Include cases with no answer. The system must say it doesn't know. If every test question has an answer, you never test the behavior that prevents the worst errors.
  • Freeze the set and version it. You will rerun it every time a model, prompt or vendor changes, long after the PoC is over.

That last file is the most durable asset a proof of concept produces. When a model gets replaced, a labeled set of real cases with an agreed definition of correct keeps its value, and it lets you change supplier with evidence in an afternoon.

The one-page AI PoC charter

If the plan doesn't fit on one page, the scope is too big. These are the fields I would insist on, each with the kind of answer that is acceptable.

  1. Decision to be made. "On day 45 we decide whether to fund a pilot in billing support." A decision, with a date.
  2. Decision owner. One name. The person who controls the budget and the process.
  3. Assumption under test. One sentence that can be false.
  4. Scope. Which cases, which channel, which language, which period. Also what is excluded.
  5. Baseline. Today's quality, time and cost, with the measurement date.
  6. Success criteria. The four groups, with thresholds.
  7. Stop conditions. The results or incidents that end the test early.
  8. Data. Sources, permission to use them, where they are processed, when they are deleted.
  9. People and time. Named experts and the hours they have actually been released for.
  10. Budget. Build, run and internal time, with a ceiling.
  11. Path to production. Who would operate it, which systems it touches, which reviews are required.
  12. What we do if it fails. The fallback, and who communicates the result.

Field nine is the one that gets faked. Subject matter experts are the scarcest input in any evaluation, because they define correct and grade the sample. "Support from the business as needed" means nobody will come. Write "Dana, six hours a week for five weeks, backfilled", with your real names in place of mine, and get their manager to sign.

Field eleven is the one that gets skipped. You don't have to build the path to production during the test. You have to know it exists. A short call with the owner of the target system in week one can save the whole budget.

Readiness scorecard: should you start this AI proof of concept?

Score each statement 0 for no, 1 for partly, 2 for yes. Do it with the process owner in the room, and don't negotiate the points.

  1. We can describe the task in one sentence, with its input and its output.
  2. We know the monthly volume from a system, without guessing.
  3. We have measured the current time, cost or error rate in the last 90 days.
  4. We can obtain at least 200 real past cases with a known correct outcome.
  5. We have confirmed that we may use that data for this test.
  6. A named manager will change the workflow if the test passes.
  7. Experts have protected hours to label and review.
  8. Security and privacy have seen the design before the build.
  9. We have written the thresholds and the stop conditions.
  10. We know roughly what production would cost and who would pay for it.

0 to 8 points: don't start. Spend three weeks on measurement and access. Building now would produce a demo with a project code.

9 to 14 points: start, but use the first two weeks to close every zero. A zero on statements 5 or 6 blocks the project whatever the total says.

15 to 20 points: you are ready. Keep the time box short, because your real uncertainty is in the pilot.

A low score is useful information about the company, and it usually applies to the next five use cases as well. If several teams land in the bottom band, the issue is upstream, and an AI readiness assessment will pay back faster than another experiment. It's also the point where an outside conversation helps: if you want a second pair of eyes on the use case and the thresholds before you commit a budget, request a consultation and bring the scorecard with you.

The economics: a worked AI PoC business case

A proof of concept has to end with a number that a finance director can attack. Here is a worked example. The inputs are illustrative, so replace every one of them with yours. The structure is the point.

A support team handles 4,000 billing emails a month. Each takes 9 minutes. A loaded hour costs $48. The test shows that 70% of emails are eligible, so 2,800 a month. When a draft is accepted, handling drops to 3 minutes, which saves 6. When a draft is rejected, the agent has lost a minute reading it.

With 80% of drafts accepted:

  • Minutes saved on 2,240 accepted drafts: 13,440.
  • Minutes lost on 560 rejected drafts: 560.
  • Net time released: about 215 hours a month, worth $10,304.
  • Less $3,000 a month for licenses, usage, monitoring and support: $7,304.
  • Investment of $20,000 for the PoC and $60,000 to reach production: payback in 11 months.

Now move one variable, the acceptance rate, and leave everything else alone.

Drafts acceptedNet monthly benefitPayback
50%$2,60030.8 months
60%$4,16819.2 months
80%$7,30411.0 months
90%$8,8729.0 months

The table tells you what the proof of concept is for. This business case lives or dies between 60% and 80% acceptance, so that is the quantity to measure precisely. A test that reported "the model is 92% accurate" and never measured acceptance would have answered a question the investment doesn't depend on.

It also shows the trap in time savings. The 215 hours become money only if something happens to them: overtime goes away, a vacancy isn't filled, the backlog that was losing customers gets cleared. Write down which one. The wider method, including the denominator most teams forget, is in my guide to AI ROI for business.

What to budget by company size

I don't publish price lists for this, because market quotes for the same scope vary too much to be useful. Effort is more stable, so budget in days and multiply by your rates. These are planning assumptions, to be corrected with your own estimates.

Startups and small companies. Use an existing product or a model API with very little code. Plan 10 to 20 person-days, at least a third of them from the person who knows the work. If it needs more, the use case is too big for a first test.

Mid-market companies. Plan 25 to 50 person-days. The extra effort goes to data extraction, access rights and the review by security, which exist at this size and should be paid for openly.

Enterprises. Plan 50 to 90 person-days for one use case, and be suspicious above that. Beyond this point you are usually building a platform inside a test budget, and the platform deserves its own decision inside your AI operating model.

In all three bands, reserve a fifth of the budget for evaluation. It's the line that gets cut first and the only one that produces the thing you are buying, which is evidence.

Data, security and risk during the test

"It's only a proof of concept" has justified more bad data handling than any technology I know. Test data is real data. A copy of the customer database in a trial account is a transfer of the customer database.

The minimum set of controls is short:

  • Use the smallest extract that answers the question, and remove identifiers the task doesn't need.
  • Read the terms of the trial tier. They are often different from the enterprise agreement on retention and on training with your inputs.
  • Keep a list of who can see inputs, outputs and logs. Logs are the copy everybody forgets.
  • Test permissions with two users who should see different things, if the system retrieves internal documents.
  • Delete the data at the end and keep the confirmation.

For the structure, the NIST AI Risk Management Framework gives four functions, govern, map, measure and manage, that translate well into a PoC checklist. Its Generative AI Profile, published in July 2024, lists the risks specific to generative systems, including confabulation, data privacy and information security. For applications built on language models, the OWASP Top 10 for LLM applications is a practical source of attack cases to put in the test set, starting with prompt injection.

Classify the use case before the build. If it affects hiring, credit, health, access to services or anything with legal effect on a person, the evidence you need and the reviews you face are of another order. An AI risk assessment takes a few hours at this stage and several months if you discover the category later. The same applies to the data itself, where ownership and permitted use should follow your AI data governance rules from the first extract.

Testing AI agents: what changes in an agentic AI proof of concept

An agent takes actions, so the test changes. Gartner predicted in June 2025 that over 40% of agentic AI projects would be canceled by the end of 2027 because of escalating costs, unclear business value or inadequate risk controls. Its analyst Anushree Verma described most agentic projects at that time as early experiments or proofs of concept, often misapplied.

When the output is an action, grading the final answer is not enough. You evaluate the path.

Judge completed tasks. An agent that gets each of eight steps right 95% of the time finishes the whole task correctly about 66% of the time, if the errors are independent. Step accuracy flatters. End-to-end success is the number that counts.

Start read-only. In the proof of concept the agent proposes and a person executes. Write access is earned in the pilot, one action type at a time, starting with the reversible ones.

Give it a budget. Set a maximum number of steps, a maximum spend per task and a timeout. An agent in a loop is a billing incident.

Attack it. Put instructions inside the documents and emails it reads, and check that it ignores them. If it can read untrusted text, access private data and send something out, assume someone will try to connect those three things.

Log every step. If you can't reconstruct why it did something, you can't correct it and you can't defend it.

Permissions, ownership and the kill switch belong to AI agent governance, and they are easier to define when there is one agent in a test than when there are thirty in production.

Running a proof of concept with a vendor

A vendor-led PoC is a sales process with a technical part. That is fine, as long as you know it and write the rules. Seven clauses do most of the work.

  1. Your data, your test set. The vendor doesn't choose the examples and doesn't see the holdout.
  2. Thresholds in the agreement. The same ones as in your charter, with the method of measurement.
  3. No training on your data, and a deletion date with written confirmation.
  4. Full cost at target volume. A production quote before the test starts, so a pass can't be used against you on price.
  5. Named configuration. Model versions, settings and any manual work the vendor performs. Human effort behind the curtain must be declared.
  6. Exit without commitment. The test ends by itself. No automatic conversion to a subscription.
  7. Ownership of results. You keep the test set, the labels, the prompts written for you and the report.

Clause five deserves a direct question in the kickoff: "What will your people do by hand during these six weeks?" An honest vendor has an answer. Some preparation is normal. You need to know it so you can price it.

If you compare vendors, give them the identical frozen set on the identical dates, and limit it to three. The fourth and fifth evaluation consume the experts' hours, and by then those hours are what you are short of.

How to run an AI proof of concept in 30, 60 and 90 days

Ninety days is enough to go from a question to a funded pilot, or to a documented no. The test itself occupies the middle.

Days 1 to 30: define and measure

  • Week 1. Choose the use case, name the decision owner, write the assumption and call the owner of the target system.
  • Week 2. Measure the baseline. Agree the rubric and have two experts grade the same 30 cases. Fix the rubric where they disagree.
  • Week 3. Assemble the test set and split off the holdout. Open the security and privacy review.
  • Week 4. Sign the charter with thresholds and stop conditions. If the scorecard is under 9, stop here and say so.

At day 30 nothing has been built, and that will make people nervous. It is the cheapest month of the project and the one that decides the quality of all the others.

Days 31 to 60: build small and evaluate

  • Week 5. Build the simplest version that can be tested. A baseline model, a plain prompt, no interface.
  • Week 6. Run the development set, read the failures one by one and group them by cause. Missing data, bad retrieval, wrong instruction and task too hard need four different fixes.
  • Week 7. Improve, then run the holdout once. Once is the rule. If you keep retesting on it, it becomes a development set.
  • Week 8. Calculate cost per completed case at real volume and update the business case with the measured acceptance rate.

Days 61 to 90: decide and hand over

  • Week 9. Hold the decision meeting with the owner, finance, security and operations. One of three outcomes, written the same day.
  • Weeks 10 and 11. If it's a go, design the pilot: users, workflow, training, support, monitoring and the control group.
  • Week 12. Put the first users on it in shadow mode, where the system works and the person still decides, or publish the closing note and release the team.

The handover is where the operating questions surface: who watches quality, who pays the usage bill, who retrains people. They are the subject of an enterprise AI adoption framework, and it's better to meet them with one use case than with ten.

The decision meeting: go, change or stop

The meeting takes forty-five minutes if the charter was honest. Put the thresholds on the left, the results on the right, and read down the page.

Go when all four groups pass on unseen cases and the business case holds at the lower end of the confidence range. Use the lower end. If the investment works only at the point estimate, you are betting on sampling luck.

Change when one criterion fails for an identified and fixable cause. A change gets one extension with a date, a budget and the same thresholds. A second extension is a stop that nobody wants to announce.

Stop when a risk gate fails, the economics don't work at a realistic acceptance rate or the owner has lost interest. Write one page: what we tested, what we found, what would need to be true to try again. Then say it publicly and without blame. The first time a company punishes a team for a clean negative result is the last time it gets one.

That final point matters more than any template here. If every proof of concept in your company succeeds, your proofs of concept aren't testing anything.

What to report to the board

Boards don't need confusion matrices. They need five lines:

  • The decision requested and the amount.
  • The measured result against the threshold, with the sample size.
  • The payback at the cautious end of the range.
  • The worst failure observed and the control for it.
  • The name of the person accountable in production.

Results that count: four examples of the right unit

These are results from my own work, shown by sector and without client names. None is a story about a model benchmark. I quote them for the unit in which each result is expressed, because that is the unit your success criteria should end in.

A sports distribution company: 30% more sales with AI marketing. The measure is sales. Creative volume, impressions and hours saved are inputs to it. A test of AI in marketing should say from the start how its metric connects to that line.

A hotel: revenue from 9 to 10 million. The measure is revenue for the property. If you test a pricing or forecasting assistant, forecast error is your technical criterion and revenue is the one that funds production.

A medical center: 20% more capacity. The measure is capacity, which an administrator can count. In a service business this is often a better target than headcount, and far easier to get staff to support.

A farm stay: twice as many guests. A small operation, and the measure is guests. Small companies can run a serious test. They need one number they already track and the patience to write it down before and after.

When someone proposes a proof of concept to you, ask which number of this kind moves if it works. If the answer takes more than a sentence, the use case isn't ready.

Common AI proof of concept mistakes

Starting from the model. "Let's do something with the newest model" reverses the order. Problem, metric, data, then technology.

Testing on the easy segment. English-language, typed, short documents pass. The business runs on scanned, multilingual, forty page ones. Report results by segment or don't report them.

Counting accuracy and ignoring review time. A system that is 95% right and needs every output checked can be slower than the process it replaces. Time the review.

Running ten at once. Parallel PoCs share the same three experts and the same security reviewer. Two or three, finished, teach more than ten open ones.

Leaving the change work to later. The people who will use the system should grade the test cases. They learn what it can do, you learn what they won't accept, and the change management for AI starts months early at no extra cost.

Letting the test become production. A useful prototype gathers users quietly. Six months later it handles customer data with no owner, no monitoring and an API key in someone's personal account. That is how shadow AI is born inside official projects. Switch the prototype off on the end date.

Where to start this week

Take the AI idea that gets mentioned most in your company and write one sentence under it: the assumption that would kill it if false. Then count how many real past cases you could use to test that sentence, and find out how long the task takes today. It's two hours of work, and it will tell you whether you have a proof of concept or a conversation.

An AI proof of concept is a small instrument for a large decision. Keep it small, make it falsifiable and honor the result. If you would like to pressure-test a use case, a vendor proposal or a business case before the budget is approved, request a consultation: send the one-page charter, even half empty, and we'll start from the blanks.

FAQ

What is an AI proof of concept?

An AI proof of concept is a time-boxed test that establishes whether a specific AI approach can perform a defined business task, on the company's own data, at an agreed level of quality, cost and risk. It ends with a decision to go, change or stop. It differs from a demo, which uses examples chosen to work, and from a pilot, which puts the system in front of a limited group of real users to measure the business effect inside the daily workflow.

What are good AI proof of concept success criteria?

Good AI proof of concept success criteria are written before the build and cover four groups: quality on unseen real cases, economics per completed case, operational limits such as response time, and risk gates that cannot be averaged away. Each needs a number, a measurement method and a baseline from the current process. An example is at least 90% of drafts passing an agreed rubric on 200 unseen cases, with no critical errors and a lower handling time including human review.

How long should an AI PoC take?

Plan about four weeks to define the question, measure the baseline and prepare the test set, then four weeks to build and evaluate. With a decision and a pilot design, the full cycle fits in 90 days. A narrow test on an existing product can be shorter. If the build and evaluation phase needs more than two months for one use case, the scope probably includes platform or integration work that should be funded and decided separately.

How much does an AI proof of concept cost?

The cost depends mostly on effort, so budget in person-days. As planning assumptions, a small company can expect 10 to 20 days, a mid-market company 25 to 50 and an enterprise 50 to 90 for one use case, plus model usage or licenses. Reserve about a fifth of the budget for evaluation. Always estimate the production cost before starting, because a test that passes is worthless if nobody can afford to operate the result.

How many test cases do I need to evaluate an AI proof of concept?

For a decision around a 90% quality threshold, 200 unseen cases is a practical minimum: an observed 90% then has a 95% confidence range of roughly 85% to 93%. With 50 cases the range is about 79% to 96%, too wide to decide. Rare critical errors need larger samples, since zero failures in 300 cases is still compatible with a true failure rate of about 1%.

Why do AI proofs of concept fail to reach production?

The usual causes are a vague problem, inadequate data, technology chosen before the use case, missing deployment infrastructure and tasks that are too hard for current AI, as RAND documented in 2024. In practice they combine with two management failures: no business owner willing to change the workflow and no threshold agreed in advance. S&P Global found that the average organization scrapped 46% of its proofs of concept before production.

What is different about an agentic AI proof of concept?

An agent takes actions, so you evaluate whole tasks and the steps taken, in addition to final answers. Eight steps that are each 95% reliable give about 66% end-to-end success if errors are independent. Start in read-only mode, cap steps and spending, test prompt injection through the documents the agent reads, log every action and grant write access gradually during the pilot, beginning with actions that can be reversed.

Should a small business run an AI proof of concept?

Yes when the task is frequent, measurable and expensive to get wrong, and no when an existing tool can be tried safely with a few licenses. A small business can keep the test to 10 to 20 person-days, use past cases from its own records and judge one number it already tracks, such as hours, capacity, revenue or guests. The discipline of a written threshold and an end date matters more than the size of the budget.