Your First AI Pilot in Finance: What to Measure Over Four Cycles

Executive summary

A first AI pilot in finance usually starts with a good demo and ends without a decision. Six weeks in, someone asks whether it worked. Nobody can say, because nobody wrote down how long the work took before, who owned the pilot, or what result would count as a pass. The licence renews anyway.

There is a quieter problem too. The time a person spends checking the tool's output is real work, and most pilots never count it. The demo shows the draft appearing in seconds. It does not show the controller spending two hours reading it. And because checking usually falls to the most senior person in the routine, a tool can save the analyst's hours while spending more of the CFO's.

A pilot answers one question: does checking the tool's work cost less than doing the work yourself, at an error rate you can live with?

The method below answers it in four cycles. Pick one routine you already run on a schedule and measure it by hand first. Name an owner who runs the pilot and an approver who signs off every change the tool proposes before it is used. Write the pass mark, the extend rule and the stop rule before cycle one. Then count everything: licence or usage, plus every hour of checking and rework. In the illustrative example below, the tool cut production time by 90% and total time by 58%, and the pilot still cost more than doing the work by hand over its four cycles. Both facts belong in the decision.

This guide sits in the AI trust and governance hub. For choosing which routine to try first, see the AI use case prioritization framework.

When to use this

Use this method when:

• A vendor demo went well and finance has been asked to "try it for a month."
• A pilot is already running and nobody can say whether it is working.
• Someone on the team is quietly redoing the tool's output by hand, just in case.
• A licence is up for renewal and the case for it rests on how people feel.

The method assumes the routine already exists and runs on a fixed cycle, such as a monthly forecast score or a weekly review of accounts payable exceptions. If it does not, you have nothing to measure against. Set it up by hand first; operating cadence: weekly, monthly, quarterly covers where it should sit.

Diagnostic questions

1. Which single routine is in scope, and how often does it run?
2. How many hours did it take by hand over the last three cycles, split into producing and checking?
3. Who owns the pilot by name, and who approves each change the tool proposes before it is used?
4. What result, written down before cycle one, would make you keep it, and what would make you stop?
5. Are the hours spent checking the tool's output being logged, or only the hours it saved?
6. Is anyone still running the old process in parallel, and are those hours counted?
7. When the approver rejects a suggestion, is the reason recorded anywhere?

If you cannot answer questions 2, 4 and 5, you are not running a pilot. You are running a demo with a longer calendar.

Framework: one routine, four cycles, three outcomes

One cycle tells you almost nothing. The first run of any new tool is slow because people are learning it, and checking is heavy because nobody trusts it yet. By the third and fourth cycle you can see a trend in checking time and in how often the approver sends work back. Four cycles of a monthly routine take three to four months; of a weekly one, a month. Both are short enough to hold one owner's attention.

The six numbers

Add a control: cycles run as planned, on the date, with the named owner. A pilot that skipped a cycle has three data points.

The trend that decides it

Production time drops in cycle one and says little after that. The number that decides a pilot is checking time across the four cycles, read next to the rejection reasons. You would not hand a new analyst the forecast without a manager reading their work for the first few months, and you would expect that reading to get lighter as they learn the business. Hold a tool to the same standard. If checking time falls and the same rejection reason stops coming back, corrections are sticking, and in time some low-risk items can move to sampled review under your decision rights. If checking time is flat by cycle four and the same reasons repeat, the approver is learning to work around the tool. The work has moved from producing to checking and settled there, and the team is now operating a tool instead of handing off a job.

Three outcomes, decided first

Pass means every criterion is met by cycle four, and the routine moves to the tool with the same owner and approver. Extend means most criteria are met and one is close and improving, which earns two more cycles, once, with one named change. Stop means a stop rule fired, or the extension failed.

Write these before you start. A 40% cut in hours sounds good until someone says they hoped for 70%. If the number was set before cycle one, nobody has to argue.

Why a pilot that works in a demo still fails

Approval fatigue. The approver sees twenty suggestions a cycle and starts clicking approve. By cycle three the approval rate is 100%, which looks like success. It may mean the tool is perfect. More often it means nobody is reading. A run of zero rejections is a warning that the approval has become a formality. The approver needs the reasoning and the sources in front of them so they can judge each request before they sign it.

Unmeasured review time. The tool produces the draft in minutes, so the pilot report says "minutes." The two hours of reading appear nowhere, and in most finance routines that is where the remaining cost sits. A common pattern, illustrative rather than measured: the vendor's pilot summary leads with suggestions accepted and minutes saved per item. Both figures look best when nobody is reading closely.

No record of what it got wrong. If rejected suggestions are thrown away, the same mistake returns next cycle and the approver fixes it again. A correction only lasts if it is written down and applied to the next cycle.

No owner after the first month. The sponsor moves on. The pilot keeps running because nobody stops it, and nobody can say what it achieved.

Step-by-step process

1. Pick one routine. It must run on a fixed schedule and produce an output someone checks. "Score last month's forecast" qualifies. "Help with month-end" does not. Owner: CFO.
2. Measure the baseline. Log the last three cycles: hours to produce, hours to check, errors found at check. If you have no log, run one more cycle by hand and time it. Owner: whoever does the routine today.
3. Name the owner and the approver. The owner runs the pilot and keeps the sheet. The approver signs off every change the tool proposes before it enters a forecast, a payment run or a report, and must be able to judge the work. See decision rights for AI actions.
4. Write the rules and the cost lines. A figure for each of the six numbers, the extend range, the stop rules, licence or usage per cycle, a loaded hourly cost and setup hours. The CFO signs the charter.
5. Run each cycle with sign-off. The tool proposes. The approver approves, rejects or sends back each proposal before anything is used.
6. Log time as it happens. Producing and checking separately, including any parallel run of the old process. Estimates made at month end are always low.
7. Record every rejection. One line on what was wrong and what the approver changed.
8. Decide against the charter. After cycle four, read each number against the rule, not against anyone's hopes. Record pass, extend or stop with the reason and the date.

A passed pilot then goes into the AI risk register and review cadence, so someone keeps looking after the novelty fades.

Worked example: a forecast-scoring pilot

Illustrative. The routine, figures and costs below are invented to show the method. They are not benchmarks.

The baseline

Each month an FP&A analyst scores last month's forecast against actuals on 40 lines, and the CFO checks the result before it goes into the monthly pack. (The scoring method is in forecast bias vs. forecast accuracy; the free forecast accuracy scorecard computes the measures from pasted rows.) Averaged over three months by hand: 5.0 hours to produce, 1.0 hour for the CFO to check, 6.0 hours in total, and 3 errors found at check out of 40 lines (7.5%). At an illustrative blended cost of $90 an hour, one cycle costs 6.0 × $90 = $540.

The rules, written before cycle one

Extend once, for two cycles, if total hours land above 3.0 and up to 4.5, or the reject rate from 20% to 35%. Stop if an error the check missed reaches the monthly pack, if total hours exceed 6.0 in cycle three or four, if the owner's seat is empty for a cycle, or if the reject rate is above 35% at cycle three. Pause if there are two cycles in a row with zero rejections, and have someone other than the approver re-check a sample. Any other result at cycle four, such as total hours between 4.5 and 6.0, is a stop.

Licence or usage: $150 per cycle. Setup: 6 hours. Both illustrative.

The four cycles

The tool made 20 suggestions per cycle: lines to flag, causes to test, corrections to carry forward.

Decisions recorded counts every rejection and send-back, each with a one-line reason, plus the corrections the approver chose to carry forward.

The full cost

Check: four baseline cycles cost 4 × $540 = $2,160, and $2,490 − $2,160 = $330. At the cycle-four rate the saving is $540 − $375 = $165 per cycle, so the extra $330 comes back in two more cycles, if cycle four holds.

What the numbers say

The demo headline: production time fell from 5.0 hours to 0.5, a 90% cut.

The honest one: total time fell from 6.0 hours to 2.5, a 58% cut ((6.0 − 2.5) ÷ 6.0). Checking time doubled, from 1.0 hour to 2.0, and is now 80% of the work (2.0 ÷ 2.5). All-in cost per cycle fell 31% ((540 − 375) ÷ 540). And over the pilot itself, setup included, the business paid $330 more than it would have by hand.

That last line is uncomfortable and normal. A pilot is an investment paid for mostly in checking time. It pays back only if checking time falls or holds, and the only way to know is to keep counting it.

The blended rate hides one more thing. The checking is the CFO's time. Before the pilot the CFO spent 1.0 hour a cycle on this routine; by cycle four, 2.0. The analyst's hours fell by 4.5 and the most senior hours in the routine doubled. That does not change this decision, because the charter set one blended rate before cycle one. It should change the next charter: price checking at the checker's loaded cost, because it is usually the most expensive hour in the routine and the hardest to find more of.

The decision after cycle four

Every criterion was met: 2.5 hours against 3.0; error rates of 5.0% and 2.5% against 7.5%; a 10% reject rate; every rejection logged; $375 against $540; four cycles, same owner. The CFO signed and dated the decision: pass, with conditions. The analyst stays as owner. The CFO stays as approver of every change the tool proposes before it enters the pack. Checking time stays on the sheet for another quarter, and if total hours rise above 3.0 in any cycle the extend rules apply again.

The decision took five minutes, because every figure in it was set before cycle one.

A pilot that stalled

This is a composite, illustrative pattern. It describes no single company.

A finance team pilots a tool on its weekly review of AP exceptions: duplicate invoices, price mismatches, invoices with no receipt. The controller sponsors it and tells the AP lead to "use it and see." Nobody times the review beforehand. Nobody writes down what would count as a pass.

The AP lead approves suggestions one by one for two weeks, then in batches. By week five the approval rate is 100%. She also re-checks the whole exceptions list by hand every Monday, just in case, and those hours go on no sheet. In week six the controller moves on to the budget.

At renewal, someone asks whether it saved time. The AP lead thinks so. The controller is not sure. The tool's log is complete, every suggestion and every approval timestamped. It recorded everything that happened and was responsible for none of what came next: nothing in it says whether the pilot passed, who owned it, or who should decide. The licence renews by default, and the Monday re-check carries on.

A pilot without a baseline cannot fail. That is also why it cannot succeed.

Templates

The one-page pilot charter

If it needs a second page, the routine is too big for a first pilot.

The four-cycle measurement sheet

The parallel-run column is the one most pilots leave out. If anyone redoes the work the old way, those hours are part of the cost.

Metrics and KPIs for a first AI pilot

Keep checking share on the sheet after the pilot ends. If it creeps up, either the output is getting harder to trust or the approver is reading more carefully, and you want to know which. To set licence against people hours across your planning stack, the free planning tool cost calculator puts them side by side.

Common mistakes

1. Starting without a baseline. Without the before number, every after number is an opinion.
2. Counting time saved and not time spent. Production hours fall fast; checking hours, usually senior ones, often rise. Log both.
3. Leaving the parallel run off the sheet. If someone redoes the work by hand "just in case," the pilot is costing double. Count those hours, or end the parallel run with the approver's agreement. Quietly doing both is the most expensive option and the least visible one.
4. Reading 100% approval as success. Sample the approved items and see what a careful reader would have caught.
5. Throwing away the rejections. Without them the approver corrects the same thing every cycle.
6. No named approver. If the tool's output goes into a forecast or a payment run without a named person approving it first, you have handed over a decision without deciding who answers for it.
7. Moving the pass mark after the results. Then it was never a test.

AI prompt assistance

Use a general AI assistant to draft and check the paperwork, not to judge the pilot. Give it figures you have already logged, and do not paste confidential data into a tool your company has not approved.

1. Draft the charter from your notes.

Here are my notes on a planned pilot: the routine, its frequency, the last three cycles' hours to produce and check, errors found at check, the owner, the approver, licence cost per cycle and loaded hourly cost. Draft a one-page charter with these fields: routine, baseline, owner, approver, pass criteria, extend and stop rules, pause rule, cost lines, decision. Where I have not given a figure, write "TO SET" instead of suggesting one.

Check: every figure in the draft comes from your notes, and the CFO sets each "TO SET" before cycle one.

2. Recompute the cost table.

Here is my four-cycle measurement sheet with hours, licence cost, setup hours and a loaded hourly cost. Recompute total hours, checking share, all-in cost per cycle, the pilot total including setup, and the difference from baseline for each. Show every calculation and flag any cell that differs from what I entered.

Check: recompute one cycle and the pilot total by hand. If they differ, find the break before anyone sees the result.

3. Group the rejection reasons.

Here are the rejection and send-back reasons from four cycles. Group them into recurring types and count each type by cycle. List any type that appears in more than one cycle. Do not say whether the pilot passed.

Check: the approver confirms the grouping. A type that repeats is a correction that did not stay corrected.

Implementation guide

The hardest part of the first fortnight is an honest baseline, checking included. During the cycles, the usual failure is a sheet that goes stale; review it at each cycle's close, not at the end.

Governance: the CFO owns the standard and signs the decision. The pilot owner owns the sheet and the timetable. The approver owns every approval, rejection and send-back, and nothing the tool proposes is used before that person signs off. For the wider policy, see AI governance for finance and operations.

Downloadable assets

The pilot charter and the four-cycle measurement sheet above are designed to be copied into a document or spreadsheet. For a forecast-scoring pilot, the free forecast accuracy scorecard produces the baseline measures in your browser. For the cost lines, the free planning tool cost calculator sets licence against people hours.

Related resources

• AI trust and governance hub: every page on trusting a system that acts.
• How to trust an AI finding: what the approver should see before signing off.
• Decision rights for AI actions: who approves what, and when review can move to sampling.
• AI risk register and review cadence: where a passed pilot goes next.
• AI use case prioritization framework: choosing which routine to pilot.
• Forecast bias vs. forecast accuracy: the scoring method in the worked example.
• AP process design: approval, matching and payment runs: the routine behind the stalled pilot, done well by hand.

Frequently asked questions

How long should a first AI pilot in finance run?

Four cycles of one routine: three to four months for a monthly routine, a month for a weekly one. Fewer gives you no trend. Many more lets the owner's attention drift before anyone decides.

What should you measure in an AI pilot?

Hours to produce and hours to check, against a baseline measured by hand first. Add the error rate in checked output, the reject and send-back rate, decisions recorded with a reason, all-in cost per cycle including licence or usage, and whether each cycle ran on time with the named owner.

Why count checking time separately?

Because that is where the remaining cost sits. In the illustrative example production time fell 90% while checking time doubled, so the real saving was 58%. A pilot that reports only production time overstates the result.

Is a 100% approval rate a good sign?

Not on its own. It can mean an excellent tool or an approver who has stopped reading. After two cycles with no rejections, have someone else re-check a sample of what was approved.

What is the difference between an AI proof of concept and a pilot?

A proof of concept shows the tool can do the task, usually once, on sample data. A pilot shows whether doing the task with the tool, checking included, beats your current routine on your own data over several cycles. One asks "can it?" and the other "should we?"

How this works in a System of Performance

Most pilot tools keep a log of every suggestion and every click, and nothing in that log is responsible for whether the pilot passed or for what happens after. A Performis routine can take on a recurring job like this, with the scope and fit agreed with you: it works from the records your team supplies, prepares the work for your named reviewer, and keeps each approval on record. To see where your own numbers stand first, bring it to us as a launch partner.