Back to all articlesForecasting discipline

Check Whether High-Confidence Forecasts Actually Close More Often

Alactic Editorial9 min read

Prepared with AI assistance by Alactic Editorial. Examples identified as hypothetical are illustrative, not customer results.

Your team labels opportunities high confidence, but does an estimate of 80% actually correspond to roughly eight wins in every ten comparable opportunities? To check, preserve the probability assigned at a specific forecast date, define exactly what must happen by a fixed deadline, and compare those estimates with verified outcomes after that deadline.

The key is to test the prediction that was made, not the probability currently displayed in the CRM. A probability edited after a deal slips or closes cannot tell you whether the original forecast was reliable.

This review is about probability calibration: whether stated likelihoods match observed frequencies. It is not a complete assessment of revenue forecasting, sales productivity, or deal quality. Start with a small, auditable worksheet before changing stage weights or asking the team to use different percentages.

1. Define the event before extracting the data

A probability is incomplete without an outcome and a time horizon. “Likely to close” could mean this month, this quarter, or eventually. Those are different predictions, even when they refer to the same opportunity.

Write one testable statement for the review. For example, in a hypothetical review:

For each eligible opportunity open at the quarter-start snapshot, the recorded probability estimates the chance that it will reach our agreed closed-won milestone by that quarter’s end.

Then specify the milestone. Your team might use a signed agreement, an accepted order, or another consistently recorded commercial event. Choose the definition that matches what the forecast was supposed to predict; do not choose retrospectively based on which produces the best result.

Record these decisions at the top of the worksheet:

  • Forecast population: which opportunities were eligible at the snapshot.
  • Snapshot timing: when the probability was preserved.
  • Target event: what counts as a win.
  • Deadline: when the event must occur.
  • Unit: one opportunity, counted once within the cohort.
  • Probability meaning: period-specific closing likelihood, not eventual win likelihood.

If the historical percentage represented eventual win probability, do not compare it with quarter-end wins and call the difference overconfidence. Either recover the original intended horizon or begin a prospective review with an explicit definition.

2. Preserve the forecast, not the latest version of it

Build a row-level archive containing opportunity ID, snapshot date, original probability, target deadline, outcome evidence, and the date the target event occurred. Keep the original estimate unchanged. Put later corrections and outcome records in separate fields.

You can assemble this in a spreadsheet from authorised records. The operating requirement is a reproducible snapshot, not a particular software feature.

For a first review, use one snapshot per opportunity. Repeated weekly snapshots can be useful later, but they answer a different question: whether forecasts made at each weekly distance from the deadline were calibrated. Counting every weekly record as an independent opportunity gives long-running deals disproportionate influence.

Include opportunities that later disappeared from the active pipeline, were reassigned, or were marked lost. Selecting only the deals still visible today can remove exactly the failures the review needs to examine.

If you have no historical probability archive, do not reconstruct it from memory. Preserve the next forecast now and schedule the outcome check after its deadline. Current deal notes can explain a result, but they are not a substitute for a contemporaneous estimate.

3. Resolve horizon mismatches and unknown outcomes first

Before calculating a win rate, separate three situations that are often mixed together.

The deadline has not arrived

The forecast is not yet assessable. Keep it in a pending cohort. Treating an opportunity as a failure while it still has time to close will make probabilities appear too high.

The deadline passed and the deal remained open

For a prediction of winning by that deadline, this is a known non-win, provided records reliably establish that the milestone had not occurred. It is not an unresolved outcome merely because the opportunity is still commercially active. A later win belongs outside the original period’s success count.

The deadline passed but evidence is missing

This is genuinely unknown. An outdated CRM status may not establish whether a contract was signed before the deadline. Investigate the source record rather than automatically treating the opportunity as lost or dropping it without explanation.

Show unknown counts beside the results. If missing evidence is concentrated among high-probability deals, a resolved-only comparison may be misleading. You can calculate bounds by treating all unknowns first as non-wins and then as wins. If either assumption changes your proposed decision, resolve more records before changing weights.

Also separate fixed-period forecasts from rolling-horizon forecasts. An estimate of closing within the next month is not directly comparable with an estimate made shortly before quarter-end for closing within the quarter. Do not pool them simply because both records contain a percentage.

4. Use a filled calibration worksheet

Fictional worked example: All dates, counts, probabilities, and results below are hypothetical. They are not customer results or benchmarks.

A team preserved quarter-start estimates for opportunities open on 1 April, with a target of reaching its agreed closed-won milestone by 30 June. The review takes place on 10 July. A separate July-start cohort has not reached its deadline and is kept out of this review.

HYPOTHETICAL CALIBRATION WORKSHEET

Prediction: closed-won milestone reached by 30 June
Snapshot: 1 April
Review date: 10 July
Unit: one opportunity at the snapshot
Weighting: one opportunity, one vote; not deal value

Cohort reconciliation
Archived April opportunities:                 208
Verified period-specific outcomes:           200
Unknown outcomes awaiting evidence:            8
Separate July cohort not yet due:             24

Resolved-outcome calibration
Band       N   Mean p   Expected wins   Actual wins   Observed   Gap*
0–29%     30    20%          6.0             9          30%     +10 pp
30–49%    40    40%         16.0            18          45%      +5 pp
50–69%    50    60%         30.0            27          54%      -6 pp
70–89%    50    80%         40.0            34          68%     -12 pp
90–100%   30    95%         28.5            24          80%     -15 pp
TOTAL    200   60.25%      120.5           112          56%   -4.25 pp

*Gap = observed win rate minus mean predicted probability.
 pp = percentage points.

Unknown records, excluded from the table above
0–29%: 0 | 30–49%: 1 | 50–69%: 1 | 70–89%: 2 | 90–100%: 4

Verified non-wins among the 200 resolved records
Closed-lost by deadline:                       58
Still open at deadline, with no milestone:     30
Total non-wins:                                88

Follow-up owner: forecast operations lead
Next task: resolve the eight unknown records and refresh the worksheet
Decision status: investigate high-band gaps; retain current weights

For each band, calculate the mean of the actual archived probabilities, not the band midpoint. Expected wins equal the sum of those probabilities expressed as decimals. An expected count can be fractional; it is an aggregate expectation, not a claim that part of a deal closed.

Observed win rate equals verified wins divided by opportunities with verified period-specific outcomes. A negative gap indicates that observed wins fell below the forecast likelihood. A positive gap indicates that they exceeded it.

The total row is useful, but it is not enough. Underconfidence in lower bands can partly cancel overconfidence in higher bands, leaving an acceptable-looking overall average.

5. Read the pattern without overstating the evidence

In the hypothetical worksheet, high-confidence opportunities did close more often: the highest band won at 80%, compared with 30% in the lowest band. That is useful ordering. It does not establish calibration. The highest band’s mean prediction was 95%, so its observed result was 15 percentage points lower.

These are distinct questions:

  • Ordering: do opportunities assigned higher probabilities win more often?
  • Calibration: do the observed frequencies match the probabilities assigned?

A team can rank opportunities sensibly while overstating their likelihoods. It can also have a reasonable overall average while failing to distinguish stronger opportunities from weaker ones.

Missing evidence does not fully explain the highest band’s hypothetical gap. Among its 30 verified opportunities, 24 won. Including its four unknown records would produce a win rate between approximately 70.6% and 82.4%, depending on their outcomes. Even the upper bound remains below that band’s original predicted range. Recovering their individual probabilities is still necessary to calculate the exact updated gap.

Small groups require additional restraint. In a hypothetical group of five opportunities, one extra win changes the observed rate by 20 percentage points. Do not translate a dramatic-looking gap in such a group directly into a new weight. Show the count, retain the individual records, and gather comparable cohorts. If reporting uncertainty intervals, use a documented statistical method and remember that shared customers, sellers, or market conditions can make outcomes dependent.

6. Diagnose the mismatch before changing weights

Start with records from the bands showing the largest gaps. Ask whether the mismatch reflects probability judgment, timing assumptions, or unreliable data.

  • Wrong probability meaning: sellers estimated eventual success, while reviewers counted period-end wins. Repair the definition before evaluating accuracy.
  • Late wins: opportunities eventually closed but missed the deadline. Investigate timing evidence rather than treating those later wins as proof the original forecast was right.
  • Stage defaults: many opportunities inherited the same probability regardless of deal-specific evidence. Check whether that default represented the intended event and horizon.
  • Selection changes: inclusion rules changed between snapshots, or deleted opportunities vanished from the export. Rebuild the cohort where possible.
  • Outcome-recording delays: milestone dates and CRM update dates were confused. Use the event date supported by evidence.

Segment only when you have a concrete reason to expect a different relationship, such as different forecast horizons or sales processes. Splitting immediately by every seller, product, and deal size can create tiny groups that invite overinterpretation.

If an audit finds that the historical data cannot support the test, label the review inconclusive. Preserve the defects found, fix the next snapshot process, and collect a clean cohort. An inconclusive result is more useful than a precise-looking number built on incompatible predictions.

7. Test a correction on future forecasts

A calibration gap is a reason to investigate, not an instruction to replace an 80% weight with the latest observed 68%. That observed rate came from one hypothetical group and may reflect temporary conditions or chance.

If comparable completed cohorts show a persistent gap, document a candidate adjustment. State the affected population, target event, horizon, supporting counts, and exceptions. Keep the original probability alongside any adjusted estimate so the next review can compare both honestly.

Evaluate the adjustment on future outcomes, rather than only on the same records used to choose it. Changing probabilities until they match the historical worksheet demonstrates fit to that worksheet, not improved future calibration.

Also keep deal-count calibration separate from revenue forecasting. This worksheet gives each opportunity equal weight. A well-calibrated opportunity model can still produce an inaccurate revenue forecast if deal amounts or timing are wrong. Do not claim that correcting one automatically corrects the other.

8. Finish with an auditable next action

For the next review, choose one completed cohort with a preserved forecast and a common deadline. Produce the worksheet with original counts, unknown outcomes, mean probabilities, expected wins, observed rates, and gaps. Assign an owner to every missing outcome.

Record a decision for each band: retain, investigate, or test an adjustment. Include the evidence behind that decision and the date the next comparable cohort becomes assessable.

The immediate success measure is a reconciled cohort and a reproducible comparison—not a smaller gap achieved by rewriting old estimates. Once those foundations hold, repeated reviews can show whether high confidence is justified, consistently overstated, or more cautious than the outcomes warrant.

Sales forecastingProbability calibrationCRM disciplineForecast review

Put the next conversation to work.

Explore Alactic Communications for WhatsApp conversations, campaigns and AI-assisted workflows. Choose the product and plan that fit your team.

Explore Communications