Can the AI pilot pass both gates?
Check whether a prototype improves the workflow it needs to serve.
The question
Two AI versions scored the same labeled tasks. The pilot requires at least 85% overall and at least 80% on multi-turn tasks. The first report leaves out the multi-turn results.
| Current version | Prototype | |
|---|---|---|
| Tasks scored | 120 | 120 |
| Correct results | 96 | 108 |
Both versions used the same tasks and scoring rule.
“Would you put the prototype into the pilot?”
Not saved by this site. Your browser may restore text when you navigate back.
A strong answer, step by step
Open each step in turn. This is one strong answer, not the only one.
Step 1: Read the overall gate
What would you do here? Open the move you’d make.
Record the overall pass and leave the pilot decision open.
The strong move. The pilot requires both the overall and the multi-turn score to meet their own threshold.
Approve the prototype on its 90% overall score.
Tempting, but it misses something. The reported 90% gives no result for the required multi-turn subset.
See how a strong candidate says it
“The current version gets 80%. The prototype gets 90%. The prototype passes the overall gate. I still need the multi-turn result.”
Why it works: The pilot requires both the overall and the multi-turn score to meet their own threshold.
Step 2: Request the subset
What would you do here? Open the move you’d make.
Ask only whether the overall score includes multi-turn tasks.
Tempting, but it misses something. Including the tasks in the total does not tell you their own pass rate.
Get the matched multi-turn counts and the scoring rule.
The strong move. A matched subset shows whether the prototype clears the workflow requirement.
See how a strong candidate says it
“I’d ask for the number of multi-turn tasks and the correct count for each version, using the same tasks and pass rule.”
Why it works: A matched subset shows whether the prototype clears the workflow requirement.
Step 3: Check the second gate
The evaluator finds 20 multi-turn tasks. The current version gets 17 right and the prototype gets 12. The same scoring rule applies.
What would you do here? Open the move you’d make.
Hold the pilot because the prototype misses the subset gate.
The strong move. The overall gain can occur while performance falls on a smaller required workflow.
Trade the overall gain against the multi-turn loss and approve.
Tempting, but it misses something. Each threshold applies on its own. The overall score cannot make up the multi-turn shortfall.
See how a strong candidate says it
“The current version scores 85% on multi-turn and the prototype scores 60%. The overall result rose 10 percentage points and the multi-turn result fell 25. The prototype misses the 80% multi-turn gate.”
Why it works: The overall gain can occur while performance falls on a smaller required workflow.
Step 4: Give the next check
What would you do here? Open the move you’d make.
Report the failed gate and ask for a matched retest after failure review.
The strong move. Neither version clears both gates. A matched retest can show whether a revision fixes the specific gap.
Use the current version for the pilot because its subset score passes.
Tempting, but it misses something. The current version’s 80% overall score is below the 85% overall gate.
See how a strong candidate says it
“I’d keep the current version as the comparator, look at the failed multi-turn tasks, then evaluate both versions again under the same definitions.”
Why it works: Neither version clears both gates. A matched retest can show whether a revision fixes the specific gap.
The answer, in short
- Decision
- Hold the pilot and review the multi-turn failures.
- Because
- The prototype passes the overall gate at 90% and misses the multi-turn gate at 60%.
- What would change it
- A matched evaluation that meets both thresholds would support a pilot recommendation.
Common traps
- Treating a combined rate as evidence for every workflow.
- Changing a gate after seeing a failed result.
Check your answer
- I computed both overall rates.
- I asked for the matched subset.
- I checked each gate separately.
- I gave the next evaluation.
Last reviewed Nov 11, 2026