What can you still tell the interview panel?
Revise a research slide after checking the data behind its headline.
The question
You present a fictional visual-inspection study. The slide compares four predefined test sets with 100 examples per version in each set. The research team needs all four sets matched and a mean lift of at least 10 percentage points to claim its target.
| Reference correct | Prototype correct | |
|---|---|---|
| Set A | 80 | 85 |
| Set B | 75 | 79 |
| Set C | 86 | 89 |
| Set D | 52 | 96 |
The four sets have equal weight. The slide lists counts, without example IDs.
“What headline would you give the panel, and what might change it?”
Not saved by this site. Your browser may restore text when you navigate back.
A strong answer, step by step
Open each step in turn. This is one strong answer, not the only one.
Step 1: Open with the comparison
What would you do here? Open the move you’d make.
Tell the panel the study has achieved its target.
Tempting, but it misses something. The slide gives equal counts per set, with no evidence yet that the examples match.
Give the observed comparison with its matching condition.
The strong move. The average summarizes the reported rows. The study rule also needs matched examples.
See how a strong candidate says it
“The apparent mean lift is 14 percentage points. That headline depends on both versions using the same examples in all four sets.”
Why it works: The average summarizes the reported rows. The study rule also needs matched examples.
Step 2: Check what was scored
What would you do here? Open the move you’d make.
Check the data behind each row and prioritize the largest apparent change.
The strong move. D contributes 44 of the 56 points summed across the four reported comparisons.
Check the totals only, because all denominators are equal.
Tempting, but it misses something. The same denominator can describe two different sets of examples.
See how a strong candidate says it
“I’d check example IDs and data origin for both versions, starting with D because it drives most of the apparent gain.”
Why it works: D contributes 44 of the 56 points summed across the four reported comparisons.
Step 3: Revise the claim
The evaluator confirms A, B and C match. The prototype’s D score comes from a different synthetic evaluation set. The reference D score comes from the original set.
What would you do here? Open the move you’d make.
Keep the 14-point headline and label D synthetic in a footnote.
Tempting, but it misses something. D compares different examples. The footnote cannot make the original headline a matched comparison.
Report the three verified sets and leave the four-set claim open.
The strong move. A, B and C contribute 12 points to the sum. A four-set mean of 10 needs 40, so D needs 28 more points.
See how a strong candidate says it
“The three matched sets average a 4-point lift. The four-set result is still open. With D’s reference at 52, the prototype needs at least 80 correct out of 100 on that same set to reach the original target.”
Why it works: A, B and C contribute 12 points to the sum. A four-set mean of 10 needs 40, so D needs 28 more points.
Step 4: Correct the slide
What would you do here? Open the move you’d make.
Drop D and treat the three sets as satisfying the original study rule.
Tempting, but it misses something. The rule requires four matched sets. The three verified sets also average below 10 points.
Update the headline, explain the gap and ask for the matched D result.
The strong move. The panel sees what the study supports and what is needed to make the original claim.
See how a strong candidate says it
“I’d present the verified result on A, B and C, disclose why D is missing and ask for a matched D comparison before reporting the full result.”
Why it works: The panel sees what the study supports and what is needed to make the original claim.
The answer, in short
- Decision
- Present the verified three-set result and request matched D evidence.
- Because
- The apparent 14-point lift includes a comparison across different data.
- What would change it
- A prototype result of at least 80 correct on the same D examples, with the other figures unchanged, would reach the target.
Common traps
- Treating equal sample sizes as matched data.
- Keeping a headline after its key input changes.
Check your answer
- I named the matching condition.
- I checked data identity.
- I revised the reported lift.
- I stated what could restore the claim.
Last reviewed Nov 11, 2026