The question we must answer

The AI marks a paper. The teacher has limited time. Which marks must the teacher check first?

A good answer makes AI marking safe. The teacher checks a small set and trusts the rest. A bad answer wastes the teacher's time, or hides a wrong mark.

The obvious lever is the AI's own confidence score. Ren records a confidence value with every mark. We tested that value against production data. It does not work.

Confidence has almost no range

We examined more than 35,000 marks from production. Each mark carries a confidence score from 0 to 100.

| Statistic | Value |

|---|---|

| Mean | 95.4 |

| At or above 95 | 73.9% |

| At or above 90 | 90.6% |

| Below 80 | 1.55% |

Almost every mark scores high. No threshold selects a useful set. A cut at 80 selects 1.55% of all marks, which is too few to matter. A cut at 90 selects 9.4%, and most of those marks are correct.

Confidence points the wrong way

The direction is worse than the range. We compared the confidence score against the teacher's decision to reject the suggestion.

| Confidence | Share of the set | Teacher rejected |

|---|---|---|

| 30–70 | 1.6% | 0.0% |

| 80–89 | 6.9% | 0.4% |

| 90–99 | 39.1% | 0.9% |

| 100 | 52.5% | 4.8% |

The lowest band is thin, so read its zero with care. The pattern above it is not thin, and it is clear. The most confident marks are rejected most often. On divergent marks specifically, the mean confidence is 93.9 against a population mean of 95.4. That separation is noise.

A model is a poor judge of its own correctness. The confidence score is a number the model writes about itself, and nothing checks it.

The teacher's clicks are not ground truth

We first assumed the accept and reject record was a labelled dataset. It holds tens of thousands of rows. Roughly three in four are accepted, and one in four rejected. It is not usable as truth.

79.8% of rejections occur in same-second groups of five or more on one submission. The largest single group holds 65 rejections.
94% of rejections carry no score. 82% name no question.
Of the rejections that do name a question, only 13% carry a replacement comment. A rejection usually means "delete", not "correct".
Of all accepted suggestions, only 0.8% ever had the score edited.

These are bulk clicks. A teacher presses "accept all" or clears the page. The record measures the button, not the marking.

Most edits compress, they do not correct

We removed the bulk clicks and kept about 2,000 marks where the teacher demonstrably acted on the content. We then split them by whether a mark was at stake.

|---|---|---|---|---|

The essay half is teachers who compress AI prose into their own voice. One representative pair carries the same substance at very low text similarity:

AI — "The claim is an important evaluative conclusion, but it is asserted rather than supported by a concrete comparison (severity, likelihood distribution, number of people affected, or why certain harms dominate)."

Teacher — "Would have been better if this claim was supported with evidence"

This is a real product signal. The AI writes too much. It is not a signal that the AI is wrong. Any measure that reads a heavy edit as an error will chase this group and find nothing.

Where the errors actually are

We read 40 divergent marks by hand, against the AI output, the student script, and the marking criteria.

| Group | Read | Genuine AI error |

|---|---|---|

| Score moved by one mark or more | 14 | 12 (86%) |

| Comment rewritten, score unchanged | 8 | 1 (13%) |

| Essay comment rewritten, no score | 6 | 0 |

| Suggestion deleted | 6 | 0 |

| Suggestion rejected and replaced | 6 | 0 |

Score movement is the signal. Comment churn is not.

The dominant failure is over-credit. The AI awards a mark the criteria explicitly withhold. Four examples from the set we read:

The criteria state "do not award the mark if no reason is given". The student gave no reason. The AI awarded the mark.
The criterion requires the gradient of the tangent at two seconds. The student used the chord gradient over the whole interval. The AI awarded the method mark.
The criteria carry an explicit presentation deduction. The AI never applied it.
The question requires a final simplified form. The student stopped at the intermediate constants. The AI awarded full marks.

One class of rule never appears in the criteria at all. General Paper comprehension penalises text lifted from the passage. No marking criteria field states this rule. The AI cannot apply a rule that nobody wrote down.

Divergence is not proof of AI error

Two of the 40 cases went the other way. Both matter.

The scheme states the correct answer is D, with full working. The student wrote D. The AI marked it correct at confidence 100. The teacher overrode the AI and recorded "wrong, the correct answer is A". The AI was right.
The AI caught an arithmetic slip and withheld the mark correctly. The teacher waived it.

A teacher edit is not automatically a correction. This is why the corpus needs human adjudication, and why we do not derive a label from the clicks.

What works instead

Deterministic checks work. They read the AI's own output and search for internal contradictions. They need no second model and cost nothing to run.

The clearest check compares the score against the comment. One production example:

Comment: "+1: correctly shows the expansion
          +1: correct final expression"
Recorded score: 1
Confidence: 99

The comment awards two marks. The score records one. The teacher accepted it unchanged.

Here are the results on the adjudicated set. "Fires on" is the share of ordinary marks the check selects, so a lower number means a smaller review load.

|---|---|---|---|

| Confidence below 90 | 100% | 7.1% | 9.6% |

| Score against comment | 100% | 7.1% | 0.8% |

| All deterministic checks together | 100% | 21.4% | 3.3% |

The combined checks triple the recall of the confidence threshold. They also fire on a third as much of the population.

We then ran the score-against-comment check across every scored mark in production. About 40% of them use the itemised format the check can read. Within that readable share, the check found a contradiction in roughly one mark in fifty. The confidence score rated many of them at 95 or above.

One trap in the measurement

The check appears to perform worse than random if you score it against the accept and reject record. That number is an artefact of the label.

The score-against-comment check fires on a few dozen marks that teachers accepted without a change. We inspected them. Almost all are real defects. The check is right and the label is wrong.

This is the most important methodological point we drew from the work. You cannot evaluate a review flag against accept and reject data. That data comes from teachers who were not reading carefully. Careless reading is the exact problem the flag exists to solve.

The honest limit

Eleven of the fourteen known errors escape every deterministic check. Those errors are semantic. Each one needs a judgement about whether a student met a written criterion.

No rule makes that judgement. A dedicated verification pass can, and that is the next piece of work.

What this means for a teacher

Three things follow.

Do not read a confidence score as a safety rating. We do not use it to decide what a teacher must check, and we do not present it as an accuracy claim.

Check the marks where the score moved first. That is where 86% of the genuine errors sit. Comment style is worth your attention for a different reason, but it is not where the marks go wrong.

Treat the AI output as a first pass. Ren is built so a teacher reviews, edits, and approves every piece of feedback before a student sees it. This study is one more reason why. We wrote about the wider case for that design in Why AI Marking Still Needs Human Oversight.

Confidence Scores Are Not a Safety Net