Why generic rubrics fail
Two experienced teachers evaluating the exact same physics script will often mark differently. One teacher awards a method mark when the algebraic working is nearly complete, while another insists on seeing the final units written explicitly. One writes three full sentences of feedback, while the other writes five concise words.
Neither teacher is incorrect. Every academic department develops internal grading conventions through years of departmental moderation. When an automated grading tool ignores those conventions, teachers spend more time rewriting AI comments than they would have spent marking by hand.
Ren learns departmental standards through two complementary mechanisms:
- The teacher style guide. Builds gradually over time by tracking a teacher's edits across assignments.
- Assignment calibration. Runs immediately on a small sample of papers to align rubric interpretations before marking an entire cohort.
How the continuous style guide learns
Whenever a teacher reviews and approves a batch in Ren, the system compares the original AI suggestions against the finalized feedback.
The system tracks five distinct teacher actions as training signals:
- Editing an AI-generated feedback comment.
- Changing a suggested numerical mark.
- Modifying or adding diagnostic tags.
- Rejecting a suggestion entirely.
- Writing a new margin note where the AI left none.
A background process aggregates these signals for each teacher. To prevent knee-jerk overreactions, the engine requires at least three consistent signals before updating a style rule.
Ren maintains two independent style guides per instructor: one for structured questions and one for essay papers. Preferences on partial credit for a two-mark calculation should never contaminate an essay rubric.
When updating a guide, the engine regenerates it from scratch against all recent signals. It drops any outdated instructions that conflict with new teacher edits, preventing the style guide from accumulating contradictory legacy rules.
Why the guide stays concise
We cap style guides strictly at 600 words, six bullet points per section, and three examples across four core areas: General Preferences, Scoring Rules, Feedback Style, and Examples.
This constraint is practical rather than aesthetic. The style guide gets injected directly into the LLM context window alongside the exam rubric, and prompt space has direct consequences on model attention.
In our internal testing, appending 570 characters of irrelevant text to a grading prompt caused final cohort scores to shift by an average of 1.2 marks without any change to the rubric. Context pollution degrades grading accuracy.
Keeping the guide concise ensures that every rule earns its place without crowding out the official mark scheme. Furthermore, the style guide remains subject-agnostic. It specifies general evaluation principles (such as "award method marks if the approach is sound despite arithmetic slips") rather than paper-specific topic facts.
Calibrating rubrics before batch marking
While the style guide learns general teacher preferences over months, calibration solves paper-specific interpretation issues in ten minutes.
Before grading a 300-student batch, a teacher can choose to calibrate the assignment. The teacher grades two or three sample scripts manually.
Ren then evaluates those same sample scripts and compares its suggestions against the teacher's markings across every question. Wherever discrepancies appear, Ren generates structured proposals to update the assignment's answer key, such as clarifying acceptable alternative phrasing, adjusting partial credit thresholds, or refining component definitions.
Requiring teacher sign-off on rubric proposals
Automated calibration never modifies a live rubric on its own.
Every proposed adjustment is staged as pending. The teacher reviews each proposal, inspects the side-by-side evidence from the sample scripts, and explicitly approves or rejects the change. Only after teacher sign-off does the system apply the revised rubric to the rest of the cohort.
Assignments progress through explicit states: awaiting calibration, marking, analysing, under review, and released. If a calibration run produces zero proposals because the AI matched human grading perfectly, the system pauses at review and asks the teacher to confirm before proceeding.
Handling analysis failures cleanly
A calibration pass makes independent LLM calls for each question on the paper. If network timeouts or schema errors cause these calls to fail, a naive system might interpret zero generated proposals as "no changes needed" and proceed with batch grading.
That would produce false confidence. In Ren, if any question-level calibration call fails, the entire analysis job halts, rolls back, and flags the error. A teacher never receives an empty proposal list that resulted from a silent system failure.
Instruction hierarchy
When generating marks, Ren applies instructions in a strict four-layer hierarchy:
- Subject instructions. Global conventions shared across the entire subject discipline.
- Teacher style guide. The learned preferences for this specific educator.
- Teacher notes. General freeform instructions the teacher saved to their profile.
- Assignment marking instructions. Specific rules set for this exact exam paper.
Paper-level instructions sit at the top of the hierarchy, overriding general style preferences. If a teacher writes "accept either US or UK spelling on this vocabulary paper," that rule takes precedence over a general preference set months earlier.
Keeping teachers in control
None of these systems treat the AI as an independent grading authority.
The style guide is generated from the teacher's own edits. Calibration proposals derive from the teacher's own marks on sample scripts, requiring explicit approval. Paper instructions come directly from the educator.
Ren's purpose is to apply that teacher's exact standard consistently across 500 scripts without succumbing to late-night fatigue. The judgment remains with the educator.
For more on our system design, read why AI marking still needs human oversight and our research on why confidence scores are not a safety net.