Review as an instructor

Instructor reviews are how the system earns trust. Every review confirms or corrects the automated assessment, feeds the agreement statistic and becomes a training label.

Review a sortie#

1. Open the sortie and scroll to Instructor review.
2. For each category, leave Agree ticked if the automated score is right, or untick it and enter your own score and a comment.
3. For each event, untick Valid if it should not have been raised.
4. Click Submit review. The page shows the fleet's updated agreement.

You can review again later; the latest review from each instructor is kept, and earlier ones stay in the audit log.

How agreement is measured#

A category agrees when your score is within 10 points of the automated one. Debrief → Instructor agreement shows agreement overall and per category against the 95% objective, and the share of events marked valid.

Where your overrides go#

• Training labels. Reviews export as CSV for model and rubric work (GET /api/reviews.csv).
• Labelling queue. Disagreements, invalid events and weak tracking queue the sortie for labelling in the learning loop.
• Rubric tuning. When instructors consistently score a category differently, the tuning view proposes thresholds that would agree better, for an administrator to adopt as a new configuration version.

Good practice#

• Review before you read the comments of other instructors, so the agreement statistic stays honest.
• Write the reason for an override; it is the most useful label there is.
• Mark an event invalid only when it should not have fired, not when it was minor.