Given who it is, I have to assume this is scripted to some degree. Also because, y'know, that's a LOT of reviews to do entirely by hand.
Sam's got a ReviewQueueHelper script, so let's assume that's what's being used here:
- Keyboard shortcuts for review action buttons
- Keyboard shortcuts for popup action dialogs (close, flag, delete, reject edit)
- Auto-focus submit button when option selected in popup action dialogs (e.g.: so you can go "2 > 2 > space" to submit)
- Automatically skips review audits
- Esc goes back a previous dialog pane
- More options available when SOMU Options userscript is installed
Keyboard shortcuts (and Esc, auto-focus) are good - that avoids a lot of potential mistakes and even hand/eye fatigue hunting for options to click. 👍
Auto-skip audits is... Not so great. 😬 Not that the audits in the close queue are particularly good either, but with that many reviews the opportunity for mental fatigue is extremely high.
Some effects I've observed in the past in folks doing this volume of question evaluation:
- over-reliance on heuristics (grammar, spelling, keywords, the visual "shape" of a question)
- decision hysteresis (once one action is taken, it stands a higher chance of being applied to the next task, requiring a much better / much worse question to appear in order to trigger a change)
- plain ol' sloppiness (hitting the wrong key, double-tapping, etc)
Audits aren't the only way to avoid these issues, or even the best ones (time-delays go far). But I'd be concerned about anyone attempting this volume without some sort of fatigue-check in place.
Sam, if you read this: I strongly recommend slowing down and taking a bit of time to implement some fatigue checks in your script.
For others: if you have 10K, you should be able to review reviews. Don't try to review 1500 of them, take samples (10 from one place maybe 900 or so reviews in, 10 more distributed randomly) and check those. That's why the functionality to view the complete review history exists after all!
One thing's for certain: SO is in absolutely no danger of running out of questions to review. Realistically, the answer to this question must come down to whether the effects are acceptable or not - which is gonna require manually auditing the reviews that are being done.