Quantcast
Channel: Is a mod unilaterally reviewing/closing 1500 questions in a single day okay, or too much? - Meta Stack Overflow
Viewing all articles
Browse latest Browse all 10

Answer by gnat for Is a mod unilaterally reviewing/closing 1500 questions in a single day okay, or too much?

$
0
0

I sampled and checked 100 of these 1500 reviews (boring sampling details are at the bottom of this answer for those interested).

To me, all of the sampled closures looked deserved (at the time of closing - that is, ignoring edits made to questions after closure).

The vast majority of close reasons looked accurate, except for 4 cases where I could imagine an alternative close option but wasn't sure if it would really be a better match, and 1 case where it felt really off and I would certainly pick a different close reason (given accuracy of reasons in other questions in my sample selection, this one looked more like an occasional misclick).

If the rest of the reviews outside of my sample size are like that, then the overall closing accuracy looks fairly acceptable to me.

As a side note, prior to starting my study I was going to support Shog and second the recommendation to turn off suppressing the audits, because–to me, personally–audits make an invaluable tool to monitor and control focus and quality of my reviews. However, observing the results of this study made me change my mind and abstain from recommending this because, as I wrote above, the quality of reviews looked acceptable as is.


Studying these reviews helped me address another interesting concern raised here, about an insufficient amount of prior close reviews.

What was special about the questions I checked is that all of them looked really clear-cut as close-worthy; very easy to decide. For the sake of completeness, one (only one!) didn't look like that to me, but when I took a closer look into it, I noticed that the reviewer has a gold badge in the question tag, meaning that–to them–it was just as easy and clear as other questions were to me.

Overall, it looked like the reviewer thoroughly skipped all questions that could be a gray area where one could have even a hint of doubt about whether it is close-worthy or not. Given the amount of reviews, I wouldn't be surprised if part of these skips were automated (say by a bot that iterates over available reviews and automatically clicks skip based on some reverse-triage heuristics, so that when one reviews manually, they have most of complicated stuff already filtered out by skips made by this bot).

Okay, now this observation made me recall that there is another review queue where one can gain relevant experience reviewing stuff like that - Triage. The Triage queue focuses on questions of exactly the kind I just checked here.

And then, it just dawned on me that this user has more than enough prior experience in triage reviews, which is really most relevant to the kind of questions I checked. This kind of completed the puzzle by making it totally understandable how they could review these 100 questions with such speed and accuracy.


Sort of a follow up, further discussion in the comments under the question made me wonder if we can somehow utilise experience of this extraordinary review marathon in more routine, typical workings of the review system. And one thing that caught my attention here is how easy it turned out to correctly perform Triage-like reviews—it felt like a miracle when I saw that the accuracy of reviews at hour 15 looked about the same as hour 1.

This naturally brings a thought: what if regular reviewers had an option to choose some sort of "easy mode" in the Close Vote Review queue where the system would only feed them reviews for questions scoring the lowest 10-20% by some quality metric? (The metric(s) currently used in selecting questions for Triage would probably work well enough)

Probably folks using such a mode would be inclined to keep reviewing over a longer term, which in turn could help us solve the old, painful problem of terrible attrition of Close Vote Reviewers,

[...] many users find it difficult to work in review queue. Drop after 250 (silver badge) suggests that even after substantial amount of reviews, many users still fail to discover a way to work productively


Finally, as promised above, here is an explanation of how I sampled reviews for this check.

I went to the Reviews tab in the user's profile (it's public by design; here is an example of how it looks in my profile). I observed that every page in this tab lists 20 reviews, so 1500 reviews would occupy 75 pages or a bit less if some questions were deleted.

I decided that I want to check 100 reviews at 5 more or less uniformly distributed pages. For that, I picked pages 1, 16, 32, 48, and 64 (page 68 and higher turned out listing yesterday's and older reviews, i.e. out of the scope of my check). I opened all links to questions on these pages (total 100) and studied these.


Viewing all articles
Browse latest Browse all 10

Trending Articles



<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>