How is the remote review of notebooks handled? How do you as a JA control the amount of time a judge gets on each notebook? I am considering this option, but without the controls to keep the review times consistant, I will not go forward with this option.
On an aside, I have had a judge not follow instructions and used almost 2 hours reviewing notebooks when times were limited to 30 minutes. I had to ignore his score, but had I known…I would have tossed him immediately because he become a cancer during deliberations.
Hi @David_Cox I’ve never done remote review of notebooks, but thought I would chime in. I think this connects to the notebook review thread you started a couple weeks back, worth linking the two since Alaina’s suggestions there apply directly here: time-box each judge to a fixed window, 10 to 15 minutes, send notebooks in smaller batches so the window actually gets enforced, and train judges to stop once they’ve found one piece of evidence per rubric row instead of reading the whole thing cover to cover.
For the consistency problem specifically, a shared visible timer plus a check-in from you partway through tends to catch a judge who’s run long before it turns into a two hour review and a scoring mess.
Hope that helps.
Hi @Jason_McKenna
For the in person we have soem timers and all which makes it easier. For remote, I have never done those before, but do we take the judges at their word? I know personally some of these really long notebooks are really interesting to read, so I would lose track of time more easily, which could lead more to an artificially high score.
So in person is easier to control times, but getting enough judges will be an issue, as always.
This is my thrid season as a JA. I know from talking to others at state last year. Consistancy between events was lacking to be kind so the application of the how and all fell apart. So I want to be consistant to my peers, but most will not speak…It appears. We should be ready and willing to compare note and how we are doing things to give the kids (are actual target market) the best results. Don’t you think?
Dave
Hi David,
I completely agree that consistency in notebook scoring is one of the biggest challenges, especially when reviews are done remotely.
One approach that has helped is spending a short time with the judges before they start scoring. Walking through one or two sample notebook entries together (or even a short simulation) and asking each judge why they chose a particular score can surface differences in interpretation early. When the Judge Advisor then follows up with clarifying questions, it gives newer judges a clearer shared reference point and usually leads to more consistent scoring across the group.
It also reinforces that we are looking for evidence against the rubric rather than personal preference, which is especially important when judges are working independently on remote reviews.
Appreciate you raising this. These kinds of conversations help all of us get more consistent for the students.
