Hi, I have a JA for a couple of years. With the changes to the rubric, timing of review and handling of the review,
How are you handling remote reviews of notebooks to keep times consistant?
Since it is highly likely the 40 minute per notebook with judges is likely to be shortened in live judging (notebook reviewed in person), what issues are you concerned with?
For interview confidentiality, and pit sizes, how are you adjusting for that?
Looking forward to some advise.
I haven’t handled remote judging yet so this is mostly just ideas (I’ve always been at competitions with in-person reviews before), but I think a lot of it will fall on training the judges on what kinds of things to look for to minimize their review time and keep things consistent and moving forward. This could be things like:
My concern would be that remote judges take a long time on each notebook and overthink each row about if something is “enough” to meet the criteria rather than taking the content at face value. If I see tests in the notebook and sometimes there is qualitative data in one place and then quantitative in another, that would be enough to mark “Data is collected” as exemplary (4). It’s easier to have judges, in my experience, err on the side of lenient against the rubric rather than punitive to teams.
I have seen in the past where different and more unlimited time being allowed for notebook reviews allowed huge notebooks an unfair scoring advantage over the teams that geared the notebook for the more realistic and declared review time which created massive amounts of discrepancies in scoring. If judges are allowed 40 minutes versus an hour or more, the scares will vary because of that time allowed. So as a small-town JA with limited volunteers for judging like everywhere, how do I make it fair, so these teams are on level footing with bigger venues? I also talked to other JAs in person and the resulting conversations confirmed that the rules for times were not applied equally and the interviews were treated as a formality rather than the tool that they are. If anything, I find the interviews to be a better measure of the team since AI is being used to either aid or create a notebook. We had an AI professional at an event I worked at. To say he confirmed my suspicions would be inaccurate. I thought it was used less. Based on talking to him, I must admit to being wrong. Still without proof, the notebooks were permitted to move through so rules were followed. My thing is simply to make it best for the kids, we all need to be on the same page not big venues giving more time than smaller locations can provide. The essence of the struggle for small towns and small venues is how do we make it fair for the teams winning our events to even have a real chance as state or worlds when the rules are applied differently.
I have had people attempt to argue what STEM is. I work in science, I bleed STEM and have for a few decades. We look for cause/effect. Coding explain what your code does and how to adjust it. Most teams I interviewed used libraries and failed at the explanations. So they scored great on the fields, but crashed and burned on the interview. My judges rightfully offset the 2 scores as the dilberations pointed to a clear lack of understanding. These are the things I would like to get all of us JAS on one page about. I hope this helps. I know timing of reviews will impact scores. This year the EP’s have more input on the time being alloted. So remote judging of notebooks needs to better reflect the in person reviews.
Something else I’d also throw in the conversation here is the Season, Code, and Credit Summaries that are new this year. I think the Summaries that teams are submitting with their notebooks this year may help to even things out, and make the review process more efficient. The intention of a document like the Season Summary is for teams to point judges to places to look in their notebooks, so a judge isn’t expected to read it cover to cover. The summaries also give teams a chance to synthesize what’s in their notebook and give it meaning (like the abstract of a research paper does) and can help guide review and make it more focused. Things like the Code Summary help teams explain their code and show their thinking, and the Credit Summary points directly at the source material for various ideas, or AI tools, so there doesn’t need to be guesswork surrounding it.
While yes, review processes should be evened out time-wise, I think the Summary usage can also help guide judges through a more streamlined notebook review.
I think the summaries can help. They can also hide areas that are not up to the level. So I am thinking having my judges review the summaries of the selected notebooks prior to arriving or looking at the notebooks for grading thus saving a few minutes. I am also interested in trying to build a consensus for time being allowed for reviews. I know firsthand that last year, the times were not held at bigger meets to the recommended or even close after talking to other JA’s. I can state with high confidence, I cannot allow more that 30 minutes, unless I can figure out how to control time for remote notebook review. As JA’s we have the hard task of reviewing some stellar work in a small window of time. So if I allow 30 minutes and others are closer to 2 hours, I think you see where the fairness collapses. So I am trying to get things out in front of us all to get some type of baseline. I have read more manuals, summaries and methods in 25+ years as a chemist. I have done most everything as a chemist. STEM is STEM regardless of field. So if things stay uncontrolled and largely unfair to smaller schools and venues, I will not waste my time trying to help kids by being part of this. We are either fair and kid centered or we are not. This year for me is all about getting it fairer and more consistant. I am also going to start a topic on interviews. As those were treated as a joke at some tournaments. I am about action and not words. So THANK YOU for speaking up. I really do look forward to developing a working solution.
Hi David,
I wanted to share a quick thought regarding how we’re tracking the review times for the engineering notebooks.
As I currently see the rules on this, the time parameters are strictly recommendations designed to help us manage the event schedule efficiently. The official Judging Q&A explicitly confirms that judges are fully permitted to go back and spend additional time on notebooks later for qualitative analysis and deliberations. We view that as a normal and expected part of the process, very much like a follow-up interview.
This same principle applies whether notebooks are physical or submitted digitally via remote links. There’s no automated electronic cutoff preventing a judge from taking as much time as necessary to thoroughly evaluate a submission, provided we as Judge Advisors oversee the overall event timeline to keep things equitable across all teams.
Hope this perspective is helpful as we keep everything running smoothly!
Best regards,
I agree with all your points. The issue I have seen is without the tighter controls. The scores will change and normally drastically. I had a judge waste over an hour and a half on just one notebook. Obviously, it was really impressive, but the scoring was so dramatically different from the smaller ones. By this I mean the judge did thorough (time wasting) reviews of notebooks small and large. The gaps in the scoring grew due to this abuse of time usage. His scoring was tossed. I would have tossed him on the spot, but I did not know I could have.
So time controls are a must to keep the field level. If the time rules are app;ied differently the kids lose. It is sad, but it is where we are. With the split between VEX and REC, teams will be deciding more next year where to go than this year. Hence why I am being vocal. I like the concept being giving here, but the application is making me wonder why bother as the small venue teams have little chance unless they can win.
The problemsI foresee are.
When I get grades that the time was not followed closely, (I know it will be missed some) how do I level this score with the others?
Would I have them email me when they start and email the scores upon completion?
I am open to trying it as it would allow more notebooks to be scored, but I really want the level field we are all striving for.
Dave, I’ll come at this from the assessment side, since that’s where I live. The official time guidance is for the judging team to answer.
Here’s what I’d push on: time is the symptom, not the cause. Two judges can each spend 20 minutes and still score very differently. Scores stay consistent when every judge is looking for the same evidence, in the same way. Schools deal with this whenever teachers grade a common assessment. It’s called moderation, and a few habits do most of the work.
1. Calibrate before anyone scores. Have every judge score the same notebook, or a few pages of one, then compare. Where scores differ, talk about why. Paul suggested this in the Remote Notebook review thread, and ten minutes of it does more than any timer.
2. Start from the Season Summary, and cite the page. The summary tells the judge where to look. For each rubric row, the judge finds one piece of evidence, writes down the page, and moves on. A judge who has to cite a page for every score can’t drift into reading cover to cover.
3. For remote review, use a form instead of email. A simple Google Form records the time each score is submitted. Release notebooks in small batches with a due time, so no one can sink 90 minutes into a single notebook.
4. Spot-check instead of re-reading. Because every score has a page next to it, you can check the top few notebooks yourself in a few minutes each. Open to the cited pages for two or three rubric rows and ask one question: does this evidence support this score? You don’t need extra judges. A judge who’s scoring high or low shows up fast, and you go into deliberation already knowing which scores need a conversation.
You can’t really level a score after the fact. What you can do is build a process where one judge’s score never carries the decision alone, without needing a single extra volunteer.
