Hi, I have a JA for a couple of years. With the changes to the rubric, timing of review and handling of the review,
How are you handling remote reviews of notebooks to keep times consistant?
Since it is highly likely the 40 minute per notebook with judges is likely to be shortened in live judging (notebook reviewed in person), what issues are you concerned with?
For interview confidentiality, and pit sizes, how are you adjusting for that?
Looking forward to some advise.
I haven’t handled remote judging yet so this is mostly just ideas (I’ve always been at competitions with in-person reviews before), but I think a lot of it will fall on training the judges on what kinds of things to look for to minimize their review time and keep things consistent and moving forward. This could be things like:
My concern would be that remote judges take a long time on each notebook and overthink each row about if something is “enough” to meet the criteria rather than taking the content at face value. If I see tests in the notebook and sometimes there is qualitative data in one place and then quantitative in another, that would be enough to mark “Data is collected” as exemplary (4). It’s easier to have judges, in my experience, err on the side of lenient against the rubric rather than punitive to teams.
I have seen in the past where different and more unlimited time being allowed for notebook reviews allowed huge notebooks an unfair scoring advantage over the teams that geared the notebook for the more realistic and declared review time which created massive amounts of discrepancies in scoring. If judges are allowed 40 minutes versus an hour or more, the scares will vary because of that time allowed. So as a small-town JA with limited volunteers for judging like everywhere, how do I make it fair, so these teams are on level footing with bigger venues? I also talked to other JAs in person and the resulting conversations confirmed that the rules for times were not applied equally and the interviews were treated as a formality rather than the tool that they are. If anything, I find the interviews to be a better measure of the team since AI is being used to either aid or create a notebook. We had an AI professional at an event I worked at. To say he confirmed my suspicions would be inaccurate. I thought it was used less. Based on talking to him, I must admit to being wrong. Still without proof, the notebooks were permitted to move through so rules were followed. My thing is simply to make it best for the kids, we all need to be on the same page not big venues giving more time than smaller locations can provide. The essence of the struggle for small towns and small venues is how do we make it fair for the teams winning our events to even have a real chance as state or worlds when the rules are applied differently.
I have had people attempt to argue what STEM is. I work in science, I bleed STEM and have for a few decades. We look for cause/effect. Coding explain what your code does and how to adjust it. Most teams I interviewed used libraries and failed at the explanations. So they scored great on the fields, but crashed and burned on the interview. My judges rightfully offset the 2 scores as the dilberations pointed to a clear lack of understanding. These are the things I would like to get all of us JAS on one page about. I hope this helps. I know timing of reviews will impact scores. This year the EP’s have more input on the time being alloted. So remote judging of notebooks needs to better reflect the in person reviews.
Something else I’d also throw in the conversation here is the Season, Code, and Credit Summaries that are new this year. I think the Summaries that teams are submitting with their notebooks this year may help to even things out, and make the review process more efficient. The intention of a document like the Season Summary is for teams to point judges to places to look in their notebooks, so a judge isn’t expected to read it cover to cover. The summaries also give teams a chance to synthesize what’s in their notebook and give it meaning (like the abstract of a research paper does) and can help guide review and make it more focused. Things like the Code Summary help teams explain their code and show their thinking, and the Credit Summary points directly at the source material for various ideas, or AI tools, so there doesn’t need to be guesswork surrounding it.
While yes, review processes should be evened out time-wise, I think the Summary usage can also help guide judges through a more streamlined notebook review.
