Hi, I have a JA for a couple of years. With the changes to the rubric, timing of review and handling of the review,
How are you handling remote reviews of notebooks to keep times consistant?
Since it is highly likely the 40 minute per notebook with judges is likely to be shortened in live judging (notebook reviewed in person), what issues are you concerned with?
For interview confidentiality, and pit sizes, how are you adjusting for that?
Looking forward to some advise.
I haven’t handled remote judging yet so this is mostly just ideas (I’ve always been at competitions with in-person reviews before), but I think a lot of it will fall on training the judges on what kinds of things to look for to minimize their review time and keep things consistent and moving forward. This could be things like:
My concern would be that remote judges take a long time on each notebook and overthink each row about if something is “enough” to meet the criteria rather than taking the content at face value. If I see tests in the notebook and sometimes there is qualitative data in one place and then quantitative in another, that would be enough to mark “Data is collected” as exemplary (4). It’s easier to have judges, in my experience, err on the side of lenient against the rubric rather than punitive to teams.
I have seen in the past where different and more unlimited time being allowed for notebook reviews allowed huge notebooks an unfair scoring advantage over the teams that geared the notebook for the more realistic and declared review time which created massive amounts of discrepancies in scoring. If judges are allowed 40 minutes versus an hour or more, the scares will vary because of that time allowed. So as a small-town JA with limited volunteers for judging like everywhere, how do I make it fair, so these teams are on level footing with bigger venues? I also talked to other JAs in person and the resulting conversations confirmed that the rules for times were not applied equally and the interviews were treated as a formality rather than the tool that they are. If anything, I find the interviews to be a better measure of the team since AI is being used to either aid or create a notebook. We had an AI professional at an event I worked at. To say he confirmed my suspicions would be inaccurate. I thought it was used less. Based on talking to him, I must admit to being wrong. Still without proof, the notebooks were permitted to move through so rules were followed. My thing is simply to make it best for the kids, we all need to be on the same page not big venues giving more time than smaller locations can provide. The essence of the struggle for small towns and small venues is how do we make it fair for the teams winning our events to even have a real chance as state or worlds when the rules are applied differently.
I have had people attempt to argue what STEM is. I work in science, I bleed STEM and have for a few decades. We look for cause/effect. Coding explain what your code does and how to adjust it. Most teams I interviewed used libraries and failed at the explanations. So they scored great on the fields, but crashed and burned on the interview. My judges rightfully offset the 2 scores as the dilberations pointed to a clear lack of understanding. These are the things I would like to get all of us JAS on one page about. I hope this helps. I know timing of reviews will impact scores. This year the EP’s have more input on the time being alloted. So remote judging of notebooks needs to better reflect the in person reviews.
Something else I’d also throw in the conversation here is the Season, Code, and Credit Summaries that are new this year. I think the Summaries that teams are submitting with their notebooks this year may help to even things out, and make the review process more efficient. The intention of a document like the Season Summary is for teams to point judges to places to look in their notebooks, so a judge isn’t expected to read it cover to cover. The summaries also give teams a chance to synthesize what’s in their notebook and give it meaning (like the abstract of a research paper does) and can help guide review and make it more focused. Things like the Code Summary help teams explain their code and show their thinking, and the Credit Summary points directly at the source material for various ideas, or AI tools, so there doesn’t need to be guesswork surrounding it.
While yes, review processes should be evened out time-wise, I think the Summary usage can also help guide judges through a more streamlined notebook review.
I think the summaries can help. They can also hide areas that are not up to the level. So I am thinking having my judges review the summaries of the selected notebooks prior to arriving or looking at the notebooks for grading thus saving a few minutes. I am also interested in trying to build a consensus for time being allowed for reviews. I know firsthand that last year, the times were not held at bigger meets to the recommended or even close after talking to other JA’s. I can state with high confidence, I cannot allow more that 30 minutes, unless I can figure out how to control time for remote notebook review. As JA’s we have the hard task of reviewing some stellar work in a small window of time. So if I allow 30 minutes and others are closer to 2 hours, I think you see where the fairness collapses. So I am trying to get things out in front of us all to get some type of baseline. I have read more manuals, summaries and methods in 25+ years as a chemist. I have done most everything as a chemist. STEM is STEM regardless of field. So if things stay uncontrolled and largely unfair to smaller schools and venues, I will not waste my time trying to help kids by being part of this. We are either fair and kid centered or we are not. This year for me is all about getting it fairer and more consistant. I am also going to start a topic on interviews. As those were treated as a joke at some tournaments. I am about action and not words. So THANK YOU for speaking up. I really do look forward to developing a working solution.
