Hi, I am obviously stirring the pot as I want to level the way we perform our work. I have been a JA for 3 years now. I have judged for an additional 3 or 4 years. These kids are impressive, but the way we as JA’s are going about things is vastly varied. This hurts the kids centered philosophy and makes for bad experiences for the kids, judges, and coaches, long and short everyone. Last year I pushed my judges to work the interviews for details. I must say few of the top note booking teams fared well under this interview style. Teams that had really amazing codes largely failed in more than a basic level of understanding while teams that struggled with code really understood where they were on the code and just ran out of time to finish the fixes. It made for some interesting debates. This year with the changes to how we proceed to press for details as we need more details for scoring and the debate, what things are you considering and why? How are you planning on moving beyond the intro question or 2. I am thinking of pressing for better more defined understanding of how the code works and why (short answers), press for 2 design features one team decides and one judge just picks. Really need to move them away for the rehearsed and practiced talking points to see if they really understand their bots. I am hoping for a minimum of 10 minute interviews with follow-ups for the top teams. What are you all thinking?
Thanks for your question. Please keep in mind that I am coming at this from the curriculum side rather than the judging side, so take this as one more voice in the room.
The pattern you saw last year is the whole argument. When the interview asks for detail, it separates teams that own the work from teams that can present it. We see the same thing in classrooms, which is why we do so much around conversation-based grading.
The judge-picked design feature is a good idea. The team-picked feature tells you what they rehearsed. The judge-picked one tells you whether they understand the robot. The same applies to code: point to a specific line or value and ask what would happen if it changed. That is a great way to elicit evidence of understanding.
I would do the same with the notebook. Instead of asking about it, ask from it. Open to a page and say, “What happened here, and what did you do next?” A team that lived the process can talk about any page. A team that wrote it up afterward can talk about three.
“Pressing” can slide into an interrogation, and a nervous student can shut down for reasons that have nothing to do with understanding. The best question I know is, “Tell me about something that didn’t work.” It is disarming, almost impossible to rehearse, and every team that actually built the robot has an answer. Here is a great article that discusses this.
I hope this helps.
I must say I agree with avoiding more of the rehearsed speeches. My teams of judges and myself have noticed some teams struggle when asked something that is not rehearsed or worse yet only 2 to 3 members of a larger team can answer. So asking but not pressing for details is a better choice. In tournaments I worked as a judge or a JA, asking for things that were not rehearsed impacted the deliberations significantly. I think tha after the event if asked by teams, coaches or parents, pointing out that all aspects of the robot and code are fair game and students must be ready for that is a fair reminder. You can’t tell them how to answer, but reminding them the whole of the data presented, robot, notebook, code and now summaries are fair game. The summeries are not graded and will take you to likely the best prepared areas. So as an adult and parent, I understand this and will look to at least one other area to get a score. Thanks for responding. The more opinions shared the better and more lelvel the playing field will be for all.
Agreed, David.
What we have always done here as judges is to first collaborate in the judges room with the advice of the JA to determine some key questions that are good openers for the individual team interviews. Generally those open-ended questions require the team to tell how they overcame a challenge, or what a particular line of code does, or what does this thing do (as we point to something out of the ordinary on their robot).
This then leads to further questions as they respond to the openers. Or, if a team is more shy, they need more friendly leads into each question as they do not want to speak, or they have a language barrier (we do not always have a translator in our judging partners, but the teams understand the interviews are in English).
Cheers! 
This was a great article Jason. Thank you!
Through this we can see how to elicit a response from a student in a positive way.
"Ask the student to explain what is wrong.
- How is the robot moving in their project?
- How should the robot be moving?"
This example can be fitted to our judges simply for the physical robot, the code, or programming questions.
The basics of teaching are pretty universal. 
Thanks, Paul. You named it. The basics are universal. A good judging interview and a good classroom conversation make the same move: ask something a student can only answer if they did the work, then listen.
Your point about shy students and language barriers matters too. We’re looking for evidence of understanding, and a student who freezes gives us no evidence either way. Friendly openers aren’t going easy on a team. They’re how you get to the real answer.
As the JA at my event maybe events, I will be telling the judges one maybe 2 easy warm up questions than go for aspects of code, notebook and design not mentioned. Interviews will be 10 minutes. So the teams will not be allowed much of the rehearsed areas. Last year, I used this approach and it really did weed out teams, even top performing teams. The judges also really enjoyed the experience more. I will have a few returning judges, which will be helpful.
I do LOVE seeing more people posting.
I did read the article. Since I am not an educator, but a trainer and long in the tooth chemist. I can say I use this concept with those I train. Knowledge is power. So knowing how to convey, confirm message is recieved as intended and having the trainee later “train you” I have found is working the best. So the reflection, editing and all play into more than just the competition, but helps to build the foundation for their futures. Keep the educator ttips coming. Thanks!
David, your teach-back idea works with students too. In my book, the top level on the student self-rating scale is “I could teach this to someone else.” Same test.
A few more from the classroom side:
Ask the student who hasn’t spoken yet. In the classroom we call it no hands up: you choose who answers. “Who drove in your last match?” gets you past the team’s designated speaker.
Ask how they decided. I wrote about a judging interview where the judge asks, “Was this your first idea for an intake, or did you go through different ideas?” and “How did you decide on this strategy?” Rehearsed answers explain what the team did. Students who did the work can tell you why.
Follow the notebook. In that same interview, the team’s notebook listed driving as their biggest weakness, so the judge asked why. The best follow-up questions come from what the team wrote down, not from a script.
Wait before you rephrase. Mary Budd Rowe’s research on wait time found that a few extra seconds of silence gets longer, more thoughtful answers. Most of us jump in too early.
Ask about something that didn’t work. “What failed, and what did you change?” is lower pressure than “explain your code,” and teams that own their process usually have a story ready.
Your plan to probe what they didn’t mention fits all of this. Conversation shows students’ thinking in a way a presentation can’t, because it isn’t rehearsed. That’s what you’re after in an interview.
I agree. As I have conducted many an interview. We typically will take notes of things observed in the notebooks. Share those items with all the judges. I encourage my judges to actively test to the entire judge team between interviews. These kids are amazing and I find blending a teacher or similar minded person with someone that is more less in the STEM professions provide for amazing interviews and awesome things during dilberations.
We make a point of trying to get everyone talking and open with everyone needs to talk. So the kids know the expectations up front. Generally all the kids have done great knowing the expectations up front and the top teams find way to get everyone involved which comes up in the dilberations.
I have had special needs interviews. Those kids have excelled when we knew how to get them to open up. So I have had to adjust for those items as well. I have person experience growing up with a sister in this arena so it comes naturally to me. I simply ask what would make them most comfortable and let them led. Those have been some amazing interviews.
Generally I find having teams roving and stationary at tables works well for getting the interviews. Where possible we do 2 interviews per team so judges awards get a much stronger look.
Any negative feedback is shared. ALL positive feedback is conveyed.
Feel free to continue to offer any and all suggestions. I would like to improve.
