Around 2021, well before ChatGPT launched, Vincent Hellendoorn suggested I try GPT-3 on the reading quizzes in my course. It produced convincing answers passing our rubric without actually seeing the assigned paper. At the time, I changed nothing. Five years later, AI agents could do all my assignments and I have redesigned most assessments in that course, even though what I want students to learn has barely changed. The strategy is always the same: No longer test understanding with anything that is done at home and instead focus on interactions with a TA, on exam, and on a video demo.
For written reflections, which I used to have as a part of pretty much every assignment, I now shifted to in-person interactions with a TA. After every assignment, each student needs to schedule a 15-minute meeting with a TA to answer a couple of questions in a live conversation. I still share the reflection prompts in the assignment as examples of the kind of questions we ask. Students can still generate an initial answer with an LLM, but they may need to memorize parts of it, and we try to challenge them with follow up questions. The check-in meetings are part of the assignment and currently worth 20% of the assignment points, graded pass/fail. Students can try again if they fail and I encourage my TAs to have fairly high standards – we usually fail quite a few students on their first attempt. For coding tasks: code + videos + in-person knowledge checks. We have weekly labs that are low-stakes small tasks to explore new tools (e.g., Kafka, Grafana, Docker, Weights and Biases). These tasks are necessarily scoped small and need to provide some help to students starting out – so they are obviously easily automateable by coding agents. We again rely on in-person check-ins – show the TA evidence that you completed the task and be able to answer a few questions, graded pass/fail. Again, we frequently send students back to read more documentation (or let their chatbot summarize the relevant part) and let them try again without a penalty until the time of the lab session runs out. For assignments, we use the same check-in with a TA discussed for reflections to let them explain part of their technical solution. In teamwork, we have longer debriefing sessions after each milestone (30-60 min per team). We award
beyond-the-comfort-zone bonus points to a team if the TA can ask any team member to explain any part of the implementation. Also having students produce a short video demoing a feature they implemented worked really well for an assignment to extend a web application, because it required the feature to actually work with a user interface in a real workflow. I think producing videos for other parts could also work, as long as it’s not just reading an AI-produced script, but actually grounded in some technical work. More of the grade now depends on what happens in the classroom. As I see in many other courses, we also shift more points from activities done at home (e.g., homework) to activities done in the classroom (e.g., exams, participation). Exams are now worth 25% instead of 15%, and I suspect I will raise this further in the future. The debriefing is 10 to 20% of the homework and group work grade.
AI is forcing me to abandon evidence-based best practices. Let’s start with this point upfront, since it is more important than what we actually do: Unfortunately, AI is actively undermining several evidence-based teaching practices (e.g., see How Learning Works and The ABCs of How We Learn). For example, the evidence favors frequent low-stakes assessments with feedback (e.g., homework, quizzes) over few high-stakes ones (e.g., exams) – but AI is undermining practice in low-stakes settings and pushing us more toward exams. Similarly, I always provided a safety net where students can make mistakes and resubmit a limited number of assignments to regain lost points (a core recommendation of specifications grading and grading for equity to focus on learning outcomes, not the process), but we felt that this process was abused with AI: first submit a generated assignment solution without thinking and only look at the issues raised in grading for a resubmission (the typical story of externalizing the cost of AI use). In response, we have since taxed resubmissions with a 10% penalty. Also in-class interactions allow engaging with materials in an early low-stakes setting, but with AI I have seen many student groups offload the discussion questions to a model. Pen-and-paper submissions could fix this, but aside from a higher grading workload, it would also raise stress for students, take away from the low-stakes environment, and delay feedback. In general, this is a balancing act and I tend to err on the side of keeping low-stakes repeated interactions even though it can be abused. Yes, some students will get through the class without much deep learning, but it provides a better environment for those students who want to learn. I don’t want to get back to the model I’ve experienced during my own studies in Germany with mostly optional homework and a single exam at the end of the semester that was responsible for 100% of the grade in the course. This was nice for students who were self-motivated and good at learning for an exam (like me, I guess), but had failure and drop-out rates of 50 to 80%.