Here’s an amazing paper by Anthropic that helps understand the mutual impact of AI usage and learning.
Especially important for educators trying to achieve the best use of AI in their classrooms while mitigating its negative impact.
The researchers assembled a group of 52 developers, mostly 25 to 35 and with bachelor’s degrees and between 1–3 and 7+ years of coding experience.
The experimental group used AI, specifically ChatGPT-4.0. The control group did not use AI but was provided with scaffolded information about a new library, Trio, which all participants were required to use to complete a coding task. Learning was assessed using a quiz on the library the participants had just worked with.
The study found that certain types of AI use undermined learning. On average, participants in the control group performed ~15% better than participants in the experimental group that used AI. But not all AI use worked the same way!
Within the experimental group, the researchers identified six distinct behavior patterns. Three of these were associated with better learning, and three were associated with worse learning outcomes. The three better-performing AI-use patterns produced results that were generally on par with the control group that did not use AI at all on the quiz. The three underperforming behaviors led to significantly weaker learning outcomes.
Thank you so much to Judy Hanwen Shen and Alex Tamkin. I believe this experiment is a great showcase of the fact that learning depends on the human behavior, not on the tool use. Staying curious and diving deeper into AI-generated content seemingly provides the same level of learning as going without AI.
The difference in quiz results between the AI group and the control group takes us to the next level. Quizzes are great when we assess remembering but are remembering and learning the same thing? Does driving a car always require an in-depth understanding of every mechanical or chemical process in your internal combustion engine?
That question on the nature of learning takes us to the next matter. Among all the observations that researchers made, I was especially surprised to see very similar results in terms of timing between the experimental group and the control group.
Using AI didn’t always result in faster work unless it was a complete “AI Delegation”. That approach brought faster results but also negatively impacted remembering. There was also a pattern named “Iterative AI Debugging” that resulted in the longest time and the worst remembering.
These results show that working together with AI can also be a trainable skill by itself and it probably includes balancing offloading the cognitive load to get results faster with the critical evaluation of both the process and the outcome. The exact strategy might differ depending on the model and the user skill, but the pattern will probably remain the same: delegate and question.
