Researchers from Carnegie Mellon University’s Human-Computer Interaction Institute(opens in new window) have known that practice is essential for learning(opens in new window). But in a new study, they wanted to test whether adding AI-generated feedback and prompts that force students to reflect on their mistakes would improve outcomes even more.
The results were surprising: People who spent more time reflecting did not learn more. In fact, they sometimes learned less.
Practice makes perfect
In earlier research, Paulo Carvalho(opens in new window), an assistant professor in the HCII, and Michael Asher(opens in new window), an HCII project scientist, explored how to keep students practicing in the first place. Using “persuasive design” strategies, like small nudges built into learning tools, they helped students push past a common barrier: giving up after failure.
“When you get something wrong, a natural thing to do is to give up and move on to do something else,” Carvalho said. “But if you don’t keep practicing, you don’t learn.”
The team found that learning by doing and getting personal feedback was more effective than traditional teaching methods, where students listen to a lecture.
In that study(opens in new window), the researchers developed an AI-based learning tool to teach introductory Python programming. Students worked through short coding exercises and received immediate, personalized feedback generated by the system, similar to what a teacher might provide one-on-one. The tool worked even if students had never tried Python before.
In one condition of the experiment, learners were able to move directly from one problem to the next, learning through repeated practice and feedback.
Others were required to slow down. After each problem, they saw their corrected code alongside their original answer and were prompted to type an explanation of what went wrong and why.
The idea was to encourage reflection, a process that Carvalho said has long been thought to deepen learning by helping students actively process their mistakes.
The reflection bottleneck
In the study, adult participants had eight minutes to complete the activity. That meant every minute spent reflecting was a minute not spent practicing. Asher said students could learn a surprising amount in just a few minutes if time is used wisely.
“When time is limited, doing more problems, even without reflection, leads to better learning outcomes,” Asher said.
Reflection helped students think more carefully about individual mistakes. But it also reduced how many new problems they attempted and how many new mistakes they could learn from.
“It’s a little bit like dwelling on something that went wrong,” Carvalho said. “At this stage, that time could instead be spent trying new things and learning new ways to solve problems. We found that the cognitive effort required for reflection didn’t translate into better performance. The practice itself was the heavy lifter for learning.”
The motivation gap
Carvalho said that there’s another secret ingredient to consider. When people are learning something they care about, they don’t always need extra help.
“Somebody that’s trying to learn piano, or dance is less likely to get discouraged by failure because they’re intrinsically motivated to do it. They are learning because they really want to,” he said.
But in a typical math class, not every student wants to become a mathematician.
“They’re just trying to get through the work, and so failure is more discouraging,” he said. “It’s a problem teachers have to teach around.”
Rethinking how learning feels
Students don’t always recognize what works best. In previous studies, some participants reported that practice-first approaches felt “unfair” because they weren’t given explicit instruction before being asked to try problems — even though the practice and feedback were the instruction, Carvalho said.
“We all grow up thinking learning means someone tells us things first. When that doesn’t happen, it feels like we’re not being taught. This ‘unfair’ feeling is a known phenomenon where high-utility learning, like practice, feels harder and less smooth than lower-utility learning, like watching a video, leading students to misjudge their own progress,” he explained.
The results show that students who actively practiced, making mistakes and learning from feedback, consistently outperformed those who passively received information, such as through instructional videos.
Asher, once a math teacher himself, said this is something that teachers can incorporate into their lessons now, even without an app or other technology.
“Don’t feel like you need to explain everything to your students before they start practicing,” he said. “Just jump in, let them make mistakes, and use those mistakes as your teaching moments.”
Carvalho and Asher will present their research at the Learning @ Scale conference(opens in new window). A preprint of their paper, Benefit or Bottleneck? Assessing the Impact of Structured Reflection on Learning from AI-Driven Explanatory Feedback(opens in new window), is available now.
“Carnegie Mellon University is a private research university in Pittsburgh, Pennsylvania. The institution was originally established in 1900 by Andrew Carnegie as the Carnegie Technical School. In 1912, it became the Carnegie Institute of Technology and began granting four-year degrees.”
Please visit the firm link to site