Assessing the Odds
Students in Paul King's Forest Grove Middle School math class were some of the first to use ASSISTments. Their feedback—as well as feedback from two other Worcester schools—helped Heffernan develop the types of questions that are the heart of the intelligent tutoring system.
Meet Ms. Lindquist. As middle-school algebra teachers go, she is remarkably calm, patient, and eventempered. Wrong answers never faze her. “Let’s break this down,” she says when presented with a first attempt that misses the mark. Step-by-step, she walks the student through the problem until the answer seems clear. Then, with that small hurdle cleared, it’s on to the next problem and the next learning milestone.
It’s a winning approach, one Ms. Lindquist has used with thousands of students across the country. Who is this remarkable teacher? “She” is a computer program, a complex, patented collection of code developed by Neil Heffernan, assistant professor of computer science at WPI, while he pursued his PhD at Carnegie Mellon University. Ms. Lindquist (named for the real-life teacher—now Heffernan’s wife—who served as the model for the program) was Heffernan’s first voyage into the field of intelligent tutoring systems, an area that has been his focus—some might say his obsession—ever since.
For the past three years, with funding from the U.S. Department of Education, the Office of Naval Research, the National Science Foundation, and other agencies, Heffernan and a small army of graduate students and undergraduates have been developing a computerized tutoring system that helps 8th-grade students learn the math skills they need to pass the MCAS (Massachusetts Comprehensive Assessment System), the state’s mastery test.
The MCAS system represents a significant advance over Ms. Lindquist. Like the earlier program, it helps students master math concepts by breaking them into steps and responding appropriately to correct and incorrect answers. But it also monitors and provides feedback on how individual students and entire classes perform on specific test items and on the 98 math skills required to complete those items. Because the system combines assistance with assessment, Heffernan calls it ASSISTments.
ASSISTments draws heavily on Heffernan’s expertise in artificial intelligence and is built on a solid foundation of cognitive psychology. Before he built Ms. Lindquist, Heffernan did “think-aloud” studies with students, a technique pioneered by his Carnegie Mellon professors. He’d give the kids algebra problems and ask them to think aloud as they tried to solve them. If they stopped talking for five seconds, he’d prompt them to start again. It was a way of peering inside the black box of the brain to see how thinking really works.
Heffernan built computer models that simulated the techniques different students used to solve math problems. In time, the models were able to predict the performance of individual students.
“If you can do that,” he says, “then you want to build a tutor that can say, ‘It looks like you are solving the problem this way; let me tutor you that way.’ Because a tutor can track an individual student’s progress, you can do mastery learning—having students practice one skill until they’ve mastered it.”
Heffernan put Ms. Lindquist on the Web and made it available for free to any school that wanted to use it. As kids used the tutor, data flowed in about how they used it, and how well they learned. The feedback enabled him to continually fine-tune the program and study its effectiveness. He calls this “data-driven education,” something he says is all too rare.
“We do a hell of a lot more work to verify that a drug is safe than we do for any educational intervention,” he says, noting that the U.S. Department of Education (most notably through the No Child Left Behind mandate) is pushing hard to get schools to focus more on data. “They want to transform education into an empirically based science.”
From the start, Heffernan says he wanted ASSISTments to revolve around the notion of feedback: to help students learn, to help teachers better understand the learning needs of individual students, and to help principals gain a more complete and immediate picture of their classrooms. ASSISTments records the results of each student session and provides reports that teachers can use to adjust their classroom teaching.
Heffernan says he is excited by the wealth of opportunities ASSISTments provides for educational research. In fact, the software has provided fodder for several undergraduate projects and a wide range of research by graduate students. One WPI student, for example, noticed that some middle school students seemed to “game” the system, just clicking through the hints or guessing randomly. He developed software that can detect the behavior and prod students to get them back on track. He also developed a graphic that appears atop every student’s screen to help teachers quickly identify and aid those who seem to be struggling.
“I want to make mathematics education research drastically better,” Heffernan says with the passion of someone who has, himself, been on the educational front lines. Indeed, he ran a tutoring program for 6th graders in Holyoke, Mass., while an undergraduate at Amherst; he also taught 8th grade math and science in Baltimore through Teach for America, and taught for a year in Sudbury, Mass., before joining WPI.
After three years of development, testing, and refinement, ASSISTments may be ready for its next big trial. Currently only available to Worcester public school students, Heffernan would like to roll out the program statewide. He points to studies that show that tutoring—even by an intelligent tutoring system—produces “insanely better” results than classroom teaching alone. “So our schools could do a lot better.”
With the assistance—and assessment—of ASSISTments, they almost certainly email@example.com
Maintained by: firstname.lastname@example.org
Last modified: Oct 19, 2006, 15:34 EDT