We end with some general observations about the dissemination of educational technology. The decision to adopt Cognitive Tutor Algebra I is not an easy one. When a school adopts the program it is not adopting supplemental software that can be added to existing practice. Instead, the school and its mathematics faculty are adopting a new course curriculum whose problem-solving emphasis differs both in content and form from traditional algebra texts. They are also adopting a teaching style that is more student-centered than current practice for most teachers, both in the computer lab and in small-group problem-solving activities. In accounting for the success of the dissemination efforts, we need to acknowledge both the opportunity that exists in mathematics education along with the design factors that contributed both to initial adoption decisions and to sustained use of Cognitive Tutor Algebra I.
Opportunity
As implied earlier, the 1990s were an opportune decade in which to offer a novel approach to high school mathematics education in the United States for two broad reasons. First, a number of comprehensive national and international assessments raised awareness of the need to improve our mathematics education. This awareness was magnified by two related trends: the call for mathematics education reform as embodied in the NCTM reports and the call for greater accountability. Second, the cost of educational technology continued to drop throughout the decade.
Growing Expectations and Accountability. In response to the mathematics education crisis and recognition that all students need to master academic mathematics courses, many school districts and state departments of education in the United States are raising the bar for mathematics achievement. Algebra and geometry are increasingly being required for high-school graduation. Statewide assessments increasingly have consequences for students and school districts. At least 48 states have or are defining statewide standards and assessments that increasingly are employed to evaluate schools and govern student graduation. To cite two examples:
• The state of Kentucky has charged its school districts with demonstrating consistent achievement gains on statewide assessments over the years, or, in the extreme case, risk yielding control of their schools to expert teachers designated by the state.
• The state of New York is changing the role of its Board of Regents exams. In the past, approximately 15% of students passed the exams and earned the distinguished regents diploma. Beginning in 1999–2000, students must pass a revised Board of Regents exam to graduate.
216 Cognitive Tutors: From the Research Classroom to All Classrooms
The NCTM reform recommendations further magnified this opportunity by arguing that traditional mathematics education is not adequately serving students. Among other recommendations, the NCTM advocated open-ended assessments and student-centered classrooms focused on learning by doing. Of necessity, these recommendations provided only a framework for reform and created a curriculum and assessment vacuum that our project (and other curriculum projects) could fill. A noteworthy observation is that all schools are looking for new solutions and improved outcomes, not just low-performing schools. Even the most successful school districts are not satisfied with how well they are serving their students. Of the first six suburban school districts that adopted the Cognitive Tutor Algebra I course, three rank among the top eight districts in Allegheny County (out of 42 districts) on the Pennsylvania System of School Assessments, and one is top-ranked in nearby Beaver County.
Technology. The computing power necessary to support cognitive tutors has become more widely available in schools over the past decade. The original Geometry Tutor was implemented on Xerox workstations that cost approximately $20,000 each. ANGLE, in turn, was developed on a Macintosh workstation at a time when most schools still had Apple IIes. Now, the typical workstation being purchased by schools is sufficiently powerful to run cognitive tutors and the cost of an entire lab for 25 to 30 students is less than $50,000. When we began disseminating the Algebra I tutor, schools typically acquired a new workstation lab to run the software. Now it is far more common that schools already have the necessary technology. Workstations are typically acquired as productivity tools to support Internet access and document production, but schools are actively seeking demonstrably effective educational software—like the tutors—to further increase workstation utility.
Adoption and Sustained Use
We believe many factors contribute both to the decision to adopt Cognitive Tutor Algebra I and to its sustained use in schools once it is adopted. Of the many sites that have adopted the program over the past five years, only three sites have since abandoned it. This total abandonment rate over five years of approximately 4% is far less than the textbook industry’s annual “churn” rate of about 15%.
Match to NCTM Standards. Cognitive tutor mathematics courses have gained acceptance not just because they employ educational technology to achieve substantial learning gains, but because they are among the first courses that are designed to match NCTM curriculum, teaching, and assessment standards. In these courses, algebraic and geometric topics are introduced in the context of authentic problem-solving activities, and the informal problem-solving knowledge students bring with them serves as the foundation for developing formal algebra and geometry knowledge. This approach is embodied in the Algebra I Cognitive Tutor interface displayed in Fig. 9.1 as well as in the paper-based activities. The courses are designed to foster learning by doing. Small-group problem-solving activities enable students to be active learners outside the cognitive tutor laboratory, and the cognitive tutors provide just the help students need to develop individual problem-solving skills. As shown in Fig. 9.6, course assessments, like the curriculum itself, emphasize the application of algebra and geometry knowledge to solve
Technology Enhanced Learning 217
problems, reason among multiple representations, and communicate mathematical conclusions.
Fully Integrated Technology. The cognitive tutor technology is fully integrated with the algebra course and plays a necessary role in the course structure. The paper text, assignment and assessment materials, and cognitive tutor activities were jointly designed to target the same educational goals, with similar, student-centered problem-solving activities. The small-group classroom activities are designed to introduce topics, while the text is designed to immediately engage students in problem-solving activities that build on their existing knowledge. The small-group activities provide students the opportunity to explore topics together and refine each other’s ideas. In the cognitive tutor lab, students apply these mutually developed constructs to develop their individual problem-solving knowledge.
Empirical Evaluations and Achievement Gains. We are able to provide empirical evidence that Cognitive Tutor Algebra I yields increased achievement gains.
Achievement gains are of growing importance and schools, teachers and students are increasingly held accountable for outcomes. For example, when the Kentucky Department of Education organized a statewide Results-Based Practices Showcase, curriculum vendors were required to demonstrate minimum achievement gains over a three-year period. Of the 450 vendors contacted across all curriculum areas, only 61 could demonstrate the results. Perhaps not coincidentally, most of the mathematics projects that could offer the empirical evidence were other National Science Foundation- funded projects.
Of course, achievement gains in each local school district will be important in sustained use in the long run. So far, we have essentially replicated the pattern of achievement gains reported in Table 9.1 in Milwaukee. In this assessment, Cognitive Tutor Algebra I students performed 25% better on standardized test questions than comparable students in traditional Algebra I and 50% better on problem solving and reasoning among multiple representations.
It should be noted that these successful year-end summations are the culmination of a sustained empirical evaluation process that is interwoven with cognitive tutor development. The cognitive tutor design, development, and piloting process is accompanied by (a) basic research in mathematics learning (e.g., the research on inductive support cited earlier), (b) detailed assessments of student problem-solving performance in the tutor environment, which serve to refine the cognitive model, and (c) pretesting and post-testing that is also designed to refine the model. Project staffing reflects these ongoing assessment activities. Cognitive tutor development is typically supervised by a cognitive psychologist, whose time on the project is divided fairly evenly between design and assessment. In addition, for each person-year of tutor development time by research programmers, we allocate a half-year of research assistant time to carry out assessments.
Professional Development. We originally provided five days of preservice professional development as part of the Cognitive Tutor Algebra I package. About half of this time was devoted to the cognitive tutor technology and teacher interactions in the computer lab. The other half was devoted to the curriculum and teacher interactions in the classroom. We have noticed over the years, though, that teachers tend to focus on the technology in preservice training, because it is both novel and tangible. However, this
218 Cognitive Tutors: From the Research Classroom to All Classrooms
comes at the expense of the curriculum and classroom issues. Once the school year begins, students have little difficulty with the technology, and teachers then call the hotline with curriculum and teaching questions and focus the site visits on these issues.
As a result, in 1998–1999 we shifted to three days of preservice training, followed by in- service professional development days during the year.
Classroom Impact. The impact of cognitive tutor technology on the classroom—
primarily on student motivation, but also on the teacher-student relationship—has been an important factor in both the adoption and sustained use of Cognitive Tutor Algebra I.
Students are actively learning by doing in the cognitive tutor classroom and the tutor is helping them solve problems successfully. As Schofield (1995) documents, students find this environment highly motivating; they are actively engaged in doing mathematics and spend more time on task than in traditional classrooms.
Palpable student engagement in mathematics problem-solving activity plays at least two roles in our dissemination success. First, it helps schools decide whether to adopt the model, because it adds credibility to our reported achievement gains. Second, it fosters sustained use, because it makes teaching more rewarding and more fun. When students are actively engaged in problem solving, most teacher-student interactions are about mathematics, not classroom management. As Schofield notes, teachers can shift their attention to students who are struggling, and since interactions are typically student- initiated, students are ready and willing to learn. The teacher can take advantage of these
“teachable moments” by engaging in extended interactions with individual students while the rest of the class is making measurable progress.
Our challenge has been to effectively convey the high level of student engagement. In 1995–1996, we asked each of our project schools to write letters describing the classroom. Figure 9.8 provides excerpts from the six letters. In addition, anecdotes continue to come in from other sites around the country. Two examples follow:
• A Milwaukee high school has a 30-minute activity period over lunchtime in which students may engage in any activity they choose, including conversation and recreation. Teachers report that during this free period the cognitive tutor lab is full of students working on algebra problems, with more students waiting for a computer to open up.
• A New York City high school teacher reported a student was in tears while working in the cognitive tutor lab one class period this semester. When he asked what was wrong, the student said nothing was wrong—she had just never understood mathematics before!
Note that the second anecdote suggests that cognitive tutors are motivating not only because they are challenging and fun (as Schofield hypothesizes), but because students may be experiencing an unfamiliar—and exhilarating—level of success in problem solving.
Technology Enhanced Learning 219
FIG. 9.8. Excerpts from letters describing the Cognitive Tutor Algebra I Classroom.
Technical Support. Technical support, both software installation and the hotline, are an important part of the site-support package. While workstation LANs are becoming common in schools, the level of technical support varies widely across sites.
Community of Users. Teachers face a variety of challenges when introducing new curricula, technology, teaching styles, and assessment styles. In a variety of ways we have tried to create a community of users to provide support in tackling these challenges.
We require a commitment from both administrators and teachers when Cognitive Tutor Algebra I is adopted and strongly recommend that at least two teachers participate.
Indeed, among the sites that have dropped the program, support was limited either to a single administrator or to two teachers; in each case, the program ended when those personnel left the sites. We encourage schools to hold family algebra nights, often in conjunction with our site visits. We hold teacher meetings in the Pittsburgh region and
220 Cognitive Tutors: From the Research Classroom to All Classrooms
hope to propagate this model to other regions of the country. Finally, we sponsor a national e-mail users group.
It is difficult to rank the importance of each of these factors in adoption and sustained use. Certainly, achievement gains and classroom impact are critical, but curriculum structure and support are also important. While the Kentucky Results-Based Showcase required demonstrable achievement gains for admission, it also developed a consumers guide rating the curricula offerings on many of the other dimensions discussed here: On- site support, complete curriculum, professional development and demanding school commitment. Cognitive Tutor Algebra I received the highest rating on all four dimensions.