An intelligent tutoring system that gene

An intelligent tutoring system that generates anatural language dialogue using dynamic multi-level planning a School of Computer Science, Kookmin University, 861-1 Chongnung-Dong, Sungbu

An intelligent tutoring system that generates a

natural language dialogue using dynamic

multi-level planning

a

School of Computer Science, Kookmin University, 861-1 Chongnung-Dong,

Sungbuk-Ku, Seoul, Republic of Korea

b

Computer Science Department, Illinois Institute of Technology, Room 236,

10 West 31st Street, Chicago, IL 60616, USA

Department of Molecular Biophysics and Physiology, Rush Medical College,

1750 West Harrison, Chicago, IL 60612, USA

Received 16 February 2005; received in revised form 14 October 2005; accepted 21 October 2005

in qualitative causal reasoning to internalize new knowledge and to apply it tively and that they learn by putting their ideas into words

effec-Methods: Analysis of a corpus of 75 hour-long tutoring sessions carried on in to-keyboard style by two professors of physiology at Rush Medical College tutoringfirst-year medical students provided the rules used in tutoring strategies and tactics,parsing, and text generation The system presents the student with a perturbation tothe blood pressure, asks for qualitative predictions of the changes produced in sevenimportant cardiovascular variables, and then launches a dialogue to correct any errors

keyboard-* Corresponding author Tel.: +1 312 567 5153; fax: +1 312 567 5067.

E-mail address: evens@iit.edu (M.W Evens).

0933-3657/$ — see front matter # 2005 Elsevier B.V All rights reserved.

doi:10.1016/j.artmed.2005.10.004

1 Introduction

1.1 Research goals

The goal of this research was to develop an

intel-ligent tutoring system capable of carrying on a

natural language dialogue with students Our system

was originally conceived by two professors at Rush

Medical College, Joel Michael and Allen Rovick,

inspired by the conviction that natural language

interaction was the most effective way for students

to learn They observed that students learn best

when required to give explanations of their thinking

and they instituted small group problem-solving

sessions and individual tutoring sessions to

supple-ment their own computer-aided instruction (CAI)

systems [1,2]

CIRCSIM—Tutor is designed to help first year

med-ical students learn to solve problems involving the

baroreceptor reflex system, which stabilizes blood

pressure in the human body The system presents a

perturbation to the cardiovascular system and asks

the student to make qualitative predictions about

changes in seven important cardiovascular

para-meters It analyzes these predictions, identifies

any errors, and assists them in correcting their

errors The design of the system is based on the

analysis of human tutoring sessions carried on by

Michael and Rovick at Rush Medical College This

analysis convinced us that planning plays a central

role in the generation of a tutorial dialogue

When CIRCSIM—Tutor asks the student to make

predictions about the behavior of various

cardiovas-cular parameters it asks only whether that

para-meter rose or fell or stayed the same, because the

focus is on qualitative causal reasoning Practicing

physicians do not generally need to know the

numeric values of these parameters but they do

need to use this kind of qualitative reasoning every

day [3] Michael and Rovick originally used a

detailed mathematical model in their CAI system,

but they found that students lost track of the

under-lying ideas when they tried to handle detailed

mod-els and tables of numbers [4] So from the verybeginning of this project one of the goals was toteach qualitative reasoning [5—7], an approach toproblem solving that reasons about the causal rela-tionships that structure our world Anderson [8]

argues that qualitative reasoning is the mostdemanding approach, one that is essential to a highperformance tutoring system He claims that it canalso maximize pedagogical effectiveness, because it

is human-like reasoning, although the tion effort is much larger than that required for thetraditional models

implementa-1.2 Evolution of computer-based instruction at rush medical college

Michael and Rovick had a great deal of experiencewith CAI in the cardiovascular domain at Rush Med-ical College Their systems evolved from HEARTSIM

[1], to CIRCSIM [2,9], to the CIRCSIM—Tutor type[10,11]and finally to CIRCSIM—Tutor, which hasitself evolved over almost 15 years[12] HEARTSIMwas a Plato program and CIRCSIM is a stand-aloneBasic program The CIRCSIM—Tutor prototype was aProlog prototype of our intelligent tutoring systemdesigned and implemented without any natural lan-guage capabilities [10] CIRCSIM—Tutor uses manyfeatures from the CIRCSIM—Tutor prototype but it iswritten in Lisp and it includes much more completestudent modeling, instructional planning, and nat-ural language facilities

proto-1.3 Natural language dialogue and tutoring systems

We made natural language dialogue the core of oursystem, because it is an especially powerful tool forlearning Chi et al [13]have now produced scien-tific verification of our belief that putting ideas intoyour own words is a central part of learning In aningenious experiment Fox Tree [14] demonstratedthat people remember ideas that they have hearddiscussed in dialogue better than a monologue on

and to probe for possible misconceptions The natural language understandingcomponent uses a cascade of finite-state machines The generation is based onlexical functional grammar

Results: Results of experiments with pretests and posttests have shown that using thesystem for an hour produces significant learning gains and also that even this brief useimproves the student’s ability to solve problems more then reading textual material

on the topic Student surveys tell us that students like the system and feel that theylearn from it The system is now in regular use in the first-year physiology course atRush Medical College

Conclusion: We conclude that the CIRCSIM—Tutor system demonstrates that gent tutoring systems can implement effective natural language dialogue with currentlanguage technology

intelli-#2005 Elsevier B.V All rights reserved

the same topic Perhaps this is why Plato’s Dialogues

have survived for 2400 years while so much other

Greek learning has been lost Fox[15] pointed out

that, in tutoring dialogues, the tutors and the

stu-dents typically construct the answer together, so the

students remain active participants and also share

ownership in the result

The builders of the first intelligent tutoring

sys-tems, Carbonell[16], Carr and Goldstein[17],

Bur-ton and Brown [18], Collins and Stevens [19], all

assumed that tutoring should be carried out via

natural language dialogue Then the difficulty of

natural language processing and the attractions of

the new graphical user interfaces drew intelligent

tutoring systems research in a different direction

When this project began, CIRCSIM—Tutor was alone

in the field with Wilensky’s[20,21]Unix Consultant,

which is really a coach and not a tutor There was

important research in dialogue-based intelligent

tutoring systems carried on by Woolf [22,23] and

Cawsey[24], but they used template-based

genera-tion and limited (partly menu) input rather than

trying to handle whatever the user typed and

gen-erating responses from scratch

Happily, CIRCSIM—Tutor is not so lonely anymore

Increases in machine capability have made natural

language dialogue much more manageable and

knowledge of text generation has increased rapidly

One notable example is Atlas[25,26], a physics tutor

at the University of Pittsburgh VanLehn started

from Andes, a successful cognitive tutor for physics,

and assembled a top team to provide natural

lan-guage interaction Freedman’s Atlas Planning

Envir-onment carried out the dialogue planning [27]

Rose´’s [26,28] Carmel parser did the parsing and

produced a logical representation of the student

input Jordan et al [29,30] used Tacitus-Lite for

reasoning about the analysis of the student’s essay

and also developed knowledge creation dialogues

for dialogue generation

Graesser and his group at the University of

Mem-phis built AutoTutor [31] based on their studies of

human tutoring, which uses latent semantic analysis

(LSA) to handle natural language understanding and

generation For natural language understanding,

they used LSA to match the student input to one

or several ideal answers They used LSA in

genera-tion, as well, to pick out the most relevant answers

to a question from a collection of texts generated by

experts

With the encouragement of a multi-university

research grant from the Office of Naval Research

the Atlas and AutoTutor project teams joined forces

to build tutors for qualitative physics using their two

different approaches The current generation of

tutors resulting from this research, Why2-Atlas

[32] and Why2-AutoTutor [33], has been shown to

be more effective than reading text; the practicalalternative for most university courses Both tutorspresent the student with a problem and ask them towrite a short essay giving an answer and an explana-tion of their reasoning Then the system critiquesthe essay and helps the student to improve itscontent VanLehn’s team has also built the Pyreneestutor [34], another physics tutor much like Atlas,except that it discusses problem-solving algorithmswith the student in explicit terms, which gives asignificant improvement Making use of theseresults, Lane and VanLehn[35]have recently devel-oped another dialogue-based tutor for introductoryprogramming students that emphasizes the under-standing of the algorithms involved These tutorsmust analyze input with longer and more complexcontent than CIRCSIM—Tutor sees, but their dialogue

is not as interactive

CATO, developed by Ashley and Aleven, isdesigned to help law students learn techniques ofargumentation Ashley is a well-known expert inlegal artificial intelligence, while Aleven contri-butes the natural language expertise[36,37] As aby-product of this research, the authors carried out

an experiment demonstrating the efficacy ofSocratic tutoring over more didactic tutoring

A series of experiments by Di Eugenio[38]showedthat improving the quality of the natural languagegeneration of an existing system can make a sig-nificant difference in learning outcomes Moore’sBEETLE[39,40], designed to teach basic electricityand electronics to Navy recruits, uses even moresophisticated generation techniques Her group isusing these excellent natural language capabilities

to encourage students to participate more fully byresponding to student affect and playing an effec-tive part in the mixed-initiative dialogues thatresult Forbus and Rose´ have combined forces togive Forbus’s well-known CyclePad tutor for engi-neering design[6]a new interface called CycleTalk

[41], which has the ability to hold tutoring dialogueswith considerable success

The systems described so far, like CIRCSIM—Tutor,all make use of written interaction, but the age ofspeech-enabled tutoring has begun Litman hasadded a speech front-end to Atlas to produceITSpoke [42] In a series of well-designed experi-ments, she has shown that ITSpoke produces evenbetter learning outcomes than Atlas Stanley Petersand his team at the Center for the Study of Languageand Information at Stanford University have addedspeech to Wilkins’ Naval Damage Control simulation

to produce SCoT[43] Now that this tutor has beenshown to be effective, it may suggest a way toprovide training for various types of emergency

response teams In summary, there are now several

groups making important contributions to our

knowledge of dialogue-based tutoring

1.4 Domain of CIRCSIM—Tutor–—the

baroreceptor reflex

The cardiovascular system consists of many

mutually interacting components, and it is

impor-tant for the student to understand the cause and

effect relationships between the individual

compo-nents of the system.Fig 1shows a causal model of

CIRCSIM—Tutor, called the ‘‘concept map,’’

designed by Michael and Rovick [44,4] Each box

in the map represents a physiological variable, such

as SV (Stroke volume) and MAP (mean arterial

pres-sure) An arrow with a plus or a minus sign between

two boxes tells the direction of the causal effects

and whether the causal relationship between the

connected variables is direct or inverse For

exam-ple, a qualitative change in one component of the

system, a decrease in CVP (central venous

pres-sure), directly causes a decrease in SV This

quali-tative change propagates to other adjacent

components of the system according to the

propa-gation rule It is important for the student to

recog-nize that when the baroreceptors sense a change in

MAP, the baroreceptor reflex kicks in and the central

nervous system (CNS in the diagram) directly

manip-ulates three neural variables, the heart rate (HR),

the inotropic state (IS), and the total peripheral

resistance (TPR), in order to regulate MAP

There are three stages in the human body’s

response to a perturbation in the system that

con-trols blood pressure The first stage is the direct

response (DR), in which a perturbation in the system

has an immediate physical, hemodynamic effect on

the other parameters The second stage is the reflexresponse (RR), in which other parameters areaffected by the negative feedback mechanism tostabilize the blood pressure The final stage is thesteady state (SS), which is achieved as a balancebetween the changes directly caused by the initialperturbation and the further changes induced by thenegative feedback process

1.5 Organization of this paper

In the next section, we describe what the systemlooks like from the user’s point of view, display asample fragment of dialogue and give a briefreport of the system trial in November 1999, whichdemonstrated that an hour with the system pro-duced larger learning gains than reading a care-fully chosen piece of text for the same amount oftime In Section3, we describe some of the specialfeatures of the CIRCSIM—Tutor system The rest ofthe paper describes how the system works toproduce the kind of dialogue shown in the exam-ple In Section 4, we describe the system archi-tecture and then in the subsequent sections wedescribe each major module in the system and how

it functions Section 5 discusses the core issue ofplanning and describes the many kinds of planningthat are needed for expert tutoring Section 6

describes the domain knowledge base and theproblem solver and Section7describes the screenmanager Section 8 discusses some differentapproaches to understanding the student input.Section 9 describes the student modeler and thedifferent types of assessment that the systemmakes of the student’s performance Section 10

describes our approach to generating output andSection11 presents our conclusions

Figure 1 The causal concept map (An arrow from box A to box B means that parameter A immediately determinesparameter B A plus sign indicates that this relationship is direct; a minus sign indicates that it is inverse.) RV: venousresistance, PIT: intrathoracic pressure, CVP: central venous pressure, CBV: central blood volume, BV: blood volume, SV:Stroke Volume, CO: Cardiac Output, MAP: mean arterial pressure, BR: baroreceptor reflex, CNS: central nervous system,IS: inotropic state, HR: heart rate, TPR: total peripheral resistance

2 CIRCSIM—Tutor in action

2.1 How CIRCSIM—Tutor interacts with

the student

CIRCSIM—Tutor begins with a brief introductory

mes-sage and then displays a list of eight available

procedures (shown in Table 1) These procedures

were developed by Michael and Rovick for use in the

CIRCSIM program and were inherited by CIRCSIM—

Tutor Each procedure (called that because they

replaced experimental procedures with animals)

describes a perturbation of the cardiovascular

sys-tem As soon as the student has made a choice, the

system brings up the screen inFig 2with a

descrip-tion of the procedure in the window on the upper

right and the prediction table underneath.Table 2

shows a larger diagram of the prediction table The

first column is used to enter qualitative predictions

for the DR phase before the baroreceptor kicks in A

popup menu allows the student to enter a ‘‘+’’ sign

to indicate an increase, a ‘‘
’’ for a decrease and a

‘‘0’’ to indicate no change

CIRCSIM—Tutor asks the student to figure outwhich variable will change first and enter thechange for that variable in the correspondingsquare If the student has difficulty in doing this,the system gives the student a hint If that hint doesnot work, it produces a broader hint If the student’sthird try is still wrong, the system tells the studentthe answer Once the student has succeeded inpredicting the first variable, the system asks forpredictions for the rest of the first column withoutgiving any feedback until the student has predictedall six remaining variables The system then marksany errors with a diagonal bar across the box andstarts a remedial dialogue with the student aboutthese errors, as shown in the figure After thestudent has corrected all the errors in the DR col-umn, the system asks for predictions for the RR

Table 1 List of available procedures

1 Decrease arterial resistance (Ra) to 50% of normal

2 Denervate the baroreceptors

3 Decrease Rato 50% of normal in a denervated

preparation

4 Hemorrhage: remove 0.5 liter of blood

5 Hemorrhage: remove an additional 1.0 l of blood

6 Decrease cardiac contractility to 50% of normal

7 Increase venous resistance to 200% of normal

8 Increase intrathoracic pressure to 2 mg Hg

Figure 2 A CIRCSIM—Tutor screen from version 2.8 (November 1999)

Table 2 The CIRCSIM—Tutor prediction table

phase, then again marks any errors, and begins

another tutorial dialogue Once the RR errors have

been corrected, the system asks for predictions

about the behavior of these parameters in the SS

phase (the third and last column) and then again

launches a tutorial dialogue

2.2 The prediction table/multiple

simultaneous inputs

CIRCSIM—Tutor begins with a prediction table, in

which the student is asked to make qualitative

predictions about the behavior of the system given

a particular perturbation CIRCSIM—Tutor inherited

the prediction table from CIRCSIM [2,9] This very

successful, widely used system asks the student to

fill in all three columns of predictions at once,

recognizes certain patterns of errors, and then

delivers one of over 240 targeted remedial

para-graphs stored in the system Michael and Rovick

were convinced that the prediction table was an

important factor in the effectiveness of this older

system Although we believe that immediate

feed-back is valuable (which is why CIRCSIM—Tutor

gath-ers only one column of predictions at a time), we

feel that the advantages of using the prediction

table outweigh that value First, the prediction

table provides the student with a simple mental

model of the task and a way of keeping track of

current progress in the solution process Second,

CIRCSIM—Tutor can make a much more detailed and

sophisticated student model It records errors and

error patterns Some error patterns violate

funda-mental equations; others suggest the possible

pre-sence of important misconceptions Based on a

careful analysis of these errors, the tutor can

gen-erate a lesson plan, and interactive tutoring begins

by using a mixed-initiative Socratic strategy in

nat-ural language Thus, the prediction table provides a

qualitative simulation environment for the student

by requiring multiple simultaneous inputs (multiple

responses to different aspects of a problem provided

by the student in a single uninterrupted turn) before

interactive tutoring begins

There are several benefits of adapting this kind of

design strategy First, the system receives enough

initial knowledge about the student so that it can

narrow the focus for tutoring Second, it can also

detect some common student misconceptions

[45,46] and probe for them further Third, the

pre-sence of a simple mental model of the entire domain

prevents the students from getting too far off the

track Elsom-Cook [47] argues that using multiple

pedagogic strategies can provide a very powerful

learning environment CIRCSIM—Tutor begins with a

coach-like environment during the Prediction Table

entry, and then moves to Socratic tutoring for theinteractive tutoring session This kind of flexibilityprovides a fourth benefit The system can adaptrapidly to the needs of individual students

2.3 A fragment of a CIRCSIM—Tutor dialogue from November 1999

Here is a fragment of dialogue between a studentand CIRCSIM—Tutor in November 1999, in which astudent (Subject #2) is tutored about a predictionerror in the change in SV in the RR phase Theprocedure that the student is solving right now is

‘‘decrease inotropic state by 50%.’’ This situationcan result from several common diseases As ithappens, this student had made all the DR predic-tions correctly The prediction table in Table 2

shows the DR and RR predictions made by thisstudent

 T1: What are the determinants of Stroke Volume?

In this example, student #2 gives the expectedanswer at each stage as we carry out a tutoringstrategy that was abstracted from expert tutoringsessions[48], as described in Section5 Now let uslook at a more typical example, where the studenthas some difficulty with some of these questions.This second example below comes from student #35,who made the exact same prediction error for SV inthe RR phase The system starts off with the samestrategy in T11 as in T1, but the student comes upwith only one of the determinants in S12, so thesystem asks for the other in T13 In S14, the stu-dent’s answer is not too brief but too long Thesystem gives an explanation in T15 and follows upwith a hint

 T11: What are the determinants of Stroke Volume?

 S12: Inotropic state

 T13: Right, Inotropic State is one determinant of

Stroke Volume The other determinant represents

preload What is the other determinant of Stroke

Volume?

 S14: Preload, afterlaod and inotr opic state

 T15: Right, Inotropic State is one MAP, or

after-load, does affect SV, but the effect is not

quanti-tatively important in this case Think about what

parameter in the prediction table represents

pre-load

 S16: Preload

 T17: What parameter in the prediction table

represents preload?

 S18: Central Venous Pressure

 T19: Right, Central Venous Pressure determines

preload, therefore determines Stroke Volume

In this second example the student’s answer

‘‘preload’’ in S16 is categorized as a ‘‘near miss.’’

The student is certainly not wrong and we do not

want the system to respond with ‘‘wrong’’ in this

situation as the first version of the system did The

system gives the student yet another nudge in the

right direction When the student comes up with the

right parameter name, the system recapitulates

what is happening and ties it all up The student

goes on to correct the prediction for SV We

dis-covered that hinting is an important strategy for

human tutors and we have analyzed human hints

[45]in some detail and implemented them in

CIRC-SIM—Tutor[49]

2.4 Brief description of the results of

CIRCSIM—Tutor experiment in November

1999

We carried out an extensive experiment to validate

CIRCSIM—Tutor in November 1998, with 50 first-year

medical students at Rush Medical College, which is

described in detail in Michael et al.[50] In

Novem-ber 1999, we carried out another experiment with acontrol group that shows that these students learnmore about solving problems in an hour with CIR-CSIM—Tutor than in reading carefully chosen textfrom a standard textbook for an hour This experi-ment demonstrated that CIRCSIM—Tutor works andled to its routine use at Rush It was carried out in aregularly scheduled 2-h laboratory All of the stu-dents took a pretest A control group containing 28students read a specially edited chapter on thebaroreceptor reflex, excerpted from Heller andMohrman’s Cardiovascular Physiology [51] by ourexperts The experimental group (with 22 students)used CIRCSIM—Tutor A third group of 23 studentsused CIRCSIM All of the students took a posttest

We had earlier developed two comparable tests,tests a and b In each group half of the students tooktest a as pretest and test b as pretest The studentswho had taken test a as pretest took test b asposttest; while those who took test b as pretesttook test a as posttest Each test had three parts,relationship questions, problem-solving questions,and multiple-choice questions A later analysisshowed that the pairs of multiple-choice questionswere not comparable and so we will not report thoseresults Finally, the students who had used CIRCSIM—Tutor filled out a brief survey form asking for theirreactions to the system More details about thisexperiment can be found in Evens and Michael[12].The system performed pretty well It did notcrash and 60% of the students completed all eightprocedures The students made 96 spelling errorsand the system corrected 91 of them It came upwith something appropriate to say in response to allbut six of the 1692 dialogue inputs In those sixcases, in spite of the inappropriate responses bythe system, the student was able to figure out how

to keep going and continue the session

A summary of the test results appears inTable 3.Using one-tailed t-tests and assuming equal var-

Table 3 Results of the CIRCSIM—Tutor experiment at rush medical college in November 1999

Pretestmean (S.D.)

Posttestmean (S.D.)

Gain (pre—post)

p value

EffectsizeControl (N = 28)

Relationship points (max 24) 14.1 (4.8) 19.9 (4.5) 5.8 <0.001 1.27Correct predictions (max 20) 12.2 (3.0) 13.8 (2.6) 1.6 0.018 0.48CIRCSIM (N = 23)

Relationship points (max 24) 11.0 (5.5) 13.7 (6.8) 2.7 0.071 0.54Correct predictions (max 20) 11.5 (5.1) 16.4 (1.6) 5.3 <0.001 1.05CIRCSIM—Tutor (N = 22)

Relationship points (max 24) 10.9 (5.5) 14.2 (6.3) 3.3 0.039 0.65Correct predictions (max 20) 11.5 (4.8) 16.8 (1.8) 4.9 <0.001 1.24

S.D.: standard deviation.

iance we can see that both the control students and

the students who used CIRCSIM—Tutor learned a

significant amount ( p < 0.05) and the effect sizes

(calculated as the difference between the means

divided by the variance of the gain scores) varied

from moderate to large While the students in the

control group did a better job of memorizing the

relationship information, the CIRCSIM—Tutor

stu-dents did significantly better on the

problem-sol-ving task ( p < 0.001) One argument for comparing

system results with those for students reading a

targeted text is that this choice of assignments is a

real problem for instructors teaching physiology

Our results are comparable with those found by

Graesser’s group [33] The results of the survey

were quite positive, as can be seen in Table 4

Students found the program easy to use; they felt

that they learned a lot; and they would recommend

the program to other students As a result of their

requests, the system was installed on a number of

computers in the open student laboratory More

details of this study can be found in Evens and

Michael [12] Another question of interest is

‘‘How does the effect of using CIRCSIM—Tutor

com-pare with the results of using the old CAI system,

CIRCSIM?’’ As Table 3 shows, the students using

CIRCSIM—Tutor have higher mean learning gains,

but the difference is not significant Since

Michael’s colleagues are now convinced that

CIR-CSIM—Tutor is the better system, it is in routine use

and we have not been able to repeat a full-scale

experiment

3 Significant features of CIRCSIM—

Tutor

Here is a brief list of some of the areas where

CIRCSIM—Tutor has pioneered The system is

mod-eled on the behavior of expert human tutors The

pedagogical knowledge was extracted from their

expert tutoring sessions and represented explicitly

as rules, lesson planning rules and discourse ning rules The rules are used to generate lessonplans and to control discourse strategies The sys-tem interprets the rules and builds the lesson plans

plan-or returns an appropriate discourse action Thediscourse is also modeled on the experts so, likeexpert human tutors, CIRCSIM—Tutor asks questionsand produces hints whenever possible, but almostnever tells the student the answers A detaileddescription of our observations on human tutorscan be found in Evens and Michael[12]

The more time we spent analyzing human ing sessions, the more planning we found As aresult the planner in our system is the centralcontroller It combines two different instructionalplanning approaches: lesson planning and dis-course planning Lesson planning produces globallesson plans The planner then puts together stra-tegies for carrying out those plans and tactics forcarrying out the strategies Plans for carrying outthe strategies are produced by the discourse plan-ning stage The planner plans dynamically based onthe inferred student model; it generates plans,monitors the execution of the plans, and replanswhen the student interrupts with a question duringthe tutoring session The planner plans at differentlevels of the hierarchy; the higher level is anabstraction of the plan (lesson goals) and the lower

tutor-is a detailed plan (subgoals), sufficient to solve theproblem The planner supports some minimal stu-dent initiatives during the tutoring session If thestudent asks a question the planner suspends thecurrent plan, carries out the student request, andthen resumes the suspended plan

The student modeler stores four different levels

of assessment to feed plans at four different levels:curriculum, procedure, phase, and topic The inputunderstander does extensive spelling correction,and then uses a cascade of finite-state machines

to handle free text answers, which are typically

Table 4 Survey and mean responses from November 1999

Your views on CIRCSIM—Tutor (1 = Definitely YES, 2, 3, 4, 5 = Definitely NO) Mean response

3 Entering predictions into the table was easy 1.8

4 Entering answers to the tutor’s questions was easy 2.1

5 The system’s use of language seemed varied and helpful 1.9

6 The tutors hints and explanations were informative 1.85

7 I would prefer that the system always tell me about my mistakes immediately 2.65

8 CIRCSIM—Tutor helped me understand the behavior of the baroreceptor reflex 2.0

9 CIRCSIM—Tutor improved my ability to predict the cardiovascular responses to disturbances

in blood pressure

1.9

10 I would recommend the program to friends taking physiology 1.9

fragmentary The natural language generation

com-ponent generates sentences from logic forms using a

lexical functional grammar approach

4 Architecture of the CIRCSIM—Tutor

system

The typical intelligent tutoring system consists of

four major components[52,53]: the domain

knowl-edge base, a collection of instructional strategies

and an algorithm for applying them, a student

modeler, and an interface Since the major goal

of CIRCSIM—Tutor was to carry on a natural language

dialogue, we divided the interface module into

three pieces, an input understander, a text

genera-tor, and a screen manager Even a few minutes

listening to an expert human tutor in action was

enough to convince us that we needed a dynamic

hierarchical planner as well as a domain problem

solver As a result, CIRCSIM—Tutor has seven major

modules: the instructional planner, the domain

knowledge base, the problem solver, the screen

manager, the input understander, the student

mode-ler, and the text generator.Fig 3shows the overall

architecture of our system as described by Woo

[54,55] The Instructional Planner is in the center

because we discovered that planning is indeed the

central issue in tutoring

Most of our decisions about the architecture were

driven by the need to respond to the students in two

seconds or less, the longest period that we could

hope the students would wait according to the

authoritative literature on interfaces[56—58] This

meant that we had to parse the input, turn it into alogic form, update the student model with theresults, plan a response, and turn it into a series

of sentences in less than two seconds on the kind ofMacintosh that was available in early 1991 As aresult, we chose to use space to save time whenever

a time/space tradeoff presented itself We ered a couple of less conventional approaches to thesystem architecture, especially a blackboard archi-tecture [59] and a more hierarchical type of dis-tributed architecture, and decided against both ofthem, because of these same time constraints

consid-4.1 How the planner and the other modules work together

We now give a brief description of how the modules

inFig 3work together, coordinated by the planner

As an example, we explain how these modulescollaborate to produce the turns S11—S19 in thesecond dialogue in Section2.3 This dialogue carriesout the first part of the lesson goal: tutor thestudent about the value of SV in RR, because thestudent has made an incorrect prediction about thechange in SV in RR

1 Planner: picks the current topic from the subgoalstack, selects the discourse tactic, question, andcombines them into one internal logical form

 current topic: (determined-by SV?), discoursetactic: question

 call Text Generator: (question (determined-bySV?))

2 Text generator: generates the sentence: ‘‘Whatare the determinants of Stroke Volume?’’ andreturns it to the planner, which passes it to thescreen manager

3 Screen manager: displays the sentence T11 in thewindow

4 Student input (S12): ‘‘inotropic state’’

5 Planner: passes the student’s input with thecurrent lesson topic to the input understander(question(determined-by SV?, answer(inotropicstate)))

6 Input understander: parses the student’s answer,checks its coherence with the dialog history, andcalls the student modeler with the logic form(answer(determined-by SV (IS)))

7 Student modeler: calls the problem solver,which, in turn uses the domain knowledge base

to get the correct answer: (CVP, IS), comparesthe correct answer with the student answer,discovers that the student’s answer is partiallycorrect, but it is missing CVP and updates thestudent model, then returns to the input under-

Figure 3 The architecture of CIRCSIM—Tutor

stander with the information: (category partial

(correct IS) (missing CVP))

8 Input understander returns to the planner with

the logic form that it sent to the student modeler

and the information returned by the student

modeler The planner puts onto the lesson goal

stack the new goal (tutor (category partial

(cor-rect IS) (missing CVP)) Then it looks for a rule to

tutor a partial answer

Unfortunately, it does not recognize that this is a

good time to respond with ‘‘and?’’ but instead it

creates three subgoals: give a positive

acknowledg-ment for IS, give a hint about CVP, and then ask for

the other determinant It chooses the top subgoal,

calls the text generator to put it into words, then

gives it to the screen manager to display Then

chooses the next subgoal and repeats the process

The result is T13:

 T13: Right, Inotropic State is one determinant of

Stroke Volume The other determinant represents

preload What is the other determinant of Stroke

Volume?

The planner deals with each of the student

answers in the same way — it hands the answer

to the input understander, which parses the input

and calls the student modeler It takes the

infor-mation from the student modeler, creates a new

lesson goal and one or more subgoals, and puts

them at the top of the stack It then executes

those subgoals one-by-one When it gets to the

end of the output T19, it finally finishes all the new

goals, uncovers the rest of the subgoals from the

original lesson plan and continues with that plan

As you can see, all the modules in the system are

involved with the production of each turn in the

dialogue

5 The instructional planner

The instructional planner is the central component

of our intelligent tutoring system; it is responsible

for making decisions about the content of the lesson

and decisions about its presentation strategy The

planning component of CIRCSIM—Tutor must carry

out both functions, since it needs to provide a global

lesson plan, and it needs to carry on a natural

language exchange with the goal of providing the

most effective instruction possible to the student

The problem of decision-making in an intelligent

tutoring system has long been viewed as a complex

planning problem[59—63] Adaptive planning

tech-niques in the tutoring domain enable the generation

of customized plans for individualized instruction

[64,65]

It was apparent from the beginning of thisproject [54,55,66] that the computer tutorrequired a dynamic hierarchical planner capable

of producing plans just in time to use them Theplanning must be dynamic because we cannotpredict what the student will say The system must

be capable of deleting old plans and making newones at any point It must be hierarchical tohandle multiple levels of planning It must becapable of long-range planning to support lessongoals and multi-turn discourse moves like directedlines of reasoning (DLR’s, see Section 5.5), but itmust postpone lower-level planning until it isneeded Since the student model affects nearlyall lower-level plans and since that model changes

at every turn, it is important to generate level plans just prior to use

lower-The instructional planner also serves as the mainprogram of CIRCSIM—Tutor It consists of three parts(Fig 4): the lesson planner, the discourse planner,and the plan controller, which has already beenbriefly described in Section4.1 The lesson plannersets the instructional goals and develops the tutor-ing strategies and tactics needed to carry them out.The discourse planner turns the tactics into dis-course plans and calls the text generator to producethe actual sentences one at a time These plannersuse explicit planning rules derived from the analysis

of expert human tutoring sessions to build theirplans

Figure 4 The structure of the instructional planner

5.1 The lesson planner

The lesson planner decides on the contents of a

lesson, based on its model of the student’s current

knowledge about the domain The planner

gener-ates the lesson goals, sequences them, and selects

the appropriate planning strategies to create a plan

for the current lesson goal Fig 5shows the

archi-tecture of the lesson planner including the

neces-sary planning steps, the student model, and the

lesson planning rules The lesson planner is a

rule-based system The result of the lesson planning is a

set of subgoals (a plan), each of which will be the

topic for a dialogue with the student

The initial lesson planning is done as soon as the

student finishes filling in a column of predictions

The planner calls the student modeler, which, in

turn, calls the problem solver to get the correct

values for the column The student modeler

com-pares the student’s entries with the correct values,

identifies the errors, updates the student model,

and returns to the lesson planner with a list of

errors The lesson planner calls the screen manager

to turn the corresponding boxes in the prediction

table red and draw diagonal lines across them Then

it looks for patterns of errors and creates a sequence

of remedial goals The student modeler also lists any

potential misconceptions triggered by the student’s

patterns of errors If there are any possible

mis-conceptions, those goals go on the top of the lesson

goal stack Next on the goal stack are any errors in

the variables controlled by the nervous system,

since a discussion of the neural variables may raise

other misconception flags Last on the goal stack are

any errors in the other variables, listed in the

‘‘logical order’’–—the order in which the solution

algorithm determines the correct prediction, since

Michael and Rovick always use that order

More lesson goals and subgoals are added as the

dialogue proceeds Most new goals are added at the

top of the stack to be executed at once This

happens when the student makes errors that reveal

a need for another lesson as in turns S12 and S14 in

Section 4 It happens when the student takes the

initiative, when the lesson goal is to respond to the

initiative, if at all possible It also happens when thesystem fails to understand the student’s input Thenthe lesson goal is to tell the student what kind ofinput the system was expecting and asks the student

to try again

In the experiments in November 1998 and 1999, ifthe student made no errors in a column, then notutoring took place In 1999 a majority of the stu-dents solved all eight procedures and a number ofthose made correct predictions for a whole column

in the last three or four procedures The result wasthat the best students were getting the least sti-mulation In order to add stimulation and also toprobe more effectively for student misconceptions,

we developed a dozen open questions to ask thestudents in this situation, which we deployed inNovember 2002 Most students responded to thequestions with longer, more thoughtful answers

[67] This gives us a strong motivation to expandthe input understander to parse, categorize, andrepresent these answers so that the system canrespond intelligently, as described in Section9

5.2 Lesson planning rules

The lesson planner uses three sets of lesson planningrules: goal generation rules, strategy rules, andtactical rules The rules are written in if-then form

as explicit production rules and interpreted by therule interpreter The system has about 50 goal gen-eration rules, 20 strategy rules, and 20 tacticalrules

The interpreter is built using Lisp macro tions, which understand and interpret the rules forthe system As a result the rules can be written,not as Lisp code, but in any free format as long asthe rule interpreter can understand them Wedesigned the rules with three parts: the namepart of the rule, the antecedent part, and theconsequent part, in the form: (Rule_name: (ante-cedent)) (consequent)) This approach makesthe system efficient in representing the rulesexplicitly

func-For example, assume that the student made anerror in predicting the variable TPR One of the goalgeneration rules applies; if the student does not knowTPR, then build the lesson goal, tutor TPR aboutthe neural control This rule can be expressed as(G_Rule1: ((do-not-know TPR)) (neural-controlTPR))) If the current lesson goal is to teach thecausal relationship between central venous pressure(CVP) and SV, and the student does not know thedirection, then this rule can be written as (S_Rule1:((causal-relation)(do-not-know direction))) (tutor-causality))) This is the strategy rule for dealing withnon-neural variables If the strategy rule is tutor-

Figure 5 Structure of the lesson planner