Decision theoretic intelligent tutoring system

The student model is explicitly represented by a set of Bayesian networks, where their random variables denote the stochastic information associated with the mastery states for the learn

DECISION-THEORETIC INTELLIGENT TUTORING SYSTEM

PEK PENG KIAT

M Sc (Distinction), The University of Sheffield

M Eng., National University of Singapore

B Eng (Second Upper), National University of Singapore

A THESIS SUBMITTED FOR THE DEGREE OF DOCTOR OF PHILOSOPHY

DEPARTMENT OF INDUSTRIAL & SYSTEMS ENGINEERING

NATIONAL UNIVERSITY OF SINGAPORE

2003

ACKNOWLEDGEMENTS

I wish to express my special appreciation to my wife Lian Kuan for her thoughtful and challenging discussions of the ideas in the research, and to my students at Singapore

Polytechnic who had volunteered to use iTutor during some of their tutorials I also wish to

express my gratitude to Associate Professor Poh Kim Leng, my research supervisor, for having trusted my ability to carry out the research and his guidance throughout the course of formalizing the ideas I would like to thank Associate Professors Ong Hoon Liong and Tan Kay Chuan (members of my Thesis Committee) for their very helpful comments during the formative stage of the research In addition, I wish to thank Ms Ow Lai Chun for her excellent administrative support Finally, I wish to express my gratitude to Singapore Polytechnic management for having supported the research, without which I would not be able to complete the work

TABLE OF CONTENTS

Acknowledgements i

Table of Contents ii

List of Figures vii

List of Tables x

Summary xii

Nomenclature xiv

Quote xvi

Chapter 1 Introduction 1

1.1 Components of Intelligent Tutoring System 3

1.2 Current Intelligent Tutoring System 5

1.2.1 Inadequate Information on Student’s Mastery State 7

1.2.2 Making Pedagogical Decision Based on Heuristics 7

1.2.3 Ineffective Use of Test Items 8

1.3 Scope of the Research 9

1.3.1 Representing Knowledge Structure as Bayesian Networks 9

1.3.2 Decision-theoretic Approach to Computerised Tutoring 9

1.3.3 Application of Item-Response Model to Tutoring 10

1.3.4 Generating Optimal Tutoring Policy 11

1.4 iTutor: A Decision-Theoretic Tutoring System 12

1.5 Organisation of this Thesis 17

Chapter 2 Literature Review 18

2.1 Construction of Student Model 21

2.1.1 Stereotypes Modelling 21

2.1.2 Overlay Modelling 23

(a) Expert-Centric Models 24

(b) Efficiency-Centric Models 26

(c) Data-Centric Models 27

2.1.3 Extended Overlay Modelling 29

(a) Enumerative Modelling 30

(b) Generative Modelling 32

(c) Reconstructive Modelling 33

2.2 Diagnosis 35

2.2.1 Model Tracing 35

2.2.2 Constraint-Based Modelling 37

2.3 Learning Theories 39

2.4 Action Selection 41

2.4.1 Heuristic Approach 41

2.4.2 Decision-Theoretic Approach 42

2.5 Assessment 43

2.6 Human-Computer Interaction 45

2.7 Approach Adopted by This Research 47

Chapter 3 Student Model 50

3.1 Forms of Knowledge 51

3.1.1 Online Resource Materials 51

3.1.2 Learning Objectives 53

3.1.3 Student Model 55

3.1.4 Buggy Knowledge 61

3.2 Completion of Probability Tables 63

3.2.1 Prior Probabilities from Expert’s Judgment 64

3.2.2 Prior Probabilities from Empirical Data 66

3.2.3 Conditional Probabilities from Expert’s Judgment 68

3.2.4 Conditional Probabilities from Empirical Data 70

3.3 Inference 72

3.3.1 Evidence Collection 72

3.3.2 Linking Evidence to Mastery State 73

3.3.3 Probabilistic Relevance Among Learning Objectives 74

3.3.4 Posterior Belief of Student’s Mastery State 77

Chapter 4 Modelling of Tutoring Strategy 80

4.1 Making Decisions with Incomplete Information 81

4.2 Learning Values 83

4.3 Action Selection 90

4.4 Item Selection 93

4.5 Tutoring Policy 99

4.5.1 Dynamic Belief Network, DBN 100

4.5.2 Dynamic Decision Network, DDN 102

4.6 Hint Generation 109

Chapter 5 Development and Evaluation of iTutor 111

5.1 Requirements for Applying Decision-Theoretic Approach 113

5.2 Architecture of iTutor 114

5.2.1 Database Management System 114

5.2.2 Human-Computer Interaction 118

5.2.3 Spiral Tutoring Strategy 119

5.3 Items Calibration 120

5.4 Evaluation of iTutor 124

5.4.1 Evaluation 1: Tutoring Policy 124

5.4.2 Evaluation 2: Adaptive Tutoring 132

5.4.3 Evaluation 3: System Effectiveness 136

Chapter 6 Conclusions 141

6.1 Related Works 141

6.2 Summary of Contributions 143

6.2.1 Bayesian Network for Modelling Student’s Knowledge Mastery 144

6.2.2 Decision-Theoretic Approach for Tutoring Action Selection 144

6.2.3 Item-Response Model for Item Selection 145

6.2.4 Dynamic Decision Network for Tutoring Policy Generation 145

6.3 Future Works 146

6.3.1 Refinement of BNs’ Structure by Learning form Data 146

6.3.2 Inclusion of Motivation to Learn in Item Response Model 147

6.3.3 Extending Decision-Theoretic Tutoring to Web-Based Activities 148

References 149 Appendices

Appendix A Learning Outcomes for Two Key Concepts ( “Forces” and “Friction

and Screw Jack” ) 169

Appendix B Set of Bayesian Networks in iTutor 170

Appendix C Table of Conditional Probability Values for an Objective with Three

Conditions 174 Appendix D WINBUG Program on Parameter Estimation 176

LIST OF FIGURES

Fig 1.1 Interactions of components in intelligent tutoring system 3

Fig 1.2 Process flowchart in computerised adaptive tutoring 6

Fig 1.3 Login screen 12

Fig 1.4 An item presented to the student 13

Fig 1.5 Topic on “Friction and Screw Jack.” 13

Fig 1.6 Feedback from iTutor when the student’s answer is incorrect 14

Fig 1.7 An item testing “Resolution of Vector.” 15

Fig 1.8 User interface for teacher to track student’s progress 15

Fig 1.9 Output of a student’s mastery states 16

Fig 1.10 A policy with three actions 16

Fig 2.1 A classification of Student Model using BN 24

Fig 2.2 A dynamic BN modelling the mastery of the student on a single knowledge unit

27

Fig 2.3 Extended overlay student model 29

Fig 2.4 A decision tree for a single decision 43

Fig 2.5 Example of a text page with free-form test items and adaptive annotation of links 46

Fig 3.1 Problem on “Acceleration of Connected Bodies.” 52

Fig 3.2 MathCAD solution 52

Fig 3.3 Interactive graphical display 53

Fig 3.4 A generalised BN in the student model 56

Fig 3.5 BN of the key concept on “Forces.” 59

Fig 3.6 Relationship between the BNs on “Forces” and “Units & Dimensions.” 60

Fig 3.7 (a) An item testing the LO on “Resultant vector.”

(b) Feedback for option (b) (c) Feedback for option (c)

(d) Feedback for option (d) 62

Fig 3.8 A subset of the BN on “Forces” with the tables of probability values 64

Fig 3.9 A solution to item testing “7.5 Angular Motion Formula.” 68

Fig 3.10 Solution by student #1 71

Fig 3.11 Two alternatives in modelling relationships among the LOs 75

Fig 3.12 Initial state of the BN on the topic “Forces” (before observing any evidence) 78

Fig 3.13 State of BN after instantiating the node “Vector Addition.” 79

Fig 4.1 Additive independence of 2-attribute utility function 89

Fig 4.2 (a) Influence diagram for a simple decision problem (b) Decision tree representation of the simple decision problem 91

Fig 4.3 Decision model for the topic on “Forces.” 93

Fig 4.4 DDN that incorporates 4 items 94

Fig 4.5 (a) Decision network for LO selection (b) Selection of an item 94

Fig 4.6 Example of item selection for the topic on “Forces.” 95

Fig 4.7 Algorithm for computing students’ mastery states for each topic 96

Fig 4.8 Three sample questions 98

Fig 4.9 Item-student response matrix 99

Fig 4.10 A two-slice fragment of a DBN for monitoring of student’s mastery state 101

(a) Generic structure (b) Specific BN Fig 4.11 Dynamic decision network for a tutoring policy {a(1), a(2), …, a (n)} 103

Fig 4.12 Search tree for part of the DDN in Figure 4.11 104

Fig 4.13 Hint generation based on the targeted LO (shaded node) 110

Fig 5.1 Architecture of decision-theoretic tutoring system 114

Fig 5.2 Data entry by the teacher on prior probability values and path length 116

Fig 5.3 Overview of Progress Tracking module 117

(a) User interface for teacher to track student’s progress

(b) Bayesian network running as background process (transparent to user) (c) Student’s mastery states for key concept “Forces.”

(d) Dynamic decision network running as background process (transparent to user)

(e) Tutoring policy with three actions

Fig 5.4 Flowchart on tutoring strategy 119 Fig 5.5 Illustration of Gelman-Rubin convergence diagnostic for the difficulty parameter

of item #1 122 Fig 5.6 BN of the key concept on “Forces.” 125 Fig 5.7 Number of actions for achieving full mastery 136

LIST OF TABLES

Table 2.1 Issues and points of discussion in the literature review of ITS 19

Table 2.2 Two approaches for action selection 41

Table 3.1 Defining learning objectives using Mager’s approach 54

Table 3.2 Defining learning objectives using Gronlund’s format 54

Table 3.3 Learning objectives for “Forces.” 58

Table 3.4 Seven difficulty categories of a typical learning objective 65

Table 3.5 Summary of students’ responses to the item shown in Figure 3.9 68

Table 3.6 Mastery states of fifty-nine student who attempted the item shown in Figure 3.10 71

Table 3.7 Conditional probability Pr(“Vector Addition” | “Vector”, “Resolution”) 72

Table 3.8 Effects of a positive evidence (v jk = 1) on the mastery states of two different initial conditions 74

Table 4.1 Latent ability distributions for three students in the sample 98

Table 4.2 (a) Tutoring policy with 4 actions (b) Utility values and optimal actions for a student 108

Table 5.1 iTutor database 115

Table 5.2 Calibration results of ten items 124

Table 5.3 Prior mastery states of the students 125

Table 5.4 (a) A sequence of tutoring actions for Case One: all correct responses (b) Posterior mastery states of student for Case One 126

Table 5.5 (a) A sequence of tutoring actions for Case Two: having difficulty in items with difficulty level 3 (b) Posterior mastery states of student for Case Two 127

Table 5.6 (a) A sequence of tutoring actions for Case Three: having difficulty in LO L2_6 (b) Posterior mastery states of student for Case Three 128

Table 5.7 (a) A sequence of tutoring actions for Case Four: learn from examples (b) Posterior mastery states of student for Case Four 129

Table 5.8 (a) A sequence of tutoring actions for Case Five: atypical responses

(b) Posterior mastery states of student for Case Five 130

Table 5.9 (a) A sequence of tutoring actions for Case Six: all wrong responses (b) Posterior mastery states of student for Case Six 131

Table 5.10 Probability of the mastery states for the simulated students 132

Table 5.11 Summary of simulation I results 133

Table 5.12 Summary of simulation II results 135

Table 5.13 Distributions for the number of tutoring actions 135

Table 5.14 Improvement scores and ranking for both groups of students 138

Table 5.15 Test Statistics 139

Table 5.16 Summary of student feedback 140

SUMMARY

Until recently, computerized tutoring research has been focused on student modelling Literature survey reveals that construction of student model can be broadly classified into stereotypes, overlay, and extended overlay Diagnosis of students’ misconceptions is generally through model tracing or constraint-based modelling Tutoring actions are usually selected based on heuristics and implemented through rule-based inference Using normative approach to pedagogical action selection has not received much attention by student modelling researchers

This thesis seeks to demonstrate that decision-theoretic pedagogical action selection is robust because of sound probabilistic reasoning and defensible decisions The student model

is explicitly represented by a set of Bayesian networks, where their random variables denote the stochastic information associated with the mastery states for the learnable units (or learning objectives) of the domain knowledge Automated student diagnosis is possible when the decision node (that contains the tutoring alternatives) and the utility node (that formalized the learning values) are incorporated to the Bayesian network The tutoring strategy commonly used by the teacher is formulated in the form of a multiple-attribute utility function These attributes include discriminatory power of a learning objective for mastery and non-mastery of the key concept, relatedness of a learning objective to mastery of the key concept, and the student’s readiness to leave the tutoring session This research also extends the diagnostic capability of the student model to include selection of items to match students’ mastery levels This ensures that the test items are challenging to the students

Tutoring policy is an overall plan intended to guide and determine actions for tutoring

a student Since each attribute is additive utility independent of the others, the attribute utility function satisfies the separability condition In addition, the attributes are independent of the historical tutoring actions Therefore, an optimal action at each time slice

multiple-of the tutoring policy can be determined Moreover, with the incorporation multiple-of the attribute on student readiness to leave the tutoring session, a tutoring policy can always be generated in polynomial time In this way, an optimal policy can be determined at any state of the student’s mastery Such a policy is personalised to the student where sequence of actions is able to direct student’s learning in the most effective and efficient manner

A prototype decision-theoretic intelligent tutoring system known as iTutor has been

developed, which possesses knowledge associated with first year Engineering Mechanics that

is taught at Singapore Polytechnic Simulated students are used to evaluate iTutor because it

can cover wide spectrum of abilities and responses effectively and efficiently The

evaluations show that iTutor is able to generate tutoring policies for all possible cases Another evaluation where each student in the treatment group is provided with iTutor has

been performed, and the result indicates that the average test score improvement of these students is better Finally, this thesis suggests some areas of possible future research, which may extend the maintainability, capability, and accessibility of computerized tutoring

ψ Set of dependencies relationship used in Student Model definition

Φ Set of dependencies relationship used in Decision-theoretic Tutoring definition

≤ Preference order

σ Standard deviation

βi The difficulty level of item i

ζj j th Student’s expected mastery value

ℑm Bayesian network of topic m

N (t) Mastery state at time slice t

a (t) Tutoring action at time slice t

a* Optimal decision

A * Optimal policy

b Number of difficulty levels

BN Bayesian network

C Set of possible outcomes

ch(n) Child node of node n

D Set of actions {a1, a2, …, a i}

E Evidential space

E (t) Evidence at time slice t

EMU Expected mastery value

EU(d) Expected utility of an action d

H Hint level appropriate to the student who seeks help

LO Learning objective

M One of the mastery states: full mastery

N G Sets of random variables for mastery of a topic (key concept)

N k The student’s mastery state of the k th learning objective

N k’ New mastery state

N L Sets of random variables for mastery of learning objectives

Symbol Description

NM One of the mastery states: non-mastery

N u Utility node

pa(n) Parent node of node n

PM One of the mastery states: partial mastery

Pr(A) Probability of A

Pr(A|B) Probability of A given B

Q Set of items available for testing student mastery state

S States space

s Number of states in S

SD Standard deviation

S P Non-numerical state in S

U(c) Utility value of an outcome c

v jk Evidence used to instantiate the student’s (j) mastery state of learning objective k

x ij Score of the j th student’s response to i th item

P lants are fashioned by cultivation, men by education

We are born feeble and need strength; possessing nothing, we need assistance; beginning without intelligence, we need judgment All that we lack at birth

and need when grown up is given us by education.

- Jean Jacques Rousseau

Chapter 1

Introduction

The roots of education are bitter, but the fruit is sweet

- Aristotle

Over the past two decades, considerable research has been reported in the development

of computer programs for effectively teaching students These programs are called

Intelligent Tutoring Systems (ITSs) The desire to build ITSs stems from observations

of the effectiveness of one-to-one tutoring Presently, one-to-one tutoring is not

feasible because the number of students greatly outweighs the number of teachers

With the advent of affordable and high-speed personal computers, the dream takes one

step closer to reality If the computer could be programmed to emulate the teacher,

one-to-one tutoring would become a possibility

The inspiration of emulating the human tutor comes from works in Artificial

Intelligence (AI) and, in particular, expert system In principle, one has to program the

computer with the domain knowledge and pedagogy that are drawn from cognitive and

instructional science Ideally, natural language processing would allow the student to

converse with the computer in the same way he could converse with a human tutor

This aspiring initial approach rapidly proves to be an overly ambitious

challenge Firstly, natural language processing, while it has made great advances in

recent years, is not ready to support natural language conversations between the tutor

and the student As a result, most ITSs have been developed with alternate interfaces

Secondly, it is realised that an ITS can never completely replace a teacher There will

always be students for whom computer-based instruction is not suitable There may

also be particular topics for which computer-based instruction is not appropriate The

goal, therefore, has been redefined to the much more realistic aspiration of supporting

the teacher, be it in the classroom, the workplace, or at home By having ITSs as

supporting tools for the teacher, students are able to learn at their own pace and more

time becomes available for the teacher to focus on one-to-one situations with students

who may not be responding well to the ITS approach

An overview of the thesis is provided in this chapter Although there are many

reported instances of ITSs, they have many things in common and their common

architecture is described in Section 1.1, and serves as the guide for understanding their

functions In Section 1.2, the types and effects of the uncertainty inherent in ITS is

discussed In addition, this thesis proposes a solution based on normative theory which

defines rational behaviour in view of the uncertainty present in the environment By

implementing normative theory in ITS, the rationality of tutoring actions can be

assured that are consistent with students’ responses The scope of this research is

summarised in Section 1.3 In Section 1.4, a fully functional ITS, calls iTutor, which

is used to demonstrate decision-theoretic tutoring is introduced, while a guide to the

rest of the thesis is discussed in Section 1.5

1.1 Components of Intelligent Tutoring System

Most ITSs may outwardly appear to be monolithic systems, but for the purposes of

conceptualisation and design, it is often easier to think about them as consisting of

several interdependent components Research by Beck et al (1997) had identified five

major components: student model, pedagogical model, domain knowledge module,

expert model, and communication module Figure 1.1 provides a view of the

interactions among the components

The student model stores information that is specific to each individual learner

Central to the development of the student model is a set of variables corresponding to

aspects of skill and knowledge that are important in the domain These variables could

be qualitative or numerical, and concern tendencies in behaviour, conceptions of

phenomena, availability of strategies, and levels of expertise In addition, these

variables may be conceived as persisting over time or likely to change at the next

problem step

The pedagogical model provides a model of the teaching process It uses

information from the student model to determine what aspects of the domain

knowledge should be presented to the learner One pedagogical concern for ITS is the

selection of a tutoring policy (a set of actions) for teaching the domain Examples of

Student Model

Pedagogical Model

Communication

Module

Expert Domain

Knowledge Model

Fig 1.1 Interactions of components in intelligent tutoring system

actions in the tutoring policy are problem selection, topic selection, adaptive

presentation of error messages, and selective highlighting Once the actions are

selected, low level issues, such as which example to present and what item to test the

student are then decided

Effective tutoring systems (Shute, 1998) contain a lot of knowledge The

domain knowledge contains information the tutor is teaching, and is an important

component since without it, there would be nothing to teach the student One issue is

how to represent knowledge so that it easily scales up to larger domain Another issue

is how to represent domain knowledge other than facts and procedures, such as

concepts and mental models A related issue of domain knowledge concerns the

explicit representation of test items These items are useful in gauging the students’

knowledge mastery and for diagnosing buggy knowledge Mitrović et al (1996)

points out that in ideal case, if an ITS is required for a different domain, the domain

knowledge module is the only component of the system that needs to be changed

The expert model is similar to the domain knowledge in that it must contain the

information being taught to the learner However, it is a model of how someone

skilled in a particular domain solves problems using that knowledge The expert

model is able to: (1) provide the source for the knowledge in explaining student’s

responses; and (2) serve as a standard for evaluating student’s performance The

extent where the two functions is realised depends on the granularity of the domain

knowledge that can be explicitly represented With fine grain size, the expert model

can be used to generate the solution paths so that intermediate steps can be compared

When this type of comparison is possible, the expert model is said to represent the

teaching goals (Wenger, 1987)

The communication module controls interactions with the learner, including the

dialogue and the screen layouts The objective of this module is on how the material

should be presented to the student in the most effective way The interface can

improve learning significantly by reducing the cognitive load If only one component

of a problem is the focus of teaching, then the rest of the problem can be “externally

stored” within the user interface This means the student does not need to remember

extraneous details and can instead target the component of problem solving that is of

importance

1.2 Current Intelligent Tutoring System

Tutorials offer opportunities for students to discuss with the teacher on concepts taught

earlier in the lectures However, teacher faces great challenges during tutorials since

students possess different levels of knowledge mastery Some students may already

understand the fundamentals, and only want to affirm their comprehension of new

knowledge This usually takes the form of the teacher checking students’ solutions to

some tutorial problems, and correcting minor mistakes along the way if there is any

Other students may have difficulties in understanding the subject matter due to

weak mastery of pre-requisite skills or faulty concepts, but may not realise their areas

of misconceptions A common decision by teacher is to identify the areas where

students require assistance so as to provide suitable remedial actions Subsequently,

these students are asked to work on a few relevant tutorial problems From the

students’ responses, the teacher looks for evidence of their comprehension of the

subject matter This tutoring process may be repeated if the desired students’

knowledge mastery is not achieved

Unlike human tutor, ITS faces greater challenges due to the restricted

observation of student’s behaviour: usually through computer keyboard and mouse

Figure 1.2 shows the typical pedagogical actions in ITS Initially, the student is

allowed to select a topic The tutor searches for an appropriate action based on

heuristics stored in the system The tutor can select an item based either on the

student’s prior or predefined mastery level The student’s response to the item is

matched with the stored model answer If his response is incorrect, the tutor searches

for an appropriate feedback Subsequently, the tutor determines whether the student is

ready to leave the tutoring session If the student has not mastered the topic, the tutor

Present the hint

Yes

Determine misconception

Present the feedback Yes

Yes

No

Present the

feedback

determines another suitable action This action selection process continues until the

desired level of mastery has been attained

Through the structured activities, the tutor attempts to personalise the student

model and provide coaching based on the student’s states of knowledge mastery

However, in order for ITS to perform near to the expertise of a teacher, three key

difficulties have to be addressed, and are discussed in the following subsections:

1.2.1 Inadequate Information on Student’s Mastery State

Several questions arise when one attempts to equip the computer for the tutor role Is

the student ready to learn new knowledge? How does the tutor know whether the

student requires hint to start the problem? What feedback is useful for the student?

The solutions to these questions and many others usually involved human judgements

and the evidence is mostly inexact or incomplete During the formative stage of

learning, using test scores as the measure of students’ abilities may not be effective

because of inadequate information on their mastery states of the learnable units (or

learning objectives) The state of student’s knowledge is dynamic since it is subjected

to learning and forgetting

In ITS, there is currently no systematic method to incrementally raise the

mastery states of students The tutor action is usually directed towards correcting the

mistakes made by students to questions When no (or minimal) error is made, the

students are assumed to have complete mastery of the knowledge Little attention is

paid to the misconception a student may possess or he may get the answers correct

without real understanding

1.2.2 Making Pedagogical Decision Based on Heuristics

Different actions are taken during tutoring to guide the students towards raising their

mastery of the key concept The tutoring actions must be selected in the same way

human tutor would, such as provides feedback to clarify student’s misconception,

gives hint when student is at an impasse on a problem, presents a learnable unit to

student when he lacks that knowledge, and promotes the student to the next topic when

he has mastered the last one

There is no consistent measurement that compares one remedial action against

another that assists students to maximise their mastery of knowledge Pedagogical

decisions are usually made based on the teacher’s training and experience, but are not

explicitly represented in most computerised tutoring system In addition, as tutoring

actions must be made in reasonable time to prevent students from being bored, this

poses a challenge if the number of alternatives is large Nevertheless, making wrong

diagnosis of student’s misconception is an unacceptable reason for choosing fast

solution to these problems

1.2.3 Ineffective Use of Test Items

Assessing student’s knowledge mastery is a necessary part of tutoring However, item

selection has not been the main focus for most ITSs Problems are arbitrarily selected

and presented to the student The emphasis is more on eliciting the student’s solutions

to items This may result in academically good students being given easy items or

academically weak students being given difficult items Information from the

student’s response, if available, will not be useful for diagnosis

The assessment questions are usually grouped under broad areas of the topic

Student's solution is matched against the model answer and mark is awarded according

to its completeness The mark serves as a gross measure of a student mastery of the

topic tested in the question However, the mark does not provide adequate information

on his mastery levels of the learning units associated with the topic

1.3 Scope of the Research

This thesis addresses the three key difficulties through integrating sound probabilistic

reasoning, decision theory, and item-response model Although decision-theoretic

technique has been around for some time, its application in education, especially in

computer-based tutoring is uncommon The scope of this research is presented in the

following subsections:

1.3.1 Representing Knowledge Structure as Bayesian Networks

It is known from expert systems development that knowledge acquisition and

representation are major tasks (Giarratano and Riley, 1998) In this research, the

knowledge component is obtained from the specifications of learning objectives (LOs)

that are written in the course syllabus (two samples are shown in Appendix A) Each

LO dictates what knowledge unit is to be taught to the student and at which level of the

cognitive domain (McCormick and Pressley, 1997)

Similar to mind mapping (Buzan and Buzan, 1993), the association among

variables in the knowledge structure is used to build the conditional dependencies of

the variables in the Bayesian network (Pearl, 1988) The approach of using the LOs

and their causal relationships to develop a set of Bayesian networks (BNs) will be

formalised in Chapter Three Such network is an efficient representation of

hierarchical information, where inclusion of new information only affects local

conditional relationships

Probability theory describes what an agent should believe on the basis of evidence;

utility theory describes what an agent wants, and decision theory puts the two together

to describe what an agent should do Decision analysis provides a philosophy that

emphasises the insights that can be gained by a decision-maker who goes through the

decision-analytical processes For example, the decision-analytic approach highlights

the distinction between a good decision and a good outcome Heckerman (1991)

defines a good decision is one that is consistent with the preferences and complete

information of a decision-maker; while a good outcome is desirable

Unlike decision-theoretic (Raiffa, 1968) model in most business problems,

utility functions in tutoring are not based on risk attitude of the decision-maker

(teacher) Similarly, the expected utility that is based on dollar equivalent is not useful

in tutoring context Decision-theoretic approach in a computerised tutoring system

aims to provide optimal action selection which maximises student learning and is

defensible Pedagogy can be incorporated into decision analysis to measure learning

value, besides also taking into consideration uncertainty in student’s knowledge

mastery (Pek and Poh, 2002) In this way, one can be sure the correct or gold standard

for the tutoring decisions can be achieved After an action is decided, other

consequential actions such as which learnable unit to present and which item to use

can be determined The utility functions that are formulated according to the learning

value for each tutoring action will be discussed in Chapter Four

1.3.3 Application of Item-Response Model to Tutoring

For adaptive tutoring to work, it is important to rank the test items according to their

difficulty This requires a sound statistical measurement of students’ mastery levels

and items difficulty levels The items parameters are estimated based on students’

responses This could be performed by applying the concept of item response theory

(Hambleton et al., 1991) to calibrate the items difficulty levels against the students’

mastery levels associated with the key concept in the BN With the existence of a

common scale for item difficulty and student’s mastery level, appropriate items can

then be presented to assess the students Moreover, this common scale forms the basis

to determine how well a particular student will perform when he is given an item

In addition to calibrated items, there is a need to associate them to the learning

objectives they are testing Further, if multiple-choice items are used, the distractors

can be designed to represent common students’ misconceptions Feedbacks can then

be provided for wrong student’s response With these features, the student model can

be constantly updated with student’s mastery level and feedback on his misconceptions

can be provided The formalisation of an item-response model and its application to

student diagnosis will be described in detail in Chapter Four

1.3.4 Generating Optimal Tutoring Policy

Student’s learning is seldom completed with one tutoring action A tutoring policy

consists of planned sequence of actions to guide student learning The computer

selects a tutoring action based on student’s current mastery state The response affects

the student’s next mastery state The goal is to determine actions that seek to

maximise information about the student’s misconceptions or faulty knowledge

Ultimately, this sequence of actions will lead the student to learn in the shortest

possible time Since information obtained from the student’s response to test item is

usually incomplete, computerised tutoring can be treated as a partially observable

Markov decision problem (POMDP) (Monahan, 1982; Lane, 1989; Cassandra et al.,

1994) It will be shown in Chapter Four of this thesis that optimal tutoring policy can

be determined by maximising the expected utilities for the sequence of actions

Moreover, the multiple attributes utility function is formulated to satisfy the

separability condition and the readiness to leave the tutoring session is constructed

This ensures that the policy can be generated in polynomial time and yet personalised

to the student’s needs

1.4 iTutor: A Decision-Theoretic Tutoring System

The ideas formalised in this research have been applied to the development of a

functional decision-theoretic intelligent tutoring system called iTutor iTutor is

currently used for tutoring Engineering Mechanics for first year students at the School

of Mechanical and Manufacturing Engineering in Singapore Polytechnic Mechanics

is the first analytical science, where some of its fundamental concepts are found in

virtually every field of engineering For example, students of chemical and electrical

engineering gain a deeper appreciation for basic concepts in their fields such as

equilibrium, energy, and stability by learning them in their original mechanical

contexts Additionally, Mechanics is representative of most engineering domains

because of its emphasis on concept formation, mathematics and problem solving skills

iTutor applies decision-theoretic approach to emulate what a teacher will do to

assist students in their learning Let us illustrate a short interaction between iTutor and

a student Figure 1.3 shows the initial iTutor screen After successful login, iTutor

prompts the student to select a topic that he likes to be tutored on The available topics

are “Units & Dimensions,” “Forces,” “Moments of a Force,” “Free Body Diagram,”

“Equilibrium Conditions,” “Friction and Screw Jack,” “Kinematics,” “Newton’s Laws

of Motion,” and “Torque and Moment of Inertia.” Consider the key concept on “6.0

Friction and Screw Jack” is selected Based on his prior mastery state, the decision is

Intelligent Tutoring System

to assess him A multiple-choice item (Figure 1.4) is selected from the item bank to

test his mastery of the learning objective on “6.11 Inclined Plane.” Figure 1.5

illustrates the location of “6.11 Inclined Plane” in the BN Assuming the student has

selected a wrong option, iTutor provides feedback that addresses his likely

misconception

Fig 1.4 An item presented to the student

Intelligent Tutoring System

Feedback will be provided after you have selected the option.

The coefficient of static friction between the block and the surface of the inclined plane is 0.5 Determine the least value of M if the system is to be at rest.

60 0 90 0

12 kg

M

(a) 13.39 kg (b) 10.39 kg (c) 7.392 kg (d) 0.806 kg

Key Concept: Friction

6.0 Friction and Screw Jack

4.0 Free Body Diagram

6.1 Definition

6.12 Screw Thread

6.9 Rough Surface

6.11 Inclined Plane

6.10 Horizontal Plane

2.0 Forces

Fig 1.5 Topic on “Friction and Screw Jack.”

The feedback is coded in Microsoft PowerPoint where simple multimedia

effects can be incorporated to illustrate the solution sequence and to alert the student

on his likely mistake Figure 1.6 shows two shaded regions indicating the student’s

likely bugs In this example, the buggy knowledge is identified to be associated with

“2.4 Resolution of Vector,” which is the learning objective in another BN “2.0

Forces.” Referring to Figure 1.5, the key concept “2.0 Forces” is a pre-requisite of

topic “6.0 Friction and Screw Jack.” The BNs of topics available in iTutor are shown

in Appendix B

Fig 1.6 Feedback from iTutor when the student’s answer is incorrect

To ascertain the student’s buggy knowledge is “2.4 Resolution of Vector,”

iTutor selects an item (Figure 1.7) on this learning objective If the student has

responded incorrectly, a brief explanation on “2.4 Resolution of Vector” is presented

Otherwise, iTutor activates the next tutoring action in the policy These processes

where the student solves problems and receives feedback continue until he is ready to

leave the tutoring sessions In conventional situation, teacher depends heavily on her

experience when she tutors her students With iTutor, decision analysis is formalised

and optimal actions are automatically selected based on student’s responses

Intelligent Tutoring System

Feedback will be provided after you have selected the option.

Resolve the given force into the specified directions.

(a) Ft= 433 N , Fn= 250 N

(b) Ft= 433 N , Fn= 250 N

(c) Ft= 250 N , Fn= 433 N

(d) Ft= 250 N , Fn= 433 N

Key Concept: Forces

Fig 1.7 An item testing “Resolution of Vector.”

Since tutorials are usually conducted with groups of 15 to 20 students, and time

allocated for tutorials is at most two hours per week, it may not be possible for teachers

to obtain accurate diagnostic profiles of all their students iTutor provides the teacher

the facility to check the student’s progress through the Progress Tracking module

(Figure 1.8) By clicking the “Advice” button, the diagnostic profile of the student is

displayed (Figure 1.9) The student’s mastery levels are presented in an easy to read

format instead of the conditional probability tables

Intelligent Tutoring System

Student ID: 01001 The stu n de t's maste ry st ates a re :

e

7 5

0 5 5

Based on the knowledge states, you may want to provide co

Student ID: s0001 The student's mastery states are : Partial Full Expected Mastery Learning Objective Non-Mastery Mastery Mastery Value

L2_2 Vector Addition 0.000 0.000 1.000 95 L2_5 Direction 0.000 1.000 0.000 55 L2_6 Angle 0.600 0.250 0.150 34 L2_7 Magnitude 0.600 0.250 0.150 34 L2_3 Resultant Vector 0.302 0.373 0.325 54 L2_4 Resolution 0.011 0.011 0.978 94 G1 Units & Dimensions 0.100 0.250 0.650 76 G2 Forces 0.148 0.215 0.637 74 The expected score for this key concept Forces is 74

Based on the knowledge states, you may want to provide coaching in Direction, Angle, Magnitude, and Resultant Vector

Fig 1.9 Output of a student’s mastery states

The teacher can also examine the tutoring strategy applied to the current

student by clicking the “Tutoring Strategy” button iTutor displays a tutoring policy as

the series of IF-THEN-ELSE statements (Figure 1.10) These rules guide iTutor in

tutoring the student The tutoring policy and the diagnostic profile are useful

information for the teacher and student to review the short term learning outcomes In

some situations, the teacher may provide further coaching to the student

Intelligent Tutoring System

Student ID: 01001 With regard to the key concept Forces, the course of action is : select average item from l2_2 (Force_002)

if response is correct then select difficult item from l2_2 (Force_012)

if response is correct then

select average item from l2_2 (Force_021)

ect easy item from l2_2 (Force_003)

(Force_017)

Tutoring Strategy

select average item from l2_4 (Force_013) e

els else sel

if response is correct then select average item from l2_2 else

select easy item from l2_2 (Force_006)

Student ID: s0001 With regard to the key concept Forces, the course of action is : test L2_5 with item 13: F_Q14.PPT with difficulty level 3

if response is correct then test L2_5 with item 17: F_Q18.PPT with difficulty level 3

if response is correct then test L2_6 with item 8: F_Q9.PPT with difficulty level 4 else

test L2_5 with item 12: F_Q13.PPT with difficulty level 2 else

test L2_5 with item 12: F_Q13.PPT with difficulty level 2

if response is correct then test L2_5 with item 17: F_Q18.PPT with difficulty level 3 else

test L2_5 with item 9: F_Q10.PPT with difficulty level 1

1.5 Organization of this Thesis

This thesis consists of six chapters A literature review on the main issues affecting

the development of ITSs, and the approach adopted by this research are presented in

Chapter Two In Chapter Three, a formal approach for creating probabilistic student

model based on learning objectives is elaborated The formal approach of achieving

optimal tutoring policy in polynomial time is formalised in Chapter Four In Chapter

Five, the development of iTutor and evaluation of its effectiveness and efficiency is

presented Finally, in Chapter Six, the lessons that we can derive from this work and

the further directions in decision-theoretic intelligent tutoring system are provided

tutoring, the literature review in this chapter focuses on student modelling, pedagogy, and assessment Table 2.1 summarises the main points for discussion in the

subsequent sections By considering these issues with reference to representative work helps to put this thesis in perspective

Holt et al (1994) defined a student model as the representation of the computer

system belief about the learner and therefore is an abstract representation of the learner In Section 2.1, we discuss the main approaches in constructing student model using the curriculum learnable units (Wenger, 1987) We focus on works in student modelling where the learner’s answer to prompted item is used to determine his

knowledge mastery As the learner’s answer may contain faulty solution, it is also necessary to consider the representation schemes for student’s buggy knowledge In addition, the student’s intermediate solution is usually not visible to the tutor This

requires non-deterministic modelling techniques such as Dampster-Shaffer certainty

factor (Shortliffe, 1976), fuzzy logic (Matsubara and Nagamachi, 1996; Tang et al.,

2000), and Bayesian network (Reye, 1996; Conati and VenLehn, 1996; VenLehn et al.,

1998; Far and Hashimoto, 2000; Gertner and VanLehn, 2000; Millán et al., 2000; Mayo et al., 2000) Since this thesis is on decision-theoretic approach, only works related to Bayesian network (BN) will be discussed in Section 2.1

Table 2.1 Issues and points of discussion in the literature review of ITS

The level of details used to represent the student solution in ITS is an important

consideration in student modelling Although a fine grain model is able to detect specific student’s knowledge mastery with less uncertainty, the search space of his possible misconceptions grows exponentially in size, resulting in intractable

inferences On the other extreme, a coarse grain model provides quick inferences but

possesses more uncertainties in student knowledge mastery In Section 2.2, we discuss two diagnostic approaches that focus on tracking observable (explicit) student’s

solution, namely model tracing (Gertner and VanLehn, 2000) and constraint-based

modelling (Mitrović, 1998)

Behaviourism, Cognitivism, and Constructivism are the common learning

theories (McCormick and Pressley, 1997) used to guide the tutor behaviour during remediation The three learning theories in the context of ITS are briefly discussed in

Section 2.3 In Section 2.4, we focus on systems that exhibit normative (Howard and

Matheson, 1983) approaches to tutoring action selection The combination of

probabilistic reasoning and decision-theoretic tutoring ensure that actions are optimal

and rational Therefore, if the system appears to be behaving irrationally, then the procedures are not at fault Rather, one should look at the probabilities assessment and

utilities (or values) specified by the model On the other hand, heuristic approaches

are descriptive in nature and can be developed easily, but are not able to isolate the causes of erroneous behaviour in this way

In Section 2.5, we discuss the methods of assessment related to ITS to capture student’s learning The traditional methods include true-false, multiple-choice, matching, fill in the blanks, and written constructed response Alternative assessment tends to emphasise simulation-based discovery learning The choice of assessment is dependent on the domain knowledge and the communication module available In Section 2.6, we further elaborate that World Wide Web provides another venue for student’s learning with no constraints on time and place Finally, the approach adopted

by this research is discussed in Section 2.7

2.1 Construction of Student Model

Along the idea of explicitly representing the knowledge to be conveyed, comes the idea of doing the same with the student, in the form of a student model Ideally, the student model should include all aspects of the student’s behaviour and knowledge that affects his performance and learning When an ITS attempts to correct the errant behaviour of a student, the student model forms representation of the student’s ability based on the observed behaviour of the student, which is usually in the form of student’s solutions to a set of given tasks However, the task of constructing a student model is not a simple one for computer-based systems In fact, it is also a difficult task for human tutor Additional handicap of computers is that their communication interface is very restricted: mainly the keyboard and screen A human tutor has access

to other sources, like voice effects and facial expressions

The student models are built from the primitives provided by the system In some modelling approaches such as stereotypes (refer to Section 2.1.1) and overlay modelling (refer to Section 2.1.2), they are assumed to be sufficient to accurately model a student’s behaviour In other modelling approaches, some basic primitives are present in the system but are insufficient to accurately model a student’s behaviour Consequently, some perturbation methods (refer to Section 2.1.3) are required to generate additional primitives so as to better model student’s behaviour

Fixed stereotyping is the simplest approach to student modelling whereby the student’s

responses to problems categorise the student into a predefined group or level For example, the composite learner model of an adaptive tutor, ATULA (Milne et al., 1996), uses learner’s attributes to determine the most suitable form of the explanatory

material for the individual learner ATULA uses cluster analysis to identify groups of

similar users, or user stereotypes Instead of deciding subjectively on the attributes for each category of users, as in Rich’s GRUNDY (Rich, 1983), cluster analysis detects groups of similar cases such that members of the same cluster are by definition similar

in terms of their psychological and background data The profile values for each cluster are based on the post-test performance of the students who take part in the preliminary trials Multivariate statistical techniques are used to create clusters of users and to use initial questionnaires to assign a new learner to suitable initial cluster However, the student’s stereotype may also change during the session For example, if the stereotype contains an assumption about the student’s ability on a particular task, and this is observed not to be the case, then it may be necessary to switch the student to

a more appropriate stereotype

As discussed by Kay (2000), the latest development using stereotypes is the scrutability of student models which allows students to fine-tune the details This is because stereotypes are constructed in terms of their accuracy and utility for a population of users However, the inferences are only valid in a statistical sense Scrutability of stereotypes allows a student to find answers to questions such as:

• Am I a beginner?

• What are the implications of being a beginner?

• How can I let the system model me as a beginner, but have it recognised some of the more advanced things I know?

These seem to be the potential for considerable benefit if learners can explore such issues Goodman et al (1998) relate this to the possibility of encouraging reflection