Using the delphi method as a form of assessment

Case Study: Construction Management Assessment Study • Its program accreditor, the American Council for Construction Education ACCE, required a submission of a self study; in the self s

Using the Delphi method as a form of assessment

Delphi Procedure

• Complete a survey in which respondents answer

a number of open-ended questions

• From these responses, create a new survey that lists all the open-ended responses, and the

number of respondents that wrote each response

• Respondents rate their agreement with each statement on a five-point likert scale, (1) meaning "strongly disagree" and (5) meaning

"strongly agree."

• Statements with means of 4.0 or higher and standard deviations below 1.0 are designated as areas of agreement

• With this information, one can compare like responses over time (pre and post test), conduct factor analyses (given sufficient sample size), and other statistical tests

Why Delphi method?

• Focus groups can be difficult to manage

– Assumption of anonymity could

be violated – “Reticent” versus “generous”

respondents – Power dynamics

Why Delphi method?

• Rating consensus based on open-ended responses might be helpful/meaningful

• Allows for a natural means of conversation about how results can be used because

participants can also be the decision makers

• When combined with other findings, can add specific information (e.g., adds meaning to state licensure exams, etc.)

• When systematically managed, might include faculty’s review of student work samples, making it more of a “direct” observation of student performance

• Might also be a way of getting employers, intern hosts, alumni, and others to participate

• Can be managed electronically, without requirement that all participants sit in one room

at the same time

Case Study: Construction Management Assessment Study

• Its program accreditor, the American Council for Construction Education (ACCE), required a

submission of a self study; in the self study, faculty had to assess eighteen student learning outcomes

• The faculty had only six months in which to conduct alumni, employer, internship studies, in addition to some direct assessment of student work

• There was no capstone course or other place in the program where students could be assessed

on all accreditation student learning standards

• The ACCE was contacted, and they responded

to proposal to use a Delphi method affirmatively (“it is not our policy to comment on the

assessment method used by programs…”

Task One: Selected representative work samples

• Faculty were asked to identify which samples of student work they would use to evaluate

students on each standard

• Used as a source document

ACCE Standard Sources of Evidence

Program Goal One: The Graduate Will Communicate Effectively

Faculty reported having reviewed the following observations of student work to evaluate student performance on this question: (1) MCT213 reports/exams; (2) MCT351 reports/ MCT 306 assignments/ MCT 320 term/ MCT 450 final projects; (3) MCT 320 term papers, MCT 418 presentations, MCT 450 projects, MCT 310 papers, MCT413 article reviews; (4) MCT 320 term papers,

418 presentations, MCT 450 projects, MCT 310 papers; (5) written reports of co-op experiences

Program Goal Two: The Graduate Will Be Aware of Important Ethical Considerations

in the Construction Industry

Faculty reported having reviewed the following observations of student work to evaluate student performance on this question: (1) 30 reports/30 exams; (2) no report; (3) MCT 450 projects and exams, MCT 413 exams; (4) MCT 450 projects, MCT 413 exams; (5) exam responses to bids

Program Goal Three: The Graduate Will Have Adequate Computer Skills

Faculty reported having reviewed the following observations of student work to evaluate student performance on this question: (1) 19 reports, 4 exams; (2) 106 assignments, MCT 351 reports, MCT

306 assignments; (3) MCT 450 project and exams;

MCT 295 work examples; (4) MCT 450 projects, MCT 295 work examples; (5) MCT450 reports, written reports of co-op experiences

Accreditation

standards listed

Sources of evidence are listed

Task Two: Identify Strengths and Weaknesses in Student Performance

• Faculty able to comment on each standard were then asked

to look at the samples of student work they identified and make a list of strengths and weaknesses

• Lists sent to analyst

• There was fairly good compliance

Task Three: Analyze comments

• Statements of strengths and weaknesses listed according to ACCE standard

• Some statement combined if sufficiently the same

• According to each standard, list is created Those with like statements (coming from separate faculty) were identified as such; for example, after statement, “Students need to understand how to use accounting

to submit bids better” (3 of you made this statement)

Task Four: Create survey

• Each item on the list was assigned a five-point Liker-scale (5=strongly agree, 1=strongly disagree):

Example

Program Goal 1: The graduate will communicate effectively

1 Graduating students need improvement in

grammar (3 of you gave this answer)

2 Graduating students need improvement in spelling

(3 of you gave this answer)

…

Task Five: Collect and Analyze Information

• Information inputted on SPSS and analyzed

• Items with higher than 4.0, and standard deviation of less than 1.0 are declared “items of

consensus”

Task Six: Report So Faculty Can Discuss and Make

Recommendations

• Results reported to faculty in a way that they can quickly use information

• Graphical representations are often the best, where standard

is listed next to assessment finding

ACCE Standard Sources of Evidence Strengths Weaknesses

Program Goal One: The

Graduate Will Communicate

Effectively

Faculty reported having reviewed the following observations of student work to evaluate student performance on this question: (1) MCT213 reports/exams; (2) MCT351 reports/ MCT 306 assignments/ MCT 320 term/ MCT 450 final projects; (3) MCT 320 term papers, MCT 418 presentations, MCT 450 projects, MCT 310 papers, MCT413 article reviews; (4) MCT 320 term papers,

418 presentations, MCT 450 projects, MCT 310 papers; (5) written reports of co-op experiences

Satisfactory performance Construction management graduates need

improvement in grammar; there are additional indications that they also need improvement in personal presentation skills

Program Goal Two: The

Graduate Will Be Aware of

Important Ethical Considerations

in the Construction Industry

Faculty reported having reviewed the following observations of student work to evaluate student performance on this question: (1) 30 reports/30 exams; (2) no report; (3) MCT 450 projects and exams, MCT 413 exams; (4) MCT 450 projects, MCT 413 exams; (5) exam responses to bids

Satisfactory performance;Construction management graduates have an awareness of ethical issues in the construction industry

No consensus weaknesses

Program Goal Three: The

Graduate Will Have Adequate

Computer Skills

Faculty reported having reviewed the following observations of student work to evaluate student performance on this question: (1) 19 reports, 4 exams; (2) 106 assignments, MCT 351 reports, MCT

306 assignments; (3) MCT 450 project and exams;

MCT 295 work examples; (4) MCT 450 projects, MCT 295 work examples; (5) MCT450 reports, written reports of co-op experiences

Satisfactory performance;Construction management graduates are proficient in SoftPlan and in basic computer skills; there are some indications that they are also proficient in SureTrak

No consensus weaknesses

Strengths and weaknesses listed next

to standard

Task Seven: List With Findings of Other Assessments to Discover Themes (Triangulate)

• Perhaps more work than usual, but process can be very

helpful, if not meaningful and potentially impactful

ACCE Standard Senior

Surveys Internship and Co-Op

Evaluation and Student Intern Evaluations

Alumni Surveys Employer (Host)

Internship Evaluations

Student Performance Report of ACCE Standards (Delphi Panel)

1 The graduate will

communicate

effectively

Senior survey response average for question 1,

"communicate effectively" was 3.38 ("agree");

median was also 3.0

A few interns mentioned that they wanted to continue developing oral and written

communication skills

Alumni survey response average for question 1,

"communicate effectively" was 3.07 ("agree"); median was also 3.0

Interns' overall ability was 4.27 Construction management graduates need

improvement in grammar; there are additional indications that they also need improvement in personal presentation skills

2 The graduate will

be aware if

important ethical

considerations in

the construction

industry

Senior survey response average for question 2, "be aware of important ethical

considerations in the construction industry" was 3.38 ("agree"); median was also 3.0

Alumni survey response average for question 2, "be aware

of important ethical considerations in the construction industry"

was 3.29 ("agree");

median was also 3.0

Interns' overall ability was 4.27 Satisfactory performance; Construction management

graduates have an awareness of ethical issues in the construction industry

3 The graduate will

have adequate

computer skills

Senior survey response average for question 3,

"have adequate computer skills"

was 3.38 ("agree");

median was also 3.0

Alumni survey response average for question 3, "have adequate computer skills" was 3.64 ("agree"); median was also 4 ("strongly agree")

Interns' overall ability was 4.27 Satisfactory performance; Construction management

graduates are proficient in SoftPlan and in basic computer skills; there are some indications that they are also proficient in SureTrak

Final Thoughts

• The “jury is still out” in respect to the use of the Delphi method as a direct assessment of student learning

• Use of this method is a good method of adding specificity to portfolio assessments that identify student performance as

“exceeds expectations;” we do not know what that means until we ask faculty evaluators to tell us what that means, or what issues derive from that judgment

• Can be a method of inviting faculty to play

an active part in both (1) assessment itself; and (2) discussing what recommendations should be made to enhance student

performance