Terry-Rhodes-Using-Value-Rubric-Results-for-Learning-Improvement

Course Level Teaching, Learning & The Traditional Assessment & Accountability Landscape Program Assessment Disciplinary and/or Professional Accreditation General Education/ Core Curricu

Assessment that Empowers Faculty to Take Risks with Pedagogical Innovation

LEAP Texas Terrel L Rhodes Association of American Colleges and Universities

March 25, 2018

Using VALUE Rubric Results for Learning Improvement, Professional Development and Equity:

Course Level Teaching, Learning &

The Traditional Assessment & Accountability Landscape

Program Assessment

Disciplinary and/or Professional Accreditation

General Education/

Core Curriculum Assessment

Institutional Assessment

Regional Accreditation

System and/or State Level Accountability

Federal Level Accountability

Other Accountability Mechanisms (e.g., VSA)

Movement from course-embedded and program-level assessment to more global, institutional assessment = increase cost, harder to

tie results directly to improving teaching & learning at the local level; quality assurance mechanisms evidence generated requires valid

and reliable measures that transcend local conditions in order to set effective policy The traditional measure preferred at the policy

level – commercially available standardized tests – lack transparency in design and the ability to disaggregate data below the

institutional level to make changes to improve teaching and learning at the course and program level.

Course-LevelRecognize and promote student agency and

faculty development and expertise in order

to improve teaching and through the

adoption of active learning pedagogies and

enhanced assignment design

Institutional LevelCreate guided learning pathways – including successful 2- to 4-year transfer - to promote retention and completion for all students, while addressing quality assurance and accountability requirements through general education and beyond

Program Level Design curricula that leverage high-Impact

practices within and across degree areas

that respect disciplinary paradigms and

professional standards while promoting the

attainment of higher order necessary

abilities to thrive in work, citizenship, and

life for all students

educational attainment for the

common good

The VALUE Model Evidence of quality student learning to:

-The Key Elements for a Compelling Quality Framework Already Are in

Hand

Learning

The VALUE Rubric Approach to Assessing Student Learning

Valid Assessment of Learning in Undergraduate Education

www.aacu.org/value

VALUE Rubric Approach - Assumptions

 Learning is a process that occurs over time

motivated learning

dimensions of learning outcomes

learning

VALUE Embraces Imperfection as Part of

the Learning Process

“Never Let the Perfect Get in the Way of the

Good”

VALUE embraces the variables that other assessment approaches control or eliminate in their consideration of

student learning, including :

the syllabus and out of the classroom There are no required common prompts.

wash out, the inherent diversity—from race, ethnicity, and socioeconomic status to the diversity of courses, credit-

levels, and disciplinary backgrounds—found on campuses.

reach a consensus score and a rich faculty development opportunity, and that are open to all faculty whether they are contingent or tenure-track, two-year or four-year,

curricular or co-curricular

www.aacu.org/OnSolidGroundVALUE

VALUE Project map: The Multi-State, Minnesota , and

Great Lakes Colleges Association Collaboratives

Multi-state Collaborative

Multi-state and Minnesota Collaboratives

VALUE Initiative to Date:

Faculty & staff saw the VALUE rubrics as valid.

Percent of scorers who reported Strongly Agree or Agree with each aspect of rubric use

Encompassed meaning of outcome

Descriptors were relevant Descriptors were understandable

Scoring levels provided sufficient range

Useful for evaluating student work

These results are not generalizable across participating states or the nation in any way

Please use appropriately

The Anatomy of a VALUE Rubric

Levels

Performance Descriptors

Profile of Scorers by Discipline and/or Institutional Role

Arts and Humanities Natural and Applied/Formal Sciences

Professions Social Sciences Administrative

MSC Profile of VALUE Scorers

These results are not generalizable across participating states or the nation in any way Please use appropriately

Potential to disaggregate by demographic characteristics

1.5

2.0

2.5

Asian Black Hispanic White

Critical Thinking scores by race

These results are not generalizable across participating states or the nation in any way Please use appropriately.

Asian Black Hispanic White

Critical Thinking scores by Pell eligibility

1.0 1.5 2.0 2.5

These results are not generalizable across participating states or the nation in any way Please use appropriately.

Not Eligible Pell Eligible Not Eligible Pell Eligible

Faculty & staff saw the VALUE rubrics as valid

Percent of scorers who reported Strongly Agree or Agree with each aspect of rubric use

Encompassed meaning of outcome

Descriptors were relevant Descriptors were understandable Scoring levels provided sufficient range

Useful for evaluating student work

These results are not generalizable across participating states or the nation in any way Please use appropriately

Interrater reliability was moderate to strong.

ACCOUNTABILITY AND ACCREDITATION

Campus Benefits and Uses

The reflective essay prepared by Southern Connecticut State University outlined the institution’s “significant” national and international assessment initiatives as well as the “major internal assessment initiatives” undertaken to assess general education

and provide support for internal program reviews and specialized accreditation

reviews We are pleased to learn that results gleaned from analyses of student work

conducted as part of SCSU’s participation in the Multi-State Collaborative have been used to inform the restructuring of the University’s access programs , developmental math curriculum, liberal education program, and writing across the curriculum

program We are also gratified to learn of SCSU’s use of the results of a longitudinal cohort study of retention rates to determine the “most important predictors of

academic success and student retention” and to develop programming to foster the

“habits of mind” that are predictive of success The essay provided evidence that SCSU graduates are successful in their chosen fields, as measured by success in clinical

placements licensure passage rates, and employer evaluations and satisfaction rates

CAPACITY BUILDING

Campus Benefits and Uses

“It’s a professional development opportunity for all of us,”

CT L

IR

Faculty

Student Affairs

Committee Chairs

“ to see how the decisions made by

an assessment committee affect how institutional research is able to collect

or analyze the data…”

“…How data are likely to be

used in conversations [with

faculty] about curriculum and

development…”

“…gather and talk about something as important as learning outcomes.”

“…a rare opportunity for all

these different players…”

The Hamline Plan is not a set number of courses Instead,

it focuses on certain skills that you can learn in a variety of

subjects.

• “I’ll also say that we are greener than [St Olaf College] and when we began,

their faculty director of assessment was very useful for us to think about how

they had built this robust assessment structure that moved from pretty effective program-level assessment to a cross-program general education system…”

one-hour assessment charrette and offered it as a companion to an

“assessment salon” – feedback on assignments and discussion of assessment

and working alongside career services, counseling, and advising to figure out if students are on the right pathway.”

Math faculty

French faculty

the faculty really enjoyed sharing what they were

PROGRAM AND COURSE IMPROVEMENT

Campus Benefits and Uses

Applying lessons learned to our local initiatives on campus

ed courses.

artifacts

uploading and scoring of artifacts

Learning Outcome & Faculty Faculty (N)

Number of Artifacts First

Year Soph Junior Senior CCSU Total Total MSC

“…data…collected over the last eight or nine years

to look at programs “more holistically” and

evaluate staffing, course sequencing, or

program-wide curricula…”

• Math, Statistics, and Computer Science

department…started using a statistical software manual “because they realized that students

didn’t have quite the competence level … as they wanted [them] to have when they graduated…”

Institutional

Data

Criterion by CriterionFocus

Sources and Evidence

De-Identified

Project and Institution

Took Apart the Rubrics

Action

Project • Project as a Whole

Institutional • Local Scoring

Course • Individual Faculty

Research highlights importance of faculty and

student success and

equity

33

REINFORCE

Assignment designed

to reinforce previously practiced

outcome

MASTERY

Assignment designed for students to demonstrate level of mastery of the outcome

Critically Important to Collect

Analysis of student work assessed using the Critical

Thinking and Written Communication VALUE Rubrics,

seniors’ work was significantly more likely to be

scored at the “Capstone” level—the highest level of

performance—when the assignment was designed to

produce work at the Capstone level

When the assignment was “easier”

seniors’ performance, on average, went down to meet the lower expectations of the assignment

When asked for less, they produced less.

non-white and lower income students

gain access to high quality and demanding assignments less often

Inclusive excellence is an affordable, feasible

goal if the highest impact high-impact practice is

high quality and appropriately demanding

assignments.

Achieving it would transform the

learning outcomes of American higher

education.

Our notion of high-impact

where increasing the

supply is costly in higher

education

But insisting that all faculty give assignments that are both demanding and intentional about higher-order learning goals not just content learning goals is not

financially costly, only

politically and managerially difficult

We need to begin thinking of high quality and demanding assignments as perhaps the highest

Lessons Learned from VALUE/MSC

• Context or landscape is important

• Local data are critical

• Data need deconstruction/disaggregation at local

level

• Interdisciplinary/integrative experience is required to

attain high quality levels associated with graduation

• What faculty/educators do is foundational to achieve

quality student learning

or Comments?

rhodes@aacu.org