use-of-rubrics-for-assessment-of-a-senior-project-design-course

This paper describes the application of rubrics to gauge the performance, skills, and competencies of students as they complete their senior projects in the EET and CET programs at DeVry

2006-2369: USE OF RUBRICS FOR ASSESSMENT OF A SENIOR PROJECT

DESIGN COURSE

Ahmed Khan, DeVry University-Addison

Ahmed S Khan, Ph.D is a senior Professor in the EET dept at DeVry University, Addison,

Illinois He received his M.Sc (applied physics) from University of Karachi, an MSEE from

Michigan Technological University, and an MBA from Keller Graduate School of Management

He received his Ph.D from Colorado State University His research interests are in the areas of

Fiber Optics Communications, faculty development, and outcomes assessment, and, Internet and

distance education He is author of “The Telecommunications Fact Book” and co-author of

“Technology and Society: Crossroads to the 21st Century” and “Technology and Society: A

Bridge to the 21st Century.” He is a member of IEEE, ASEE, ASQ, and LIA

Robert Lawrence, DeVry University

Professor Robert Lawrence has been teaching writing and speaking at DeVry for over 20 years

He also serves as a General Education advisor for the EET/CET senior project class He received

a B.A in English from the State University of New York at Stony Brook and an M.A in English

from the University of Iowa Frag: Fragments in Context, educational software he wrote and

programmed, was published by the Learning Seed He was recently selected by the Chicago

Poetry Society to present some of his poetry at Chicago’s annual Around the Coyote festival

Use of Rubrics for Assessment of a Senior Project Design Course

Abstract

Rubrics are becoming an essential link between instruction and assessment This paper

describes the application of rubrics to gauge the performance, skills, and competencies of

students as they complete their senior projects in the EET and CET programs at DeVry

University, Addison, IL

ABET’s requirement for accredited programs to implement outcomes-based models has

stimulated the growth of formalized assessment programs within the engineering and

engineering technology communities

The use of rubrics as an assessment tool allows faculty to: (a) Improve student

performance by collecting data on student skills and competences, and (b) validate that

students are achieving course and program objectives

The senior project is a two-semester course sequence in which the students synthesize

their previous coursework Students are required to plan, design, implement, document,

and present the solution to a software/hardware engineering problem

Faculty use rubrics for the assessment of project proposal development in the eight

semester and for project implementation in form of prototype development and

demonstration in the ninth semester Feedback from the rubrics is used to take corrective

action to improve the course sequences, program objectives, and instructional delivery

I Introduction

A rubric is an assessment tool that allows instructors to enhance the quality of direct

instruction by providing focus, emphasis and attention to details as a model for a

completed product, project or behavior

There are two types of rubrics that are used for assessment: Analytic and Holistic An

analytic rubric identities and assesses the components of a completed project, and a

Holistic rubric assess student work as a whole (see Table 11)

Table 1 Analytic vs Holistic

Inter-rater reliability High/more difficult Moderate/less difficult

The use of Rubrics as an assessment tool offers a number of advantages:

• Student learning can be gauged effectively

• Student’s areas of strengths and concern can be detected

• Accomplishments of the various tasks of a project can be evaluated effectively

II CQI/Assessment Processes at DeVry University

Student success is measured by student performance, satisfaction, retention, and

placement

EET/CET students learn the specialty technical knowledge necessary to enhance or

launch their careers, as well as acquiring the general education competencies, skills, and

values that help sustain their learning throughout their careers and add meaning to their

lives

Like other baccalaureate programs at DeVry, the EET and CET programs include an

integrative senior project, where students work in teams to solve a real world problem

related to their major Students demonstrate a wide range of competencies during the

course of the project, making the direct measurement of student academic achievement

via the senior project (EET-410L) and a companion general education capstone course,

(HUMN-432) a major part of assessment effort, and overall continuous quality

improvement (CQI) process at DeVry Student outcomes assessment at DeVry serves as

the "check" function in the "Plan-Do-Check-Act" model for CQI (see Figure 1), which

emphasizes the iterative and ongoing nature of the process A number of direct and

indirect indicators are also established against which the student learning/performance

outcomes are assessed

Direct indicators of learning

• Pre- and post testing

• Oral examination/Research presentations

• Electronic portfolios

• Evaluation of capstone projects (EET-410L & HUMN-432)

Indirect indicators of learning

• Alumni survey

• Employers survey

• Graduation rates

• Retention rates

• Job placement data

Data collection methods:

• Assessment tools for EET-410L

• Essays and writing samples

• Portfolio collection of student work

• Surveys

• Focused group interviews

• Industrial advisory committee (IAC) input

• Use of external evaluators (TAC/ABET and NCA)

DeVry’s assessment activities have evolved naturally, in parallel with the changes in

DeVry's mission and resulting changes in programs The primary components of DeVry’s

assessment processes are listed in Table 2

Table 2 Assessment Processes at DeVry University

Performance gaps revealed

Institute Review Team/deans reviews data and recommend corrective actions

Deans review data and recommend corrective actions

Industrial Advisory

Board

problems and proposes solution

Ongoing process for program/course revisions and improvements

Figure 1: Deming's Plan-Do-Check-Act Cycle

CHECK ACT

Figure 2: Assessment Processes for EET/CET program

Figure 3 Assessment Process for EET Senior Project (EET-410L)

EET senior projects for EET-410L are assessed using national assessment tool

Assessment data is entered into the national assessment database (DeVry-OBT)

DeVry-OBT generates reports based on EET-410L assessment data and sends them to EET Dean for review / corrective actions

EET Dean / Course sequence committees analyze assessment data, and recommend and take corrective actions to close the loop for the CQI process

III Use of Rubrics for the Senior Project Course Sequence

The senior project is a two-semester course sequence in which the students synthesize

knowledge and skills learnt in the previous courses In the first course (EET-400, Project

management) students research, plan and develop a project proposal And in the second

course (EET-410L, Senior Project Laboratory) students implement the project plan by

building and testing a prototype The project involves a solution to a

software/hardware-based engineering problem

In both course rubrics are used to evaluate students’ accomplishments of various tasks of

project design, planning and implementation phases

The senior project course sequence also presents an excellent opportunity

to directly measure the competencies (program objectives) of EET/CET graduating

students Two Rubrics, a national and a local, are used to evaluate each student on

achieving program objectives (competencies) based on direct observation The national

assessment rubric is designed to gauge the student performance in achieving the program

objectives, and the assessment data is used to take corrective action in terms of

curriculum design and implementation The local assessment tool is designed to identify

student strengths and weaknesses at course sequence level; the assessment data obtained

is used to take corrective action at local level (campus) by revising the course contents

and teaching methodologies at the lecture and laboratory levels.(see Rubric E and Rubric

F) Table 3 presents a summary of the use of Rubrics for the assessment of EET/CET

senior project and program objectives

Table 3 Use of Rubrics for Assessment of EET/CET Senior Project and Program

Objectives

Assessment for Project

Phase

Semester/Time for completion

of Project Phases

Assessment Tool

Project Faculty approve/disapprove the project

idea/concept

Project Phase-II

Project Proposal/Plan

Development and Defense

Eight /5-14 Rubric B Data reveals the

viability of project concept

Project Faculty approve/disapprove the project

Faculty advises students to take corrective action in terms of

accomplishment of project with respect

Faculty advises students to take corrective action in terms of

accomplishment of project with respect

of project objective

Faculty determines

if student sussessfully demonstrated the accomplishment of project

implementation

Rubric E

Rubric F

Strengths and weaknesses of students’

competencies revealed

Performance gaps revealed

at course sequence level

Program director and curriculum Manager

Deans / sequence committees review data and take corrective action

Conclusion

The paper presented an overview of the use of Rubrics in the assessment of course and programs

objectives Rubrics are easy to use and allow faculty to directly assess student knowledge-base,

skill levels and competencies in senior project course sequence The faculty and deans find the

feedback useful for improving the CET/EET curriculum and student performance

References

1 Rogers, Gloria (2005) ABET TEI Assessment Workshop 2.0, Baltimore, September 24, 2005

2 Khan, Ahmed (2002) Culture of Assessment at DeVry University 2002 Connecting Classrooms,

Communities & Careers "Reform with Results" 10th International Conference (June 29 - July 2,

2002),Beaver Creek, Colorado

Rubric A EET-400 Week 4 Project Idea/Concept Generation Presentation

Proposed Project Title / Topic:

Presented by:

Name of Evaluator: _

Please circle appropriate score

Introduction/ Clarity of project

CET/EET-400 Week 14 Project Proposal Presentation

Proposed Project Title / Topic:

Presented by:

Name of Evaluator: _

Please circle appropriate score

Introduction/ Description of the

aspects(description of all reporting

requirements, legal aspects,

permissions etc)

Description of Project Schedules

Personnel (project team members’

assigned tasks) and Resources

Rubric C CER/EET-410L Week 4/11 Project Implementation Status Presentation

Proposed Project Title / Topic:

Presented by:

Name of Evaluator: _

Please circle appropriate score

Project on-schedule Yes/No _

Recommendations for corrective action/Suggestions for improvement:

Rubric D CET/EET-410L Week 14 Senior Project Final Presentation & Demonstration

Proposed Project Title / Topic:

Presented by:

Name of Evaluator: _

Please circle appropriate score

aspects(description of all reporting

requirements, legal aspects,

permissions etc)

Personnel (project team members’

assigned tasks) and Resources

Written Project Report (Compliance

with the required criteria)

Comments: _

Rubric E EET/CET COMPETENCY EVALUATION FORM / LONG

Faculty teaching this course

USE THIS FORM

Objective #1: Conduct experiments involving electronic systems using modern test

equipment, interpret test results and use them to improve products or methodologies

(0 - 4)

Improvement

Performs Needs Analysis -

define the problem

Identifies the problem to be solved Identifies tests needed to isolate the cause of the problem or to provide additional information toward solving the problem

States goals and objectives of

the experiment

States objectives of the investigation States how the investigation will help toward problem solution

Identifies resources to conduct

experiment (parts, equipment,

data sheets, etc.)

Identifies instruments, parts, software, etc needed

to set up the test Draws any schematic diagrams, flow charts, etc of the system to be constructed

Collects any technical data sheets, equipment manuals, etc as needed

Develops a procedure and

collects data using modern test

equipment

Plans stages of operation Plans the sequence of tests to be performed Conducts tests and gathers data

Outcomes Ratings

(0 - 4)

Improvement Analyzes test results and

Objective #2: Create, implement high-level, and Assembly-language programs in support of technical

activities

(0 - 4)

Analyzes the problem

logically

Performs feasibility studies Determines the output desired Determines the input needed Identifies the processing required Draws a flow diagram

Defines the objective of each module

Selects appropriate data structures Develops the logic for each module

in an algorithm

the problem Codes the algorithm into an extendable reusable software Creates good documentation

Tests and debugs each module Tests and debugs the program as a whole

Refines the program

Objective #3: For EET: Uses the principles of science, mathematics, and engineering

technology to design, implement, and evaluate hardware and software solutions to complex

Selects and defines a

meaningful problem taking

safety, ethical, social,

economic, and technical

constraints into consideration

Proposes a problem for investigation Identifies criteria for the proposal (taking safety, ethical, social, economic, and technical constraints into considerations)

Applies criteria in defining and specifying the problem

Identifies scope of the problem including a problem statement and solution criteria

Devises process to solve

Organizes tasks around available technology, personnel and financial resources effectively and efficiently

Applies appropriate

knowledge of scientific,

mathematical, and engineering

design tools toward the design

and analysis of problem

solutions

Displays a working knowledge of modern engineering design tools (e.g., HDL, Pspice and Matlab) and applies it to design and analyze problem solutions

Integrates knowledge of fundamentals in selecting system components using appropriate technology (including manufacturers' catalogs and network databases)

Displays the ability to acquire a working knowledge of new design tools

Identifies key issues in

designing and building a

prototype

Identifies materials necessary to build the prototype including power source Prioritizes procurement of design components Define steps to building a prototype

Builds, tests and troubleshoots