Ongoing and systematic academic program review

Chapter 7DOI: 10.4018/978-1-5225-0672-0.ch007 ABSTRACT Systematic program review is proposed as a semi-formal means to proactively involve higher education faculty, staff, students and a

Chapter 7

DOI: 10.4018/978-1-5225-0672-0.ch007

ABSTRACT

Systematic program review is proposed as a semi-formal means to proactively involve higher education faculty, staff, students and administrators in analyzing and making decisions about the future of their programs The chapter first examines issues facing higher education, issues that provide a rationale for annual program reviews The chapter positions program review as a form of participant-oriented program evaluation, and describes features of annual program reviews A case study illustrates how

a program review was conducted Summary benefits and implementation guidelines are provided for administrators and faculty.

INTRODUCTION

Current accountability approaches in higher education focus on the accreditation of the overall institu-tion with specialized accreditainstitu-tions for many curricular programs Academic administrators, particularly department chairs and program coordinators, are tasked out with these reporting requirements, instead

of using data for program improvement and better understanding who their students are and the impact

of academic programs on students

Within the culture of academia, the prevailing stance is the status quo, a stance that resists any at-tempt to add to the current responsibilities of faculty and any change in familiar habits and practices Administrators who have attempted to instill innovative practices, particularly any strategic planning

or program evaluation efforts, have faced resistance While higher education remains fixed in view of curriculum, outcomes, and faculty work, continual “disruptions” are occurring in demography, econom-ics, and culture (McGee, 2015) How can higher education administrators, faculty, and staff jointly face these ever-changing realities and re-think how to serve the needs of a learning population and overall re-frame how they look at their work in academic programs?

Ongoing and Systematic

Academic Program Review

Neal Shambaugh

West Virginia University, USA

Rather than formal program evaluation or strategic planning, this chapter proposes a less formal approach or program review The purpose of program review is to periodically discuss how academic programs have provided good value to students and to answer important questions, such as “Who are our students?” and “What changes need to be made or what gaps exist in our programs, gaps that students need?” The focus on annual program reviews is to better understand the student and the needs of the student, how the program meets those needs, and uses this understanding to direct future work Program review is an annual activity conducted by faculty, staff, and students In this way, program review be-comes a regular fixture and responsibility of faculty members

The chapter first summarizes four issues facing higher education, issues which suggest the need for

a means to understand students, societal changes, enrollment changes, and accountability requirements The second section describes program review as form of participant-oriented program evaluation, while

a third section identifies features of program review A case study of program review is provided so the reader can see how the process might work and what questions guide the annual review A final section summarizes major benefits from annual program reviews, as well as implementation guidelines for faculty members, program coordinators, and dean-level administrators

ISSUES FACING HIGHER INSTITUTIONS

Four specific issues facing higher education provide a rationale for why ongoing and systematic program review may be helpful and necessary

First, the complex physical and psychological nature of the learner is always changing The brain’s neurological system undergoes constant change and whatever we put in front of people impacts their thinking and behavior (Restak, 2003) Ultimately what we are motivated to pursue is based on what we choose to pay attention to (Gallagher, 2009) Schools and institutions are organized around stable fea-tures, such as a well-defined management-worker structure, workplace needs were known and roles were clear, and technology was less an influence than today According to his “then” versus “now” perspective Chester (2005), identified values as one major category of differences in young people These value shifts, according to Chester, involve digital choices, self-expression, immediate involvement, and a free-agent work ethic Digital thinking embraces a re-boot approach to behavior rather than a traditional analog view that behavior has consequences Self-expression differences in “then versus now” can be seen with the notion of personal respect, which was traditionally viewed as earned over time as opposed to now where respect by young people is expected immediately Rather than working “up the ladder” today’s young people want to make a difference Their motivation to stay in the classroom or on the job may be keyed to the opportunities they have at becoming immediately engaged in real world issues Traditional views saw life as an unfolding sequence, but life is viewed by young people as immediate involvement in games of reaction Today’s young workers see themselves in numerous occupations over their lifetime,

as opposed to one career One can see this value shift in higher education in what enrollment managers label as “swirl,” in which students enroll in multiple institutions and attend classes intermittently (Adel-man, 2006) Parents see their identify influenced by their work, while younger people see their identity

as a network of relationships and work as a changing reality over their lifetime

A second issue is that educational structures have not adapted to these individual and social changes Digital media production and learning, for example, occurs by young people in public network “hang outs” outside of adult control The skills and literacies learned are not configured as learning outcomes

in public schools or higher education (Ito, 2010) Educational reform lags behind fast moving techno-logical change, such as social media, so that understanding the individual in school and the workplace may increasingly need collaborative structures for learning (Shirky, 2009, 2010) One feature of pro-gram review advocated in this chapter is to build in ways to continually learn from human feedback in academic and workplace settings

Given the first two issues of evolving personal characteristics and workplace needs, traditional higher education institutions are facing a third issue, that of smaller enrollments Changing demographics, such

as dropping numbers of high school graduates and tighter family budgets, have resulted in higher educa-tion institueduca-tions tapping internaeduca-tional students and out-of-state students (Selingo, 2013) to pay the bills Meanwhile, higher education institutions are also facing lower statewide support and competing service providers who are attempting to reach these audiences (NCES, 2011) Students are partially drawn to for-profit educational services, which can give young people the personalized treatment they expect (i.e., the customer orientation) and the specialized, personalized and flexible educational programs desired

by older students (Selingo, 2013) Ongoing program review, the focus of this chapter, becomes a means

to help answer questions like “Who the student is?” and “What gaps exist in our programs?”

A related pressure and the fourth issue facing higher education is the call for increased accountability

to show evidence that higher education is a good value and worthy of investment Formal accountability, one means to demonstrate this value, is addressed by accreditation of the entire institution, a process that is required for students to qualify for Federal aid, but also specialized accreditation of programs, a process that is not required but can be very expensive for institutions (Selingo, 2013) Despite the cost, administrators use this accreditation to enhance their reputation, while there are efforts to transform the process into a means for program improvement (e.g., Driscoll & Noriega, 2006) One related set of accountability measures for institutions is known as persistence (term-to-term return), retention rates (fall-to-fall student numbers) and graduation or completion (time to graduation) (Hundrieser, 2012) Such data may be traced to societal and curricular issues, which can be openly acknowledged and discussed

in ongoing program reviews by not only administrators but by faculty and staff who remain closest to students In addition to understanding why enrollment in some programs is flat or declining, program administrators, parents and even donors would find of interest any intent to understand why students do not continue to graduation Citizens, too, value increased accountability, owing to the growing expense

of higher education and its value for future employment

In summary, higher education faces changes in the student and worker, changes in the workplace, increased competition for student enrollments, and accountability pressures These four issues provide a rationale for why institutions could begin ongoing program review activities in order to learn more about how academic programs provide value and what changes need to be made Program review provides

a means for everyone involved in the educational process to discuss the changing student, workplace needs, dropping enrollment, and program success data One of the challenges to considering program review is to educate faculty and staff on the purposes for review as opposed to more formal program evaluation approaches

PROGRAM REVIEW AS PARTICIPANT-FOCUSED EVALUATION

Program evaluation consists of a broad set of approaches to make judgments about programs, depending

on their primary focus Some program evaluation approaches focus on the program itself, such as program

objectives, information needs, consumer ratings, and expert judgments, and are based on a rationalistic approach to evaluation (Fitzpatrick, Sanders, & Worthen, 2004) Program evaluation also includes a continuum of informal and informal approaches (Schwandt, 2001) Program review as discussed in this chapter can be considered a type of program evaluation known as participant-focused evaluation, which keeps students and faculty as the purpose for the review process Faculty, staff, and students are considered as first-hand sources of experience and naturalistic inquiry approaches (e.g., case studies) are typically used Historically, Stake’s (1967, 1995) work on responsive evaluation provided early guidance for participant-oriented program evaluation, while and Guba and Lincoln’s (1989) focus on stakeholders served to bring people into the evaluation process

Specific approaches to evaluating academic programs in which all stakeholders participate in apprais-ing student learnapprais-ing plus faculty and staff work (Allen, 2004) share several features First, as with any

evaluation activity, it is necessary to obtain consensus on the purpose for any review or evaluation The

purpose for evaluation in participant-oriented approaches is to understand how students, faculty and staff

contribute to program success A second feature has data linked to institutional goals, as they exist in unit

and institutional mission statements and strategic plans A third feature of participant-oriented program

evaluation in the review of academic programs is determining that ongoing program improvement is a

priority and key to overseeing accurate accreditation and certification requirements (Ramaley, 2006) Such a focus re-frames program review as a part of faculty work and takes a broader view of evaluation

as more than an often heard statement that “this is the year we have to do that.”

A fourth feature of human-centered reviews is that all constituents are involved Middaugh (2009)

advocates that all members of the university community must participate in systematic and sustainable assessment, which is tied to strategic planning Faculty members, in their discussions about curriculum and the work they do, must continually ask and answer the question “How can our academic programs and experiences help to prepare the citizens responsible for a globally-connected world?” and more fundamentally ask this question: “How can faculty work processes evolve for ongoing re-examination?” Underlying these questions is the pragmatic question of “How do we change from “what we are” to

“what we need to be?”

FEATURES OF ONGOING AND SYSTEMATIC PROGRAM REVIEW

Ongoing Review Provides a Strategic Approach

People tend to avoid strategic conversations or activity because the common practice is to develop a strategic plan and then nothing more is done about the plan A major failure is not connecting the plan

to actual work and not having a process to evaluating the plan itself As strategic plans take time for coordination, discussion, and decisions, department chairs and faculty members are reluctant to repeat the process Any strategic decisions must specify how faculty work supports the plan, a problematic connection as the customary practice in academic settings is the primacy of the faculty member’s agenda and assigned workload for teaching, research, and service responsibilities A problem with strategic plans in academia is that faculty evaluation is not connected to the strategic plan A second failure in many strategic plan documents is not specifying how the plan will be monitored on a continual basis and evaluated periodically Thus, strategic plans are cut off from academic work and without an evaluation procedure the plan is guaranteed to “sit on the shelf.”

Program review labels the process as a more accessible activity to faculty and administrators, because the activity addresses the work that faculty do, the teaching, advising, research, and service activities within curricular programs Program review, given the suggested framework and questions suggested in this chapter, examines what could be viewed as strategic priorities, overall program directions as well as making judgments about priorities given available resources Program review questions prompt faculty to comment on how their programs serve constituent needs as well as to what extent these programs serve

a presence in their communities or state, as well as national and international contributions Program review asks the broad questions of “What are we known for?” “What do we want to be known for?” And

“How shall we spend our day?” These are pragmatic questions of interest to faculty members Program review, once the benefits are experienced through actual use, is then understood as an ongoing activity as opposed to a periodic challenge of re-writing a strategic plan when the present plan was never attended to

Program Review Develops a Systematic Process

The case study that follows describes a systematic and doable process to conduct yearly program re-views The program review document summarizes current programs, displays enrollment data, and uses analysis questions to lead to synthesis decisions As a result of this discussion, lasting 60-90 minutes,

a program area group collectively becomes informed as to how programs are meeting student needs, where the gaps are, what changes need to be made, and what resources are needed if programs are to be changed or added to in some way Several issues may be identified that warrant future meetings to ad-dress them Thus, a record of discussion can be archived and retrieved as needed to document progress towards identified priorities A key question asks faculty, students and staff to suggest improvements to the process of annual program reviews

A tool that can be used to map the benchmark progress of projects that have been identified by the group is logic modeling Logic modeling has been used by grant funding agencies and non-profit organizations to map progress toward a goal Logic models prompt users to identify the outcomes and the specific activities to reach those outcomes, and display progress within those activities Thus, logic models provide a communication tool that is not a strategic plan but rather a working map of progress continually updated along benchmark activities The components of a logic model include resources, activities, outputs, outcomes, and goals (W K Kellogg, 2004) An attribute of logic modeling is first determining the goal or long-term result of the project or program (Knowlton & Phillips, 2013) The model specifies resources needed by the program and activities that tap these resources, including fi-nancial and social capital Outputs prompt faculty to identify short-term, intermediate, and long-term results, as well as impacts and benefits from these results (Knowlton & Phillips, 2013) Logic models are sometimes viewed as tools for program evaluation, but they are viewed here as tools for mapping progress on faculty-identified priorities, rather than as an evaluation mechanism

Program Reviews Examine Who the Learners Are

Program reviews prompt faculty members to re-examine assumptions about students One of the reali-ties of today’s world is that the learner and demographic characteristics of the population are always changing Higher education institutions have focused on high school graduates, adolescents who make

up the traditional student population, but with dropping enrollments, administrators are beginning to pay attention to attracting adult learners who have a job and need to complete a program for career

ad-vancement These potential students may have started a program but did not finish because of family or military responsibilities Another group of potential students are those who need ongoing professional development to build knowledge and skills as well as those who are motivated as life-long learners Program reviews question the assumptions that higher education faculty and administrators have of who the students are They help to identify gaps in program offerings or re-examine primary programs

as well as niche program offerings In the program review document introduced in the case study, an initial set of analysis questions ask “Who do we serve?” Asking this question helps faculty members to match existing programs to current students Asking the question “Who are we missing?” attempts to match potential students to existing programs Answers to both questions prompt program area group faculty to identify gaps in program offerings and to make prioritized decisions about addressing those gaps Here discussions of resource needs will arise Program reviews serve to initiate dialogue between faculty members about program changes in light of available resources and what might be needed A strategic benefit occurs out of these discussions as decisions about program changes can result in resource adjustments and project scheduling

PROGRAM REVIEWS DISCOVER THE BASIS FOR PROGRAM PERFORMANCE

Program review helps to establish the criteria for program performance, such as student knowledge and skill competency, as well as enrollment and graduation rates An example worth elaborating on is competency-based education, an approach to curriculum which identifies what we want graduates to know and be able to do upon graduation from a program Rather than seeing education as needing a degree or certificate and organized around the student credit hour completed, a competency model as-sesses student learning as opposed to degrees granted Such an approach will gain significant traction if financial-aid restrictions allow this form of education Features of such competency programs may still tied to the credit system, but typically enable credits earned for life experience

Critical issues in such programs are the evaluation of courses (e.g., adequate time with instructors) and assessment of student competence Competency assessment may involve demonstration of knowledge and skills in an actual workplace setting Lumina Foundation published its report on what college gradu-ates should know from an associate, undergraduate, and master’s degrees (Adelman, Ewell, Gaston, & Schneider, 2014) In the study learning outcomes were organized around five categories of proficiency, including specialized knowledge, broad and integrative knowledge, intellectual skills, applied and col-laborative learning, and civic and global learning In its report, the Lumina Foundation acknowledged the value of affective goals, such as integrity, personal initiative, and professionalism, to be considered

by individual program groups

Program reviews would provide a mechanism for ongoing discussions of program goals, including quality of curricular delivery and determining what quality means The identification of competencies, for example, might already be in place from accrediting bodies for individual programs Student compe-tencies may take the form of competency-based education but could also follow traditional summative portfolio or capstone approaches to documenting what students can do

Another opportunity for the discussion of program goals, in particular those that address student learning, is to involve employer feedback and to consider their viewpoints on what constitutes students’ capabilities upon graduation While there may be resistance from faculty as seeing their programs as workforce development, such discussions on what constitutes expertise open the door for input by the workplace sector in tangible ways

CASE STUDY OF ONGOING PROGRAM REVIEW

An example of systematic program review of undergraduate and graduate programs in Child Develop-ment is used to illustrate one way to impleDevelop-ment this participant-oriented approach Child DevelopDevelop-ment curricular programs included one undergraduate program with four areas of emphasis attached to the program, three undergraduate minors available across the university, two undergraduate certificates, one master’s program, and a child development specialization for an across the college PhD program

Front-End Data and Program Review Document

An associate dean for Academic Affairs, a former member of the department, briefed Child Development faculty on the merits of the review and the process, which would include a 90-minute meeting where faculty members responded to questions The department chair and program area faculty approved the idea for the review and scheduled a meeting

Prior to the meeting the associate dean provided the group with summary information on program offerings and historical enrollment numbers Five-year enrollment numbers for both undergraduate and graduate programs were summarized in tables and visualized in graphs to depict five-year trends Sum-mer enrollments for five years were also supplied Also distributed in advance was a program review document template, which was used to record the responses to the questions and ensuing discussion The review document included the following elements and discussion topics:

UNDERGRADUATE PROGRAMS (REPEAT FOR GRADUATE PROGRAMS)

Enrollment and Explanations

• Program name, Classification of Instructional Programs (CIP) number, and the institution’s cur-riculum code

• Five-year enrollment tables and graphs

• Discuss reasons for program enrollment trends/numbers (up/down/flat)

• Discuss societal/cultural, legislation, and state/local context as well as university and program is-sues underlying the enrollment numbers

Analysis: Program Students – Who Do We Serve?

1 Who are your students in these programs?

2 Who are we missing for existing programs?

3 Who are we not serving?

4 What is needed to start any new programs?

5 What is our statewide and national presence? How might we serve our constituents?

Synthesis – Decisions: Program Goals with Resources

1 What summer courses are needed?

2 What improvements need to be made to existing programs?

3 For any new program or initiative provide the following information: Identify rationale/students, resources, time frame, and enrollment targets

4 What program features might attract donor support?

The above questions were repeated for the graduate program offerings A final question was asked on how the program review process could be improved

Results of Program Review Meeting

Reasons for Increased Program Enrollment

• The undergraduate program in Child Development was organized by four Areas of Emphasis (AOE) Increased enrollment numbers for each AOE were based on changes in state policy and specific job placement in the workforce The Pre-school AOE provided students with Preschool-Kindergarten certification The Child Development AOE was a Plan B certificate program for students who opted out of teacher education testing A third AOE, Family and Youth Studies, was not a certification track, but a program for older students with families Finally, a fourth AOE, Special Needs for Preschool-Kindergarten, was developed for in conjunction with the undergradu-ate Child Development degree

• A staff person hired to coordinate programs was cited as key to addressing student issues

• Admitting students with a GPA of 2.5 attracted students who had dropped below higher GPA requirements in other programs

• A general education course for students across the university was cited as an effective introduction

to the Child Development program For example, students were required to conduct an observa-tion in the programs nursery school

• Certificate enrollment was now being tracked by the university registrar instead of the college advising office

Context for Enrollment

• Program coordinators in Child Development are responsible for staying current with changes in state policy requirements for graduates to teach in the state’s public schools This policy awareness prompted program design changes to enable graduates to obtain higher-paying jobs

• Two certificate programs were added because of the U.S Head Start Reauthorization Act, which required more focus on infant/toddler development

Analysis

• Who do we serve: Four Areas of Emphasis (AOE) matched four job placement categories in state

public schools

• Who are we missing in existing programs: Three minors, which were added to support student

interests, increased enrollment Courses for an Early Childhood Education Director’s credential

are fully enrolled each summer Program certificates, certifications, and minors improved stu-dents’ employability

• Who are we not serving: As certified teachers were needed for pre-school teaching and Head Start

programs, the program review reported on progress to offer an online undergraduate Bachelor of Science program in Child Development It was reported in the meeting that all courses except one were already being taught online

• What is needed to start the online program: Faculty reported that these online courses were

be-ing readied for in-house review as required by state policy for any online course Faculty requested support to assist on this detailed requirement A suggestion was made to initially cap enrollment Faculty suggested that another staff person may be needed to coordinate the program in the long-term if higher enrollments are achieved

• Statewide presence: The group cited their accomplishments of being responsive to state students

by providing programs aligned with existing job needs

Synthesis – Decisions

• During the meeting it was decided to begin the online Bachelor of Science program for Pre-School Certification, as preschool teachers and Head Start staff would be needing this degree Faculty set

up a future meeting to agree on a program development timeline and discuss marketing support from the Dean’s office

• A specific 200-level course was reported to be required in the new online program The group agreed that this course needed to be taught in the summer to enable students who may need this course to graduate

• A future meeting was scheduled to identify other needed courses for summer

Program Features for Donor Support

• The Associate Dean who was facilitating the meeting recommended that the college’s Director for Development and Director of Recruitment be invited to a program meeting to discuss program offerings that might support statewide employment

Summary of Program Review Meeting

After a 90-minue meeting the program review accomplished the following:

• Understanding enrollment increases is just as important as explaining flat or declining enrollments

• Connecting program decisions to potential student needs (i.e., employment) means students have

a purpose for enrolling in a program as opposed to offering programs and expecting students to apply

• Discussing additional program offerings is conducted in the context of available and needed re-sources Faculty members have a say in what programs can be offered given existing rere-sources

• Scanning the horizon for changes in state policy, career interests, workplace priorities give faculty information early enough to schedule program development and put resources in place

• Producing information for department and college recruiting as well as the college advisory group made up of individuals from the private sector

• Involving staff responsible for development and recruiting serves to increase both donor support and student numbers

• Addressing program “presence” across the state and how faculty, students, and staff contribute to the wellbeing of local, statewide, and national and international audiences

BENEFITS OF ONGOING PROGRAM REVIEW

• Dedicating time for faculty, staff and students to discuss how one’s academic program helps to prepare students to enter the workforce and develop habits of ongoing learning and improvement

• Establishing a long-term objective that may be broad in scope (e.g., establish a Learning Sciences research and teaching program) but captures the direction a program wants to follow in reaching that long-term objective Faculty-determined decisions provide direction and a timetable for fu-ture work in light of resources

• Challenging assumptions held about a program’s current learner profile

• Documenting the performance of students, faculty and staff within the program

• Providing enrollment and matriculation data for short-term decisions, such as curriculum changes (courses, practicums, labs, studios, etc.) and instructor assignments

• Ensuring that data is used for program improvement as well as program accountability for certifi-cation and accreditation bodies (Driscoll & Noriega, 2006)

• Producing timely information to inform various constituents, including advisory groups, donors, students, and community leaders

• Developing a data collection system, which should be ongoing, routine, and invisible if possible Data needs to be transformed for understanding (e.g., visuals, graphs) by users and made available (e.g., dashboards)

GUIDELINES FOR DEANS AND DEPARTMENT CHAIRS

An ongoing and systematic program review will not work unless the activity is supported by the academic unit’s dean or director along with the department chairs, who may resist and view this activity as “one more thing for faculty to do.” Consequently, what are needed is discussion of the values and purposes for program review, trials by volunteer program areas, and input from faculty members during and after the reviews Department chairs and faculty may initially view program review as an artifact of strategic planning that never goes anywhere for “hunting for programs to cut.” Department chairs, in particular, will need answers to “Why do this?” Major reasons to conduct program reviews can be pragmatically characterized do deans and department chairs in the following ways:

• Enrollment in the unit continues to drop over an x-year period This trend will continue without some form of intervention

• Improving enrollment is not just about recruitment, but also program improvement, keeping the students we have, giving them a quality experience, and attracting new students