Full+article-+Do+Simulations+Enhance+Student+Learning-+An+Empirical+Evaluation+of+an+IR+Simulation

Volume 2, 2006 - Issue 1 Submit an article Journal homepage Journal of Political Science Education " 1,099 Views 153 CrossRef citations to date 2 Altmetric Thematic Issue: Simulations in

Trang 1

! Cart

Enter keywords, authors, DOI, ORCID etc

Advanced search

This Journal !

Volume 2, 2006 - Issue 1

Submit an article Journal homepage

Journal of Political Science Education "

1,099

Views

153

CrossRef citations

to date

2

Altmetric

Thematic Issue: Simulations in Political Science

Do Simulations Enhance Student Learning? An Empirical Evaluation of an IR Simulation

&

Stephen M Shellman # Kürşad Turan

Pages 19-32 | Published online: 24 Feb 2007

$ Download citation

%

In this article

ABSTRACT

Introduction

Teaching &

Learning Goals

Overview of the

Simulation

Research Design

The Results

Conclusion

Footnotes

References

ABSTRACT

There is a nascent literature on the question of whether active learning methods, and in particular simulation methods, enhance student learning In this article, the authors

evaluate the utility of an international relations simulation in enhancing learning objectives Student evaluations provide evidence that the simulation process enhances their knowledge of such abstract theories and concepts Moreover, the results suggest that the simulation increases students' interest in political science/international

relations and provides a memorable experience that they will not forget

! Keywords: active learning simulations international relations terrorism negotiation technology

Introduction

Recently, in the ALIAS Notes, Krain (2005, 1) charged that “Too few of us take the time or make the e!ort to carefully situate our work in the rich literatures to which our

respective pedagogical projects potentially speak Even fewer of us make even rudimentary attempts (qualitative or quantitative) at assessing the e!ectiveness of our endeavors.” We take this charge seriously There is a growing "eld of interest in active techniques and their e!ectiveness to enhance and/or reinforce learning, yet no

universal view exists that active learning techniques are e!ective In fact, few studies attempt to assess the impact that particular techniques have on learning Our study contributes to this literature primarily by assessing an international relations

simulation's impact on students' learning A secondary contribution consists of highlighting the use of technology-enhanced techniques to improve and enrich the learning environment and assessing the utility that technology brings to the classroom

or simulation exercise in this case

The use of classroom simulations and other active learning techniques are on the rise

in political science curricula Such strategies can immerse students into environments and involve them with course materials that traditional techniques like reading and lecturing cannot (Caruson 2005, 305) As many point out, there are multiple ways to engage students with active learning strategies: e.g., study abroad trips (Thies 2005; Bowman and Jennings 2005); Kolb's (1984) experimental learning model; web-based exercises (Cogburn and Levinson 2003); classroom simulations (Smith and Boyer 1996);

and even "lm (Kuzma and Haney 2001) Our focus is on the use of simulations in the classroom and their e!ectiveness for encouraging and facilitating a student's ability to learn Herein, we concentrate on simulations, their design, the use of technology within the simulation itself, and the simulation's ability to improve the quality of learning

Skeptics of such active learning approaches argue that “student-centered” learning is a waste of class time and creates “village idiots” in the process (Rochester 2003, inside back cover; Raines 2003, 432) They argue that traditional lectures and case approaches are more e!ective than active learning approaches In contrast, Brock and Cameron (1999, 251) counter that individuals learn and process information in di!erent ways, and that while lectures may be necessary techniques to convey important information, they are insu#cient alone to teach a group of students with varied learning preferences (Shellman and Turan 2003) To address varied learning preferences, many di!erent

simulation and gaming professionals have developed active learning simulations to teach particular concepts and processes like election campaigns (Caruson 2005);

proportional representation electoral systems and coalition formation processes (Shellman 2001); international negotiations (Shellman and Turan 2003 2005; Newmann and Twigg 2000); and many others

However, little evidence suggests that active learning exercises facilitate learning and in many cases the designers of such exercises simply assert that their exercises enhance learning Gosen and Washbush (2004) along with Krain (2005) argue that more

emphasis should be placed on testing and evaluating the fruits of experiential learning techniques It only makes sense that we would want to know whether the techniques

we use are e!ective or not (Gosen and Washbush 2004) We can develop exams and assign papers, but it is hard to break down what combinations of teaching methods, study habits, intelligence, and ambition generate the grade With this in mind, Wolfe reviewed studies on the validity of games utilized in strategic management courses and found that simulations facilitated learning Washbush and Gosen (2001) and Feinstein (2001) found that managerial economic simulations and food service operation

simulations, respectively, were valid learning techniques as well However, Gosen and Washbush (2004) contend that the evidence is provisional, especially given some of the research design decisions They caution that we can only draw tentative conclusions that simulations are valid learning techniques More research must be done to assess the validity of our exercises to evaluate their utility in the classroom

This study seeks to add to the body of evidence for and/or against the validity of simulations as teaching and learning techniques by testing whether or not simulations enhance learning We argue that simulations open alternative learning paths to

participants who do not respond well to conventional lecturing approaches (Newmann and Twigg 2000, 835) We contend that our international relations simulation enhances students' knowledge of international relations theories, concepts, organizations, and processes It does so by creating an environment where students apply and make use

of theories and concepts to achieve their goals Below we describe the goals of the simulation in more detail, assert how our simulation tackles these goals, explicate the research design for testing our hypotheses and discuss the results

Teaching & Learning Goals

Raines (2003) contends that teachers should “create a lesson plan that maximizes student learning, encourages critical thinking, aids information retention, and allows students to apply key concepts and knowledge gained through readings and lecture to real (or realistic) problems.” We agree and so we designed an active learning simulation

to create such a lesson plan Simulations allow students to relate their book knowledge

to the real world (Brock and Cameron 1999, 252); they essentially provide laboratories, which soft sciences often lack, to recreate complex processes (Woodworth and Gump

1994) Simulating these processes allows students to experience and apply the concepts and theories from the course to actual scenarios and events and ultimately to understand the concepts and theories more deeply (Asal 2005) In short, it facilitates the development of critical and analytical thinking and problem-solving skills Our design also fosters understanding of negotiation processes and dynamics evident in international relations and various international organizations Moreover, it facilitates understanding and analysis of countries' and organizations' societies, economies, and foreign policies We elaborate on these points below and explain how our simulation aids in accomplishing the goals we set

To begin, the majority of students can memorize de"nitions and regurgitate them back

on a test Yet, many struggle with problem solving and application skills The 9/11 commission report argues that the number one intelligence failure was the “failure of imagination.” We wish to develop students to invent new solutions to novel problems

Our goal as educators is to develop techniques to teach content in ways that also develops critical and analytical thinking, problem solving, and lifelong learning skills In the real world, analysis is often more important than the ability to give “programmed”

responses All individuals are expected to solve a variety of novel problems in their lifetime that require the use of di!erent skills, techniques, and procedures Although students may learn what these skills, procedures, and techniques are in a book or from

a lecture, a major challenge is to learn how to use them and just as important when to use them In our exercise, students must experiment with, adapt to, and alter strategies and tactics during simulated processes This leads to learning that strategies and tactics are most e!ective given the contextual conditions Moreover, students often lack the ability to make connections across concepts in the discipline Without understanding the connections, students cannot fully engage in reasoning and problem solving

Simulating political processes forces students to make such connections and apply their knowledge to solve novel problems Our simulation forces students to apply theories and concepts to simulated events they are generating and reacting to The design allows for constant evaluation of students' “application skills” and for improvement over time as they update their knowledge of what works and what does not Finally, the

exercise, by allowing students to tackle new problems, to apply their knowledge to events and processes, and to re$ect on their choices and decisions, develops critical and analytical thinking and problem-solving skills

In addition to developing such skills, it provides deeper substantive knowledge on concepts and theories than lectures and readings One can read about the strategic problems associated with the Cuban Missile Crisis, but until students are placed into strategic interactions of their own, in which they must take responsibility for their actions and connected consequences, they do not fully understand the associated quandaries Simulations allow students to immerse themselves into processes and experience the dynamics, constraints, and consequences of particular situations that lectures and case studies cannot As a result, such immersion leads to deeper

understanding of international relations concepts and theories

In particular, our simulation provides insight into negotiation processes and theories of negotiation Until students engage in negotiations, they have little understanding of the dynamics, problems, strategies, tactics, and give-and-take associated with them By

organizing the negotiation process into a game, students are able to break it down easier and analyze its component parts and strategic nature For example, it highlights

a game board (the environment), the players (negotiators), the stakes (issues), and the moves (tactics) each player can choose from Students have a certain amount of

resources, a wide variety of tools and options to utilize, and their success depends on the choices of others Through research students enter the game with various

motivations and must attempt to achieve their goals given resource and situational constraints By actively participating in negotiations, students are able to more fully understand the strategic environment, various constraints, and the dynamic give-and-take process Using this experimental framework, students actively explore concepts like mediation, public goods, hostile diplomacy, audience costs, commitment problems, signaling, and the prisoner's dilemma among others Moreover, by including various organizations in the game, students learn how organizational constraints hinder certain tactics from being implemented For example, students often complain about the

unanimous consent rule required for NATO policy implementation, the hindrance of veto power in the United Nations Security Council, and the quantity of votes rich countries can cast in the World Bank Until students try to achieve their goals under such constraints, they do not fully realize the constraints of organizational rules and decision-making procedures

Finally, our simulation exposes students to cultures and institutions they are all too unfamiliar with In order to understand international relations, one must understand the domestic and international histories associated with the actors and their ideologies

as well as their institutional and additional contextual constraints Within the context of our simulation, students research their assigned countries and organizations as well as their individual positions They then develop explicit goals to achieve during the

simulation The research component enables students to understand the history of their actors and analyze the key decision points over the course of time, while also introducing them to current events Within their research they uncover the domestic and international contexts associated with key political decisions, processes, and outcomes In a standard survey International Relations (IR) course, students rarely get exposed to speci"c characteristics about particular countries' economies, institutions, and foreign policies Now we turn attention towards the exercise itself

Overview of the Simulation

We provide the step-by-step instructions for executing the simulation elsewhere (Shellman and Turan 2005) However, we o!er a brief overview here We base the entirety of the exercise on a revised version of Kolb's (1984) experimental learning model to accommodate varied learning preferences Kolb's model consists of four stages: abstract conceptualization, concrete experience, re$exive observation, and active experimentation (Kolb 1976 1984 1988) Herz and Merz (1998) found that students believed that participation in simulation contributed to all four stages of Kolb's learning cycle In the context of our simulation, students learn abstract theories and concepts in lecture, readings, and discussions (abstract conceptualization) Second, students research their roles and develop goals to achieve (concrete experience) Third, students develop strategies to achieve their goals and experiment with their strategies (active experimentation and concrete experience) Finally, they re$ect on their actions, choices, and the outcomes (re$exive observation)

In particular, our simulation di!ers from other IR simulations in that it tries to model multidimensional aspects of IR that many others neglect such as the use of military force, domestic constraints, such as a domestic opposition groups, and the inclusion of nonstate actors like clandestine terrorist networks and ethnic groups Finally, rather than just focus on a single process/problem, country, or region, our simulation focuses

on multiple processes at work in the global system and across multiple regions

Our simulation pits students against one another on international terrorism, the future

of Iraq, and globalization Students are assigned various roles in the international system, given a budget, asked to research their roles and asked to develop goals to achieve The instructor weights each goal by assigning a numerical value to each one

Students' grades are determined by how many goals they accomplish and by the total points earned Students research their country positions on multiple issues, discuss those positions and develop an overall strategy to achieve their goals Each group's budget forces them to consider the costs of their actions and plan strategies to increase their combined resources To take an action (publicly or secretly), students must "ll out

an action form The action form records the action taken, describes it and indicates the costs/bene"ts to perform the action Most important from a learning perspective,

students must apply a theory or concept from the course content to explain and/or justify each action taken This requires students to constantly connect abstract theories and concepts to each action they take in the simulation The media then reports the public actions and any news they dig up on their own After each simulation day, summaries of the events are posted online and discussion questions are posted along with particular events At the end of the three-day simulation, we hold a debrie"ng session and ask students to re$ect on their experiences, to comment on their strategy, and to analyze their actions

One way to improve the simulation and the reality of the game is to use technological innovations Many instructors use new technological advancements both inside and outside of the classroom to teach these days We use technology in a few di!erent ways Students use e-mail to communicate and negotiate during the simulation, we use the Internet to post important documents and the event summaries, and we use

spreadsheets to manage all of our "nancial and military transactions In addition, we introduce web casts to the mix The media report their stories from a news desk that is broadcast over the Internet to a particular URL address every 45 minutes or so These reports keep students noti"ed of important developments outside their immediate actions and negotiations Media often show video footage of events like terrorist attacks, military battles, and/or natural disasters The goal is to create a more realistic environment With that brief overview of our simulation and our use of technology, we turn attention towards our evaluation of the simulation as an e!ective teaching and

learning technique

Research Design

We argue that our simulation enhances students' knowledge of international relations theories, concepts, organizational processes, countries' foreign policies as well as

facilitating the development of critical and analytical thinking skills To assess the impact that our simulation has on student learning, we follow the work of Brown and King

(2000) and Kuzma and Haney (2001) by administering an anonymous questionnaire to our students to probe how much they learned from their simulation experience The survey questions appear in Table 1 Based on our goals, we asked students to rank how much the simulation enhanced their understanding and knowledge of theories,

concepts, international organization processes, and substantive knowledge of their assigned country/organizations Additionally, we asked if the simulation exercise facilitated the development of critical and analytical thinking skills We also asked how much the technology enhanced their learning experience and whether or not the

simulation was enjoyable For all of these questions, students were asked to rank, on a scale from 1 (low) to 5 (high), the intensity of the simulation's enhancement in each of these areas as well as its overall enhancement of the IR course material In addition we asked students to rate the exercise in terms of other learning exercises they had

participated in at The College of William & Mary Was it the worst exercise, the best exercise or somewhere in between given their prior experiences? Finally, we asked each student to report their class ranks, grade point averages, majors, whether or not the course caused them to change their majors, and if they would recommend running the simulation in future classes

There are advantages and disadvantages to this type of survey For starters, simulations are generally fun and exciting and so evaluations from simulations may tend to

overestimate its e!ects We concede the point and to address the issue we separate the two concepts in our questionnaire by asking about both enjoyment and enhanced

learning independent of one another In terms of its strengths, the survey is able to gauge the independent impacts that the simulation has on student learning For example, the survey allows us to probe whether or not the simulation is more or less e!ective at enhancing knowledge of theories over concepts or vice versa It reveals where improvements can be made to make the simulation more e!ectively address one of our goals Finally, we are able to evaluate how the simulation stacks up to other learning exercises students have participated in

We gave the survey at the end of spring semester 2005 for two sections (41 students each) of Introduction to International Politics at The College of William & Mary The 82 person sample consists of 60% rising sophomores, 20% rising juniors, 10% rising

seniors, and 10% graduating seniors Over 36% of the class are IR majors Non-social science majors make up 28% of the class

The Results

Table 2 presents the frequencies of 82 students' responses to the core questions of interest For ease in conveying the results, we refer to each of the "ve-point scale values

as “enhancement levels,” where one represents no enhancement and "ve represents very high enhancement We begin by evaluating the results for “overall IR”

enhancement and the results from question 1 The mean for question 1 is 4.15, while the median is four The mode, however, is "ve Only one person ranks the

enhancement level at two, while over half the class ranks it at four or above Our exercise simulates most aspects and processes associated with international relations rather than focusing on any one single issue, con$ict, or organization The results from question 1 suggest that it does a good job of enhancing students' overall knowledge of IR

Questions 2 through 4 break international relations and learning into three subareas:

theories, concepts, and organizations and their processes The associated results indicate strong support for the simulation enhancing all of the above For example, each median response for each equation is equal to four, indicating that over half the class felt that the simulation enhanced their learning of international relations theories, concepts, and organizations While over 53% of the students ranked the theoretical

enhancement level at four or above, 70% or more of the students ranked the simulation's conceptual and organizational knowledge enhancement levels at four or above In the cases of the two latter categories, more students reported a "ve than a four; "ve is the modal value for each category

Our simulation includes many international organizations such as the World Bank, the

UN Security Council, the Arab League, NATO, the European Union, Amnesty International, etc Most students are involved in or connected to an international organization in some way or another, whether it is indirectly through a mutual group international organization representative or directly as the Secretary General of NATO

By the end of the simulation we feel and the results infer that students can identify with the problems international organizations address, the processes that govern at least one of the organizations included, and the constraints that are placed on countries' sovereignty We "nd that students also feel that the simulation enhances knowledge of international organizations and their processes

In contrast, the modal value is equal to four for theoretical enhancement These results convey that while the simulation enhances knowledge in all three areas, it does a better job enhancing the knowledge of concepts and organizations and their processes than IR theories Perhaps assigning discussion papers and asking more theoretically based

questions during the debrie"ng could bring this number up Moreover, some of the theories we cover in class have a domestic basis like Fearon's (1994) audience cost model and Milner's (1997) domestic institutions and interest groups based theoretical framework The simulation, at the present time, unfortunately lacks proper

mechanisms to apply such theories to decisions, processes, and outcomes and to analyze their utility Thus, the simulation constraints may explain why the modal value for theories is four and not "ve

That said, overall, we "nd support that the simulation enhances knowledge of substantive course material The results "t with Brown and King (2000) who "nd that students' knowledge of international relations increases as a result of participating in the International Communications and Negotiations Simulations (ICONS) Also

relevant are the results from Kuzma and Haney (2001) They show that the use of "lm in

a U.S Foreign Policy summer course to stimulate active learning works Using self-assessment surveys, Kuzma and Haney (2001) show that students' increase their knowledge of U.S foreign policy through "lms, discussions, and written exercises Our results are consistent with those discussed above in that they support the hypothesis that active learning exercises enhance learning

Both the modal and median values calculated for country/organization enhancement are equal to "ve More than 90% of the students stated that the simulation enhanced knowledge of their assigned country or organization at a level four or above on the "ve-point intensity scale This "nding is consistent with Brown and King (2000) who found that participation in ICONS enhanced knowledge of the country assigned and U.S

foreign policy In a typical introductory international relations course, professors rarely expose students to characteristics of particular countries and/or their current foreign policies Our simulation forces students to research a particular country's foreign policy

or a particular international organization's rules, history, and structure In order to actively engage in the exercise, one needs to know what to talk about, who to talk to, and what to o!er and not to o!er in order to get what one wants The research

assignment can be eye-opening to many students who are unfamiliar with international relations, particular countries, their foreign policies, and their regional a!airs We infer from our data that the simulation enhances students' knowledge of particular countries and/or organizations

In terms of critical and analytical thinking skills, over 74% reported enhancement levels

at four or above for question 6 This supports the hypothesis that our simulation aids in the development of critical and analytical thinking skills Our simulation compels

students to make their own decisions and anticipate and/or react to the consequences

For example, students in the past have created fake Yahoo e-mail accounts using the instructor's name and have used it to gather information from other groups Another individual used her friendship with the head terrorist's roommate to acquire

information about the terrorists operations These are some of the more intelligent and sneaky tactics students innovate and implement during the simulation The students ultimately create solutions to resolve international problems given constraints placed

on their countries and organizations These are skills that are di#cult to teach by reading and lecturing Rather, we argue that simulation is the best method for teaching such skills, because students are confronted with novel problems and they are forced

to experiment with strategies and tactics to accomplish them Then, they are able to learn from their successes as well as their failures Our "nding is consistent with Leonard and Leonard (1995) who found that business school graduates believed courses involving simulation exercises prepared them better for their future jobs than courses that only used case methods We want to produce professionals in all "elds who can analyze and solve novel problems, and the survey results suggest that our simulation develops these skills

In terms of enjoyment, it is not surprising that the majority of students had fun The results for question 7 illustrate that over 62% ranked the enjoyment level at the highest level possible Previous research, by Dedeke (1999), shows that experiential learning courses increase enjoyment levels From our experiences and observations, our

students for the most part have a blast and this is captured in the results from the survey

With respect to technology, most stated that the technology associated with the simulation enhanced their simulation experience, but, given the frequencies in comparison to the other questions, it did not seem to drive the “learning” or “fun”

aspects of the simulation Yet, the median and modal values are equal to four revealing that it did impact students' simulation experience

Finally, over 51% of the class said that the simulation was the best learning exercise that they had experienced during their tenure at The College of William & Mary Given this large percentage, we calculated chi-square statistics to see if this e!ect only held for sophomores and/or IR majors Our tests revealed that the responses were independent

of class standing and GPA Like Poplin and Weeves (1992, 16) argue, students want activities that require choice and action Our simulation is one of the few exercises where college students get to make such choices and take such actions About 95% of the students recommended using the simulation in future classes The results taken together indicate that students do in fact enjoy active learning exercises, they learn something from them, but at the same time such exercises are rare events in their college tenure We hope articles like this will encourage teachers and professors to create new active learning curricula and assess their e!ectiveness on student learning

In sum, we sought to develop a simulation that would enhance knowledge of IR theories and concepts as well as teach substantive knowledge about countries' foreign policies and political situations The results from the survey suggest our simulation and design accomplishes the goals we set Very few students report low scores for any of the questions Moreover, over half the class reported that the simulation was the best teaching tactic they experienced during their tenure at William & Mary If nothing else the simulation generated interest in the study of IR For example, in our sample, more than 22% of the class is changing their major to IR after taking the course While 50% of other social science majors are changing their major to IR, about 13% of non-social

science majors are changing their major to IR Overall, the results in Table 2 suggest that, according to the students' self-assessment, the simulation enhances their

knowledge of international relations, they enjoy the exercise, and the simulation stimulates interest

Conclusion

Stice (1987) and Boyer et al (2000) both show that students learn more by “doing” and

“saying” things than by reading and listening In response, we developed a simulation that incorporates more “doing” and “saying” to enhance student learning Speci"cally,

we wanted students to apply their book knowledge to real situations in order to garner

a deeper understanding of the concepts, theories, and processes discussed in our readings and lectures Furthermore, we wanted to design an exercise relevant to all students, not just IR majors Speci"cally, we wanted to develop an exercise that would facilitate the development of critical and analytical thinking skills and that would enable them to have fun at the same time

The results from our survey suggest that we accomplished the goals we set Speci"cally, our survey results reveal that the simulation enhances both substantive knowledge and critical and analytical thinking skills Most of all, the simulation provides a fun and

memorable educational experience We hope that other professors and teachers take Krain's (2005) charge seriously and not only develop new and innovative active learning exercises for their students but also design studies to test their e!ectiveness

1

2

3

4

5

6

7

8

9

Table 1 Survey questions

10

11

Table 2 Distribution of student rankings on simulation enhancement (introduction to IR, College of William &

Mary, Spring 2005)

CSV Display Table

/

12

Định dạng
Số trang	2
Dung lượng	230,68 KB