In this chapter, the following key information is provided in preparation for evaluating an assessment instrument to measure a middle school students’ proficiency with systems thinking f
Pipher (2013) expresses hope that breaking the silence around global climate change will initiate meaningful conversations about our planet, emphasizing the importance of public awareness In *The Green Boat*, she highlights that, “it is the most practical among us who come out of denial first,” stressing that early acknowledgment is crucial for taking meaningful action Educating the public and students about environmental issues is essential, as it provides the “knowledge, skills, and experiences” needed to understand complex natural systems and their impacts on future generations (Kleckner, 2010) In 2012, representatives from 98 countries reaffirmed the original vision for environmental education through the Tbilisi Communiqué, emphasizing the importance of educating today for a sustainable future.
The objectives from the 1977 Tbilisi Conference—awareness, knowledge, attitude, skills, and participation—remain relevant today, serving as essential components for promoting sustainable development The primary goal of aligning human behaviors, actions, practices, and social conditions toward a sustainable future continues to be an ongoing challenge that has yet to be fully achieved (Tbilisi Communiqué, 2012, p 1)
The purpose of this chapter is to value educators’ efforts to realize the main goal of the
Tbilisi Conferences using Gough’s (2013) four principles from the International
Handbook of Research on Environmental Education:
• to recognize that knowledge is partial, multiple, and contradictory;
• to draw attention to racism and gender blindness in environmental education;
• to develop a willingness to listen to silenced voices and to provide opportunities for them to be heard; and
• to develop understandings of the stories of which we are a part and our abilities to deconstruct them (p 10)
In this chapter, the following key information is provided in preparation for evaluating an assessment instrument to measure a middle school students’ proficiency with systems thinking for environmental literacy:
The Oregon Environmental Literacy Plan’s (OELP) systems thinking strand provides educators with the responsibility to teach and assess middle school students' decision-making skills Incorporating decision-making into classroom instruction helps students interpret complex, partial, and conflicting data, fostering critical thinking By integrating these skills, educators and students can learn to make sense of diverse experiences and data, ultimately enabling students to act thoughtfully and responsibly in environmental contexts.
“[align] human behaviors, actions, practices and social conditions towards a sustainable future” (Tbilisi Communiqué, 2012, p 1)
Systems thinking concepts and tools equip middle school students with essential skills to model complex systems and engage in decision-making processes from their own perspectives These tools serve as a vehicle, empowering students to confidently express their ideas and make their voices heard By mastering these skills, students enhance their ability to analyze, communicate, and collaborate effectively in various conversations and contexts.
3 “Artificial boundaries” describes by former Oregon Governor Kitzhaber (2013) between Career and Technical Education (CTE) and Science, Technology,
Engineering and Math (STEM) have shaped perceptions of the value of diverse educational experiences, highlighting the importance of integrating formal and non-formal learning The Smith-Hughes Act of 1911 inadvertently created boundaries between vocational and general education through national funding, impacting educational opportunities Addressing these historic injustices requires educators to work towards reconciliation, ensuring that vocational education respects cultural and natural systems and promotes equitable access to all learners.
Citizen science and teaching practices that emphasize both inductive and deductive reasoning inspire educators and students to actively reconstruct their communities By employing observational and logical inquiry methods—individually and together—students develop a deeper understanding of their roles within larger systems This dual approach encourages critical thinking and empowers learners to make informed decisions that impact their environments.
A Plan for Environmental Literacy in Oregon
Oregon’s Environmental Literacy Task Force was established through House Bill 2544, known as the No Oregon Child Left Inside Act (NOCLI), signed into law on July 22, 2009 This legislation aims to provide Oregon youth with comprehensive environmental education, including classroom instruction on vital natural resources and opportunities for outdoor field investigations Governor Ted Kulongoski emphasized that NOCLI would ensure students gain hands-on experience and understand the importance of natural resources Chairperson Traci Price of the Freshwater Trust and the Environmental Education Association of Oregon (EEAO) played a pivotal role in drafting the Oregon Environmental Literacy Plan (OELP) as a mandate of H.B., guiding the state's efforts to promote environmental literacy among students.
Oregon became eligible for federal funding with the approval of the federal No Child Left Inside Act, part of the reauthorization of the National Elementary and Secondary Education Act The Oregon Department of Education (ODE) coordinated public meetings for the task force, whose members were appointed from various state agencies including the Department of Environmental Quality, Department of State Lands, Department of Fish and Wildlife, Department of Forestry, Department of Agriculture, and Parks and Recreation.
The project involved collaboration between the Department, Metro Regional Government, Oregon State University’s College of Science, and Sea Grant programs Citizens, representatives, and both formal and non-formal educators actively participated as members of the working groups established by the task force Formal educators included those providing students with academic credit through school districts, the Oregon University System, and community colleges, while non-formal educators offered educational experiences for students and families via organizations like the Oregon Forest Resources Institute, Oregon Zoo, Oregon Coast Aquarium, and World Forestry Center The working groups focused on areas such as Educational Standards and Diploma Requirements, as well as Teacher Professional Development, ensuring a comprehensive approach to education improvement.
The 2010 OELP emphasizes the importance of developing, implementing, and assessing science literacy that aligns with Osborne's (2007) concept of considering plural alternatives, fostering a comprehensive understanding of diverse scientific perspectives It also advocates for a system-wide commitment to science literacy, as highlighted by Cutter-Mackenzie and Smith, to ensure sustainable and inclusive educational initiatives.
The international accord outlined in the 2012 Tbilisi Communiqué aligns with the Oregon Legislature's 2010 definition of environmental literacy, as established by the No Oregon Child Left Inside (NOCLI) task force This shared vision emphasizes the importance of fostering environmental understanding and awareness among youth to promote sustainable actions Both frameworks underscore the need for comprehensive environmental education to empower individuals to address global ecological challenges By promoting environmental literacy, these initiatives aim to cultivate a generation equipped to make informed decisions for a healthier planet.
An individual’s understanding, skills and motivation to make responsible decisions that considers [one’s] relationships to natural systems, communities and future generations” (Oregon Department of Education, 2010, p 4)
The NOCLI task force has identified Environmental Literacy Strands that establish a comprehensive framework of content and skills By high school graduation, students are expected to be environmentally literate, capable of applying systems thinking concepts and tools to support informed decision-making in environmental contexts.
Understand and apply systems thinking concepts and tools a Systems as context for thinking and action b Implications and consequences c Strategic responsibilities of systems thinking d Shifting mental models and paradigms (p 16)
The tools for using systems as a “context for thinking and acting” included:
The Oregon Environmental Literacy Plan integrates systems thinking concepts and tools to help educators design engaging learning experiences and assessments tailored to middle school students’ ability to envision possibilities and take action Systems thinking goes beyond problem-solving, inspiring students to articulate their reasoning and become active participants in solutions that work harmoniously within natural systems (See Appendices A-F for templates of systems thinking tools from Waters Foundation's "Systems Thinking in Education," watersfoundation.org).
Systems Thinking Concepts and Tools
While systems thinking may initially appear too complex for middle school students, it is essential to introduce such approaches given the intricacy of natural phenomena and social structures Educators have a responsibility to teach decision-making strategies that cross academic boundaries in the sciences, fostering critical thinking and understanding at an early age.
In Western culture, the context is often assumed or considered outside the problem solver's responsibility, leading to a focus on well-developed analytical methods across sciences Systems dynamics stands out as the only synthetic scientific method, which gained prominence with the advent of computers and is now widely adopted in various fields.
With, or without computers, systems thinking concepts and tools help individuals synthesize information and articulate the kind of reasoning for decisions characteristic of environmental literacy
To effectively assess middle school students' environmental literacy, it is essential to align evaluation methods with systems thinking, environmental education guidelines, and science standards outlined in Chapter 1 Reviewing relevant literature highlights the importance of developed assessment instruments, shaped by educational leaders experienced in environmental literacy Osborne emphasizes that the research community should focus more on assessment to advance environmental literacy practices.
Effective practice in education relies on the integrated triad of curriculum, pedagogy, and assessment, though research has historically focused less on assessment In an era of growing accountability, teachers primarily rely on assessment to identify the intended curriculum rather than solely examining the curriculum content itself (Osborne, 2007, p 182).
Hollweg et al (2011) emphasize the importance of researching students’ confidence in decision-making and tracking their progress over time to enhance environmental literacy They highlight the critical need for national and international assessment data that detail students' understanding of environmental concepts across different components and developmental levels Additionally, Elder (2003) underscores the importance of establishing standardized assessments based on common standards to effectively measure environmental literacy at the national level.
To strengthen environmental literacy within national priorities, it is essential to establish a foundational baseline through comprehensive national goals, benchmarks, and standards (p 93) By 2013, Oregon educators successfully developed learning strands for environmental literacy that can be used for assessment purposes The next step involves utilizing these Oregon Environmental Literacy Program (OELP) strands to measure each student's proficiency level using appropriate assessment tools.
Chapter 2 explores two key components: first, it discusses educational theories for assessing science and environmental literacy to understand the reasoning behind assessment development It emphasizes that educators design assessments to measure specific aspects of environmental literacy tailored to particular groups, locations, and time periods The chapter evaluates existing assessment instruments aligned with the environmental literacy strands in the OELP Secondly, it provides a detailed comparison of eight assessment tools, examining their validity, reliability, and equity to ensure comprehensive evaluation and effectiveness.
The educational theory appropriate for developing an environmental literacy assessment instrument was constructivist because it “empowers” educators (Guba & Lincoln, 1998, p 210) For Guba and Lincoln (1998) constructivism entailed
A constructivist approach involves understanding and reconstructing the initial beliefs held by individuals, including inquirers, with the goal of reaching consensus while remaining open to new interpretations as information and understanding evolve This methodology ensures that implications for policy and practice are grounded in validity and reliability, which are derived from community consensus about what is 'real' and meaningful, especially for informing future actions By adopting constructivist theory, educators can confidently communicate about middle school students’ proficiency in systems thinking for environmental literacy, as outlined in the 2010 Oregon Environmental Literacy Plan (OELP), to diverse stakeholders such as school boards, principals, and funding agencies.
The constructivist approach gained prominence through public discourse initiated in 2009 when the Oregon legislature established the No Child Left Inside Task Force to develop an Environmental Literacy Plan Enhancing mutual understanding of students’ and educators’ constructs is essential for improving the credibility of environmental literacy assessment tools According to Maddock (1999), citing Ellsworth, educational strategies are shifting from presenting multiple perspectives, enabling students to choose one, toward fostering deeper understanding and critical engagement with diverse viewpoints.
The article highlights how certain discourses reveal the partial, interested, and potentially oppressive nature of knowledge, which can influence alliances by fostering environments where “difference can thrive.” Guba and Lincoln (1998) emphasize measuring credibility through “authenticity” and whether findings act as a “catalyst for action,” promoting ongoing efforts toward consensus Educators and students aim to understand their world by interpreting the meanings of their experiences, viewing these as essential to comprehending the “processes of interactions between individuals” that reflect underlying cultural norms shaping their lives.
This article discusses the adoption of a constructivist approach to assess educators' understanding of systems thinking, proficiency, equity, and environmental literacy in Oregon, utilizing the Delphi technique to develop a scoring guide that highlights different conceptual understandings It emphasizes the importance of social constructivist theory, particularly Vygotsky's Zone of Proximal Development (ZPD), to explain how children learn through guidance and encouragement from skilled partners Additionally, the development of children's "counterfactual thinking," or the ability to imagine alternative worlds and actions, as described by Gopnik (2010), informs the developmental appropriateness of engaging middle school students in explaining their reasoning using systems thinking tools.
Research on assessing environmental literacy has expanded since Osborne (2007) highlighted the need for increased educational accountability and curriculum-based assessments The Oregon Environmental Literacy Plan (OELP) suggests several measurement tools, including the OAKS, student work samples, and an adapted version of the 2008 NELA test used by NAAEE to establish a baseline for middle school students’ environmental literacy To effectively demonstrate student proficiency across the five OELP learning strands, educators can also use validated assessment methods such as teacher-scored work samples and targeted instruments like MEERA, designed to evaluate specific environmental education programs Applying these reliable and equitable assessment strategies ensures honest reporting of students’ environmental literacy levels and aids in providing clear performance data to the public.
• Consider a student's ability to show their reasoning and evidence in evaluating students’ decision-making skills
• Verify the understanding of the construct of environmental literacy among those using the scoring guide
• Recognize that the understanding of scientific principles and personal experience influence the development of environmental literacy over time
This study highlights the importance of considering the level of generalization when interpreting assessment data Due to its research design, findings are limited to a small, representative sample of middle school students and educators skilled in systems thinking, science concepts related to natural phenomena, and experience working with community and natural systems across Oregon's diverse eco-regions.
Review of the Methodological Literature
Construct validity is essential in ensuring accurate measurement of systems thinking in environmental literacy According to Trochim (2006), two key threats to construct validity are a preoperational understanding of the construct before measurement and the mono-operation bias from using a single setting, time, and group Trochim emphasizes that construct validity is an overarching quality that underpins all other measurement validity types To address these challenges, he recommends using Pattern Matching Theory to align theoretical patterns with empirical data, thereby enhancing the accuracy and robustness of construct measurement.
“observed pattern” (para 2) In short, he suggested a researcher investigate whether the
Educators’ "ideas" or "hunches" about how a student demonstrates a specific environmental literacy strand align with what can be "observed" and "measured" through targeted assessment tools This approach ensures that assessments accurately reflect students’ environmental understanding, supported by examples such as Cronbach and Meehl’s work on measurement validity Using appropriate assessment instruments helps verify whether students’ demonstrated knowledge matches educators’ expectations, enhancing the validity and reliability of environmental literacy evaluations Incorporating these principles into assessment design improves the accuracy of measuring students’ environmental literacy skills effectively.
A nomological network, introduced in 1955, serves as crucial evidence in psychological testing research by representing the interconnected concepts, their observable manifestations, and relationships within a study To minimize mono-operational bias, it is essential to apply consistent scoring methods across different raters, such as educators assessing middle school students' knowledge and skills in systems thinking This consistency can be achieved by providing uniform scoring instructions during field testing, whether in formal or informal settings, ensuring reliable measurement of the constructs of interest.
The OELP Task Force enhanced the face validity of the study by clearly defining the meaning of environmental literacy, ensuring a more accurate assessment This improvement was supported by input from a diverse group, which fostered a shared understanding of the construct Prioritizing face validity in the drafting process was crucial, especially in alignment with the principles outlined in H.B., to strengthen the study's credibility and relevance.