LIST OF TABLES Page 1 Five Main Categories Used in Formation of Attitude Survey ...21 2 Correlation of "I would voluntarily take an additional CS courses if I were given the opportunity"
Significance
Rapid growth in technology has created an urgent need for well-trained scientists and engineers in the United States, prompting NSF-funded programs to support education projects One grant, CPATH CB: Computing Concepts Across the Curriculum (NSF ID: 0829563), funds Drs Vicki H Allan, Donald Cooley, Chad Mano, and Joel Duffin to develop interactive learning modules (ILMs) that help students grasp computer science concepts Inspired by the National Library of Virtual Manipulatives (NLVM), ILMs replicate classroom activities with virtual representations, since computer science lacks many proven physical teaching props Evaluation is critical and takes multiple forms, including student evaluations of ILMs, teacher evaluations of ILMs, and a computing attitude survey, with the aim of establishing a process for evaluating future ILMs.
Problem
Introductory computing classes are often unappealing because they emphasize tedious details An AAUW study shows that in schools girls are more likely to learn spreadsheets and word processing than computer programming, limiting early exposure to the field When programming is the first encounter with computer science, it frequently fails to engage students, contributing to high dropout rates Many students find the initial CS material intimidating and unwelcoming As Stanford’s John Gardner warned, "Twenty years from now we will look back at education as it is practiced in most schools today and wonder that we could have tolerated anything so primitive." This suggests that new approaches are needed to make computer science more accessible, engaging, and relevant to students.
Entry into the computer science field often hinges on the ability to program, making coding the gatekeeper and leaving little room for alternative paths Despite calls for reform, many so‑called new approaches to teaching CS still adopt a programming‑driven model with calculus as a prerequisite, thereby preserving the same barrier for learners seeking a different route into tech.
Traditional learning relies on formal classroom instruction, reading, solving practice problems, and experimentation, but many students find that text-heavy study and written homework have predictable limitations This approach often lacks a method for independent experimentation or for exploring practical questions such as, “What happens if I do this instead of that?” To close these gaps, educators are increasingly embracing hands-on, inquiry-driven, and self-directed learning strategies that encourage experimentation, immediate feedback, and real-world problem solving.
To address these challenges, interactive learning modules (ILMs) across a range of computer science topics are designed to deliver engaging, experiential education These modules introduce computer science through hands-on learning, helping students move beyond the notion that CS is “just programming.” The ILM approach presents intriguing problems and the fundamental concepts that computer scientists study, inviting students to experiment with tools built on those concepts in an interactive and enjoyable environment.
Although many ILMs have been created, little evaluation has been done on their usability and effectiveness To gather feedback, each ILM includes an optional student survey, and there is also a survey for teachers to describe their experience with the ILMs To assess students' attitudes toward computer science, a survey was conducted using questions drawn from [2] The survey questions are provided in Appendices A, B, and C.
1 All of the ILMs are available on web at csilm.usu.edu
This chapter describes the methods used to collect the data, our first step for improving ILMs.
Student Survey
For the assessment of the ILMs, surveys were administered to Logan High School students focusing on two of the ILMs to gauge user enjoyment and educational value The study sought to determine whether students found the ILMs enjoyable to use and whether they perceived them as educational, with the results intended to inform future refinements.
Student surveys reliably capture student likes and dislikes but are less effective at determining whether interactive learning modules (ILMs) actually teach concepts In high school, it is uncommon for students to be asked whether an activity is a good use of their time or whether it helped them learn the material Our preliminary results indicate that high school students are more drawn to the entertainment value of ILMs than to whether the activity meets its educational objectives.
Teacher Survey
The basic research question is, “Are ILMs an effective teaching technique?” However, there are so many variables that this is hard to determine Variables include:
1 Enthusiasm of teacher for the ILMs
2 Expertise of teacher in the subject area
3 Variation in time on task
Due to the inability to control the external variables and limited data points, our evaluation of effectiveness relies on teacher perception of effectiveness
When instructors show acceptance of interactive learning with ILMs, students are more likely to be encouraged to use them, whereas if teachers are not convinced of ILMs’ value, student adoption is unlikely; surveys that measure teacher willingness to adopt ILMs in classroom settings help explain these dynamics Briscoe notes a situation where teachers did not implement new techniques in class despite indicating a willingness to do so, illustrating the gap between attitude and practice Moreover, curriculum and teacher attitudes shape student attitudes, with studies showing that senior students’ attitudes are less variable than those of freshmen Finally, teachers themselves provide valuable insights into the problems students face when working with ILMs.
Usability Testing
To evaluate the usability of the Intelligent Learning Modules (ILMs), feedback was collected from a software‑testing specialist instructor, and five computer science majors were asked to use the ILMs and provide their reviews The data were collected from these participants to capture their perceptions of usability and overall user experience with the ILMs.
Two groups of students were surveyed after using two ILMs—the Boolean Logic ILM and the Loops ILM—both designed by Russ Weeks, a computer science and math teacher at Logan High School, during a Research Experiences for Teachers program in the Computer Science Department at Utah State University These ILMs were chosen for evaluation because they allowed access to students in the classroom, and a high school setting with computers provides an ideal environment for observing use, whereas college classrooms often rely on ILMs mainly for demonstrations or homework due to tighter schedules and fewer individual computers To observe authentic use, the Logan High classroom with computers was selected Students completed surveys and were observed by the instructor, the two ILM designers, and the grant’s principal investigator Descriptions of the ILMs follow.
Boolean logic ILM
Developed by Kyle Feuz, a Computer Science student at Utah State University and a member of the CPATH team, the Boolean Logic ILM was created as part of the CPATH grant from the National Science Foundation (NSF) The ILM is designed to teach how Boolean operators function, with its basic layout illustrated in Figure 1.
An interactive learning module presents a UI where users select shape properties on the bottom left (isRectangular, isRound, isStriped, isSpotted, isTall, isWide, hasRed, hasGreen, hasBlue) and build a Boolean expression using operators from the upper left to be typed in the text field at the bottom; clicking Apply Expression prompts the ILM to evaluate the expression and move the shapes that evaluate to true to the selected side In Let me Try mode, you can drag shapes between the selected and unselected sides to verify results, and the Apply Expression button becomes Check Expression to provide feedback on whether your selected shapes match the expression The tool offers adjustable difficulty levels based on your understanding of Boolean operators, and in basic mode you can form Boolean expressions using only core operators such as AND, OR, and NOT.
NOT is a fundamental operator In normal mode, you gain access to additional options such as XOR and SET DIFFERENCE, and you can use parentheses to alter operator precedence In advanced mode, you can employ a broader range of more powerful operators.
Explore NAND, NOR, and XNOR operations in this interactive tool You can control how many shapes the Boolean expression is applied to by selecting a number from one to six in the drop-down menu labeled "Num of Shapes," giving you flexible, scalable testing across multiple shapes.
Loops ILM
I developed the Loops ILM as part of the NSF CPATH grant, a block-based educational tool that lets students experiment with for loops by creating blocks and adjusting loop parameters to arrange these blocks into a quilt-like pattern The basic layout of the ILM is shown in Figure 2.
In this tool, users can finely tune loop parameters and other properties to customize pattern design, including border color, background color, pattern type, block colors, rotation of the pattern to any degree, and block size The effects of multiple loops can be combined by selecting the “Add to existing project” option, enabling more complex patterns To view the result, click “View Project” to execute the pattern-creating loop(s), as illustrated in Figure 3.
Figure 3 Output created from the loop in Figure 2
Use the Rotate block by angle feature to rotate the colored block to your desired degree, then click View Project to print the block on the quilt You can adjust the print size with Block size (in Pixels), ensuring the block fits your quilt precisely The Pseudo Code button reveals the pseudocode for all the For loops used to create the pattern, while the Use Border option lets you add or remove a border on the quilt Customize the look further by selecting Background color and Border Color from the color dropdown, choosing from twelve standard colors or selecting More Colors for a broader palette Finally, the Sample Pattern button opens a window with several sample patterns for students to generate.
Appendices D and E contains instructions provided on the side bar of the Boolean logic ILM and the Loops ILM, respectively
This chapter describes how the surveys were conducted.
First Session
The first session, held on May 5, 2010 at Logan High School, welcomed a class of 18 students enrolled in the beginning programming course The session used the Boolean logic ILM, and most students already had some knowledge of Boolean logic and Boolean operators.
Initially, students received a brief introduction to using the ILM and a review of Boolean operators, followed by a practical quiz that challenged them to apply Boolean expressions to real-life prompts, such as "Stand up if you are blond AND tall," to gauge understanding Ice-cream coupons were distributed as a reward for cooperation in the ILM evaluation The students then worked with the ILM on their own for the remainder of the period Before starting, they were informed about an anonymous end-of-ILM survey designed to collect feedback and help improve the ILMs.
Out of 18 students, 14 filled out the survey While some students enjoyed the ILM, some thought that it would have been much better if it were more visually appealing
To test our hypothesis that many students do not attempt to read instructions, we asked the students, “Who prefers to learn by reading the instructions and who does not?”
Only 3 of 18 students reported that their standard learning method involved first reading the instructions In the study, these three were asked to read the instructions before starting the ILM, while the others were told not to read them We observed that students who indicated they did not generally read instructions were not tempted to read them, even when they did not understand something In the survey, they complained that something was awkward or did not work, even though reading the instructions would have solved the problem.
Second Session
On May 25, 2010, a second survey was conducted after school due to scheduling difficulties, requiring voluntary participation with motivations of pizza and extra credit Twelve students signed up, and the arrival of the pizza served as an unplanned test of the ILM's level of engagement, since students who had not completed all the sample patterns were busy generating them and did not begin eating until finished One difficulty in evaluating ILMs is finding a situation where students have the prerequisite explanation but desire additional experience; this testing environment was not ideal for evaluating ILM teaching effectiveness, but it was extremely useful in judging popularity and engagement The first session with the Boolean Logic ILMs suggested several changes, including more shapes and more descriptors, which were implemented before the second evaluation The Boolean logic ILM is currently depicted in Figure 4.
Figure 4 Improved Boolean logic ILM.
The new Boolean logic ILM now includes more graphics and shapes in response to requests from several students in the first session, enhancing visual learning Let Me Try mode is now the default, and two separate buttons are provided for Apply Expression and Check Expression instead of one The difficulty selector has been changed to "Boolean Operators" and moved to the top of the ILM for quicker access These updates improve usability and engagement for users exploring Boolean logic and expression evaluation.
Use the Objects button to select the number of objects (between 1 and 6) and to choose the object type The object type can be either geometrical shapes or fun images If you select images as the object type, new Boolean operands become available—for example, isLiving and isHuman—to tag and filter images more precisely.
“isAnimal”, “isPlant”, “isCartoon”, “canMove”, “isElectronic”, “smells” are provided
We offered students a choice between the Boolean logic ILM and the Loops ILM, and Meagan Beavers, a technology teacher at Thomas Edison Charter School in North Logan, Utah, and a RET program member, noted that students are more motivated by a challenge—“See if you can produce X”—than by following step-by-step instructions to produce X Acting on this insight, sample patterns were added as an activity for students to generate The various patterns were chosen to exercise a variety of features of the loops, including rotation, increments, starting values, and even variable starting values such as beginning with the row index Figures 5 through 10 show the sample patterns used.
Figure 5 Loops ILM sample pattern 1.
Figure 6 Loops ILM sample pattern 2
Figure 7 Loops ILM sample pattern 3
Figure 8 Loops ILM sample pattern 4
Figure 9 Loops ILM sample pattern 5.
Figure 10 Loops ILM sample pattern 6
To address students’ tendency to ignore instructions, videos were prepared for each ILM Each video is approximately 4 minutes long and gives basic information about the functioning of the ILM, but does not contain information motivating the topic
In this session, initially, the students watched both the videos and then were free to try either or both ILMs The students were given a sheet of paper for each ILM to direct their activities One contained questions on Boolean logic involving simplifying the expression, determining the output of an expression, and comparing two expressions to determine if they are logically equivalent The other sheet was merely a check off list for the various sample patterns of the Loops ILM Students had to get an instructor’s signature after creating each sample pattern There was no point value associated with producing a given pattern or any reward for progress At the start, the students were almost equally distributed in the two groups of ILM users, but eventually nearly all the students started working on the Loops ILM In the end, we received the survey response from eight students out of which one was for the Boolean ILM, six for the Loops ILM, and one student did not mention the ILM he used.
Third Session
Heersink and Moskal [2] identify the factors behind the decline in students’ interest in computer science and information technology through an attitude survey In this study, the attitude survey used is an extension of the original Heersink and Moskal instrument, designed to capture broader attitudes toward CS and IT.
The survey used in this study followed the original design implemented by the authors of [2], with the addition of a few questions on students’ access to computers and their GPA The survey instrument, Computing Attitude Survey (Long), is available at http://csilm.usu.edu It is based on the same five categories as those used in the original study by the authors of [2].
In this discussion, we replace professional category with a more precisely labeled
“Stigma” category The five categories for the survey are described in Table 1 Refer to Appendix C for the questions used in the survey
Table 1 Five Main Categories Used in Formation of Attitude Survey
Confidence Category Students’ confidence in their own ability to learn computing skills Interest Category Students’ interest in computing
Gender Category Students’ concept of computing as a male field
Usefulness Category Students’ belief in usefulness of learning computing, &
Stigma Category Students’ belief about computing as a lonely field to work
The motivation behind our study is to provide insight to the following questions:
1 What predicts whether a student will want to take more CS classes?
2 Do students believe that a computing career is appropriate for women?
Fourth Session
To evaluate the usability of ILMs, a small cohort of computer science graduate students was invited to use the ILMs and provide feedback College students were chosen to capture perspectives that differ from those of high school students For usability testing, we adopted an Observation Approach in which testers observe individual users interacting with the ILMs, either by recording video or by taking detailed notes Participants were asked to verbalize their thoughts while using the product to reveal their decision processes and usability issues Video observation was considered advantageous for capturing a range of factors, and in practice a video was used to evaluate the ILMs with two students, while the other participants were monitored through notes during their use of the ILMs.
This chapter states the things learned from the surveys and some assumptions.
Students Do Not Read Instructions, and Video Seems to Work Well
A total of 22 surveys were collected across both ILMs In the first session, 14 of 18 students completed the survey, and in the second session 8 of 12 did so Observations and survey responses show that only four of the 22 students read the instructions before starting the ILM, indicating a low tendency to read instructions To reduce navigation discomfort when instructions are on a separate page, the ILM instructions are now displayed in the right-hand side pane of the ILM, as depicted in Figure 11.
Placing the instructions in the same view as the ILM may be convenient, but it isn’t clearly effective A key factor appears to be peer pressure: when students see nearby peers taking a more active role, they quickly abandon passive reading of the instructions In the initial survey, four of fourteen students reported not understanding the operators well, even though the sidebar provides explanations for all operators.
Figure 11 Boolean logic ILM and its instruction on the right
In our testing environment, the classroom is split into two zones—a lecture area with chairs in rows facing a smart whiteboard and a separate computer lab—so instructions and the video presentation are delivered in the classroom before students move to the lab, ensuring they cannot start using the ILM until the preliminary briefing is complete This two-zone setup is effective, and a similar outcome can be achieved by locking students out of the computers during the presentation.
Video-based instructional content quickly orients students and follows a problem-solving, start-to-finish approach, while printed instructions tend to be more functional, outlining every option The video format is more likely to be viewed in its entirety, whereas ILMs are largely self-explanatory, which can make reading the instructions less compelling; to address this, tooltips on the ILM should be provided wherever possible In the first survey, four of thirteen students reported difficulty understanding Boolean operators and suggested adding tooltips on the buttons (for Boolean operators and operands) to improve comprehension.
An instructor's comment highlighted that the demo video was too long to maintain student interest, and analysis of student responses found that two learners struggled with understanding how to add loops over another and why a design flaw arises when adding a new loop over another, even though those points were clearly addressed in the video A possible reason is that the video introduced all ILM concepts before students gained practical experience, making it hard to retain information they had no immediate need for A practical solution is to produce both an introductory ILM video and a separate advanced-features video; however, in classroom settings, many computers lack speakers, so having each student access the appropriate video on demand for additional instruction would not be feasible.
Overall, instructional videos are more effective than written instructions for guiding learners For example, one student watched the video after the demo had been shown to the entire class, then began working on the ILM.
5.2 Activities on the ILM Help Capture Students’ Attention
During the first session, students working solo on the Boolean logic ILM completed most tasks within about fifteen minutes and then proceeded to the survey despite the ample time available In the second session, which integrated both the Loops ILM and the Boolean ILM, students showed remarkable interest in the Loops ILM Generating sample patterns with the Loops ILM proved highly engaging, teaching 'for loops' and cultivating a competitive spirit even though there was no reward for finishing first; even when pizza arrived, attention stayed on pattern generation Two students who could not complete all patterns remained deeply engaged until the period ended and the computer shut down, preventing them from completing the survey Although there is no measurement explaining why the Loops ILM appealed more than the Boolean logic ILM, several possible reasons are proposed for this difference in appeal.
Creating sample patterns proved more challenging for students than pure self-experimentation, and the Boolean logic worksheet felt like homework Six students completed the Loops ILM survey, and each participant mentioned, at least once, that they enjoyed the activity of creating patterns The Loops ILM results were additive, with students recognizing steady progress as they moved closer to their goal.
2 In a group setting, students tend to be affected by what others are doing Watching neighboring student generate patterns created interest in the student.
Loops ILM offers greater play value and is highly creative and colorful In the present age when students use computers for gaming, the Loops ILM proved to be more game-like than the Boolean logic ILM, enabling learners to create something even if they can't produce a correct pattern.
Beyond the points already discussed, this article presents a structured comparison of the differences between Loops ILM and Boolean Logic ILM The differences are organized under distinct subheadings for each ILM, detailing factors such as pedagogy, user experience, assessment methods, and practical applicability Students often favor Loops ILM over Boolean Logic ILM due to its interactive loops, immediate feedback, modular content, and emphasis on iterative problem solving that mirrors real-world workflows Conversely, the Boolean Logic ILM offers a rigorous theoretical foundation, a formal logic framework, and clear criteria for logical reasoning that attract learners seeking deep conceptual understanding Additional differentiators include interface usability, adaptability to diverse learning paths, and the depth of contextual examples provided This comparative analysis helps educators and learners identify which ILM best aligns with their goals, with Loops ILM frequently preferred for hands-on engagement and adaptive learning, while Boolean Logic ILM is valued for structured theory and analytical rigor.
The Loops ILM activity was goal directed
Active feedback engaged students in the learning process, inviting them to participate and reflect on their own work They could see the difference between the pattern they generated and the pattern they were asked to develop, making gaps in understanding visible By identifying where their approach went wrong and deciding how to correct it, students practiced error analysis, self-assessment, and corrective strategies that strengthen their pattern-design skills.
By visually comparing their results with the desired output, students could spot subtle differences, such as an incorrect border color or rotation, and they tended to strive for perfection rather than settle for “close enough.”
4 Problem conjures up ideas for the solution
When problems arise, people often describe them with language that points to a solution rather than the root issue A student might say, "I don't have the right number of columns" or "I haven't rotated it enough," instead of admitting, "Gee, it’s wrong and I have no idea why." This problem-framing—focusing on the fix rather than the underlying cause—can mask the real challenge and slow down learning and effective problem solving.
5 Immediate feedback involving teacher approval
Immediate visual feedback on the output saves time by clearly showing results, while the instructor’s check-off of completed solutions provides an opportunity for teacher praise and positive reinforcement.
The patterns were arranged in increasing order of difficulty Starting with simple patterns helped students to build confidence to help them in moving to more complex patterns
Once students completed all the patterns, they were able to help their peers, turning finished work into peer support Even after finishing the patterns, some students kept designing patterns on their own while others continued to assist their classmates with new designs.