work-in-progress-assessment-of-automation-labs-to-facilitate-continuous-improvement

Results and Discussion For each lab, the survey questions are presented, box plots of the data by question, box plots for the combined question data, a table of the Wilcoxon Ranked Sum

Mr Bradley Lane Kicklighter, University of Southern Indiana

Brad holds a BS in Electrical Engineering from Rose-Hulman Institute of Technology (1989) and an MS

in Electrical and Computer Engineering from Purdue University (2001).

His past work experience includes eleven years at Delphi (formerly Delco Electronics) as an Advanced Project Engineer, eleven years at Whirlpool Corporation as a Lead Engineer/Solution Architect, and three years at Ivy Tech Community College as an Instructor/Program Chair of Pre-Engineering Since 2015,

he has been employed at the University of Southern Indiana as an Assistant Professor of Manufacturing Engineering Technology.

He holds three patents, has served as an IEEE section officer since 2004, and has been a Licensed sional Engineer in the State of Indiana since 2005.

Profes-c

Work in Progress: Assessment of Automation Labs to Facilitate Continuous

Improvement

Abstract

With the implementation of new equipment and laboratory activities in an automation course, it

is desired to assess the effectiveness of the equipment and labs so that improvements may be made This paper covers the assessment methods used and the results to date

New Programmable Logic Controller (PLC) trainers have been introduced in SCADA

(Supervisory Control and Data Acquisition) Systems Design The labs for this course have either been modified or new ones created to work with the new equipment To objectively assess the effectiveness of the equipment and laboratory activities, pre- and post-lab surveys are

administered to the students Students are surveyed about their understanding of the various objectives of the labs The survey results, along with direct assessment data in the form of lab scores, are used to identify areas of improvement for the equipment and the labs

Introduction

In the fall of 2020, a new PLC trainer was introduced to SCADA Systems Design, which

required several labs to be modified or replaced with new ones Surveys administered to the students and lab scores were used to assess the effectiveness of the equipment and the labs [1], [2] The objectives of the labs formed the basis of the survey questions The survey results are used to identify areas of improvement

Course

ENGR 382 SCADA (Supervisory Control and Data Acquisition) Systems Design is an division course taught to students in the Engineering Department as an elective or required course, depending upon the program The following are the course learning outcomes:

upper-1 Understand common Industrial Automation concepts, methods, and control

algorithms

2 Understand sensors and actuators used in Industrial Automation tasks

3 Design Piping & Instrumentation Diagrams (P&IDs) for simple process systems

4 Measure process variables in response to process parameters and analyze the resulting

process behavior

5 Understand Programmable Logic Controller (PLC) components, signal interface

methods, and applications

6 Design and write PLC control programs Recognize other control program language

• Lab 1: Automation Device Specifications

• Lab 2: CLICK Programming Software Introduction

• Lab 3: Basic Ladder Logic Programming

• Lab 4: Sequential Programming and Looping

• Lab 5: Car Lot Management System

• Lab 6: Stop Lights

• Lab 7: LabVIEW

• Lab 8: Analog I/O

• Lab 9: Analog Calculator

• Lab 10: Reaction Timer

• Lab 11: Process Trainer Closed-Loop Control

• Lab 12: Process Control System Design

Seven of twelve laboratory activities use the new PLC Trainer (Lab 1, Lab 2, Lab 7, Lab 11, and Lab 12 do not) See the Appendix for the labs referenced by this paper

PLC Trainer

The PLC trainer (see Figure 1) used for this course is based around an AutomationDirect CLICK PLC, a 16-point discrete input module, and 8-point relay output module [3] There are six lighted momentary pushbuttons, two potentiometers (connected to analog inputs), and a voltmeter (connected to an analog output)

Figure 1: CLICK PLC Trainer

Methods

Blackboard is the learning management system used by our university The surveys used for this assessment were administered in Blackboard For each lab being assessed, a pre-lab survey is administered to the students Completing the survey makes the lab available using adaptive release in Blackboard Submitting the lab assignment to Blackboard makes the post-lab survey available The surveys consist of Likert Scale questions about the objectives of the lab and a place for students to make comments about the lab Note that for a given lab, the pre- and post-lab survey questions are the same

Since the surveys are Blackboard surveys, this means that they are anonymous Thus, the pre- and post-lab survey cannot be paired Since the data are non-parametric (see Shapiro-Wilk Normality Test results in the appendix) and that the pre- and post-lab data cannot be paired, the Wilcoxon Ranked Sum Test is used to test differences between the pre- and post-lab data to a 95% confidence level [4], [5] Box plots for each lab are prepared

R is used for the Schapiro-Wilk Normality Test and for the Wilcoxon Ranked Sum Test

Microsoft® Excel® is used to prepare the box plots

The Likert Scale results are mapped to numeric scores per Table 1

Table 1: Likert Scale Result to Score Mapping

The hypothesis is that the post-lab survey values are greater than the pre-lab survey values

Results and Discussion

For each lab, the survey questions are presented, box plots of the data by question, box plots for the combined question data, a table of the Wilcoxon Ranked Sum Test results, and the comments entered by students Note that comments were not asked for until Lab 5

A discussion of results follows each lab’s results

Following the survey data is direct assessment data in the form of scores on the labs For a direct assessment item, success is defined as the average student performance being greater than 70% The labs assessed indirectly and directly are Lab 3 to Lab 6 and Lab 8 to Lab 10 The other labs

in the course did not involve the new PLC trainer

Lab 3

Questions:

Q1: I feel confident in using the CLICK Programming Software to create simple ladder logic programs

Q2: I understand how logic gates as implemented in ladder logic work

Q3: I am able to translate a Boolean expression into a ladder logic program

Q4: I understand what a seal in and a seal breaker are

Figure 2 shows the box plots of pre- and post-lab survey questions and Figure 3 shows the box plots of the combined pre- and post-lab data

Figure 2: Lab 3 Box Plots for Q1 to Q4 Pre- and Post-Lab Survey Results

Figure 3: Lab 3 Box Plots for Combined Pre- and Post-Lab Survey Results

Table 2 lists the Wilcoxon Ranked Sum Test p-values

Table 2: Lab 3 Wilcoxon Ranked Sum Test p-values

Comments were not asked for in either the Pre-Lab or Post-Lab surveys

While the p-values indicate that it is likely that post-lab scores are higher than the pre-lab ones, box plots for question 4 (I understand what a seal in and a seal breaker are) indicate that the pre- and post-lab survey scores are not statistically different

Q1: I understand how to initialize variables in a ladder logic program

Q2: I understand how to use an on-delay timer in a ladder logic program

Q3: I understand how to create a sequential ladder logic program to automate a set of steps Q4: I understand how to create a sequential ladder logic program that loops

Figure 4 shows the box plots of pre- and post-lab survey questions and Figure 5 shows the box plots of the combined pre- and post-lab data

Figure 4: Lab 4 Box Plots for Q1 to Q4 Pre- and Post-Lab Survey Results

Figure 5: Lab 4 Box Plots for Combined Pre- and Post-Lab Survey Results

Table 3 lists the Wilcoxon Ranked Sum Test p-values

Table 3: Lab 4 Wilcoxon Ranked Sum Test p-values

Comments were not asked for in either the Pre-Lab or Post-Lab surveys

Both the p-values and the box plots indicate that it is likely that post-lab scores are higher than the pre-lab ones

Q1: I understand how to use counters in a ladder logic program

Q2: I understand how to use n copy instructions in a ladder logic program

Q3: I understand how to use basic math instructions in a ladder logic program

Q4: I understand how to use comparison instructions in a ladder logic program

Figure 6 shows the box plots of pre- and post-lab survey questions and Figure 7 shows the box plots of the combined pre- and post-lab data

Figure 6: Lab 5 Box Plots for Q1 to Q4 Pre- and Post-Lab Survey Results

Figure 7: Lab 5 Box Plots for Combined Pre- and Post-Lab Survey Results

Table 4 lists the Wilcoxon Ranked Sum Test p-values

Table 4: Lab 5 Wilcoxon Ranked Sum Test p-values

Pre-Lab Survey Comments:

• Satisfied with the lab process

Post-Lab Survey Comments:

• I can build the system and apply these techniques, but I'm not sure they will 100% work the way intended

• Challenging Lab but helpful

Both the p-values and the box plots indicate that it is likely that post-lab scores are higher than the pre-lab ones

Q1: I understand how to design a ladder logic program from a set of requirements

Q2: I understand how to create a ladder logic program using advanced sequential programming techniques

Figure 8 shows the box plots of pre- and post-lab survey questions and Figure 9 shows the box plots of the combined pre- and post-lab data

Figure 8: Lab 6 Box Plots for Q1 to Q2 Pre- and Post-Lab Survey Results

Figure 9: Lab 6 Box Plots for Combined Pre- and Post-Lab Survey Results

Table 5 lists the Wilcoxon Ranked Sum Test p-values

Combined 0064

Table 5: Lab 6 Wilcoxon Ranked Sum Test p-values

Pre-Lab Survey Comments:

• Have read the Advanced Sequential Programming and gained a better understanding Post-Lab Survey Comments:

• Good Lab

• I enjoy trying to figure these labs out I usually try nontraditional approaches since all of this is new to me, but once I get a push in the right direction it usually makes sense While the p-values indicate that it is likely that post-lab scores are higher than the pre-lab ones, box plots for question 1 (I understand how to design a ladder logic program from a set of requirements) indicate that the pre- and post-lab survey scores are not statistically different

Q1: I understand how to set up analog inputs and outputs on a PLC

Q2: I understand how to use analog inputs and outputs in a ladder logic program

Q3: I understand how to use floating-point values in a ladder logic program

Q4: I understand how to translate requirements into a ladder logic program

Figure 10 shows the box plots of pre- and post-lab survey questions and Figure 11 shows the box plots of the combined pre- and post-lab data

Figure 10: Lab 8 Box Plots for Q1 to Q4 Pre- and Post-Lab Survey Results

Figure 11: Lab 8 Box Plots for Combined Pre- and Post-Lab Survey Results

Table 6 lists the Wilcoxon Ranked Sum Test p-values

Table 6: Lab 8 Wilcoxon Ranked Sum Test p-values

Pre-Lab Survey Comments:

• Expecting to improve these skills during lab

Post-Lab Survey Comments:

• This lab has increased my knowledge and experience in PLC ladder logic programming Both the p-values and the box plots indicate that it is likely that post-lab scores are higher than the pre-lab ones

Q1: I understand how to use subroutines in a ladder logic program

Figure 12 shows the box plots of pre- and post-lab survey question Since this lab only has one survey question, there are no combined data box plots

Figure 12: Lab 9 Box Plots for Q1 Pre- and Post-Lab Survey Results

Table 7 lists the Wilcoxon Ranked Sum Test p-value

Table 7: Lab 9 Wilcoxon Ranked Sum Test p-value

Pre-Lab Survey Comments:

• Expecting to expand experience on using subroutines in a ladder logic program

Post-Lab Survey Comments:

• I enjoyed this lab I'm not sure what about it made me enjoy it more than others, but it was a fun lab and I didn't ever feel lost like I have with a few of the other labs

• I have a much better understanding of the use of subroutines and their advantages Both the p-value and the box plots indicate that it is likely that post-lab scores are higher than the pre-lab ones

Lab 10

Questions

Q1: I understand how to use edge contacts in a ladder logic program

Q2: I understand how to generate pseudo-random numbers

Figure 13 shows the box plots of pre- and post-lab survey questions and Figure 14 shows the box plots of the combined pre- and post-lab data

Figure 13: Lab 10 Box Plots for Q1 to Q2 Pre- and Post-Lab Survey Results

Combined < 001

Table 8: Lab 10 Wilcoxon Ranked Sum Test p-values

Pre-Lab Survey Comments:

• I expect to learn more about edge contacts and pseudo random numbers

Post-Lab Survey Comments:

• Had a lot of trouble with this lab, not sure how to do the MOD or input a rising edge contact Also had trouble with the internal time clock and using that within the logic program

• Good lab

• I'm slowly getting better at this! I talked to my advisor Dr McDonald about the

possibility of a SCADA 2 class, as these topics really interest me

Both the p-values and the box plots indicate that it is likely that post-lab scores are higher than the pre-lab ones

Direct Assessment – Lab Scores

Box plots of lab scores are shown in Figure 15 and score averages by lab are show in Table 9 The number of points possible for each lab is fifteen and n = 16 for each lab

Figure 15: Box Plots of Lab Scores

Lab 3 4 5 6 8 9 10

Average 94.9% 85.1% 93.3% 97.5% 95.2% 96.1% 93.1%

Table 9: Averages of Lab Scores

Since the average on every lab is greater than 70%, the success criterion is satisfied for each lab and this data does not suggest the need for any improvements in the labs

Conclusions

Based on the p-values, box plots, and lab score averages, most of the labs do not appear to

require changes to improve effectiveness

Lab 3

The survey box plots for Lab 3 would indicate that improvements need to be made The material used to teach the concepts of “seal in” and “seal breaker” need to be improved by adding a step-by-step example of how the seal in and seal breaker work

From the comments, one student had trouble with the modulus operator and edge contacts in Lab

10 The description of the modulus operator in the lab document needs an example added and the lecture material on edge contacts needs an illustration to show signal behavior over time

Surveys

As far as methods go, the surveys need to be changed away from anonymous ones so that pre- and post-lab survey data may be paired This would also allow pairing with direct assessment data

Acknowledgements

I thank Dr Tom McDonald for providing invaluable advice on statistical analysis for this paper

https://peer.asee.org/35060 [Accessed: 10 Oct 2020]

[2] B Smyser, “Voice of the Students: Continuous Lab Course Improvement Using Student

Feedback,” in 2018 ASEE Annual Conference & Exposition, June 23, 2018 [Online] Available:

[3] “CLICK Series Programmable Controllers,” AutomationDirect [Online] Available:

https://www.automationdirect.com/adc/overview/catalog/programmable_controllers/click_series

[4] “Unpaired Two-Samples Wilcoxon Test in R,” STHDA [Online] Available:

2021]

[5] D Levine, D Stephan, and K Szabat, Statistics for Managers Using Microft® Excel, 7 th Ed

Upper Saddle River, NJ: Pearson, 2014, pp 448-452

Appendix

The Appendix contains a summary of survey response numbers, the test results for the Wilk Normality tests of the survey data, and the labs referenced by this paper in their entirety

Shapiro-Summary of Survey Response Numbers

Table 10 provides a summary of the number of responses each question on each survey received There were 16 students in the class

Lab Survey Q1 Q2 Q3 Q4 Combined