A study of energy monitoring and road adaptive vibration reduction system for electric bicycles

It is not just simply showing the data but through the integration of immersive sensations, which are perceived as natural parts The purpose of this dissertation is to help students enha

Instilling Computational Thinking through making Augmented Reality application

by The Vinh Nguyen

A Dissertation

In Computer Science Submitted to the Graduate Faculty

of Texas Tech University in Partial Fulfillment of the Requirements for the Degree of DOCTOR OF PHILOSOPHY

Approved

Dr Tommy Dang Chair of Committee

Dr Kwanghee Jung Co-chair of Committee

Dr Yuanlin Zhang

Dr Susan A Mengel

Dr Akbar Siami-Namin

Dr Yo Woon Chong Graduate Dean’s representative

Mark Sheridan Dean of the Graduate School

October, 2020

Copyright 2020, The Vinh Nguyen

ACKNOWLEDGMENTS

The purpose of this page is to recognize scholarly and professional aid and advice; however, the inclusion of references to persons who provided clerical help, help with field studies, financial assistance, and permission to use copyrighted

materials is also acceptable

Acknowledgments should be brief, in a professional style, and should not exceed two pages

TABLE OF CONTENTS ACKNOWLEDGMENTS II ABSTRACT VI LIST OF TABLES VII LIST OF FIGURES VIII

1 INTRODUCTION 1

1.1 Motivation and Issues 1

1.2 Research Contributions 5

2 BACKGROUND 8

2.1 Framework for presenting complex models 8

2.2 Potential capability of generating web-based AR/VR applications in the classroom setting 10

2.3 Alternative ways of producing Web-based AR/VR applications through a 3-part use case study 12

2.4 Instilling Computational Thinking through using a Visual Programming Interface 13

3 FRAMEWORK FOR PRESENTING COMPLEX MODELS 15

3.1 Contributions 15

3.2 Introduction 17

3.3 Related work 19

3.4 Design 21

3.4.1 Material contents 22

3.4.2 Vuforia package for Unity3D 24

3.4.3 Google Cardboard Package 25

3.4.4 Unity3D 26

3.4.5 Application 26

3.5 Evaluation 28

3.6 Challenge and Discussion 29

3.7 Conclusion 30

4 POTENTIAL CAPABILITY OF GENERATING WEB-BASED AR/VR

APPLICATIONS IN THE CLASSROOM SETTING FROM LEARNERS’

PERSPECTIVES 31

4.1 Contributions 32

4.2 Introduction 33

4.3 Related work 36

4.4 Methodology 37

4.4.1 Goal and objectives 38

4.4.2 Study design 38

4.4.3 Project assessment 40

4.4.4 Survey 41

4.4.5 Case Study 42

4.5 Results 48

4.6 Lessons learned and challenges 54

4.7 Recommendations 56

4.8 Conclusion 57

5 ALTERNATIVE WAYS OF PRODUCING WEB-BASED AR/VR APPLICATIONS THROUGH A 3-PART USE CASE STUDY 58

5.1 Contributions 59

5.2 Introduction 61

5.3 Related work 65

5.4 Methodology 66

5.4.1 Goal and objectives 66

5.4.2 Study design 67

5.4.3 Participants 70

5.5 Use Case Study 71

5.5.1 Use Case 1: Developing a WebVR application 71

5.5.2 Use Case 2: Developing a VR/AR application on a given topic 75

5.5.3 Use Case 3: Developing a VR/AR application on any topic 78

5.6 Results and Discussion 79

5.6.1 Research question 1: What VR/AR development framework/library best-allowed users (novice to expert) to stimulate their interest in creating and sharing VR/AR content in both perceived utility and ease of use? 79

5.6.2 Research question 2: Did the library/framework that students favored afford them the ability to solve more complex problems? 83

5.6.3 Research question 3: When given the choice, which library/framework did students employ to develop a VR/AR application (based upon their

interests)? 84

5.6.4 Research question 4: Based upon reported learners' perspectives, what were the pros and cons of WebVR compared to other app-based VR/AR tools? 84

5.6.5 Lessons learned and discussion 88

5.7 Conclusion and Future work 91

6 INSTILLING COMPUTATIONAL THINKING THROUGH MAKING AUGMENTED REALITY APPLICATION 93

6.1 Contributions 94

6.2 Introduction 95

6.3 Related work 99

6.4 Methods 101

6.4.1 System Design 101

6.4.2 Use Case 114

6.4.3 Evaluation 116

6.5 Results 121

6.5.1 Qualitative Analysis 121

6.5.2 Quantitative Analysis 122

6.6 Discussion 125

6.7 Conclusion 127

7 CONCLUSION 128

REFERENCES 130

ABSTRACT

It is widely recognized that instilling and inculcating computational thinking skills (CTS) such as problem formulation, effective representation of big data, and identifying, analyzing and implementing possible solutions are essential for succeeding

in STEM disciplines There is also a recognition that technology and human behavior are tightly interrelated and leveraging computational thinking to understand complex human-computer interactions is vital to foster systemic sustainable developments Augmented Reality is a technology that expands the physical world with additional digital information The central value of AR is that the components of the digital world blend into a person's perception of the real world It is not just simply showing the data but through the integration of immersive sensations, which are perceived as natural parts

The purpose of this dissertation is to help students enhance computational thinking skills for a successful future career through making an Augmented Reality application To tackle the aforementioned issues, we provide students with an interactive web-based tool that allows them for experimenting, testing, abstracting, modularizing, reusing, and remixing the application ideas

LIST OF TABLES

4.1 Pearson correlation test scores produced by SPSS software 54

5.1 Participant distribution by gender vs graduate level 71

5.2 Research questions for the survey 73

5.3 Survey questions for peer evaluation in Project Case 2 75

5.4 Pearson Correlation test scores produced by SPSS 82

5.5 A summary of the VR/AR application types and hardware 83

6.1 Construct and items 119

6.2 General information about the participants 120

6.3 Means and standard deviations of TAM measures (N = 66) 122

6.4 Internal Consistency and Convergence Validity 123

6.5 Estimates of loadings 124

6.6 Estimates of path coefficients 125

LIST OF FIGURES

Figure 3.1 A comprehensive framework to build VR/AR application 21

Figure 3.2 Example of some free3D models that will be used in the application 22

Figure 3.3 Heightmap generator from terrain.party 23

Figure 3.4 Terrain generated from heightmap in Unity 24

Figure 3.5 Birch tree 3D model 25

Figure 3.6 Example of using tree in the inventory to plant and build the city 25

Figure 3.7 Main menu of the VR/AR game that allows to switch between VR/AR mode 27

Figure 4.1 A collage of WebVR applications created by sampled students 44

Figure 4.2 Sampled Students' WebVR project examples: (a) A moon dream house (b) A 2-level dream house (c) A New York skyline dream condo 47

Figure 4.3 Students' grade distribution of the WebVR dream house project: Level 1 is equivalent to a C while Level 3 is equivalent to a A 48

Figure 4.4 Responses from students (regarding questions 1-3) 49

Figure 4.5 Responses from students (regarding questions 4-6) 50

Figure 4.6 Responses from students (regarding questions 7-8) 52

Figure 4.7 Responses from students (regarding questions 9-10) 53

Figure 5.1 The study design: 16-week activities 68

Figure 5.2 A collage of WebVR applications created by sampled students 74

Figure 5.3 Three good WebVR project examples: (a) A moon dream house (b) A relaxing dream house (c) A bar dream house 74

Figure 5.4 Students' grade distribution of the WebVR dream house project: Level 1 is equivalent to a C while Level 3 is equivalent to a A 75

Figure 5.5 A collage of VR/AR applications created by students 77

Figure 5.6 A collage of VR/AR applications created by students in Project 3 78

Figure 5.7 Survey results from students in Project 1 from question 1 to

10 80Figure 6.1 The visual interface of Blockly for generating JavaScript

from blocks 100Figure 6.2 The coding editor of BlocklyAR: it enables users to drag and

drop a palette of commands into the working space 103Figure 6.3 Different shapes of blocks allow users to stack or wire up

components together 105Figure 6.4 The visual AR component enables enthusiasts to experience

their coding schemes in the mixed 3D space 113Figure 6.5 Tutorial section where learners are guided on how to use

blocks and the connections among them 114Figure 6.6 Use case of using BlocklyAR to recreate the Palmito Battle

Ranch 116Figure 6.7 The conceptual research model with extensions of Task-

Technology Fit and Visual Design variables Each set of ellipses represents a construct and an arrow denotes a hypothesis 118

CHAPTER 1

1 INTRODUCTION

1.1 Motivation and Issues

It is widely recognized that instilling and inculcating computational thinking skills (CTS) such as problem formulation, effective representation of big data, and identifying, analyzing and implementing possible solutions (Sykora, 2014; Wing, 2006) are essential for succeeding in STEM disciplines (Assaraf & Orion, 2010; Kokkelenberg

& Sinha, 2010) Evaluation and understanding of highly complex systems with extensive inter-connection and feedback has become an integral focus of many STEM fields as the world we live in is increasingly becoming dynamic, self-organizing and continually adaptive (Assaraf & Orion, 2010; Bower et al., 2017) A strong grounding

in CTS is an essential requirement to study these complex adaptive systems There is also a recognition that technology and human behavior are tightly interrelated and leveraging computational thinking to understand complex human-computer interactions

is vital to foster systemic sustainable developments

Computer programming is not only writing a code but the process of analyzing

a situation, identifying its key components, modeling the data and processes, and creating or refining a program through an agile design-thinking approach to accomplish

a specific computing result These strategies could be considered under the umbrella of the CT concept (Wing, 2006), thus having and understanding creative programming skills would improve computational thinking Basic concepts in computer programming include sequence (identifying an ordered series of steps), loop (running the same set of sequences multiple times), parallelism (running multiple sequences at the same time), events (one thing causing another thing to happen), conditional (making decisions based

on criteria), operator (mathematical and logical expressions), data (storing, retrieving, and updating values)

Human Computer Interaction (HCI) focuses on the interaction between human and computer and it has been applied in my fields such as user customization, embedded computation, augmented reality, social computing and knowledge and knowledge driven (Biseria & Rao, 2016) Augmented Reality (AR) is a technology that expands the physical world with additional digital information such as sound, image, model, etc (R T Azuma, 1997) The central value of AR is that the components of the digital world blend into a person's perception of the real world It is not just simply showing the data but through the integration of immersive sensations, which are perceived as natural parts

of an environment In recent years, the growth of AR applications can be attributed to solutions that focus on contextualizing information (e.g., annotating different parts of a physical object (Bruno et al., 2019), displaying artifacts at a given place (K Jung et al., 2020), aligning virtual objects with the real world (Norouzi et al., 2019) that is, automatically position an object on the detected table or floor) In the educational setting, AR technology can be incorporated in the classroom to enhance teaching/learning efficiency and the motivations of both educators and learners (R Azuma et al., 2001; V T Nguyen, Hite, & Dang, 2018)

Traditional approaches involving making an AR application are heavily dependent on a programming language in which the syntax of programming is not easy

to master for non-computer science users For example, developers rely on available tools/frameworks such as ARCore, ARKit, Vuforia, Unity (Linowes & Babilinski, 2017), and ARToolkit (Kato, 2002) to create and manipulate AR applications ARCore

is a framework from Google for building augmented reality applications for both Android and iOS devices Apple also provides ARKit for making AR apps and games for their iOS devices Both of these frameworks use a handheld device's sensors for motion tracking, light estimation, and environmental understanding Unlike ARCore and ARKit, ARToolkit and Vuforia are computer tracking libraries that overlay virtual objects on the real world based on markers The above frameworks and libraries can be integrated in Unity for porting an AR application on different operating system devices (e.g., Android, iOS) To use these frameworks, it is necessary to have an integrated

development environment (e.g., XCode for iOS and Android Studio for Android, or Unity for both iOS and Android), a compatible device (e.g., ARCore requires that the device must be running Android at least 7.0, or ARKit requires device to be run on iOS

11 with an A9, A10, or A11 processor), and knowledge of a specific programming language (e.g., Objective-C, C#, C/C++, Java) These requirements are obstacles for beginners who want to create an AR experience on their own In their study, Nguyen et

al (Vinh T Nguyen, Jung, & Dang, 2019) showed that API version incompatibility in the integrated development environment (IDE) is a major obstacle students face while working with the application in terms of both coding and deploying The study also indicated that students faced difficulty in analyzing “800+ line scripts”

To alleviate difficulty of having an IDE and a compatible device, web-based AR (e.g., WebVR, WebXR) is an alternative approach for users to experience virtual objects

in the real world only by using a web browser on their handheld device A web browser engine supports Hypertext Markup Language (HTML), Cascading Style Sheets (CSS), and scripting languages such as JavaScript that can be programmed with a simple text editor, thus releasing users from the need for an IDE Furthermore, web-based AR toolkits such as ARToolkit for web (Kato, 2002) are compatible with a wider range of devices (e.g., running on OS 4.0.3 or higher for Android and 7.0 or higher for iOS) The use of ARToolkit has been widely adapted in other libraries such as ThreeJS (Danchilla, 2012), and A-Frame.io (Mozilla, 2019) Currently, although web-based AR has some limitations due to its low framerate and not leveraging the full capacity of in-app AR, the continual development of other technologies will help web-based AR keep growing

in the future For example, the presence of an open binary instruction format WebAssembly (WebAssembly, 2020) would allow the browser engine to run native code in a browser and this provides the capability to access the in-app AR features through a web-based VR

Recent research has produced some insights that describe how to lessen the issue

of mastering a certain programming language for young learners and enthusiasts based programming (Weintrop, 2019) is a type of programming language where

Block-instructions are mainly represented as blocks (or visual cues) and users drag and drop the cues to form a set of instructions This programming paradigm enables developers

to focus on logical programming rather than memorizing the syntax of coding Scratch (Resnick et al., 2009) is an example of using this visual paradigm in K-12 education

By using Scratch, K-12 students are able to program a 2D game and experience it immediately on a web-browser Radu and Blair (Radu & MacIntyre, 2009) extended Scratch to make it possible to use to create AR However, their work stopped at rendering 2D images on the screen only; 3D objects and spatial position manipulation have not been implemented CoSpaces (CoSpaces, 2020) is a commercial product created for education; it enables students to create virtual reality (VR) applications through context-based language It also can be used to create a simple AR app by superimposing 3D objects onto the physical environment However, the interactions between 3D objects are limited In addition, Laine (Laine, 2018) pointed out that the majority of AR apps were developed through Vuforia SDK with a few occurrences of ARToolkit, which would not be suitable for non-programmers, but expert programmers Furthermore, Wu et al.(Wu, Lee, Chang, & Liang, 2013) indicated that in some AR systems, the learning content and the teaching sequence are rather ''fixed'' such that teachers are not able to make changes to accommodate students' learning needs As such, an authoring/storytelling tool is highly needed for teachers and students to create

AR applications (Klopfer & Squire, 2008) In addition, learning programming with activities such as generating prime numbers or sorting a list/array is not of interest for many young learners And finally, there is a little guidance when things go wrong or encourage deeper explorations when things went right in some context of programming These limitations create a barrier to fostering computational thinking in STEM education

The purpose of this dissertation is to help students enhance computational thinking skills for a successful future career through making an Augmented Reality application To tackle the aforementioned issues, we provide students with an interactive web-based tool that allows them for experimenting, testing, abstracting,

modularizing, reusing, and remixing the application ideas To meet this goal, the following objectives was carried out during the implementation of the dissertation:

 We propose a framework and a setup for presenting complex models for curriculum contents in both augmented reality and virtual reality environment

 We study the potential capability of generating web-based AR/VR applications in the classroom setting from learners’ perspectives

 We investigate other alternative ways of producing AR/VR applications through a 3-case study

 We provide an interactive web-based tool that allows students to generate an

AR application toward fostering computational thinking skills

1.2 Research Contributions

The contributions in this dissertation for each of the objectives are as follow:

Framework for presenting complex models: We propose a framework and a

setup for presenting complex models for curriculum contents in both augmented reality and virtual reality environment After constructing some three-dimensional models representing real world objects such as trees, stones, rivers, dams, and buildings, our workflow uses the Unity engine in combination with Virtual Reality headset devices to create interactive applications for both Virtual Reality and Augmented Reality environments to support students understanding the curriculum contents through their surroundings Typical challenges are addressed when creating 3D curriculum contents, integrating these models into Unity and solutions are proposed where possible

Potential capability of generating web-based AR/VR applications in the classroom setting from learners’ perspectives: Web-based VR (WebVR) has

emerged as a platform-independent framework that permits individuals (with little to no prior programming experience) to create immersive and interactive VR applications Yet, the success of WebVR relies on students' technological acceptance, the

intersectionality of a user's perceived utility of and ease of use with said technology In order to determine the effectiveness of the emerging WebVR tool for computer science students of varied experience levels, this work presents a case study of 38 students tasked with developing their dream house using WebVR Results showed that students exhibited technological acceptance by not only learning and implementing WebVR in

a short time (one month), but were also capable of demonstrating creativity and problem-solving skills with classroom supports (i.e., pre-project presentations, online discussions, exemplary projects, and TA support) Results as well as recommendations, lessons learned, and further research are addressed

Alternative ways of producing Web-based AR/VR applications through a 3-part case study: To ascertain the viability of Web AR/VR tools for software

engineering undergraduates in the classroom, this paper presents a 3-case contextual investigation of 38 undergraduate students tasked with creating AR/VR content In each use case, students were provided increasing choices in their AR/VR content development parameters Results indicated that students demonstrated elements of technological acceptance in their selection of Web AR/VR and other platforms, and successfully creating rich and robust AR/VR content (PU), but also executing these projects in a short period of time (PEOU) Other positive externalities observed were students exhibitions of soft skills (e.g., creativity, critical thinking) and different modes

of demonstrating coding knowledge, which suggest further study Discussed are the lessons learned from the WebVR and AR/VR interventions and recommendations for Web AR/VR instruction This work may be helpful for both learners and teachers using AR/VR in selecting, designing, and developing coursework materials, tools, and libraries

Instilling Computational Thinking through making Augmented Reality application: State of the art tools for creating augmented reality (AR) applications often

depend on a specific programming language and the deployed target devices Typing syntax of a program is error-prone, and device dependency makes it difficult to share newly created AR applications This work presents BlocklyAR, a novel web-based

visual programming interface for creating and generating an AR application This tool

is intended for non-programmers (young learners and enthusiasts) who are interested in making an AR application The goals of this tool are: 1) to help young learners and enthusiasts express their programming ideas without memorizing syntax, 2) to enable users to perceive their expressions, 3) to enable learners to generate an AR application with minimal effort, and 4) to support users by allowing them to share newly created

AR applications with others BlocklyAR uses Blockly for creating a palette of commands and AR.js for transcribing commands into AR experience The applicability

of BlocklyAR was demonstrated through a use case where an existing AR application was recreated by using our tool The result showed that our tool could yield an equivalent product We evaluated the visual tool with the help of 66 users to gather perspectives on the specific benefits of employing BlocklyAR in producing an AR application The Technology Acceptance Model was adapted to assess an individual’s acceptance of information technology

The rest of the dissertation is organized as follows: Chapter 2 discusses the background of this research, including state-of-the-art AR/VR approaches in STEM education Chapters 3, 4, 5, and 6 describe how each objective was carried out in detail Finally, chapter 7 concludes the dissertation with critical discussion and future research directions

CHAPTER 2

2 BACKGROUND

This chapter provides an overview of the existing studies for integrating AR/VR in STEM education; acquiring 3D models and presenting them in virtual spaces; the feasibility of producing web-based AR/VR applications in the classroom setting from learners’ perspective; alternative ways of generating AR/VR applications; and visual programming interface to foster computational thinking skills

2.1 Framework for presenting complex models

Serafin et al (Serafin, Adjorlu, Nilsson, Thomsen, & Nordahl, 2017) provide a number

of approaches in integrating virtual reality (VR) and augmented reality (AR) in musical education from an academic and a commercial perspective The study showed that the quality of learning and the retention rate of students in the classroom had improved significantly Several potential applications taking into account of are suggested to help students gain better performance and feel more confident in public spaces However, this paper only points out some available approaches and promising research directions without any further guiding on how to build a real application

Miyata et al (Miyata, Umemoto, & Higuchi, 2010), on the other hand, create an educational learning environment for developing VR applications Teamwork and collaboration are emphasized through competition for creating different VR applications Similar to this approach, Dinis et al (Dinis, Guimarães, Carvalho, & Martins, 2017), provide a more a practical approach for engaging students in the learning process During a 6-week introductory class project, first year students of the master’s in civil engineering will create virtual environments in which the models are created in CAD drawings and imported into Unity3D The output results then will be viewed in Oculus Rift headset device Augmented reality is also implemented for mobile device to recognize the image of an object This is very interesting approach since it helps students engage in the whole process, from creating a model, incorporating the model into Unity3D, and exporting results to VR/AR headset devices However, this

useful report does not outline the overall framework of how to build the application from technical perspective No-Savvy technology readers in the same field find it difficult to reconstruct the learning process

Chen et al (Chen et al., 2011) provide an AR-based system to increase the spatial awareness and interest of learning for Engineering Graphics Education 3D virtual objects are overlaid on 2D drawing to help students quickly access to 3D solid structure and spatial detail information This application greatly reduces time spent by the instructors and students in the classroom Messner et al (Messner, Yerrapathruni, Baratta, & Whisker, 2003) provide an expensive approach by using the CAVE-like projection system for undergraduate Architectural Engineering students to create VR interfaces The application has a great influence on the student performance since students are able to develop a plan of the construction of a nuclear power plant within a short time (1 hour) with very little experience concerning buildings and infrastructures

Regarding to the geosciences education, Jiayan Zhao et al (Zhao, LaFemina, Wallgrün, Oprean, & Klippel, 2017) create an immerse environment platform for students by using LiDAR (Light Detection and Ranging) technology and images to reconstruct the Iceland's Thrihnukar volcano Data imported from OpenTopography.org portal then transcribed into Unity3D The simulation result is rendered and view in HTC Vive headset device The 3D model of the volcano is created in the Agisoft PhotoScan Pro, a rapid 3D modeling software that intuitively stitches together photos to form 3D geometry This study described very detailed necessary steps to get data, construct models and put them into virtual reality environment In addition, this work leverages existing advanced technology to enhance student learning process without going to dangerous places such as inside the active volcano

A very similar approach to our study is the work proposed by Parmar et al (Parmar et al., 2016) Based on the similarities of dance with programming, Parmar creates a Virtual Environment Interactions (VEnvI) application that allows students to learn complicated concepts of programming such as sequences, loops, conditionals, variables, functions and even parallelization The study results showed that students are

motivated by learning activities and remembered what they saw in the VR context rather than laboratory-based demonstrations

Obtaining free 3D models and then customize them to use in VR/AR is also investigated by Voinea et al (Voinea, Moldoveanu, & Moldoveanu, 2017) The aim of this study is to animate the avatar of the musculoskeletal system 3D model is obtained from medical datasets with the help of the supporting tool (Simpleware Scan IP), then skeletons are added to the model to simulate animation

There are still more available VR/AR applications in the literature targeting educational purposes; however, most of these approaches focus on programmed scripts

or a set of static data Understanding material contents through real life surrounding environment and playing with contents are the main concerns that should be taking into account In addition, a comprehensive framework for guiding learners and educators to set up similar environment is highlighted in this paper

2.2 Potential capability of generating web-based AR/VR applications in the classroom setting

The VR concept was introduced in the early nineteenth century when a French general

painter Louis-Francois Lejeune created a panoramic painting of The Battle of Borodino

(Duffy, 1972), whose purpose was to extend the vision of the entire battlefield for the viewer Since then, technologically enhanced hardware and software have been developed for VR-based entertainment, training, and learning This paper does not intend to survey all existing VR work, rather focusing solely on educational approaches

In education, Nguyen and Dang (V T Nguyen & Dang, 2017) have presented a model as a structural guideline for educators building applications using VR and AR This model suggested the use of both technologies (i.e., VR and AR) to engage students

in the programming process (using Unity3D), to create vivid, realistic environments The authors suggested using the Unity3D engine to create the application then exports

to smartphone devices; so, VR content can be created manually or obtained from free online providers This approach is novel in significantly reducing the amount of time

needed to create 3D models, yet still required a heavy demand for users' programming skills

Another education-focused approach is the study of Miyata et al (Miyata et al., 2010) where students collaboratively (via group-work) designed a VR application In engaging students in creating VR application cooperatively, students reported feeling creative and knowledgeable, developing as learners both individually and as a group, throughout the shared activity However, it is difficult to replicate this model, provided the variability in time on task, students' efforts, the complexity of VR content, and various supporting devices Taking a further step, Hafner et al (Häfner, Häfner, & Ovtcharova, 2013) engaged students in the creation of a 3-year industrial VR project, similarly resulting that task specification and group formation played the central role for the success of the VR product Takala et al (Takala, Malmi, Pugliese, & Takala, 2016) conducted a similar, yet more in-depth study, where they shared their experience

of teaching VR development courses over a span of five years Several efforts have been made to bring VR into classroom, from concepts (Bell, 1996), content (Zara, 2006), (Cliburn, Miller, & Doherty, 2010), and hands-on learning experiences (Kamińska et al., 2017),(Miyata et al., 2010) The challenges and difficulties found in most studies are the availability of HMDs, a standard hardware platform for VR, as well as the multifaceted needs for programming knowledge and specialized software tools in VR development Many reasons cause these difficulties, like those reviewed here, due to the cost of the VR headset for testing; the compatibility of the application on the target devices, and the professional skills required for creating 3D, VR content

This work attempts to elaborate on these existing problems when helping students develop VR content in the classroom Specifically, this work provides an insight into these issues with a novel pedagogical approach which will be presented in Section 4.4.2

2.3 Alternative ways of producing Web-based AR/VR applications

through a 3-part use case study

The concept of Virtual Reality or VR is not new; it first was introduced in the early nineteenth century by a French general and painter Louis-Francois Lejeune (Duffy, 1972) Since this time, and only very recently, has the idea of VR become a (virtual) reality VR has been hampered by the absence of VR-ready technologies (e.g adequate hardware and software) and early equipment costs This paper does acknowledge the interesting and rich history of VR development yet does not intend to review all current

VR research, rather concentrating on instructive methodologies using VR technologies for content development, specifically

In education, several efforts have been made to bring VR into classroom, from understanding complex concepts (Bell, 1996), teaching content (Zara, 2006), and providing hands-on learning experiences (Kamińska et al., 2017; Miyata et al., 2010) Specifically, VR content development has been integrated into a computer graphics course (Cliburn et al., 2010), focusing on students using existing libraries and frameworks (V T Nguyen & Dang, 2017) To explore more deeply the affordances of these experiences, Miyata et al (Miyata et al., 2010) examined skill development and group work performance when students created VR applications cooperatively Throughout the course, this study showed that students' creativity and knowledge gradually increased A similar study was performed by Hafner et al (Häfner et al., 2013), where students were engaged in developing a 3-year industrial VR project, finding that task specification played the vital role for students' successes in creating a final VR product A more in-depth study was conducted by Takala et al (Takala et al., 2016) where the authors shared their teaching experience of VR development courses throughout five years They emphasized that face-to-face meeting between students and teaching assistant played a vital role in the students' performance Stansfield (Stansfield, 2005) presented a study of a VR course that combined both traditional teachings (lecture) supplemented with hands-on experiences in VR development The author argued that the format of this VR course provided more and varied experiences in

communication skills, research skills, and presentation skills, the oft-described “soft skills” needed for student success beyond graduation

Yet there remain challenges and issues among the research, such as the accessibility and cost of HMDs hardware and the multifaceted programming knowledge and specialized libraries needed for VR content development Therefore, this work endeavors to mitigate these presented issues when aiding learners (as designers) in creating VR content in a classroom setting In particular, this work provides insight into this process by relating novel pedagogical strategies as assessed by their technological acceptance

2.4 Instilling Computational Thinking through using a Visual

Programming Interface

In this section, we briefly review the use of block-based visual programming, using Blockly (Inc, 2020) as an example While Blockly has been used in many different domains, we only discuss the research that is relevant to our study, particularly in making AR applications

Blockly (Inc, 2020) is a client-side library, a project of Google, for creating block-based visual programming languages and editors One interesting feature of Blockly is that it can run in a web browser Visual cues in Blockly can be linked together

to make writing code easier The central value of Blockly is the ability to generate code

in many different languages such as JavaScript, Lua, Dart, Python, or PHP A typical application that uses Blockly is Scratch (Resnick et al., 2009) This web-based visual programming language is targeted primarily at children who are between the age of 8 to

16

Radu and Blair (Radu & MacIntyre, 2009) developed a first AR Scratch tool that allows children to create programs that mix real and virtual spaces In their work, they customized the Scratch environment by adding an AR feature to the interface ARToolkitPlus library was used for detecting markers' position and orientation Once the markers were found, 2D actor sprites (or images) were overlaid on to the corresponding markers The pilot study result showed that young learners were

enthusiastic and returned to interact with the tool even after the study finished As indicated by the authors, the main drawback of this tool was not adding a 3rd dimension

to the Scratch environment due to its complexity in specifying relationships and interactions between objects

Mota et al (Mota, Ruiz-Rube, Dodero, & Arnedillo-Sánchez, 2018) built an app AR tool called VEDILS (which stands for Visual Environment for Designing Interactive Learning Scenarios) Android users can leverage this tool in order to create

in-AR applications The VEDILS's in-AR components were developed using the Vuforia for image recognition and tracking Like Scratch, VEDILS relied on the Blockly library for generating visual blocks

The idea most similar to our work in the literature was presented recently by Clarke (Clarke, 2019), in which the author extended VEDILS for working on iOS devices His tool enables users to work with 20 augmented reality primitive components, including basic shapes such as box, capsule, cone, sphere, text, 3D models, etc A tutorial for using the tool was provided to help participants get familiar with the AR components and the visual interface The pilot study result showed that participants felt empowered by working with the AR components and they could build AR applications after using the AR components As the author pointed out, API incompatibility was one

of the main issues in the study since the tool required iOS 12+ to run In addition, features such as animations and movements in the AR environment had not developed and these components were put in the future work

Although there are still a number of tools in the literature for generating AR applications, they are out of the scope of this dissertation since we are focusing on block-based programming language The aforementioned work still faced the same issues as its predecessors when deploying on a device or they lacked interactions in the AR environment Our work overcomes the limitations of existing studies by implementing the tool on the web-based environment In addition, animations of the 3D models and interactions in the AR scene are added

3 FRAMEWORK FOR PRESENTING COMPLEX MODELS

It is widely recognized that instilling and inculcating computational thinking skills such

as problem formulation, effective representation of data, and identifying, analyzing and implementing possible solutions are essential for student success in science, technology, engineering and math (STEM) subjects Current science standards advocate for K-12 teachers to use disciplinary-specific ideas to explore scientific phenomena and develop problem solving skills Game-based interventions have been recently lauded as a viable vehicle to not only engage students in STEM content but also develop computational thinking skills along with traditional teaching and learning methods Recently, with the advancement of modern technologies, virtual reality and augmented reality have emerged as a vehicle to help students achieve specific knowledge While virtual reality creates a programmed environment that simulates reality, augmented reality on the other hand integrates digital information into the real environment in which people are living

in Many of the available applications toward virtual reality learning based have centered on the hard sciences such as anatomy, biology, and astronomy with a set of pre-defined objects and programmed processes Although this approach greatly engages students in understanding the object and its phenomenon, students are mostly limited in the modelled environment without exploring in the real world Furthermore, the idea of having students create and design their own experiences is becoming more feasible as new technologies pave the way for virtual reality going its journey into the K-12 education sector Therefore, there is a need to develop an educational support system using both virtual reality and augmented reality techniques to demonstrate how computational thinking and problem solving skills learned in high school can be used

in practical engineering and environmental management applications and help students connect theoretical concepts with real world phenomenon

3.1 Contributions

 We propose a practical comprehensive structural framework for building an application using virtual reality and augmented reality techniques

 We demonstrate how each component of the framework can be integrated together in Unity game engine

 We address some typical technical challenges when integrating components and provide solutions where possible

Our own contributions in this chapter are published in:

Nguyen, V T., & Dang, T (2017, October) Setting up virtual reality and augmented reality learning environment in unity In 2017 IEEE International Symposium on Mixed and Augmented Reality (ISMAR-Adjunct) (pp 315-320) IEEE

Setting up virtual reality and augmented reality learning environment in

Unity Abstract

We propose a framework and a setup for presenting complex models for curriculum contents in both augmented reality and virtual reality environment After constructing some three-dimensional models representing real world objects such as trees, stones, rivers, dams, and buildings, our workflow uses the Unity engine in combination with Virtual Reality headset devices to create interactive applications for both Virtual Reality and Augmented Reality environments to support students understanding the curriculum contents through their surroundings Typical challenges are addressed when creating 3D curriculum contents, integrating these models into Unity and solutions are proposed where possible The overall structure of the project is described with some functionalities added to Unity for visualization and interaction with the models

3.2 Introduction

It is widely recognized that instilling and inculcating computational thinking skills such

as problem formulation, effective representation of data, and identifying, analyzing and implementing possible solutions (Sykora, 2014), (Wing, 2006) are essential for student success in science, technology, engineering and math (STEM) subjects (Assaraf & Orion, 2010), (Kokkelenberg & Sinha, 2010) There is a growing recognition that technology-human behavior and environmental impacts are tightly inter-related and leveraging computational thinking to understand complex human-environmental interactions is vital to foster systemic, sustainable solutions (Easterbrook, 2014)

Current science standards advocate for K-12 teachers to use disciplinary-specific ideas to explore scientific phenomena and develop problem solving skills (Campbell & McKenna, 2016) Game-based interventions have been recently lauded as a viable vehicle to not only engage students in STEM content but also develop computational thinking skills (Berland & Duncan, 2016), (Lee, Mauriello, Ahn, & Bederson, 2014),

(Repenning, Webb, & Ioannidou, 2010) along with traditional teaching and learning methods Recently, with the advancement of modern technologies, virtual reality and augmented reality have emerged as a vehicle to help students achieve specific knowledge While virtual reality creates a programmed environment that simulates reality, augmented reality on the other hand integrates digital information into the real environment in which people are living in Many of the available applications toward virtual reality learning based have centered on the hard sciences such as anatomy, biology, and astronomy (Reede & Bailiff, 2016) with a set of pre-defined objects and programmed processes Although this approach greatly engages students in understanding the object and its phenomenon, students are mostly limited in the modelled environment without exploring in the real world Furthermore, the idea of having students create and design their own experiences is becoming more feasible as new technologies pave the way for virtual reality going its journey into the K-12 education sector

Therefore, there is a need to develop an educational support system using both virtual reality and augmented reality techniques to demonstrate how computational thinking and problem solving skills learned in high school can be used in practical engineering and environmental management applications and help students connect theoretical concepts with real world phenomenon

In response, this paper aims to take the initial step toward building computational thinking game application for students with the following contributions:

 We propose a practical comprehensive structural framework for building an application using virtual reality and augmented reality techniques

 We demonstrate how each component of the framework can be integrated together in Unity game engine

 We address some typical technical challenges when integrating components and provide solutions where possible

The idea of this game can be divided into categories in the same application: Augmented reality game where students go out in the real environment and use mobile phone camera pointing to a real object (tree, for example) A 3D model of this tree will pop up; students click on this model to add to their inventory The collected 3D models will later on be used in Virtual Reality application where students are assigned a random land Students will use collected assets in the inventory to build the city

3.3 Related work

Serafin et al (Serafin et al., 2017) provide a number of approaches in integrating virtual reality (VR) and augmented reality (AR) in musical education from an academic and a commercial perspective The study showed that the quality of learning and the retention rate of students in the classroom had improved significantly Several potential applications taking into account of are suggested to help students gain better performance and feel more confident in public spaces However, this paper only points out some available approaches and promising research directions without any further guiding on how to build a real application

Miyata et al (Miyata et al., 2010), on the other hand, create an educational learning environment for developing VR applications Teamwork and collaboration are emphasized through competition for creating different VR applications Similar to this approach, Dinis et al (Dinis et al., 2017), provide a more a practical approach for engaging students in the learning process During a 6-week introductory class project, first year students of the master’s in civil engineering will create virtual environments

in which the models are created in CAD drawings and imported into Unity3D The output results then will be viewed in Oculus Rift headset device Augmented reality is also implemented for mobile device to recognize the image of an object This is very interesting approach since it helps students engage in the whole process, from creating

a model, incorporating the model into Unity3D, and exporting results to VR/AR headset devices However, this useful report does not outline the overall framework of how to build the application from technical perspective No-Savvy technology readers in the same field find it difficult to reconstruct the learning process

Chen et al (Chen et al., 2011) provide an AR-based system to increase the spatial awareness and interest of learning for Engineering Graphics Education 3D virtual objects are overlaid on 2D drawing to help students quickly access to 3D solid structure and spatial detail information This application greatly reduces time spent by the instructors and students in the classroom Messner et al (Messner et al., 2003) provide an expensive approach by using the CAVE-like projection system for undergraduate Architectural Engineering students to create VR interfaces The application has a great influence on the student performance since students are able to develop a plan of the construction of a nuclear power plant within a short time (1 hour) with very little experience concerning buildings and infrastructures

Regarding to the geosciences education, Jiayan Zhao et al (Zhao et al., 2017) create an immerse environment platform for students by using LiDAR (Light Detection and Ranging) technology and images to reconstruct the Iceland's Thrihnukar volcano Data imported from OpenTopography.org portal then transcribed into Unity3D The simulation result is rendered and view in HTC Vive headset device The 3D model of the volcano is created in the Agisoft PhotoScan Pro, a rapid 3D modeling software that intuitively stitches together photos to form 3D geometry This study described very detailed necessary steps to get data, construct models and put them into virtual reality environment In addition, this work leverages existing advanced technology to enhance student learning process without going to dangerous places such as inside the active volcano

A very similar approach to our study is the work proposed by Parmar et al (Parmar et al., 2016) Based on the similarities of dance with programming, Parmar creates a Virtual Environment Interactions (VEnvI) application that allows students to learn complicated concepts of programming such as sequences, loops, conditionals, variables, functions and even parallelization The study results showed that students are motivated by learning activities and remembered what they saw in the VR context rather than laboratory-based demonstrations

Obtaining free 3D models and then customize them to use in VR/AR is also investigated by Voinea et al (Voinea et al., 2017) The aim of this study is to animate the avatar of the musculoskeletal system 3D model is obtained from medical datasets with the help of the supporting tool (Simpleware Scan IP), then skeletons are added to the model to simulate animation

There are still more available VR/AR applications in the literature targeting educational purposes; however, most of these approaches focus on programmed scripts

or a set of static data Understanding material contents through real life surrounding environment and playing with contents are the main concerns that should be taking into account In addition, a comprehensive framework for guiding learners and educators to

set up similar environment is highlighted in this paper

3.4 Design

In this section, we propose a practical, comprehensive structural framework for building

an application using virtual reality and augmented reality techniques and describe how each component in the framework is incorporated into Unity3D As depicted in Figure 3.1 our framework includes five main components namely: Material contents, Vuforia package for Unity3D, Google Cardboard package for Unity3D, Unity3D game engine and Application (VR/AR)

Figure 3.1 A comprehensive framework to build VR/AR application

3.4.1 Material contents

In our application, material contents are the most important factors to engage and motivate students in the learning process Creating contents for a classroom is very time consuming so we come up with an idea of getting free available contents without the need to reinvent the wheel For example, rivers, trees, animal are those models we import from the internet In general, our materials come from two main sources: free 3D models and manually created models There are plenty of free 3D models available such as free3D.com, tuborsquid.com, clario.io, cgtrader.com, archive3d.net or assets store in Unity We choose four main sources as indicated in Figure 3.1 because they provide most of the assets related to our project Figure 3.2 illustrates some 3D models that we will integrate into our application (trees, house, river, dam, cow)

Figure 3.2 Example of some free3D models that will be used in the application

A virtual environment is another important factor needed to take into account Instead of building the surrounding environment manually, we reconstruct the virtual environment close to where the student lives in by using height map A heightmap is a raster image used to store values, such as surface elevation data Black color represents the lowest point and white color represents the highest point A survey conducted by Smelik (Smelik, De Kraker, Tutenel, Bidarra, & Groenewegen, 2009) showed that heightmap is often used as the basis of a terrain model Natural, mountainous-like

structures are gained by adding and rescaling several levels of Perlin noise (Perlin, 1985)

to each point in the heightmap Input data for heightmap is generated from the terrain.party portal website as depicted in Figure 3.3 We find this website quite useful because it allows users to generate a heightmap for a given area anywhere in the world The square box overlaid on top of the screen lets the user select the desired area; this box can be expanded or contracted based on user's preference The main drawback of this portal is the size of the selection box since it is limited to only 60 km For our application, the average number of students in the classroom is around 25 students; each student will be assigned a land randomly of 500m So 60km in our case study is far more enough and reasonable Note that in every dimension, the size of the heightmaps should be 1 pixel larger than the resolution of the terrain texture, that is the power of two plus one Because each pixel from the texture must be mapped to a polygon which consists of two triangles, not a pixel, in the terrain mesh

Figure 3.3 Heightmap generator from terrain.party

Figure 3.4 Terrain generated from heightmap in Unity Figure 3.4 (b) shows the actual terrain generated from heightmap Figure 3.4 (a)

As mentioned earlier, the majority of the assets in our application are imported from the internet However, some of the assets need more work such as animation, rigging, texture, or modification to suite our need In addition, unavailable models for our application also need to be created such as typical kind of trees or birds Among much other 3D software, we choose Blender for this purpose because it is free, powerful, rich

of community involvement and compatible for many platforms Cow, for example, is a free static 3D model We import this model into Blender and add some skeletons for rigging and animation that allows it to move around the certain area

3.4.2 Vuforia package for Unity3D

For the creation of Augmented Reality applications in mobile devices, we use Vuforia (Linowes & Babilinski, 2017) Software Development Kit package for Unity3D It is a free development kit and supports multiple platform such as Android, iOS, or Unity To make it works, users have to register and download license key from Vuforia website

The most used functionality of this package is its ability to recognize and track planar images in real time Each object in the real world will be labeled with an image, and the application can recognize this image then position the corresponding 3D model about real world object Figure 3.5 illustrates the use of augmented reality to collect real world objects Students will go outside of their classroom to the watershed and collect the objects they like (trees, dams, rivers ) The collected objects will be put in their inventory in the virtual reality game

Figure 3.5 Birch tree 3D model

3.4.3 Google Cardboard Package

While Vuforia package helps to create Augmented Reality application through using a mobile camera, Google Cardboard package, on the other hand, provides some pre-programmed features such as user head tracking, side-by-side stereo rendering, detecting user interaction with the system which we will use to build our application Side-by-side stereo rendering splits mobile phone screen into two parts Users will look into this virtual environment using Google Cardboard

Figure 3.6 Example of using tree in the inventory to plant and build the city Downloading and importing Google VR SDK for Unity is straightforward Users go to the Google Developer page and download the appropriate SDK Kit for their Environment (Android or iOS) After the package is downloaded, users will import into

Unity in the Assets folder Our application runs on iOS device, so additional software package (XCode) is installed to port the application on iPhone

3.4.5 Application

Unlike other 3D game applications in which users collect objects that are already available in the game Users are not required to go outside for assets Our VR/AR game comes up with a new idea by combining both VR and AR techniques To reconstruct the virtual world, users need some assets These assets can be bought from other players

or can be collected from a real world Users use a camera on their mobile phone pointing

to a real object A three-model corresponding to this object will appear, and then users can get this object and put it into their inventory When students come back to Virtual Reality game, they can use the assets in their inventory to construct their environment

To switch between VR and AR mode, users rotate their phones to the right to go back

to the main screen and then select the desired mode Interactions between users and objects in the game are performed using a gaze system, which includes a small circle (reticle) on the screen moving along with user's head (small white circle in Figure 3.7) Movement in the game is performed through a button on the Google cardboard, users point the reticle to the desired location and keep pressing that button The central idea behind the watershed development game is to introduce the students to systems thinking and help them visualize how human activities and natural events (climate) are

intrinsically coupled and how actions of one in the watershed have implications on others downstream

Figure 3.7 Main menu of the VR/AR game that allows to switch between VR/AR

mode

To use our watershed-based games, users only need to have a Google cardboard (which only costs a few dollars) and a smartphone with an integrated web browser (no additional software is required) Currently, there are many available VR devices on the market, such as Oculus Rift, Gear VR, HTC Vive, View-Master, Google Cardboard to name a few Out of these devices, Google Cardboard is chosen due to its low-cost VR headset and simple design, making it feasible to implement in most mainstream U.S classrooms Figure 3.7 illustrates of the main menu in the game, when the user taps

“Explore world” on the screen, the application will switch to AR mode, where as “Enter your game” will go to VR game

The watershed-based games will contain a set of units with tasks which engage students in learning and understanding water features The immerse and interactive experiences make students feel more attached to the environment as being the center of all consequences of their observations and actions

3.5 Evaluation

To evaluate the effectiveness of our application, we conduct a use case with two students (one Master student and one Ph.D student in our lab) who are not involved in any process of the development The aim of this study is to help students understand their surrounding environment The experiment is conducted on the campus where there are several kinds of trees and bushes The expected outcomes of this study are the ability to find, recognize, name, and collect some objects in real life

We setup the environment by labeling some typical trees where they are easily

be spotted Each student will be given an iPhone with installed app and a Google Cardboard headset Students are required to go around the campus and pick any real-life objects they like by using the application in Augmented Reality mode After collecting their desired assets, they are asked to position these objects in the Virtual reality game

The study shows that the students are highly motivated when using the application because it helps them recognize and naming some kinds of trees that students saw before but do not know trees' names or vice versa Being immersed in the virtual environment, students find it interesting to build the territory in the way they want

However, there are some technical problems of the application that need to be improved First is the ability to recognize the labeled objects Since our application use printed paper to label objects, students have to go very close to the objects and point mobile phone camera toward the images at a certain angle This problem can be solved

by scanning target image as an object for recognition Although this process is consuming when designing material contents, we find it feasible and useful for a large class We argue that, when the size of the game is big, users do not want to spend too much time to grab a single object

time-The second problem that students encounter is loading 3D objects in the VR game When the number of objects increases, the loading process is slow and moving

in the scene is lagging We find that all the objects in our model are rendered upon the

application starts, so that is the reason why it is slow The solution to overcoming this problem is to “simplify” the scene, meaning that the application only renders objects that are close to the player or visible to the camera Distant objects will not be rendered

3.6 Challenge and Discussion

The first challenge in our project is to incorporate 3D models from different websites into Unity3D Some of the models are not fully supported by Unity3D, for example, the Wavefront format This 3D model format includes two files with extension: obj and mtl where the obj holds information of the object such as the position of each vertex, the UV position, vertex normals, and the faces that make each polygon, and texture vertices The MTL File Format describes surface shading (material) properties of objects In other words, it holds texture information of an object Unfortunately, this extension is outdated and not supported by later technology such as Unity One possible approach is to use another 3D software that can read these files then exports these files into another format We use Blender to import these files then export to Unity3D readable format such as blend or fbx

The second challenge is to use a mobile phone camera to recognize an object in nature While Vuforia package allows developers to pre-defined Image Targets by scanning the object, it has too many limitations that we cannot use in a real situation First, it requires to print out the Object Scanning target that is a small image Second, the object being scanned should be aligned with the position of the printed image This will result in the capability of scanning only small objects Also, objects should be scanned under some lighting conditions, so scanning big objects for recognition is not feasible Our solution to overcome this problem is to label an object with a name tag Because our focus is going out to collect the object, not the object recognition We found this solution is utmost simple and easy to deploy

The third challenge is the VR contents; it is the most time-consuming part of the application The more they look realistic in the environment, the longer it takes to produce them To alleviate this problem, importing available models with texture from the internet is the first solution The second solution is to get a similar poly mesh of the

object we want to reconstruct and modify it in 3D software And the third solution is to try to create a low poly mesh as much as we can to the acceptable level and use high-resolution texture

3.7 Conclusion

In this paper, we have presented a comprehensive framework for building an application using virtual reality and augmented reality techniques How each component of the framework can be integrated together in Unity3D This framework will work as a guideline to help students and instructors in the other fields to build and construct their own material and application Several challenges in each step in the workflow are discussed, and we suggest some possible solutions And an environment for the application is setup so that students can first understand their surroundings

This is the on-going project toward enhancing computational thinking for students In future work, we will apply mathematical model into the game, for example, excessive building many factories can cause pollutants into streams and creeks that will affect other players A set of compliance points (CP) is set up on the stream segment to monitor water quality in the stream When critical thresholds are reached at CPs, the player causing the pollution will be asked to pay the penalty In some instances, several players with land holdings upstream will be asked to pay the penalty directly proportional their pollutant loadings and inversely proportional to the distance from their parcel to the point of concern on the river These penalties are apportioned among downstream players based on their distance from the polluted point

Further investigation on balance between complexity and fidelity of 3D model contents should be carefully considered because complex models can give correct material information On the other hand, they could overload the application