Learn ARKit, Apple’s New Augmented Reality Development Framework With ARKit, you can create immersive, engaging experiences by mixing virtual 2D and 3D content with the live camera feed of the world around you. If getting started with this new framework sounds difficult and timeconsuming, don’t worry, we’ve got you covered. ARKit by Tutorials helps you learn ARKit the quick and easy way: by following fun and easytoread tutorials. Who This Book Is For This book is for intermediate to advanced developers who want to create ARKit apps and learn about the intricacies of developing for Augmented Reality on iOS. Topics Covered in ARKit by Tutorials Introduction to ARKit: Get a basic introduction to Augmented Reality and the ARKit framework concepts. Session Management: Discover what’s required to manage an AR session properly; this includes error handling and tracking issues. Adding Objects: Learn how to place 3D Objects into augmented space. Materials Lighting: Explore the concepts of Physical Based Rendering, textures and light estimation. Creating 3D Assets: Get creative with the SceneKit editor and make your own 3D assets right inside Xcode. Facebased Apps: Find out what it takes to develop and ship FaceBased AR apps. And much, much more
Trang 2Notice of Liability
This book and all corresponding materials (such as source code) are provided on an
“as is” basis, without warranty of any kind, express of implied, including but not limited to the warranties of merchantability, fitness for a particular purpose, and noninfringement In no event shall the authors or copyright holders be liable for any claim, damages or other liability, whether in action of contract, tort or otherwise, arising from, out of or in connection with the software or the use of other dealing in the software
Trademarks
All trademarks and registered trademarks appearing in this book are the property of their own respective owners
Trang 3Table of Contents: Overview
About the Cover 13
What You Need 18
Book License 19
Book Source Code & Forums 20
Section I: Getting Started with ARKit! 22
Chapter 1: Hello ARKit! 23
Chapter 2: Your First ARKit App 35
Chapter 3: Basic Session Management 61
Chapter 4: Adding 3D Objects & Textures 75
Chapter 5: Detecting Surfaces 100
Chapter 6: Adding Physics 120
Section II: Portal Apps 142
Chapter 7: Building a Portal 143
Chapter 8: Adding Objects to Your World 155
Chapter 9: Geometry, Textures & Lighting 171
Section III: Interactive AR Apps 192
Chapter 10: Detecting Placeholders 194
Chapter 11: Beginning User Interaction 213
Chapter 12: Advanced User Interaction 241
Chapter 13: Location Tracking & Beacons 268
Section IV: Face-Based AR 298
Trang 4Chapter 14: Getting Started with Face-Based AR 299
Chapter 15: Tracking the User’s Face 311
Chapter 16: Creating Assets for Face-Based AR 326
Chapter 17: Working with Blend Shapes 345
Chapter 18: Recording Your ARKit Experience with ReplayKit 354
Section V: Game Physics 368
Chapter 19: Vehicle Physics, Part I 369
Chapter 20: Vehicle Physics, Part II 386
Section VI: Persistent & Shared Experiences 400
Chapter 21: World Mapping & Persistent AR Experiences 401
Chapter 22: Collaborative AR Experiences 420
Section VII: Image Tracking, Object Detection & AR Quick Look 437
Chapter 23: Introducing AR Quick Look 438
Chapter 24: Introducing USDZ 452
Chapter 25: Detecting Images & 3D Objects 470
Section VIII: Introduction to RealityKit 494
Chapter 26: Introducing RealityKit & Reality Composer 495
Conclusion 511
Trang 5Table of Contents: Extended
About the Cover 13
About the Authors 15
About the Editors 16
About the Artists 17
What You Need 18
Book License 19
Book Source Code & Forums 20
Section I: Getting Started with ARKit! 22
Chapter 1: Hello ARKit! 23
Augmented reality 24
Introducing ARKit 26
What can ARKit do for you? 27
Limitations of ARKit 30
Technology behind ARKit 32
Challenge 32
Key points 34
Chapter 2: Your First ARKit App 35
Getting started 36
Key points 60
Chapter 3: Basic Session Management 61
Getting started 62
Starting an AR session 64
Controlling an AR session 67
Updating the status message 68
Handling AR session errors 68
Handling AR session state changes 70
Trang 6Challenge 72
Key points 74
Chapter 4: Adding 3D Objects & Textures 75
Getting started 76
Adding 3D objects 76
Creating the dice scene 78
Shaders, materials and textures 79
Physically based rendering (PBR) 81
Texturing the 3D Objects 89
Loading 3D objects 91
Placing 3D objects 93
Challenge 97
Key points 99
Chapter 5: Detecting Surfaces 100
Getting started 101
Adding game states 102
Anchors 104
Detecting surfaces 106
Creating new planes 106
Updating planes 109
Creating a focus node 110
AR Coaching Overlay View 115
Challenge 118
Key points 119
Chapter 6: Adding Physics 120
Getting started 121
Adding physics 123
Plane physics 127
Randomizing rotation 129
Force 129
Trang 7Lights and shadows 131
Wrapping things up 134
Hit testing 138
Challenge 139
Key points 140
Section II: Portal Apps 142
Chapter 7: Building a Portal 143
The portal app 144
Getting started 144
Setting up ARKit 145
Plane detection and rendering 148
Key points 154
Chapter 8: Adding Objects to Your World 155
Getting started 156
Hit testing 160
Adding crosshairs 162
Adding a state machine 166
Key points 170
Chapter 9: Geometry, Textures & Lighting 171
Getting started 172
Building the portal 174
Adding the doorway 186
Placing lights 190
Key points 191
Where to go from here? 191
Section III: Interactive AR Apps 192
Chapter 10: Detecting Placeholders 194
Introducing Razeware Mobile Kiosk 195
RazeAd 195
Trang 8Getting started 196
Plane detection vs object detection 197
Detecting a rectangle 197
Creating the billboard 202
Displaying placemarks 206
Adding SceneKit nodes 208
Handling interruptions 210
Key points 211
Chapter 11: Beginning User Interaction 213
Getting started 214
Detecting a QR code 219
Showing an image 222
Showing an image carousel 225
Playing a video 230
Removing the video player 235
Using the data encoded in the QR code 236
Key points 239
Where to go from here? 240
Chapter 12: Advanced User Interaction 241
The updated starter project 242
Interacting with the billboard 245
Going fullscreen 254
Detecting reference images 259
Key points 267
Chapter 13: Location Tracking & Beacons 268
Getting started 269
Locating the user 270
Geofencing 276
Beacons 286
Testing 294
Trang 9Key points 297
Where to go from here? 297
Section IV: Face-Based AR 298
Chapter 14: Getting Started with Face-Based AR 299
ARKit for face-based AR 300
The starter project 301
Starting a face tracking session 305
Handling errors and interruptions 307
Key points 310
Chapter 15: Tracking the User’s Face 311
Detecting the user’s face 312
Working with face geometry 313
Handling updates 316
Adding additional masks 318
Connecting the switch button 322
Setting up a basic environment 323
Adjust the lighting 323
Key points 325
Chapter 16: Creating Assets for Face-Based AR 326
Creating 3D Content 327
Creating the Glasses class 329
Adding the glasses option 331
Creating Pig 333
Creating the Pig class 342
Key points 344
Chapter 17: Working with Blend Shapes 345
Animojis and blend shapes 346
Animating the brow 347
Animating the eyes 349
Trang 10Animating the mouth 352
Key points 353
Chapter 18: Recording Your ARKit Experience with ReplayKit 354
Getting started with ReplayKit 355
Adding ReplayKit to FaceCase 357
Enhancing the user's experience 362
Key points 367
Where to go from here? 367
Section V: Game Physics 368
Chapter 19: Vehicle Physics, Part I 369
Getting started 370
Building the monster truck 373
Loading the monster truck 379
Adding vehicle physics 380
Key points 385
Chapter 20: Vehicle Physics, Part II 386
Getting started 387
Creating an infinite ground plane 387
Adding engine force 390
Adding steering 393
Limiting the speed 396
Resetting the game 397
Challenge 398
Key points 399
Section VI: Persistent & Shared Experiences 400
Chapter 21: World Mapping & Persistent AR Experiences 401 Getting started 402
Trang 11Saving your world map 406
Loading and restoring content 410
Resetting content 417
Key points 418
Where to go from here? 419
Chapter 22: Collaborative AR Experiences 420
Getting started 420
Establishing a peer connection 421
Sending data to peers 429
Receiving data and relocalizing 431
Key points 435
Where to go from here? 436
Section VII: Image Tracking, Object Detection & AR Quick Look 437
Chapter 23: Introducing AR Quick Look 438
What is AR Quick Look? 439
Experiencing AR Quick Look 439
AR Quick Look features 442
AR Quick Look for web 444
AR Quick Look for apps 446
Key points 451
Where to go from here? 451
Chapter 24: Introducing USDZ 452
What is USD? 453
What is USDZ? 454
USDZ tools 455
Converting a 3D model to USDZ 457
Converting USDZ to USDA 461
Inspecting and validating USDZ files 462
USDZ support for Unity 464
Trang 12Key points 468
Where to go from here? 469
Chapter 25: Detecting Images & 3D Objects 470
Image Tracking 471
Detecting 3D Objects 484
Key points 492
Where to go from here? 493
Section VIII: Introduction to RealityKit 494
Chapter 26: Introducing RealityKit & Reality Composer 495
What is RealityKit? 496
Creating a RealityKit project 497
RealityKit API components 500
What is Reality Composer? 502
Creating a new Reality scene 504
Importing custom content 505
Adding behaviors 507
Code generation 508
Loading Reality scenes 509
Key points 510
Where to go from here? 510
Conclusion 511
Trang 13A About the Cover
Plesiosaurs were ancient marine reptiles; there are records of Plesiosaur fossils being studied as far back as the 1600’s, which makes them one of the first orders of extinct reptiles to be recognized as such By the mid 1800’s, researchers had a pretty good idea of what Plesiosaurs looked like and how they behaved But a string of recent discoveries of new types of Plesiosaurs have helped scientists clarify how they looked, how they evolved, and even how they hunted prey
Just as paleontologists bring the past alive through their discoveries, ARKit brings things real or imagined into reality through your smartphone Whether you’re bringing dinosaurs to life in your dining room, creating portals to play in, or simply want to play Pac-Man on your city streets, ARKit can help make sure your great idea for an AR app will never become extinct!
Trang 14"To my wife Corné and my daughter Marizé Thank you for
your patience, support, belief, and most of all, your love
Everything I do, I do for you."
— Chris Language
"To Dhaval, for always supporting me.To Ginger, for all the
laughs and hugs!"
Trang 15About the Authors
Chris Language is a seasoned coder with 20+ years of experience,
and the author of 3D Apple Games by Tutorials He has fond memories of his childhood and his Commodore 64; more recently
he started adding more good memories of life with all his Apple devices By day, he fights for survival in the corporate jungle of Johannesburg, South Africa By night he fights demons, dragons and zombies! For relaxation, he codes You can find him on Twitter
@ChrisLanguage Forever Coder, Artist, Musician, Gamer and Dreamer
Namrata Bandekar is a Software Engineer focusing on native iOS
and Android development When she's not developing apps, she enjoys spending her time travelling the world with her husband, SCUBA diving and hiking with her dog She has also spoken at a number of international conferences on mobile development Say
hi to Namrata on Twitter: @NamrataCodes
Antonio Bello is an author of this book Antonio has spent most of
his life writing code, and he’s gained a lot of experience in several languages and technologies A few years ago he fell in love with iOS development, and that’s what he mostly works on since then, although he’s always open for challenges and for playing with new toys He believes that reputation is the most important skill in his job, and that “it cannot be done” actually means “it can be done, but it’s not economically convenient." When he’s not working, he’s probably playing drums or making songs in his small, but well fitted, home recording studio
Tammy Coron is the co-author and final pass editor on this book
Tammy is an independent creative professional and the host of Roundabout: Creative Chaos She’s also the founder of Just Write Code Find out more at tammycoron.com
Trang 16About the Editors
Jerry Beers is a tech editor of this book Jerry is co-founder of Five
Pack Creative, a mobile development company specializing in iOS development He is passionate about creating well-crafted code and teaching others You can find his company's site at
fivepackcreative.com
Joey deVilla is a tech editor on this book Joey is a developer
turned accordionist turned tech evangelist turned developer herder A Canadian turned Tampanian, he works at Sourcetoad, blogs at globalnerdy.com and joeydevilla.com, and does what he can to avoid becoming a “Florida Man” news story
cat-Morten Faarkrog is a tech editor on this book cat-Morten is an app
developer from Denmark who works at the digital agency Adapt When not developing apps he enjoys reading, travelling, and running To connect with Morten you can find him on LinkedIn
Matthew Morey is a tech editor on this book Matt is an engineer,
author, hacker, creator and tinkerer As an active member of the iOS community and CTO at MJD Interactive he has led numerous successful mobile projects worldwide When not developing apps
he enjoys traveling, snowboarding, and surfing He blogs about technology and business at matthewmorey.com
Trang 17About the Artists
Vicki Wenderlich is the designer and artist of the cover of this
book She is Ray’s wife and business partner She is a digital artist who creates illustrations, game art and a lot of other art or design work for the tutorials and books on raywenderlich.com When she’s not making art, she loves hiking, a good glass of wine and
attempting to create the perfect cheese plate
Jake Nolt is an artist for this book Jake is a 3D designer working
for a decade in video game and application development He is also
an assistant professor of animation and the proprietor of a small indie game studio When not developing or teaching, Jake enjoys spending time with family, gaming, and being outdoors
Trang 18W What You Need
To follow along with the tutorials in this book, you need the following:
• A Mac running macOS Catalina 10.15 or newer
• Xcode 11.0 or newer
• An AR-capable iOS device with iOS 13 or later installed Any iOS device with an A9 chip or newer that has iOS 11 or newer installed will support ARKit However, the latest features, such as motion capture and people occlusion, will only run on a device with an A12 Bionic processor or newer and iOS 13 or newer
Trang 19L Book License
By purchasing ARKit by Tutorials, you have the following license:
• You are allowed to use and/or modify the source code in ARKit by Tutorials in as
many apps as you want, with no attribution required
• You are allowed to use and/or modify all art, images and designs that are included
in ARKit by Tutorials in as many apps as you want, but must include this
attribution line somewhere inside your app: “Artwork/images/designs: from ARKit
by Tutorials, available at www.raywenderlich.com”.
• The source code included in ARKit by Tutorials is for your personal use only You are NOT allowed to distribute or sell the source code in ARKit by Tutorials without
prior authorization
• This book is for your personal use only You are NOT allowed to sell this book without prior authorization, or distribute it to friends, coworkers or students; they would need to purchase their own copies
All materials provided with this book are provided on an “as is” basis, without warranty of any kind, express or implied, including but not limited to the warranties
of merchantability, fitness for a particular purpose and noninfringement In no event shall the authors or copyright holders be liable for any claim, damages or other liability, whether in an action of contract, tort or otherwise, arising from, out of or in connection with the software or the use or other dealings in the software
All trademarks and registered trademarks appearing in this guide are the properties
of their respective owners
Trang 20B Book Source Code &
Forums
If you bought the digital edition
The digital edition of this book comes with the source code for the starter and completed projects for each chapter These resources are included with the digital edition you downloaded from https://store.raywenderlich.com/products/arkit-by-tutorials
If you bought the print version
You can get the source code for the print edition of the book here:
• https://store.raywenderlich.com/products/arkit-by-tutorials-source-code
Forums
We’ve also set up an official forum for the book at forums.raywenderlich.com.This is
a great place to ask questions about the book or to submit any errors you may find
Digital book editions
We have a digital edition of this book available in both ePUB and PDF, which can be handy if you want a soft copy to take with you, or you want to quickly search for a specific term within the book
Buying the digital edition version of the book also has a few extra benefits: free updates each time we update the book, access to older versions of the book, and you
Trang 21Visit our ARKit by Tutorials store page here:
• https://store.raywenderlich.com/products/arkit-by-tutorials
And if you purchased the print version of this book, you’re eligible to upgrade to the digital editions at a significant discount! Simply email support@razeware.com with your receipt for the physical copy and we’ll get you set up with the discounted digital edition version of the book
Trang 22Welcome to the first section of the book where you’ll get your feet wet by learning all about the amazing power of ARKit
• Chapter 1: Hello ARKit!: You’ll start at the very beginning with a guided tour of
what augmented reality is and how ARKit can help You’ll also learn about the limitations of ARKit
• Chapter 2: Your First ARKit App: Once you’ve covered the basics, you’ll get your
hands dirty by creating your own ARKit app This will be the start of creating a fun augmented reality poker dice game
• Chapter 3: Basic Session Management: In this chapter, you’ll learn about
session management, an essential part of every ARKit app Not only will your app comply with the basic requirements, but it will also provide the users with a
seamless AR experience
• Chapter 4: Adding 3D Objects & Textures: Next, you’ll add some basic 3D
objects to your project You’ll also create materials and learn about the different types of textures used in PBR-based materials
• Chapter 5: Detecting Surfaces: In this chapter, you’ll learn how to detect and
manage surfaces — like floors, walls and tables — with ARKit Once you know how
to do that, you’ll know where and how to place AR content within the scene
• Chapter 6: Adding Physics: Finally, you’ll add physics to your virtual content
With physics, you can make things bounce off of surfaces and interact with other virtual content as if they were real
Well, what are you waiting for? Jump in!
Trang 231 Chapter 1: Hello ARKit!
By Chris Language
Augmented reality (AR) is a technology that will undoubtedly play a large part in our digital evolution as it begins to turn science fiction into science fact Coupled with the recent surge of interest in artificial intelligence (AI) — specifically machine learning — we’re only beginning to understand how these technologies can work in concert to seamlessly blend the real world with the digital world
Not bad for a bunch of primates living on a tiny blue sphere hurtling through space, eh?
Trang 24Let’s use a visual aid to explain the basic concept behind augmented reality This is
my pet chameleon, Elon — as in Cham-E-Leon, get it? Erm moving on
Elon is a cute little fella, wouldn’t you agree? Well, the truth is, Elon isn’t real In fact, he’s nothing more than a realistic-looking 3D model
Starting with the photo on the left, you can see my boring old iPad cover Suddenly, Elon the chameleon appears out of thin air As I move the camera around, Elon stays put; however, his head and eyes follow me around When I move out of his view, he moves around so that he can keep an eye on me
How on earth is this possible? Is it some kind of advanced Jedi mind trick?
Augmented reality
As cool as it would be, AR is not a Jedi mind trick Instead, the illusion comes from overlaying the virtual content on the live image feed coming in from the camera
That is to say, AR enhances (augments) the real world (reality) To make the illusion
seamless, the position, scale and rotation of the virtual content updates constantly using real-time motion data from the device as it moves through real-world space
The result? Mind-blowing effects that make you question your own reality.
Let’s look at how this works
Marker-based tracking
AR is not a new concept; many modern, camera-equipped smart devices have been
capable of doing it for the past few years This is usually achieved by using tracking
Trang 25The tracking card serves as an anchor point on a real-world surface, and a special algorithm analyzes the camera data to determine the card’s position, scale and orientation That tracking information is then used to project, scale and place 3D content which is superimposed over the camera’s image A good example of this is
Om Nom Candy Flick, a surprisingly fun augmented reality game
Unfortunately, the tracking is lost the moment the card goes beyond the camera’s view, spoiling the whole AR experience The next frontier of AR — which is where we find ourselves now — is to create AR content that does not require tracking physical markers
Markerless tracking
Imagine you have a pair of magical glasses, capable of superimposing virtual 3D elements over everything you see If you put them on, you can seamlessly meld your real-world experience with that content, making everything appear so real that you can almost reach out and touch it
Well, I’ll let you in on a little secret: Those magical glasses already exist, and one such example is the Microsoft HoloLens
Trang 26The HoloLens is far from magical, though Instead, it’s a highly sophisticated device capable of tracking your every move It’s also spatially-aware, which means it not
only sees but understands the world around you It tracks movement through
real-world space and actively maps out the environment, creating a virtual 3D “real-world space”
Since the HoloLens is a wearable device, a standard human-machine interface, such
as a joystick or a mouse, didn’t work well for input To fix this problem, Microsoft came up with some clever ways for users to interact with virtual content:
• Gaze: Uses a focus cube placed in the middle of the display, which acts as a virtual
mouse cursor You tilt your head around to move the cursor on top of objects to interact with them
• Gesture: The HoloLens can recognize a pinch gesture made with your hand to
simulate a mouse click
• Voice: You can also speak commands to the HoloLens, much like you would with
Siri or Microsoft Xbox Connect
The HoloLens is a fantastic device, but it’s not quite perfect Big and clunky, it’s a far cry from the sporty sunglasses-like headsets that we were promised in science fiction stories Plus, the device is still quite expensive, which renders the technology out-of-reach for most people
However, those barriers to entry dissolved when Apple announced ARKit
Introducing ARKit
ARKit is Apple’s mobile AR development framework With it, you can create an immersive, engaging experience that mixes virtual 2D and 3D content with the live camera feed of the world around you
What sets ARKit apart from other AR frameworks, such as Vuforia, is that ARKit performs markerless tracking That’s right! No tracking cards are required On top of
Trang 27into a markerless AR-capable device Think about that: At this very moment, millions
of Apple users already have a sophisticated AR device right in their pockets
Note: Some of the more recent ARKit features are limited to devices with the
A12 or newer processor This includes the iPhone XS, XS Max and XR, the 3rd generation iPad Air and the 5th generation iPad mini
What can ARKit do for you?
If you’ve worked with any of Apple’s other frameworks, you’re probably expecting that it will take a long time to get things working But with ARKit, it only takes a few lines of code ARKit does most of the heavy lifting for you, so you can focus on what’s important: creating an immersive and engaging AR experience
Let’s look at what ARKit can do for you
World tracking
Similar to the HoloLens, ARKit tracks the device’s position and orientation in time through real-world space It uses sampled motion data from the device to automatically update the virtual content transforms It does this by moving a virtual camera’s point of view in the virtual scene to match that of the device
real-Scene understanding and persistence
To achieve markerless tracking, ARKit creates and manages its own world map of the detected surfaces Special event hooks in ARKit trigger each time it detects a new surface, when a surface is modified or when a surface is removed This gives you incredible control over your scene, including how your content interacts with these surfaces
ARKit can also load and save world maps for quick recognition of previously scanned spaces With the same world map loaded on multiple devices, you can easily create a shared experience with a basic network
You can also keep virtual content persistent across multiple ARKit sessions Now, tell
me, where did I put my mug again? :]
Trang 28ARKit can track up to three faces simultaneously on devices equipped with a
TrueDepth camera, which is used to power front-facing camera experiences
Motion capture
Real-time motion capture allows you to track a persons body position, movement and pose and produces a 3D skeleton consisting of joints and bones You can use this information as input into your own AR experiences, or you can store the captured information to animate your own game characters
People occlusion
Thanks to machine learning, ARKit lets your AR content pass in front and behind of real-world human figures by means of depth-based occlusion This creates a much more immersive experience for your AR content
Multi-camera tracking
You can use both the front and back-facing cameras for simultaneous world and face tracking, which means you can use your face as input to control your AR characters, for example
Trang 29Collaborative sessions
Create shared AR experiences with live collaborative sessions between multiple users ARKit builds a collaborative world map between multiple users, allowing them
to get into a shared AR experience much faster and ultimately making the
development of shared AR experiences much easier and faster too
Light estimation
ARKit uses the current video frame to estimate the environmental lighting
conditions in real-time It can then adjust the brightness levels of your virtual content automatically to match the environmental conditions, provided your virtual content uses physically-based materials
Real-time environment map
ARKit generates a real-time environment map by sampling your immediate space This allows you to give your PBR-based objects a realistic-looking reflection, along with real-time lighting information
Scene interaction
ARKit provides hit-test functions that let you interact with the 3D content in your scene in unrestricted ways You can also couple this interaction with traditional touch-based gestures, which is what Apple recommends in its Human Interface Guidelines for AR-based apps However, because you’re inside Apple’s ecosystem, there’s nothing stopping you from using any of Apple’s other great frameworks, such
as CoreML or Vision You can even use SiriKit and command the AR experience with your voice
Metric scaling system
For calculating distances and sizes in your scene, ARKit uses a fairly accurate metric scaling system It doesn’t matter which graphics API you’re using; in ARKit, 1 unit equals 1 meter
Trang 30Rendering integrations
ARKit does not have its own graphics API; it only provides you with tracking
information and surface detection capabilities While you still need to provide the graphics in the scene, ARKit makes it easy to integrate with graphics frameworks such as SpriteKit, SceneKit, RealityKit and Metal
Apple’s relationships with the gaming development community has also resulted in some excellent plug-ins for the popular game engines Unity and Unreal
Limitations of ARKit
Although ARKit presents a new paradigm for user experience on mobile devices, it has a few limiting qualities that impact how you develop ARKit apps
Let’s look at some of the limitations of ARKit
Surface detection takes time
Surface detection and real-world understanding take a noticeable amount of time — it’s hardly instant Therefore, you need to manage user expectations and guide them through the surface detection stage of your app Otherwise, your app will get binned quite quickly
Motion processing lag
Excessive device motion can cause stuttering and blurred imagery, making it difficult for ARKit to understand what it’s looking at When the real-life scene gets shaky, ARKit processing will get a bit shaky too It’s important to take it slow over those bumpy roads
Trang 31Low-light conditions
Low-light conditions are problematic The on-device camera cannot discern light scenes as well as the human eye, and that means ARKit will struggle with scene detection in low-light conditions; it may even fail to recognize any surfaces at all
low-If you’re struggling to see things in low-light conditions, ARKit is finding it ten times worse
Smooth textureless surfaces
Smooth, textureless surfaces make it difficult for ARKit to detect and track surfaces The more textured a surface is, the easier it will be for ARKit to detect and track the motion of those surfaces
As a developer, your job is to make sure your users understand this before using your AR-enabled app; otherwise, they’ll get frustrated when they can’t place virtual furniture on their polished, jet-black marble floors
Ghost effect
After using AR for a while, you might notice some virtual content disobeys the laws
of physics For example, the virtual content might pass through real-world objects, much like a ghost would pass through a closed door
Since situations like this can break the AR experience, it’s a good idea to encourage users to use your application in wide-open spaces or on top of a nice clean table.Apple is definitely hard at work to improve the overall experience with the
introduction of people occlusion Who knows, perhaps Apple is already working on how to occlude those pesky pets that are always getting in the way! :]
Trang 32Technology behind ARKit
When you look under the hood of ARKit, you might be surprised that some of the technologies are quite familiar and have been around for a while now
ARKit uses Visual Inertial Odometry (VIO) to track the motion of the device and
the world around it VIO fuses the AVFoundation-based input from the camera sensor with the device motion data captured via CoreMotion ARKit also makes use
of CoreML to help speed up the surface tracking and object recognition processing If you’ve used any of these frameworks before, you’ll instantly recognize how they work together to support ARKit’s scene recognition and tracking
Since this is your first challenge, we’ve made it simple: Read through the following resources and bookmark them Because it can be difficult to find resources on ARKit through Googling alone, we’ve curated a list for you
Trang 33Official ARKit documentation
This is Apple’s official site for ARKit documentation Here, you can find up-to-date information about ARKit and its related technologies
• Apple Developer Documentation for ARKit: http://apple.co/2sE0rUq
Human Interface Guidelines for Augmented Reality
Apple also provides a set of Human Interface Guidelines to coach you on the best practices when designing your ARKit apps This guide shows you exactly what to do
— and what not to do Failure to comply with these guidelines may result in Apple rejecting your app submission
• Human Interface Guidelines for AR: http://apple.co/2xOwp1Q
Official WWDC videos on ARKit
Apple always packs a good deal of information into its WWDC presentations Their videos on ARKit are no exception and are definitely worth watching more than once
• ARKit video’s from WWDC 2017 to 2019: https://apple.co/2NxL2TX
Interactive content with ARKit
This is one of Apple’s official ARKit demos, and it’s a fun way to get some hands-on
experience with ARKit Follow the link, and look for the Download Sample Code
button Once you find it, download the zip file, extract the contents and build the project using the latest version of Xcode You’ll need to run it on a physical device with an A9 processor or better (iPhone SE, iPhone 6s and later models) At this time, you can not test AR apps on the simulator
In this demo, you’ll get to meet Elon the chameleon, but be careful — he’s always watching!
• Interactive Content with ARKit: http://apple.co/2yI4gi2
In this demo, you’ll learn all about creating shared AR experiences
Trang 34Key points
You’ve reached the end of the first chapter, and you’ve taken the first step on your journey to creating your own AR-enabled games and apps
Here’s a quick recap of what you did in this chapter:
• You learned what augmented reality is and how smart devices use their onboard camera and motion sensors to implement AR
• You discovered the differences between marker-based and markerless tracking
• You were introduced to ARKit You learned what it is, what it does, and what some
of its limitations are
• You learned ARKit is not a graphics library; instead it’s a framework that does the heavy lifting of AR for you
In the next chapter, you’ll learn how to create your first ARKit application using Xcode See you soon!
Trang 352 Chapter 2: Your First
ARKit App
By Chris Language
It’s time to create your first ARKit application using Xcode’s built-in ARKit
application template You’ll also learn how to modify your app to accommodate basic
UI elements such as labels and buttons to provide user feedback and receive user input
It’s time to get your hands dirty!
Trang 36Getting started
Xcode comes with several different types of application project templates box
out-of-the-These application templates are essentially boilerplate applications that are
designed to save you time, effort and get you up and running as fast as possible
Creating an ARKit project
Open Xcode and select File ▸ New ▸ Project from the main menu If you’re an Xcode ninja, you can use the keyboard shortcut: Shift-Command-N.
Select the iOS ▸ Application ▸ Augmented Reality App template and click Next to
continue
Trang 37You need to provide some basic details about your project Enter ARPokerDice for the Product Name, select Swift for Language, SceneKit for Game Technology, uncheck the unit tests and click Next:
The final step is to choose a location to save your project Pick a directory and select
Create Xcode will work its magic and generate your project.
Building the ARKit project
Now that you’ve generated your ARKit project from the template, it’s time to see it in action
Unfortunately, the Xcode simulators are of no use to you when it comes to testing, as they don’t have support for ARKit applications The only way you’ll be able to test is
to use an actual mobile device that has an A9 or better processor in it
Here’s a list of supported devices:
• iPhone SE
• iPhone 6s and 6s Plus
• iPhone 7 and 7 Plus
• iPhone 8 and 8 Plus
• iPhone X, XR, XS and XS Max
• iPhone 11, 11 Pro and 11 Pro Max
Trang 38• iPad Pro (1st, 2nd and 3rd generation)
• iPad (5th+ generation)
• iPad Mini (5th+ generation)
Grab your iWhatever and connect it to Xcode Select it as the active output device
from the toolbar Press the Play button at the top to build and run your project, or just press Command-R if you’re lazy like me!
Behold, your first ARKit app will compile and deploy to your device
Look, Ma! Someone parked a spaceship in the garden
Note: The first time the app launches, it requests access to the camera; this is
a requirement for all augmented reality apps More on this a little later
Take a moment to test things out Point your camera at a big empty space, then walk around until you see the giant spaceship Try to get a view of all the possible angles
of the craft Do you notice how the spaceship stays anchored in its location in the real world? That’s the magic of ARKit
When you start your app, ARKit uses the device’s current location in space as its world anchor point It then loads the SceneKit scene and places the scene into augmented space; this is a quick and dirty way to get anything displayed in AR
OK, time to put away your spaceship, kids There’s still a lot to do
Trang 39Reviewing the project files
Before moving on, there are a few essential project files you need to review; these were created automatically for you
The launch screen storyboard
The launch screen is a standard part of every app It’s the first thing the user will see when they launch your app
Select LaunchScreen.storyboard to open it.
This is where you can place a beautiful splash image to show off your AR app
The main storyboard
The main storyboard is the view component of your AR application; this is what the user sees when your application is running
Select the Main.storyboard to open it.
Trang 40Take particular note of the ARSCNView view class; it lets you overlay a 3D scene on a live background image from the camera It provides seamless integration between ARKit and SceneKit.
Note: There’s also a view class dedicated to ARKit with SpriteKit and
RealityKit integration: ARSKView and ARView In this chapter, you’ll focus on SceneKit, hence the reason why you’re using ARSCNView
The view controller
Think of the view controller class as the code sitting behind your main storyboard
Select ViewController.swift to open it.
Take note of the frameworks at play here:
It also adopts the ARSCNViewDelegate protocol from ARKit, which contains methods you can implement to synchronize your SceneKit content with your AR session
The SceneKit asset catalog
This is a special folder designed explicitly for SceneKit assets SceneKit is a robust, high-level graphics framework capable of creating and managing all the graphical and audio content of your app; this is where you’ll store things like scenes, textures,