Tutorial: Introduction To ARKit With SceneKit

Tutorial: In this tutorial, you’ll learn about ARKit and how it integrates with SceneKit.


First Steps

No one can deny that WWDC 2017 was a smash hit! Apple introduced tons of new features and frameworks, causing developers everywhere to rejoice.

One of our favorites is ARKit — and the community agrees. Everywhere you look, people are talking about ARKit. That being said, let’s jump right in and whip up some of our own augmented reality!

What is Augmented Reality?

Augmented Reality (AR) is not a new concept; it’s been around for awhile. The basic idea is to fuse virtual reality with reality reality (that’s a thing, right?).The way it works is like so: AR lets us view a real-world space through a camera; it then introduces virtual objects into that space, giving us the impression that those virtual objects actually exist within that same space. The resulting effect is Augmented Reality.

Traditionally, AR uses tracking cards as anchor points within the real-world space. You point the camera at the tracking card, and once recognized, the tracking card serves as an anchor point, which in turn provides transform information about the real-world space. A 3D object can then be placed into the real-world by making use of this transform information. As the camera moves around the tracking card, the transform information gets updated, which updates the 3D object accordingly. This creates the impression that the 3D object is real. Everything is good… as long as you have that tracking card in view of the camera.

Games like Om Nom Candy Flick make use of tracking cards.

With the introduction of Microsoft’s HoloLens, things got way cooler. The HoloLens is spacial-aware, which means it has the ability to identify real-world surfaces and can use those surfaces to track movement – which means a tracking card is no longer required. The HoloLens is a fantastic specialised device, but it comes at a price. And it’s not a small price!

At WWDC 2017, Apple announced ARKit and showed us some impressive demos. The first thing we noticed was how ARKit has the ability to turn your iPhone or iPad into a HoloLens! That’s right! ARKit is also spacial aware, and it doesn’t need tracking cards to track movement.

Introducing ARKit

The ARKit framework is a mobile augmented reality platform that you can use to develop your own AR experiences. It’s a high level API that provides a simple interface with powerful features. ARKit is supported on A9 and higher capable devices.

ARKit provides the following capabilities:

  • Tracking

    It tracks your device position in real-world space in realtime. It makes use of visual inertial odometer, which combines camera tracking with motion sensor data. As an added bonus, no tracking cards are required!

  • Scene Understanding

    It’s spacial-aware and has the ability to identify surfaces in the real-world environment. Things like floors, tables, and walls. This is also referred to as plane detection.

  • Rendering

    It provides easy integration into SpriteKit, SceneKit as Metal. As an added bonus there’s even support for it in Unity and Unreal Engine. Yeah baby! Now we’re talking.

Basic ARKit Concepts

Now that you know what ARKit is all about, let’s take a closer look at a few fundamental concepts you’ll need in order to understand and make use of it.

  • Session

    ARKit is a session based framework. To start a new ARSession you need to provide it with an ARSessionConfiguration. After that, you just need to run the session. This process runs at 60 frames per second and will start gathering information from the camera and the motion sensors. You can either poll for information or you can make use of the available session update delegates.

  • Configuration

    ARKit provides tracking based on device capabilities. ARSessionConfiguration is used for less capable devices, and it only tracks device orientation. ARWorldTrackingSessionConfiguration is used for fully-capable devices, and it tracks device orientation, relative position, and scene information. Additional AR session features can be enabled or disabled through these configuration classes.

  • Frames

    As the AR session runs, you can access the ARFrame for information. It contains a captured image, detailed tracking, and scene information which contains the current tracking points and lighting conditions.

  • Feature Points

    This is a list of real-world tracking points gathered by ARKit over time, which relates to trackable features within the real world. These points provide depth information and gives ARKit its spacial-aware capabilities.

  • Anchors

    An anchor is a position and orientation in real-world space. ARKit will maintain that position and orientation for you as you move the camera around. You need to add anchors to an AR session.

  • Planes

    A plane is a detected surface like a floor or a table surface. When enabled, ARKit will detect, maintain, and track available surfaces in the real-world space. You can then use these surfaces to place your content, which will appear on top of the surface.

  • Lighting

    ARKit also tracks the current lighting conditions, which can be used to properly light virtual content in a more believable fashion.

Using ARKit

Apple made it simple to work with ARKit. Let’s take a closer look at the steps required to get ARKit up and running:

  • STEP 1: Configuration

    First, create an ARConfiguration and set all its properties. Also, make sure you set a configuration based on the device capabilities (not all devices work with ARWorldTrackingSessionConfiguration):

    
    if ARWorldTrackingSessionConfiguration.isSupported {
        configuration = ARWorldTrackingSessionConfiguration()
    }
    else {
        configuration = ARSessionConfiguration()
    } 
    
  • STEP 2: Create and Start the AR Session

    Now that you have an ARConfiguration all sorted out, you need to create and start the ARSession. This will start the tracking process:

    
    // Create an AR Session
    let arSession = ARSession()
    
    // Nominate self for session delegation
    arSession.delegate = self
    
    // Run the session
    arSession.run(configuration)
    
  • STEP 3: Managing an AR session

    Once the session is up and running you can manage the session as follows:

    
    // PAUSE. This will suspend all AR processing.
    session.pause()
    
    // RESUME. This will resume the AR session and continue processing.
    session.run(session.configuration)
    
    // CHANGE. This will change or update the configuration, should you want to do so.
    session.run(newConfiguration)
    
    // RESET. This will restart the session tracking process from scratch.
    session.run(configuration, options: .resetTracking)
    
  • STEP 4: Handle Session Updates

    While the AR session is running you can make use of the ARSessionDelegate protocol to react to various events:

    Frame Updates. func session(_: ARSession, didUpdate: ARFrame) provides a detailed frame by frame update. The ARFrame contains a captured image, tracking and scene information, feature points, and lighting conditions.

    Anchor Added. func session(_: ARSession, didAdd: [ARAnchor]) gets called when an anchor gets added to the AR session. Anchors can be added by calling session.add(anchor: newAnchor).

    Anchor Updated. func session(_: ARSession, didUpdate: [ARAnchor]) gets called when and anchor requires an update.

    Session Tracking State Change. func session(_ session: ARSession, cameraDidChangeTrackingState camera: ARCamera) provides the current state of the AR session. While the session is still being established, it will be in a Not available state. When there’s insufficient features available to track it, it will be in a Limited state. Finally, it will be in a Normal state if the session is actively running without any issues.

    Session Failure. func session(_: ARSession, didFailWithError: Error) provides session errors that may occur, allowing you to present the user with useful information. Or if you’d like, you can keep them guessing! Come on, you know that’s fun to do!

SceneKit Integration

ARKit seamlessly integrates into SceneKit. The best part: Apple made it so simple that even a cat can use it! -[:)]

There are a few things you need to be aware of when dealing with ARKit along with SceneKit. Just don’t share these with the cat. We need to hold them off as long as possible!

  • ARSCNView.

    So traditionally for SceneKit, you had to use an SCNView. There’s now a new view available for you to use: ARSCNView. It includes everything you’ve been using in SceneKit, but has been combined with the ARKit features. It contains an ARSession and an SCNView. All you need to do is provide it with an ARConfiguration and kickoff the ARSession.

  • ARSCNViewDelegate.

    This protocol provides all the basic session delegates from ARSessionDelegate along with a few delegates related to the SceneKit renderer.

    Let’s take a closer look at some of the available delegates:

    Node Added. func renderer(SCNSceneRenderer, didAdd: SCNNode, for: ARAnchor). Take special note that every anchor added to the current AR session will not only create an ARAnchor, but it will also automagically create an equivalent SCNNode within the SceneKit scene. This is all managed by ARKit internally. Simply add your new content to this new node… and your done. Bazinga! -[:)]

    Node Updated. func renderer(SCNSceneRenderer, didUpdate: SCNNode, for: ARAnchor) gets called when the SCNode’s properties were updated due to an update on the related SCNAnchor.

    Node Removed. func renderer(SCNSceneRenderer, didRemove: SCNNode, for: ARAnchor) gets called when the SCNode was removed due to removal of the related ARAnchor.

With all that finally out of the way, it’s safe to say – You now know ARKit! -[:)]

ARKit Demo Project

To end things off, here’s a fun little demo project you can mess around with. You’ll find the project under the resources/projects/ARDragon folder.

Run the project on your iPhone, point the camera at a table, and then sway it around a bit so the AR tracking kicks in. Then, tap anywhere on the screen.

Press play to find out what happens:


Whoa! There’s a friggin’ FIRE BREATHING DRAGON on my desk! Be careful and try not to get burned! -[:)]

When you’re looking through the project, take a look at these things:

  • Hit-Testing Feature Points

    When you tap anywhere in the scene, a hit-test is performed to find the closest feature point. Once found, that feature point location in space is used to create a new anchor.

    The code looks a little something like this:

    
    // Get the first touch location on screen
    if let touchLocation = touches.first?.location(in: sceneView) {
        
        // Perform hit-test against all feature points 
        if let hit = sceneView.hitTest(touchLocation, types: .featurePoint).first {
             
             // Add new anchor to AR session
             sceneView.session.add(anchor: ARAnchor(transform: hit.worldTransform))
        }
    }
    
    

  • New Anchors Creates New Nodes

    Remember, when you add a new anchor to the AR session, an equivalent SCNNode will be created. So all that needs to be done is to handle the delegate and add your own content to that SCNNode.

    The following code clones a dragon:

    
    // Delegate for SCNNode added
    func renderer(_ renderer: SCNSceneRenderer, didAdd node: SCNNode, for anchor: ARAnchor) {
            
        // Clone a new dragon
        let dragonClone = dragonNode.clone()
        dragonClone.position = SCNVector3Zero
            
        // Add dragon as a child of node
        node.addChildNode(dragonClone)
    }
    
    
  • Following The Camera

    Did you notice the dragon always follows you around the room as you move? That’s accomplished by adding a LookAt constraint on the SCNNode, using the camera’s point of view as a target.

    Here’s how you do that:

    
    // Create a LookAt constraint, point at the cameras POV
    let constraint = SCNLookAtConstraint(target: sceneView.pointOfView)
    
    // Keep the rotation on the horizon
    constraint.isGimbalLockEnabled = true
    
    // Slow the constraint down a bit
    constraint.influenceFactor = 0.01
    
    // Finally add the constraint to the node
    node.constraints = [constraint]
    
    
  • Light Estimation

    Did you notice when the light is switched off the dragon also dimmed to blend into the darker scene? There’s an ARConfiguration property available to control this: isLightEstimationEnabled. You don’t have to enable it, because it’s enabled by default. Just make sure you make use of PBR-based shading, and the rest will automagically work!

  • Hit-Testing Nodes

    When you tap on top of a dragon in the scene, a hit-test is performed to see if you tapped on an object. If so, that object is removed from the scene. See you later alligator… er, uh dragon.

    To do that, you can use the following bit of code:

    
    // Get the first touch location on screen
    if let touchLocation = touches.first?.location(in: sceneView) {
                
        // Perform hit-test against all objects in the scene
        if let hit = sceneView.hitTest(touchLocation, options: nil).first {
       
            // Remove the node
            hit.node.removeFromParentNode()
        }
    }
    

Next Steps

Well folks, that brings us to the end of this tutorial, we hope you enjoyed it, and we can’t wait to see what you create with ARKit and SceneKit. When you’re done, share your creations with us here at DayOfTheIndie.com.

If you want to learn more about SceneKit, be sure to check out my book, 3D Apple Games by Tutorials.

Stay tuned! And remember, if you’re not having fun… you’re probably not doing it right!

Chris Language

Co-Founder at Day Of The Indie
A long time ago, Chris fell in love. She wasn't the prettiest, but Chris loved her anyway. He was mesmerized. She taught him the true meaning of love. But, sadly, like most love stories, this one came to an end. One day, Chris simply walked out the door. He just left her there... waiting, wanting, and missing him. To this day, Chris regrets his foolishness, but he will never forget his beloved Commodore 64.

Member of the Apple Game Frameworks Team @ RayWenderlich.com

Author of 3D Apple Games by Tutorials.

Forever Coder, Artist, Musician, Gamer, Dreamer Indie!

3 thoughts on “Tutorial: Introduction To ARKit With SceneKit

  1. Hi, thanks for the excellent tutorial. What I don’t understand is why do we need to:
    sceneView.session.add(anchor: ARAnchor(transform: hit.worldTransform))
    then add the node via the delegate?

    Can’t we just do something like:
    node.position = SCNVector3(hit.worldTransform.columns.3.x, hit.worldTransform.columns.3.y, hit.worldTransform.columns.3.z)
    sceneView.scene.rootNode.addChildNode(node)

    1. Hi there, when adding an anchor point to the ARSession, you are making ARKit aware that it should keep track of that point in Real-World space. As an added bonus, the ARSession will create an empty SCNNode, linked to that anchor point. The delegate is simply used to get access to that newly added SCNNode, so that you can use it to add additional content to it.

      ARKit keeps track of anchor points, and it will make sure that the related SCNNode is kept in-sync with the anchor point transform as the device moves through real-world space.

      Nothing stops you from simply adding content to the scene. Just keep in mind that if it’s not linked to an anchor point, ARKit will not keep track of it.

      Glad you enjoyed the tutorial! I hope this explanation helped.

      Cheers! -[:)]

      1. Thanks for the reply. I can understand if the anchor point is a plane – as it grow as shrink – but if you are adding says a cube? In what way does the anchor point transform the cube? I am still trying to think of a scenario that needs ARKit to keep track of the cube?

        Thanks, I am a newbie so trying to grab the importance of ARAnchor (ARPlaneAnchor I can understand).

Comments are closed.