Tutorial: 3D Picking & Dragging With SceneKit

Tutorial: Learn how to 3D pick and drag SceneKit objects by converting 2D coordinates into 3D coordinates, and 3D into 2D.

scenekit


First Steps

Now that everyone’s making cool ARKit applications using SceneKit, it’s important to understand how to reach out and touch virtual objects in the world. This is where the ability to pick and drag objects comes in handy!

3D Picking

We’re all familiar with nose-picking, right? Heck, some of us may even be at an expert level. -[:)] But what the heck is 3D picking?

3D picking is the ability to “pick” or select an object in 3D space, from a 2D viewport, like a flat screen on a device. The main problem with that is dealing with two different coordinate systems.

When you touch the display, your screen can only provide an X and Y coordinate. That 2D coordinate has to be converted into a 3D coordinate. Think of it like shooting a virtual ray into the 3D scene using the 2D coordinate from the camera’s point of view. To do that in SceneKit, you need to perform a basic hit-test. When the ray “hits” a 3D object, you can use the coordinates of that 3D object as the location you’re making contact with in the 3D scene.

Project & Un-project

Ok, great! You’re making contact with an object in 3D space. But what if you want to move that object around?

Because you’re only access into this 3D world is through the 2D display, you’ll need some way to convert the 2D coordinates into 3D. You might be thinking of using a hit-test again, but that won’t help here. The solution is to make use of a technique known as project and un-project.

    scenekit

  • Project

    Project shoots a ray, in reverse, from 3D space back to the 2D camera viewpoint. It essentially provides you with the 2D screen coordinate of the 3D object in the scene. Not only does it provide the 2D screen coordinates, it also provides the distance/depth of the 3D object from the camera’s view point.

    The depth of the object depends on the camera’s viewing frustum. The provided value will range from 0.0 to 1.0, where 0.0 is equal to your camera’s zNear frustum and 1.0 is equal to zFar.

  • Un-Project

    Un-project is used to convert a 2D coordinate into a 3D coordinate. You need to provide the 2D coordinates and also the depth at which you want your coordinate converted to.

Demo Project

Now that you know all there is to know on the subject, grab yourself a copy of the demo project.

In summary—with a few code extracts—to perform basic picking and dragging you need to do the following:

  • Pick.

    On touchesBegan(), do a hit-test to determine which object in 3D space was touched.

  • 
    override func touchesBegan(_ touches: Set<UITouch>, with event: UIEvent?) {
      let touch = touches.first!
        if let hit = sceneView.hitTest(touch.location(in: sceneView), options: nil).first {
          selectedNode = hit.node
          ...
    }
    
  • Project.

    Use projectPoint() on the touched object to project the 3D coordinates into 2D coordinates. Basically, getting the objects depth from the camera.

  • 
    zDepth = sceneView.projectPoint(selectedNode.position).z
    
  • Un-Project.

    On touchesMoved(), use unprojectPoint() to convert the 2D touch coordinates, along with the stored depth value, back in to a proper 3D coordinate. Simply update the 3D object position with this new coordinate. Boom!

  • 
    override func touchesMoved(_ touches: Set<UITouch>, with event: UIEvent?) {
      guard selectedNode != nil else { return }
      let touch = touches.first!
      let touchPoint = touch.location(in: sceneView)
      selectedNode.position = sceneView.unprojectPoint(
        SCNVector3(x: Float(touchPoint.x),
                   y: Float(touchPoint.y),
                   z: zDepth))
    }
    
Next Steps

Well folks, that brings us to the end of this quick tutorial. We hope you enjoyed it! Now, go forth and touch and drag some objects! -[:)]

Want to learn more about making games with SceneKit? Now’s your chance.

Go check out my book, 3D Apple Games by Tutorials, which was recently updated to work with Swift 4 and Xcode 9.

Stay tuned! And remember folks, if you’re not having fun… you’re probably not doing it right!

Chris Language

Co-Founder at Day Of The Indie
A long time ago, Chris fell in love. She wasn't the prettiest, but Chris loved her anyway. He was mesmerized. She taught him the true meaning of love. But, sadly, like most love stories, this one came to an end. One day, Chris simply walked out the door. He just left her there... waiting, wanting, and missing him. To this day, Chris regrets his foolishness, but he will never forget his beloved Commodore 64.

Member of the Apple Game Frameworks Team @ RayWenderlich.com

Author of 3D Apple Games by Tutorials.

Forever Coder, Artist, Musician, Gamer, Dreamer Indie!