3D Picking & Dragging with SceneKit – FREE

First Steps

Now that everyone’s making cool ARKit applications using SceneKit, it’s important to understand how to reach out and touch virtual objects in the world. This is where the ability to pick and drag objects comes in handy!

3D Picking

We’re all familiar with nose-picking, right? Heck, some of us may even be at an expert level. -[:)] But what the heck is 3D picking?

3D picking is the ability to “pick” or select an object in 3D space, from a 2D viewport, like a flat screen on a device. The main problem with that is dealing with two different coordinate systems.

When you touch the display, your screen can only provide an X and Y coordinate. That 2D coordinate has to be converted into a 3D coordinate. Think of it like shooting a virtual ray into the 3D scene using the 2D coordinate from the camera’s point of view. To do that in SceneKit, you need to perform a basic hit-test. When the ray “hits” a 3D object, you can use the coordinates of that 3D object as the location you’re making contact within the 3D scene.

Project & Un-project

Ok, great! You’re making contact with an object in 3D space. But what if you want to move that object around?

Because your only access into this 3D world is through the 2D display, you’ll need some way to convert the 2D coordinates into 3D. You might be thinking of using a hit-test again, but that won’t help here. The solution is to make use of a technique known as project and un-project.

  • Project

    Project shoots a ray, in reverse, from 3D space back to the 2D camera viewpoint. It essentially provides you with the 2D screen coordinate of the 3D object in the scene. Not only does it provide the 2D screen coordinates, it also provides the distance/depth of the 3D object from the camera’s view point.

    The depth of the object depends on the camera’s viewing frustum. The provided value will range from 0.0 to 1.0, where 0.0 is equal to your camera’s zNear frustum and 1.0 is equal to zFar.

  • Un-Project

    Un-project is used to convert a 2D coordinate into a 3D coordinate. You need to provide the 2D coordinates and also the depth at which you want your coordinate converted to.

Demo Project

Now that you know all there is to know on the subject, grab yourself a copy of the demo project.

In summary—with a few code extracts—to perform basic picking and dragging you need to do the following:

  • Pick.

    On touchesBegan(), do a hit-test to determine which object in 3D space was touched.

    override func touchesBegan(_ touches: Set, with event: UIEvent?) {
      let touch = touches.first!
        if let hit = sceneView.hitTest(touch.location(in: sceneView), options: nil).first {
          selectedNode = hit.node
  • Project.

    Use projectPoint() on the touched object to project the 3D coordinates into 2D coordinates. Basically, getting the objects depth from the camera.

    zDepth = sceneView.projectPoint(selectedNode.position).z
  • Un-Project.

    On touchesMoved(), use unprojectPoint() to convert the 2D touch coordinates, along with the stored depth value, back in to a proper 3D coordinate. Simply update the 3D object position with this new coordinate. Boom!

    override func touchesMoved(_ touches: Set, with event: UIEvent?) {
      guard selectedNode != nil else { return }
      let touch = touches.first!
      let touchPoint = touch.location(in: sceneView)
      selectedNode.position = sceneView.unprojectPoint(
        SCNVector3(x: Float(touchPoint.x),
                   y: Float(touchPoint.y),
                   z: zDepth))

Next Steps

Well, folks, that brings us to the end of this quick tutorial. We hope you enjoyed it! Now, go forth and touch and drag some objects! -[:)]