Take a closer look at how Apple is leveraging familiar gestures for new interactions with Vision Pro

Apple has put a lot of thought into how users interact with the Vision Pro. Traditional inputs from a keyboard, trackpad, controller, and voice are all options, but Apple really wants its users to use their eyes and hands with its upcoming device, creating a more personal and connected experience. Of course, there are a lot of unanswered questions at this point, and there have also been a lot of people chiming in with their thoughts on the device. So it’s always interesting to get a perspective from someone who’s invested heavily in the space, like Andrew Hart, founder of Hyper, a company that creates indoor augmented reality experiences for retail.

Hart shared his views on the Vision Pro, Post in a thread What he found interesting. With Vision Pro, Apple creates new forms of interaction for its unique product, but this does not mean that it needs to move away from the actions that we are used to every day using computers and smartphones. In order to create a sense of familiarity with the Vision Pro, Apple makes use of these gestures so that users can focus more on the experience and less on the way they interact. By using familiar gestures, like pinch and pinch to zoom, Apple is able to reduce the learning curve for the new experience. But what really makes this work so well is that the experience is so well done that interactions and actions feel seamless.

In the developer session, an example shows how a user can zoom in on an image simply by looking at an area, then pinching with a hand gesture. Interaction comes naturally if you’ve ever used a smartphone; Best of all, it doesn’t take you out of the experience. Apple also highlights how important it is for developers to keep objects centered in the frame to reduce eye and neck strain. But it also shows how elements in the terminal can be used with actions that can quickly move a writing instrument from one area to another with just a simple click of the user, much like how it works on a PC or smartphone.

But from what we’ve seen and heard from those who’ve tried it, Apple got things right, and this way of interacting is unlike anything we’ve seen before. Our very own Ben Sin was present at WWDC and got hands-on time with Vision Pro, sharing his thoughts, and discussing how easy it is to navigate the user interface using his eyes and hands. Perhaps an important difference with Vision Pro compared to other products that came before it is that these types of interactions felt seamless, giving users familiar tools but new ways to interact with content.

Apple understands that any interaction with the hands will lead to fatigue and recommends minimizing this as much as possible. But sometimes these types will be needed to create a better experience. While these types of gestures are important, Apple also has built-in accessibility controls that let users select the action with their eyes if needed. Perhaps one of the biggest caveats with Apple’s Vision Pro is that it can’t provide any kind of haptic feedback because it doesn’t use physical controllers. Now, this is a big problem because when we usually interact with things in the real world, we don’t just see them, we feel them as well.

This is perhaps one of the tricky hurdles when building objects in a digital space, and Apple is discussing overcoming this problem by compensating with other types of sensory information. A good example of this is the keyboard in VisionOS, where Apple lifts the keys off the base of the numeric keypad to create a sense of depth. Apple also benefits from lighting, as keys are highlighted when you hover a finger over them. The light becomes brighter as the finger approaches and dimmer as it moves away. When the user interacts and presses a key, there is a visual indication that the key is being pressed, which is also accompanied by a sound that is recorded as confirmation of the press.

For the most part, this is just a small part of what to expect when Apple finally releases its new headset. The company highlighted the available hardware, software, and tools. From here, it will be up to the developers to create exciting apps to get people hooked. This is a highly specialized product that costs a lot of money. While no one expects this to be an overnight success, many other companies have tried to offer similar products and they are yet to gain traction. Will Apple be the first to change this product category and make it popular, as it has done in the past with many other products? Only time will tell, but if all of this sounds interesting, I’d recommend checking out the developer session For full details on how Apple uses gestures.