Apple's Bet On AR & The Future of UI Design

AR vs. VR

First, let’s break down the difference between augmented reality (AR) and virtual reality (VR). AR is a technology that layers computer generated enhancements atop an existing reality in order to make it more engaging. VR is an artificial computer-generated simulation or recreation of a real life environment.

Image Credit: The Verge and IFA

With the iPhone 8 release on the horizon, the potential for mobile augmented reality design has never been greater. Rumors are circulating about 3D sensing modules, a rear-facing VCSEL, and iris scanning. Snapchat and Pokemon Go are redefining the potential for AR design in mobile apps. Google has already released the Daydream View and Google Cardboard, where you can hook your phone up to a VR headset. Even Amazon is bouncing between the two, testing out AR & VR in their e-commerce and retail stores.

Google Cardboard, Image Credit: Google

While it appears that Apple has let Microsoft, Google, and Amazon lead the way in the past few years, Apple filed their first patent for a “hyper reality” product in 1996 and are rumored to have a secret research unit with hundreds of employees working on AR and VR.

“I think AR is big and profound. This is one of those huge things that we’ll look back at and marvel at the start of it.” — Tim Cook, CEO of Apple

They’ve also acquired multiple AR/VR companies in the past year. Apple even released ARKit in iOS 11, a new framework that allows developers to create augmented reality experiences for iPhone and iPad.

But how does this redefine design? It’s easy to say this is a defining moment because the tools we use to design interfaces will shift. But AR design has more potential than just layering a virtual filter on top of our faces in Snapchat. It’s shortsighted to think that the phone is where the AR frontier begins and ends.

As the home of 2D design becomes one of the first vessels for virtual and augmented reality, designers will have to re-evaluate the playing field.

UI Design Will Evolve

As humans increasingly let technology into more areas of their lives, their interactions with technology will begin to evolve. It’s not too far off that we may live a life without the screens we know now. AR will become an integrated part of our environment, with the information we need placed just at the right position.

Data currently belongs inside a box within a screen, and designers must produce new and different ways to visualize that data in 2D. With AR, data becomes a texture layered into your world, personalized for your own experience.

Video Credit:Hyper Reality

Controls & Interactions:
Currently, we control our interfaces with the touch of a finger or a cursor on the computer. What will this look like in AR?

It could be with the wave of the user’s hand, like in the gif above. Some companies are working on tracking the user’s eye movements, battling to overcome human nature of distraction. Then there’s the element of voice, which could be used to issue commands to the AR device.

Position & Environment:
Something that will be new in AR design is the position of the user in their environment. How does AR take into account the user’s height, the lighting, the environment, and the location of other users?

A person who is 6’2” will undoubtedly see the world differently than a person who is 4’9”. Compare the experience of a grown man to a child or a person with a disability. These are design problems designers already encounter, but the solutions will deal with a medium entirely new to us.

2D inside a 3D Interface:
AR is layering a 2D UI in the user’s reality. The normal problems, like readability, contrast, and information hierarchy, just became much harder to solve. For the mobile phone, the most extreme environmental conditions most designers deal with are bright daylight and complete darkness.

With AR, designers will have to think about the environment in an entirely new way. There could be rain, high speed travel, crowds, and more in the users view, and designers will have to come up with something that can withstand them all.

Ads & Data Collection

Augmented reality will allow advertisers to personalize an ad experience like never before. Let’s imagine you’re wearing a pair of AR glasses. It starts with your morning routine, when your AR assistant realizes you are low on toothpaste, and 3 different ads for toothpaste brands appear before you as you brush your teeth. You shake your head in annoyance, and the ads disappear noting your “disinterest”.

Image Credit: Imgur and Business Insider

On the left is the controlled simplistic version of ad displays in AR. With so much data collection and so little regulation, it’s entirely possible that ads can take over the user’s screen, overwhelming their visual field like the example on the right. Back in the early 2000s during the rise of the .com bubble, one type of advertising took over. Pop-up ads.

Now, I’d like to say we’ve learned from the past. But the reality is, the first pop-up ad was probably wildly successful. By the 50th, they started to lose traction. In AR, the same rules will apply. The first type of any new advertisement experience will be successful. The 50th of the same ad type will become a nuisance.

Until there are defined systems for how AR functions, the right version above is much more likely than the left. Once these boundaries are set, it could be anything from physical districts and buildings to a pay-per-month model that determines how many ads you see. Once thing is for sure though, the physical world is about to become our digital playground and designers will have the first go at shaping this new world.