The first time people don the new HoloLens 2 on their heads, the device automatically gets to know them: It measures everything from the precise shape of their hands to the exact distance between their eyes.

The artificial intelligence research and development that enabled those capabilities “was astonishingly complicated” but essential to making the experience of using the device “instinctual,” said Jamie Shotton, a partner scientist who leads the HoloLens science team in Cambridge, United Kingdom.

“We want you to know how to use HoloLens without having to be taught how to use it,” he said. “We know how to interact with things in the real, physical world: We pick things up, we press buttons, we point to things. We aim, as far as possible, to translate that directly into mixed reality.”

Microsoft today announced the HoloLens 2 is now shipping to customers. The sensor-packed holographic computing headset uses AI to displace space and time, creating a mixed reality of people, places and things in order to facilitate one of our most basic human impulses: exchanging knowledge.

Microsoft Technical Fellow Alex Kipman said the headset defines the highest watermark of intelligent edge devices – AI-capable technologies that can collect and process data even without a reliable internet connection – and that can then share some or all of that data with the intelligent cloud when connected.

On a recent day, Kipman sketched a diagram of this ubiquitous computing fabric on a digital whiteboard in his office.

“HoloLens,” he said, “is the first native device to be invented from the ground up with this worldview in mind.”

The marriage of the AI in HoloLens 2 with the AI capabilities of Azure, Microsoft’s cloud computing platform, allows heads-up, hands-on workers to learn skills that advance their careers, and makes it possible for people on opposite ends of the Earth who speak different languages to collaborate with a shared sense of physical presence.

“You can do really interesting things with HoloLens, and you can do really interesting things with the cloud,” said Julia White, Microsoft corporate vice president of Azure marketing. “But when you see these two things come together, it changes the game in terms of what people can actually do.”

Delivering mixed reality with AI

To enable instinctual interactions with HoloLens 2, Shotton and his colleagues developed, trained and deployed AI models onto the device that track people’s hand motions and eye gaze so that, for example, they can perceive a hologram floating in front of them and reach out to resize it or reposition it.

To build the hand tracking system, the team built a rig with a dome of cameras pointing inward that they used to record a diverse range of people’s hands. Then the team used offline cloud processing to build a 3D model capable of representing all human hand shapes and motions.

From this 3D model, the team was able to use computer graphics to render realistic, synthetic images of hands along with synthetic labels to make the model robust across a variety of hand shapes, poses and movements.

“You can generate effectively unlimited quantities of training data,” Shotton said.

The team used this data to train a compact deep neural network, a type of AI algorithm, that fits on the HoloLens’s onboard processor and runs efficiently on every frame coming from the device’s depth sensor.

When a new customer puts on a HoloLens 2, the system uses this neural network to help fit a personalized 3D model to the customer’s hands, enabling the precise tracking required to allow instinctual interaction with holograms.

Shotton’s team took a similar approach to build and train the eye tracking model, paying close attention to what’s called interpupillary distance, or the distance between the centers of the pupils of the eyes, which varies across people and affects how a person sees near or distant objects.

The result is an eye tracking model that allows the HoloLens 2 to precisely display holograms in front of customers for interaction and manipulation with their hands and eyes.

“Without eye tracking, aligning holograms to the real world – especially the person’s physical hand – would just be impossible to the level of precision needed to allow instinctual interaction,” Shotton said.