Astronauts Are Getting a ‘Holographic Instruction Manual’

Astronauts on the International Space Station will soon get a boost from augmented reality.

This week, NASA is sending a pair of HoloLens devices, which allows the user to place digital objects in the world around them, to the ISS as part of the next commercial resupply mission.

The crew aboard the space station has a great deal of work to do on a daily basis, most of which involves keeping the station running. To do this, they must receive a lot of instruction from people on the ground. That’s where the HoloLens comes in. NASA teamed up with Microsoft, the maker of the HoloLens, on a “holographic instruction manual,” in the words of Jeff Norris of NASA’s Jet Propulsion laboratory, who lead the project along with Alex Kipman, a Microsoft technical fellow.

At first glance, HoloLens resembles the typical virtual reality device, much like Oculus Rift. However, the two technologies are very different. HoloLens is basically a pair of see-through glasses that runs on Windows 10, displaying mixed or “augmented” reality. This means that when wearing the device you see holograms mixed in with the real world.

Alex Kipman discusses Microsoft’s partnership with NASA on HoloLens. Photo: Amy Thompson

“As opposed to virtual reality which removes you from the real world, with HoloLens the real world plays just as much into the experience as the digital assets,” Kipman said. “We’re adding photons to the back of your eye. They feel anchored and pinned in the real world, but are see through.”

HoloLens features two different modes of operation: remote expert mode and procedure mode. In remote expert mode, ground control teams are able to see through the eyes of the astronaut.

“It is operationally significant to see an astronaut’s perspective as they work,” Norris said. “Historically this is not a perspective that’s been easy to get until now.”

Ground crews can also draw annotations, like which button to push, onto the world the astronaut sees while talking with them via Skype.

Astronauts are also responsible for carrying out hundreds of research experiments, as the space station is a massive orbiting laboratory. With the help of HoloLens, the astronauts can be connected directly to leading experts or scientists and get the help they need with the remote expert mode.

Procedure mode is basically a holographic instruction manual. Back in the shuttle era, crews relied on paper manuals to help figure out certain tasks. Today, those manuals have been replaced by laptops and tablets. HoloLens goes above and beyond today’s computers by placing animated holograms on top of what an astronaut is working on.

Microsoft and NASA spent a lot of time developing this product and ensuring that the user would not experience any vestibular discomfort—like nausea, headache, etc.—as can sometimes happen with virtual reality platforms. As a result, they have a product which stabilizes holograms in both space and time, ensuring all of the user’s senses are rooted in the real world. This means the holograms are not jittery or flickering, allowing the user to see them just as they would in the real world. The device is designed to be worn several hours a day, several days in a row, meaning your eyes will never strain or fatigue.

HoloLens uses spatial mapping to determine where people are in relation to their environment. HoloLens has a holographic processing unit (HPU) in the headset that works much like a human brain. Like the brain, HoloLens is equipped with sensors that work like the visual cortex. The HPU then processes the incoming data, automatically making adjustments to ensure the user sees the digital objects just as they would if they were real.

Over 45 research experiments on HoloLens were conducted in microgravity on NASA’s zero-g aeroplane, the Weightless Wonder, to ensure the device would be able to operate in microgravity.

HoloLens is set to fly on the newly enhanced Cygnus spacecraft as it returns to flight following a fiery explosion in October of last year. This flight marks the first time since June that an American cargo ship will launch to the space station. HoloLens was originally scheduled to launch on a SpaceX Dragon spacecraft as part of the company’s 7th commercial resupply mission back in June, but the rocket disintegrated during launch and all the cargo was lost.

This may be just the start of HoloLens in space. JPL and Norris have also developed project OnSight, which allows scientists and engineers to work side-by-side with the Mars rovers on the surface of Mars. By donning a HoloLens, scientists on Earth will be able to explore Mars from the comfort of their offices.

Project OnSight. Photo: NASA

Historically, scientists have explored Mars by looking at images on a screen, now with the help of holographic computing, they will be able to explore Mars much like geologists would on Earth. Surrounded by the Martian landscape, scientists can virtually examine rocks and other surface features, then program tasks for specific instruments to carry out.

Not only will this type of technology streamline station processes and allow astronauts to develop new ways to complete complex tasks, it will also help transform the way we explore the Solar System.