What if medical students could project bear-sized holographic brains in their dorm rooms when studying for an anatomy exam? What if surgeons could use those holograms to prepare for complex surgeries or even support the operations themselves? HoloLens appears to be the platform for the medical application of mixed reality, so I was curious how it performs and whether it is already fit for healthcare. Check out my HoloLens review below!
An average day of Matt Woratzki in 2056 – Or how to imagine the future with HoloLens!
At 6.30 in the morning, the smart sleep alarm smoothly indicated that it’s time to wake up. Matt was almost half-awake anyway. He got out of bed, had his morning smoothie (his GP told him to avoid coffee after his genetic test showed some risks for high blood pressure), dressed up and went for a morning run. The sensors in his active wear and the Fitbit on his wrist measured every vital sign and health data, the optimal run time and speed.
After he got home, he realized that a couple of messages are buzzing on his HoloLens. Today, he had to prepare for a complicated brain surgery, and his colleague was already at the hospital waiting for his input. He sent over some MRI images, where a smart algorithm indicated the problematic areas for discussion. Matt didn’t want to check the messages in a hurry, so instead, he got into his Tesla, and looked at the images carefully while his car drove him to his workplace.
HoloLens for work and playtime
The colleague was already waiting for Dr. Woratzki. He and the team all had their HoloLenses on, and the holographic image of the patient’s brain was vibrating in the room in the size of an elephant to clearly see where an intervention had to be made. After the consultation, Matt had some patient visits, where the administration was taken care of by the hospital’s EHR system supported by voice recognizing smart algorithms and his Hololens. After lunch with colleagues, he had a walk in the green area of the hospital, where he could figure out the details of the operation he will do the next day.
When he got back, the surgical robot was ready for practicing the right moves and he also had some words of wisdom for the med students trying to learn the tricks of the trade next to him. After reading the latest study in his field and visiting more patients, he had to pick up his son from school. His wife was abroad at a business meeting, but he didn’t mind spending some time with him, so when they got home, they joked about the teachers and played some Minecraft via HoloLens together. Matt went to bed thinking about how lucky he is with his life.
Augmented, virtual and mixed reality
With the development of augmented reality (AR), virtual reality (VR) and mixed reality (MR) and their healthcare applications, we are closer to the above scene as you might believe. The AR/VR device markets are skyrocketing. According to the latest forecasts, the AR device market is expected to reach $659.98 million by 2018, while its counterpart, VR is also expected to boom in the next couple of years.
The three technologies are often mixed up, although there are significant differences. While AR lets users see the real world and projects digital information onto the existing environment, VR shuts out everything else completely and provides an entire simulation. It is a logical consequence that VR is more immersive. Mixed reality is closer to AR in a sense that it also projects synthetic content on the environment that is anchored in reality. However, unlike AR, MR interacts with the world. It means that while AR will soon be able to project the price of an apartment in the building in front of me, MR first senses what is around and projects the requested data adjusted to the given environment.
Google Glass versus Microsoft HoloLens
The most widely used platform for AR is the well-known Google Glass, while mixed reality is paired with Microsoft HoloLens. The latter is the first self-contained, holographic computer which enables the user to engage with digital content and interact with holograms around. Kinect, an add-on for Microsoft’s Xbox gaming console that was introduced in 2010, could be considered as the predecessor of HoloLens, while the pre-production version was first shipped in 2016. Developers and testers might buy the HoloLens for more than $3000 which is definitely not for the purses of the earthborn, average geeks, but rather for companies and tech-savvy millionaires.
There are significant differences between Google Glass and HoloLens. When I tried the Google Glass wearable computer with the optical head-mounted display, I saw a simple „window” in front of me wherever I looked. I was able to take pictures, make videos, carry out web searches and use voice commands. Microsoft HoloLens was different as it scanned my environment first and I could actually place the appearing „window” on the objects around me. The device interacts with the environment, and unlike with Google Glass, it matters what is around.
The question might arise whether a third competitor could appear on the mixed reality market next to Microsoft and Google. In June 2017, TechRadar reported that Apple might prove as a contender. At WWDC 2017, the Cupertino company revealed ARKit, an app that helps developers create AR experiences, replete with “fast and stable motion tracking.” However, there was no news about a possible AR headset from Apple down the line.
HoloLens in medical use
As augmented reality, mixed reality has a bright future in healthcare. HoloLens opens up radically new ways for medical education as it is able to project the human body in its full size in front of med students. Thus, the organs, veins or bones will be visible accurately in 3D, and future medical professionals will be able to analyze their shape, remember their characteristics more vividly than it is possible when studying from a book. There are already some universities who plan to introduce the new technology: Case Western opens its new health education campus in collaboration with the Cleveland Clinic in 2019, where students won’t learn anatomy from cadavers either, they’ll learn it from virtual reality.
Another way of making use of HoloLens is applying it in the pre-operative planning phase of operations. Physicians could plan their entire intervention using 3D holograms, where they could accurately see the spaces for making incisions and also clearly envision the consequences of their moves. There are some hospitals where HoloLens is already in use for planning surgeries. Researchers in Oslo have developed a way of turning traditional two-dimensional medical images into 3D augmented-reality models for planning surgery and navigating around organs during operations. Moreover, HoloLens could also find its way into the OR. Medical technology firm Scopis has created the first mixed-reality interface for surgeons on Hololens, primarily for open and minimally invasive spinal surgeries. Scopis’ AR-powered tech claims to improve the accuracy and speed of surgeons wearing the HoloLens by showing precise angles and positions of equipment.
Knowing about so many mind-blowing innovations, I was really curious how the HoloLens works in practice, whether the average doctor could sense the magnificent technology right away and whether it could be already used in practice.
Imitating the RoboCop
When I put on the HoloLens, I immediately felt the power of technology. I was glad the huge, black goggles did not exclude everything else, though, but I found myself excited to experience something brand-new. The HoloLens itself is unusually heavy, but you can adjust to the extra weight fairly fast. The goggles can be adjusted with a wheel at the back of the headband, which also helps distribute the weight of the unit equally for comfort. The design reminds me of the RoboCop, the futuristic movie from 1987, which is awesome, but I slightly hope that in the future, the headband will weigh less and becomes more transparent.
I didn’t wear my glasses with the HoloLens, I did not find space to fit them into the headgear, although I would appreciate if I could see my surroundings more clearly also when using the HoloLens. It is great, though, that it allows me to still see my environment clearly which made me easy and spared me a headache. As digital information (an actual window) only occupied a fraction of my sight, therefore, I could walk around easily. The weight of the device was a bigger obstacle in this. The side effects I experienced after the heavy use of VR was missing with Hololens. The challenge was getting used to the place of the window in my sight and the way it interacts with the environment while I moved my head.
The device looks and feels like something precious and premium, to be handled with utmost care. As mixed reality alongside with augmented reality moves into more and more sectors, such as architecture, construction, tourism or real estate, it is time to think about whether these gadgets could be used outdoors. I do not see the application of HoloLens on construction sites yet, for me, it seems to be rather delicate for it, but the future might hold a more robust device for that sector.
Open, Sesame! Show me the future through HoloLens
The gadget has a fairly good battery life: you can use the HoloLens for 3-4 hours without plugging in. The device itself is very user-friendly, as you only have to press the power button at the end of the left arm alongside a row of five, small individual LED nodes to indicate system status.
HoloLens uses voice commands with the help of Microsoft’s digital assistant, Cortana. So, after the device connected to the nearby WiFi network, and saying „Hey, Cortana”, you can search the web or start certain apps. The gadget uses speakers which do not obstruct external sounds to allow you to hear both virtual and natural sounds. You are able to command the entire system through various hand gestures, but a clicker is also available for your use. After I got used to the clicking hand gesture, interacting with the apps became quite straight-forward. It felt like using Microsoft Windows without a mouse or a keyboard. If I wanted to talk to someone while having the Hololens on, I could easily do that as I could keep eye contact and only look at my window when I wanted to.
When I tried apps about anatomy, I saw the app launching in my window and then a 3D anatomy structure appeared in front of me. I could walk closer, zoom in and out, click on parts of it to learn more and it felt like I’m looking at a real anatomy structure. It took time to acknowledge that this is a hologram no matter how realistic it looks. It was simply amazing!
Verdict: HoloLens on the right track into the future
Although Microsoft HoloLens is far from perfect, and the Redmond Headquarter has to work on a lot to improve its current version, the gadget is on the right track to becoming a significant medical device in the future. Right now, not only its price, but also its technical characteristics make it unfit for commercial use, but I’m sure that this is an appropriate basis for the next generation of smaller, smarter and more resistant HoloLens headbands or some similar devices such as the Meta family.
There is also no question about mixed reality’s place in healthcare – but certainly not with the present devices. If physicians had to use Hololens today, they would get to hate mixed reality forever. Yet, I’m sure we don’t have to wait long before amazing HoloLens innovations overswarm medicine. Beyond the listed fantastic innovations in medical education and surgical planning, there are many small teams working on HoloLens to figure out solutions which could make healthcare better. For example, Dr. Bence Németh working for Immed.AI, a Hungarian start-up, told me during our Facebook Live session on HoloLens that they work on augmented and mixed reality applications that could facilitate the work of radiologists by showing them the results in front of their eyes instead of the need to look at a screen.
Needless to say that I’m already excited about all the future HoloLens applications and I look forward to testing the future generations of such devices.