Home System concept Augmented reality, superhuman abilities and the future of medicine

Augmented reality, superhuman abilities and the future of medicine


Missed a GamesBeat Summit 2022 session? All sessions are available to stream now. Look now.

Earlier this month, I participated as a panelist at the Digital Orthopedics Conference in San Francisco (DOCSF 2022) where a major theme was imagining the medical profession in the year 2037. In preparation for the event , a small group of us reviewed the latest research on the clinical uses of virtual and augmented reality and critically assessed the current state of the field.

I have to admit that I have been deeply impressed with the progress Augmented Reality (AR) has made over the past eighteen months for use in medicine. So much so that I don’t expect we’ll have to wait until 2037 for AR to have a major impact on the ground. In fact, I predict that by the end of this decade, augmented reality will become a common tool for surgeons, radiologists, and many other healthcare professionals. And in the early 2030s, many of us will go to the family doctor and be examined by a doctor wearing AR glasses.

The reason is simple:

Augmented reality will give doctors superpowers.

I’m talking about superhuman abilities to view medical images, patient data, and other clinical content. The costs associated with these new capabilities are already quite reasonable and will rapidly decrease as augmented reality hardware is produced in higher volumes in the years to come.

The first superpower is x-ray vision.

Visualization and augmented reality concept.

Augmented reality will give doctors the ability to peer directly into a patient and see signs of trauma or disease in the exact spot on their body where they reside. Of course, the ability to look under the skin already exists with tools like CT and MRI, but currently doctors view these images on flat screens and have to imagine how the images relate to the patient on the table. This kind of mental transformation is an impressive skill, but it takes time and cognitive effort, and isn’t as informative as it would be if doctors could just look inside the human body.

With AR headsets and new techniques for recording 3D medical images on a patient’s real body, the superpower of X-ray vision is now a reality. In an impressive study from Teikyo University School of Medicine in Japan, an experimental emergency room was tested with the ability to capture full-body CT scans of trauma patients and immediately allow the medical team , all wearing AR headsets, to look at the patient on the exam table and see the trauma exactly where it resides. This allowed the team to discuss injuries and plan treatment without having to refer to flat screens, saving time, reducing distractions and eliminating the need for mental transformations.

In other words, AR technology takes medical images off the screen and places them in 3D space exactly where they are most useful to doctors – perfectly aligned with the patient’s body. Such a capability is so natural and intuitive that I predict it will be quickly adopted in all medical applications. In fact, I expect that in the early 2030s doctors will see the old way of doing things, peeking at flat screens, as clumsy and primitive.

Going beyond x-ray vision, augmented reality technology will provide physicians with assistive content superimposed on (and in) the patient’s body to aid in clinical tasks. For example, surgeons performing a delicate procedure will receive navigation cues projected onto the patient in real time, indicating exactly where procedures need to be performed with precision. The goal is to increase accuracy, reduce mental effort and speed up the procedure. The potential value of surgery is extreme, from minimally invasive procedures such as laparoscopy and endoscopy to freehand surgical efforts such as placing orthopedic implants.

The concept of augmented surgery has been an aspiration of AR researchers since the basic technologies were first invented. In fact, it dates back to the first AR system (the Virtual Fixtures Platform) developed at the Air Force Research Laboratory (AFRL) in the early 1990s. The goal of this project was to show that AR could boost dexterity human in precision tasks such as surgery. As someone involved in this early work, I must say that the progress the field has made over the decades is remarkable.

Consider this – when testing that first AR system with human subjects in 1992, we asked users to move metal pegs between holes two feet apart to quantify whether virtual overlays could improve handheld performance. Now, thirty years later, a team from Johns Hopkins, Thomas Jefferson University Hospital and the University of Washington performed delicate spine surgery on 28 patients using AR to help position in place of metal screws with an accuracy of less than 2 mm. As published in a recent study, the screw placement system achieved such an accurate registration between the real patient and virtual overlays that surgeons achieved 98% on standard performance measures.

In the future, we can expect augmented reality to impact all aspects of medicine as accuracy has reached clinically viable levels. Additionally, major breakthroughs are underway that will make it faster and easier to use AR in medical settings. As described above, the biggest challenge for any precision augmented reality application is the accurate recording of the real world and the virtual world. In medicine, this currently means attaching physical markers to the patient, which takes time and effort. In a recent study from Imperial College London and the University of Pisa, researchers tested a “markerless” AR system for surgeons that uses cameras and AI to precisely align the real and virtual worlds. . Their method was faster and cheaper, but not as accurate. But that’s just the beginning – in the coming years, this technology will make AR-assisted surgery viable without the need for expensive markers.

Additionally, camera-based recording techniques will take AR out of highly controlled environments like operating rooms and into a wider range of medical applications. In fact, I predict that by 2030, GPs will commonly see patients with the benefits of AR headsets.

This brings me to another superpower that I expect doctors to have in the near future – the ability to go back in time. Doctors will be able to capture 3D images of their patients using AR headsets and later view these images aligned with their patient’s body. For example, a doctor could quickly assess the healing progress of a skin lesion by viewing the patient through AR glasses, interactively scanning in time to compare the current view with what the lesion looked like during previous visits.

Overall, the advances made by researchers in the medical uses of virtual and augmented reality are impressive and exciting, with important implications for both medical education and medical practice. To quote Dr. Stefano Bini of UCSF’s Department of Orthopedic Surgery, “The beneficial role of augmented reality and virtual reality in healthcare workforce development cannot be overstated.”

I agree with Dr. Bini and would go even further, as I see augmented reality impacting the workforce far beyond healthcare. After all, the superpowers of x-ray vision, navigational landmarks, dexterity support and the ability to look back in time will come in handy for everything from automotive construction and repair to engineering, manufacturing , agriculture and of course education. And with AR glasses being developed by some of the biggest companies in the world, from Microsoft and Apple to Meta, Google, Magic Leap, HTC and Snap, these superpowers will almost certainly arrive at mainstream consumers within the next five to ten years, improving all aspects of our daily lives.

Louis Rosenberg, PhD is CEO and Chief Scientist of Unanimous AI and has been awarded over 300 patents for his work in VR, AR and AI.


Welcome to the VentureBeat community!

DataDecisionMakers is where experts, including data technicians, can share data insights and innovations.

If you want to learn more about cutting-edge insights and up-to-date information, best practices, and the future of data and data technology, join us at DataDecisionMakers.

You might even consider writing your own article!

Learn more about DataDecisionMakers