The immersive school bus: Hubs-built journeys into the body for medical education

Kristen Ramirez and Greg Dorsainville of the NYU Grossman School of Medicine write about how instructors use Mozilla Hubs, a VR platform that’s accessible and private by design, to teach students about the human body.

Kristen Ramirez is a research Instructor at NYU Grossman School of Medicine. She is a human evolutionary biologist by training and anatomist by trade, publishing on the evolution of human joints and advances in anatomy medical education. Over the last two years she has introduced the use of immersive XR and VR teaching tools to the anatomy curriculum. 

Greg Dorsainville explores how new media and immersive computing can unlock the potential of both educators and learners at the NYU Grossman School of Medicine’s Institute for Innovations in Medical Education. His focus is on how XR can expand the opportunity for experiential active learning experiences.

The kid who sat wide-eyed every Saturday morning as Miss Frizzle loaded her class into the magic school bus has grown into today’s medical student. They have been primed to learn from engaging, demonstrative and immersive visualizations pioneered by PBS’s first-ever fully animated series.

Now, these budding doctors are entering into a medical field shaped by the recent revolution in minimally invasive procedures and advances in scanning techniques and technologies. A physician’s anatomical knowledge is increasingly being put to use beyond the physical exam and operating room, meaning students need to be prepared to view the body in novel ways. While they won’t be shrinking down to microscopic scales and sailing through the gut anytime soon, advances in medical technologies will connect doctor and patient in yet unseen ways.

Instantaneous renderings of 3D digital models of patients, pin-head cameras and externally controlled surgical tools were once the musings of science fiction. Not anymore. These medical advances are appearing in conjunction with new commercial technologies for entertainment and education alike. Studies have correlated video game playing with better performance on laparoscopic surgery skills. Now, adoption of various eXtended Reality (XR) teaching and training resources is occurring at medical schools and teaching hospitals across the country.

‘To the bus!‘

In XR, the magic happens through the technology used to access experiences combining real and virtual elements. These experiences extend the sensory perception of the user beyond their physical environment into a virtual space. This occurs along a spectrum of real-to-digital interaction ranging from single sense inputs like adding a cartoon filter on Facetime or Snapchat to complete immersion in a far off world with a virtual reality (VR) headset. 

Unsurprisingly, the human body is really complex. Miles of nerves, arteries and veins weave around muscles, bones and organs. Not only do future doctors need to know the names of each arterial branch and intestinal fold, but also where all these structures sit relative to each other, how they interact and most importantly, how conditions afflicting one affect the other. Traditional dissection techniques allowed students to slowly descend layer by layer deeper into the body. This mirrored conventional approaches during surgery. Now, biomedical advances revolutionized the approach to invasive procedures, and anatomy faculty have needed to adjust their preparation of their future physicians to match these new interactions and visualizations. As digital assets supplement the doctor’s whole view of their patient, XR teaching tools will supplement the role of the cadaver, connecting students with the body like never before. 

After reviewing commercially accessible hardware and software options we encountered the many gaps still left to fill in this emerging market. We’re not yet ready to completely abandon our physical models and anatomical donors for a plug-and-play XR module. Instead we identified key regions where an immersive view would be most impactful. Recognizing we needed to custom-build our experiences, we turned to Mozilla Hubs.

Mozilla Hubs allows you to create immersive virtual worlds on the web. With Hubs, you can customize many aspects of your 3D space and invite others to join you with a link on a mobile device, desktop computer or VR headset. Hubs is open source and extensible, backed by a community of creative web developers, artists, designers and event hosts who contribute to improving and building on Hubs.

It was amongst this community that we were able to acquire the skills and leverage the newest tools developed in the platform to create our own custom XR experiences. On one side, our faculty was able to leverage the intuitive controls afforded by VR programs to model these anatomical spaces before passing them over to our multimedia developer for optimization and use in the virtual world. Together we set the scenes mixing multimedia, subtle interface cues and open access resources within and outside of Hubs to create spaces steeped with knowledge. Design sessions iteratively layered educational content with multimedia accessories creating scenes that balanced the awe VR can invoke with the learning objectives. Once we, as architects and builders, were satisfied, we were ready to, as Ms. Frizzle would say, “take chances, make mistakes, and get messy” by bringing students into our creations.

Field trips

Our first journey into the virtual space with our students brought them into the chambers of the heart, with blood cells whooshing past to guide the way through the circulatory system. This was the fall of 2020; our students were learning remotely due to the pandemic. Not only were they trying to figure out how to connect their 2D computer screens to the 3D anatomy content, they were trying to connect with each other and their identities as incipient medical students. Hubs promised to be the answer to both dilemmas. We envisioned bringing them all together within a 3D educational environment allowing them to speak to each other and to faculty while interacting with a complex anatomical space on their computer screens. 

As the maiden voyage, we had very few expectations. But throughout the event, we learned as much about our students’ hopes, needs and desires as they did about the human heart, and we began to formulate best practices for orientation, design and interactions for our spaces. Later that year we developed an “escape room” through the GI tract. The Hubs virtual space was synced with an outside resource guide of questions, clues and instructions to assist students in getting from the stomach to the intestines, saving the gallbladder along the way. 

Once our students had returned to campus we were excited to more intimately connect the immersive learning with other anatomy lab content and more fully immerse the students into the heart. The fact that Hubs is web-based and accessible on numerous devices allowed us to translate this virtual experience to an in-person event. We added in more media like videos of heart valves beating and faculty-recorded narrations to completely transport students into the heart itself for private tours past labeled structures – the same structures students had just held in their own hands minutes before in the anatomy lab. 

The most recent space we developed is, in reality, smaller than a pea. It sits somewhere in the middle between the eye, mouth, cheek and nose. To make things more challenging, it has seven openings in and out, connecting it to most regions of the head. Generations of students have grumbled over the pterygopalatine fossa, lovingly nicknamed the terrible-palatine fossa. In the VR headset, students can figure out where this tiny space actually sits by grabbing and enlarging a transparent skull with a blinking “you are here” sign pointing to a relatively small divot between two bones. They then look around and see direction labels at their feet to self-orient. Go up to the eyes, right to the nose, turn around for the brain. As they progress they watch arteries and nerves course through the head’s very own Grand Central Station, appreciating their path and interactions along the way. 

Learning as educators

Even in these early stages, initial feedback from students is positive, and we plan to build new experiences based on student-reported difficulties with other anatomical regions. While the use of XR as educational tools is promising, there is always a cost-benefit analysis to be had for any new XR intervention. Is the new information, or the new means to present this information, worth the investment of time and money to get both faculty and students able to access this tool? With an ultimate goal of education, an XR experience must first hit those objectives before layering on cool aesthetics and advanced interactions. 

Beyond the value of the tool itself is the context in which it is deployed. Like any new technology still in its infancy, users need to be slowly introduced and acclimated to XR. New adopters should consider the environment outside of the headset including temperature, space and other distractions, just as important as the assets within the virtual world. Concerns over discomfort and disorientation for students highlights the need for experts in user experience and scene optimization as a part of the development team, proactively mitigating problems by leveraging proper design principles. 

In our practice, XR is a resource that complements the traditional lectures, texts, physical models and cadavers used in anatomy education. Our deployments are in conjunction with these other resources to show our students the human body from all angles and perspectives. By recreating physical spaces in a virtual environment, it furthers the real-to-digital connection among various teaching tools, or what we call eXtended Learning. These tools extend our students’ learning outside of the physical lab and outside the confines of our course, providing a depth of understanding and long-lasting memorable experiences that will make them the physicians of tomorrow.  

Now, if only we could get Lily Tomlin to narrate our next field trip inside the human body.

Read more about Mozilla Hubs here:

• You don’t have to be an astronaut to explore space, Mozilla Hubs can take you there
• From art portfolios to new hobbies, here are 3 ways to use Mozilla Hubs

The post The immersive school bus: Hubs-built journeys into the body for medical education appeared first on The Mozilla Blog.

Original article written by Mozilla >