Other companies can probably do this, too, but they won’t understand why we need it. By contrast, CrossComm has that 1% extra push to make things happen, and it makes the entire experience phenomenal. It takes a company like them to understand and deliver the nuance we require.
Client
FocuSStem NextGen, Karmanos Cancer Institute
Services
AR/VR, and Mobile
Each year, FocuSStem NextGen and the Karmanos Cancer Institute hold a summer intensive that introduces underrepresented students to STEM careers. The intensive allows students to meet with leading cancer physicians, researchers and clinical staff, and connect with peers to work together in real-world scenarios during a summit event.
In 2020, CrossComm collaborated with FocuSStem NextGen to develop several virtual reality (VR) environments consisting of a simulated research lab, ICU, and operating room. Through a VR headset, students were transported into realistic medical and research scenarios.
In 2021, CrossComm was challenged by Dr. Elisabeth Heath of Karmanos Institute to move the needle further. She wanted even those students without a VR headset or reliable high-speed internet and wifi to have access to the programming.
Many children in America have limited wifi connectivity and these disparities are strongly associated with socioeconomic status and racial-ethnic identity. Likewise, children living in rural areas have poor home connectivity due to less-than-ideal infrastructure, with only 47% in some areas having any high-speed internet at all. The FocuSStem team wanted to reach as many students as possible despite this digital divide, while still keeping the remote, immersive nature of the previous summit.
Our Approach
In order to reach a broader audience, CrossComm needed to build virtual and augmented environments to host a live extended reality (XR) summit. The live summit would allow medical doctors, researchers, and scientists to present 3D objects (like an MRI machine, the heart, and the brain) on a virtual stage. Access from home was simple through the augmented reality (AR) app available to anyone with a mobile phone. Students at home could then explore the objects, interact with the presenters, and follow along live without the need for high-speed internet or VR headsets.
Leveraging Cellular Technology for Remote Learning
In the initial brainstorming sessions, the CrossComm team identified cellular connection as an acceptable alternative to wifi-based high-speed internet. Additionally, many of the interested students did not have access to a VR headset, but the majority did have access to a smartphone. Since modern smartphones come equipped with the technology needed to participate in an XR platform (such as cameras, microphones, and high-resolution screens), it was an excellent solution to bridge the digital gap.
Developing Access in AR and VR
To achieve an immersive alternative to VR-only experiences, our team built an asymmetrical XR platform on the Photon multiplayer game engine. The asymmetry refers to the difference in access, display, and interaction experienced by different AR and VR users. Student learners tuned into the live event via the AR platform, while the speakers and presenters held the summit using the VR platform. The live summit required developing access across three user groups:
Students: Learners from across the region who tuned in live for the event.
Speakers: Leading technologists, cancer physicians, and researchers who discussed STEM topics.
XR Presenters: CrossComm CEO Don Shin and Dr. Elisabeth Heath from Karmanos who presented 3D objects correlating with the speakers’ talks.
The Student AR Experience
The students were logged into a web-based video conference platform where they could see and hear the speakers and presenters. Meanwhile in the freely available mobile application, they could see avatars of the XR presenters as well as visualize the STEM 3D models that were presented in VR. The mobile app was able to identify a user’s spatial surroundings and project the simulated 3D objects into their real-world environment. Just like with real objects, students could move closer to the virtual objects, or even circle around them for a more detailed inspection. Within the app they had a chat bar to submit questions and a preloaded set of emojis to react with.
Presenters Showcase in VR
Meanwhile, Dr. Heath and Don acted as the presenters and were in control of the main visual aspect of the live summit. They accessed the platform via VR headsets that took them inside a virtual presentation hall. Within the VR space, they could spawn and interact with 3D models to support the scientists and physicians in their presentations.
Since Don and Dr. Heath were in the VR environment, they couldn’t directly see the engagement from the students participating through the AR mobile app. To solve this, CrossComm built a virtual chalkboard within the VR presentation space that automatically showed the questions from the live chat. This allowed Don and Dr. Heath to easily see and respond to the questions or ask the speakers to elaborate further on their topics.
Simple Access for Speakers
The researchers, technologists, and physician speakers accessed the space via a simple webpage and could speak and watch the summit in a standard 2D view. This simpler, more traditional style allowed them to share their topics without having to worry about maneuvering or presenting in 3D virtual space. They could speak comfortably in their area of expertise and instruct the XR presenters to showcase and manipulate the chosen 3D objects in time with their talk. The speakers could even see the live emoji reactions from students and hear any questions that were asked.
The asymmetrical XR platform helped keep a smooth and moderated flow of engagement between all three parties, and the live summit was informative and interactive. Students could engage with the speakers and presenters via emojis and questions. The speakers could interact with the presenters and students by talking and seeing emojis, and the presenters could tie it all together by talking, manipulating models, and fielding questions.
Running a Smooth AR/VR Live Event
There are a lot of moving parts to a live event, so our team made sure to have contingency plans in place. First, we wanted to make the XR experience easy to access. By uploading the mobile app to respective app stores, we streamlined the process for distributing it to students down to simply providing a link or QR code.
It was also important to make sure the platform we built could handle a large amount of users at once without crashing. To be sure, our developers ran a load test of the XR experience by pushing it beyond the limits of its expected operation. Essentially, they opened dozens of instances of the app, all connected to the same server. They even tested interaction by sending a robust amount of emojis at once—replicating an audience of overzealous students— to ensure all features were in working order for the live summit event.
Results
Despite the continued social distancing and lockdown hurdles of the pandemic, the nature of this event allowed FocuSStem to continue with their summer summit. The one-day event drew in students from around the region via computer and mobile phone access, overcoming hurdles caused by limited internet connectivity or lack of access to XR hardware.
The 2021 virtual summit won a silver Telly Award in the “Virtual Events & Experiences” category and a bronze award in the “Craft-Use of AR” category.
If you’re interested in taking the next step in your project or idea, we would love to hear from you and discuss whether CrossComm would be the right technology development team for you. Tell us more and receive a free assessment from CrossComm.