Meta and BMW teamed up to get a Quest Pro working in a moving vehicle.
If you try to use inside-out tracking in a car today, in a headset like a Quest 2 or even on a phone, you'll notice the virtual world (if using VR) or virtual objects (if using AR/MR) drifting through the cabin as the car turns and accelerates - the content won't be stable.
That's because these systems use both the inertial measurement units (IMU) and cameras to determine your position. While the cameras see the car's interior as stationary, the accelerometer and gyroscope in the IMU sense the car's acceleration and steering through the outside world.
The two companies first announced a partnership on this problem in late 2021, with the stated goal of exploring how AR glasses could be eventually integrated into cars, including to "eventually help drivers navigate their surroundings".
Today Meta researchers presented the progress made so far. They began by connecting the car's IMU to Project Aria research glasses to feed through acceleration & steering data in real-time so the tracking system can filter it out. The Aria glasses don't have any form of display though, so they transferred this approach to a Quest Pro - Meta's first headset with color mixed reality - to test out anchoring virtual objects to the interior of the car.
Meta claims the virtual content stays stable even when the car is making rapid turns, accelerating quickly, or going over road bumps.
The researchers say the next step is to access the car's precise positioning system to enable seeing world-locked virtual objects outside the windows, such as markers on landmarks and other points of interest.
Meta and BMW aren't the first companies to explore in-car headset usage. In fact, in Germany owners of some Audi cars can already buy a HTC Vive Flow based in-car VR entertainment system called Holoride, which we tried out earlier this year. But Holoride avoids the tracking problem by disabling the headset's positional tracking and using custom content that uses the car's real steering, accelerating, and braking as virtual movement.
BMW's partnership with Meta is currently solely in the realm of research, not product development. There was no announcement of any current or future BMW car actually supporting Quest headsets.
“It is too early to tell exactly how or when this technology will make it into customers’ hands", said Claus Dorrer, Head of BMW Group Technology Office USA. "The research partnership with Meta will allow us to discover what immersive, in-vehicle XR experiences could look like in the future and spearhead the seamless integration of such devices into cars."
In the long term, Meta sees research like this having the potential to "revolutionize travel in cars, trains, planes, and beyond, unlocking new forms of hands-free communication, entertainment, and utility — giving us far more value than the screens and instruments we’re used to seeing in vehicles today".
via Mint VR