These researchers came up with a solution for one of VR’s biggest issues: tracking your legs

By Rachel Metz, CNN Business

If you’re a newcomer to virtual reality, you’ll notice something odd about your appearance: Depending on the app, you may look down to see you’re represented by an avatar that’s just a floating torso or nothing but a pair of hands.

While wireless headsets and handheld controllers laden with sensors and software do a good job of tracking your upper body in VR, actual leg tracking is still rare. That’s because it tends to require an array of sensors on or around you to do it well, which most consumers don’t want to buy. (Some apps, such as VRChat, do offer full-body avatars, but they tend to use software to approximate lower-body motions.)

Now, a group of researchers at Carnegie Mellon University (CMU) are offering a possible solution to the leg tracking problem that has so far arguably limited how immersed people can really feel in virtual spaces.

The researchers came up with a way to track a person’s entire body by affixing cameras with fisheye lenses to a pair of Facebook-parent Meta’s Quest 2 controllers, which is currently the most popular VR headset by far (and, like most headsets on the market, does not currently offer leg tracking). As a Quest 2 headset wearer held the controllers, the cameras captured images of their body and were used to control a full-body avatar in several VR demos the researchers created.

With the prototype, users could make their avatar kick and stomp blocks in “Feet Saber” (a tongue-in-cheek reference to the popular VR game “Beat Saber”), or twist and turn into the right position to pass through an obstacle in a sort of human-sized “Tetris” game. The researchers also came up with demos that let users kick a soccer ball and defend a goal from incoming hockey pucks.

“I think the legs are expressive. I think they are important to see how people are moving around — not just for locomotion but body pose for human-to-human interactions,” Chris Harrison, an associate professor of human computer interaction at CMU and a coauthor of the study, told CNN Business.

The prototype, called ControllerPose, isn’t precise enough to capture fine movements, Harrison said, but it works “quite well” for tracking coarser actions. “If you want to look down and see if you’re squatting or lifting a leg or kicking a ball, it’s more than accurate enough to capture those kind of cases,” he said.

On average, the system is off by 6.98 centimeters (2.75 inches) when estimating where to show a joint in VR versus where it actually is, he said.

To make it work, researchers streamed images from two fisheye-lens cameras on each controller — one facing upwards, one facing downwards — to a computer. (Initially, they used a desktop computer; later on, they used a small computer connected to a controller to show the whole process could be done right in the palm of your hand.) Software filtered out bad pictures and fixed lens distortion, then stitched camera images together so they could be used to estimate the headset wearer’s real-life poses. Those estimates determined the poses of the full-body avatar in VR.

Harrison’s lab at CMU, the Future Interfaces Group, has worked on paid collaborations with Meta in the past, he said, but this was not one of those projects.

Meta has been considering for years how to make avatars more realistic. In an Instagram “Ask Me Anything” session in February, Meta CTO Andrew Bosworth acknowledged the difficulty of the task while saying the company is considering how to solve it. “Tracking your own legs accurately is super hard and basically not workable just from a physics standpoint with existing headsets,” Bosworth said.

Skip Rizzo, director of the Medical Virtual Reality Group at the University of California’s Institute for Creative Technologies, thinks leg tracking is useful for certain applications, such as when using VR for rehabilitation after a stroke. (One of the researchers’ demos was a balance-training exercise that had users stand on one leg at a time.) As with video game accessories — be it a steering wheel for a racing game or a vest that vibrates if you’re shot in a first-person-shooter game — he knows that most people don’t want to pay for extra sensors to use with their VR headsets.

“You gotta be real hardcore to buy that,” he said.

Still, he noted that adding cameras and a small computer to hand controllers would suck up more power (and since the Quest 2 controllers run on batteries, that could mean replacing them more often). Harrison, too, sees power consumption as a challenge; in the future, the researchers would like to increase the speed at which they can process images on the handheld controller, but they don’t want to run down its battery life too quickly.

While the ControllerPose project is meant to get people interested in the potential for using controllers to track the body, Harrison thinks it’s likely we’ll see such a feature included with an upcoming VR headset. (Meta declined to comment.)

“It’s not that it’s the killer app for VR,” Harrison said, “but we’re just going that one step closer to unlocking that vision of really teleporting yourself into another reality.”

The-CNN-Wire
™ & © 2022 Cable News Network, Inc., a WarnerMedia Company. All rights reserved.