Imagine you're driving through a tunnel in an autonomous vehicle, but unbeknownst to you, an accident has brought traffic ahead of you to a standstill. Normally, you'd should depend on the automotive ahead of you to know to brake, but what in case your vehicle could see across the automotive ahead of you and brake even sooner?
Researchers at MIT and Meta have developed a pc vision technique that would someday enable an autonomous vehicle to do exactly that.
They have presented a technique that creates physically accurate 3D models of a whole scene, including the occluded areas, using images from a single camera position. Their technique uses shadows to find out what’s in occluded parts of the scene.
They call their approach PlatoNeRF, based on Plato's Allegory of the Cave, a passage from the Greek philosopher's novel The Republic during which chained prisoners in a cave perceive the fact of the skin world through shadows solid on the cave wall.
By combining light detection and ranging (LIDAR) technology with machine learning, PlatoNeRF can create more accurate reconstructions of 3D geometry than some existing AI techniques. In addition, PlatoNeRF is healthier in a position to easily reconstruct scenes where shadows are difficult to detect, akin to in strong ambient light or dark backgrounds.
In addition to improving the security of autonomous vehicles, PlatoNeRF could also increase the efficiency of AR/VR headsets by allowing the user to model the geometry of a room without having to walk around and take measurements. It could also help warehouse robots find items faster in cluttered environments.
“Our core idea was to bring together these two things which have already been done in several disciplines – multibounce lidar and machine learning. It seems that if you bring these two together, you discover many recent ways to explore and use one of the best of each worlds,” says Tzofi Klinghoffer, an MIT graduate student in media arts and sciences, member of the MIT Media Lab, and lead creator of a Paper on PlatoNeRF.
Klinghoffer co-authored the paper along with his supervisor Ramesh Raskar, associate professor of media arts and sciences and head of the Camera Culture Group at MIT; lead creator Rakesh Ranjan, a frontrunner in AI research at Meta Reality Labs; and Siddharth Somasundaram at MIT and Xiaoyu Xiang, Yuchen Fan and Christian Richardt at Meta. The research shall be presented on the Computer Vision and Pattern Recognition conference.
Shedding light on the issue
Reconstructing an entire 3D scene from a camera perspective is a posh problem.
Some machine learning techniques use generative AI models that attempt to guess what’s within the hidden areas. However, these models can fake objects that should not actually there. Other techniques attempt to infer the shapes of hidden objects based on shadows in a color image. However, these methods may cause problems when the shadows are difficult to see.
For PlatoNeRF, the MIT researchers expanded on these approaches, using a brand new sensing method called single-photon lidar. Lidars map a 3D scene by emitting pulses of sunshine and measuring the time it takes for the sunshine to bounce back to the sensor. Because single-photon lidars can detect individual photons, they supply higher resolution data.
The researchers use a single-photon lidar to light up a goal point within the scene. Some of the sunshine is reflected from that time and returns on to the sensor. However, many of the light is scattered and reflected from other objects before returning to the sensor. PlatoNeRF relies on these second reflections of sunshine.
By calculating how long it takes for the sunshine to reflect twice after which return to the lidar sensor, PlatoNeRF captures additional information in regards to the scene, including depth. The second reflection of sunshine also accommodates details about shadows.
The system tracks the secondary light rays – reflected from the goal point to other points within the scene – to find out which points are in shadow (as a result of lack of sunshine). Based on the position of those shadows, PlatoNeRF can infer the geometry of hidden objects.
The lidar illuminates 16 points one after the opposite and captures several images which might be used to reconstruct the complete 3D scene.
“Every time we light a degree within the scene, we create recent shadows. Because we’ve got all these different lighting sources, there are plenty of rays of sunshine shooting around, so we work out the world that’s hidden and outdoors the visible eye,” says Klinghoffer.
A winning combination
The key to PlatoNeRF is the mix of multibounce lidar with a special variety of machine learning model called a Neural Radiance Field (NeRF). A NeRF encodes the geometry of a scene into the weights of a neural network, giving the model a powerful ability to interpolate or estimate recent views of a scene.
This interpolation capability, together with multibounce lidar, also results in highly precise scene reconstructions, says Klinghoffer.
“The biggest challenge was determining mix these two things. We really needed to think in regards to the physics of sunshine transport with multibounce lidar and model that with machine learning,” he says.
They compared PlatoNeRF with two common alternative methods, one using only lidar and the opposite using only a NeRF with a color image.
They found that their method outperformed each techniques, especially when the lidar sensor had a lower resolution. This would make their approach more practical to be used in the actual world, where lower-resolution sensors are common in business devices.
“About 15 years ago, our group invented the primary camera that would 'see' around corners. It worked by exploiting multiple light reflections, or 'light echoes.' These techniques used special lasers and sensors and used three light reflections. Since then, lidar technology has develop into more common, which led to our research on cameras that may see through fog. This recent work uses only two light reflections, which suggests the signal-to-noise ratio may be very high and the standard of the 3D reconstruction is impressive,” says Raskar.
In the long run, the researchers would really like to trace greater than two light reflections to see how this will improve scene reconstruction. They are also excited about applying more deep learning techniques and mixing PlatoNeRF with color image measurements to capture texture information.
“Camera images of shadows have long been studied as a way of 3D reconstruction. This work now revisits the issue within the context of lidar and shows that the accuracy of the reconstructed hidden geometry has been significantly improved. The work shows how clever algorithms combined with bizarre sensors can enable extraordinary capabilities – including the lidar systems that a lot of us carry in our pockets today,” says David Lindell, assistant professor within the Department of Computer Science on the University of Toronto, who was not involved on this work.
