Neuralangelo Nvidia AI Lets You Form 3D Models From 2D Videos

Nvidia unveiled an AI model that can generate three-dimensional objects from two-dimensional videos. It is called Neuralangelo, a mix of the words “neural” from neural networks and “Michelangelo,” the famous Renaissance artist. It compiles each 2D frame into details for a 3D object, significantly facilitating the work of artists and other creatives.

Artists could take a video of a model from every angle and submit it to this AI program. It will provide a 3D model so the artist can immediately edit it for a project. The best part is Nvidia has been leading in AI hardware due to the skyrocketing demand for graphics cards.

That means we would likely see this feature active soon. This article will discuss how the new Nvidia Neuralangelo AI model works. Then, I will cover its various applications and explain when you can see it in action!

How does Neuralangelo work?

Nvidia compared Neuralangelo to the historical artist Michelangelo who carved intricately-detailed marble sculptures. The company said the technology adopts instant neural graphics primitives to capture these details. Here’s how it works:

1. The AI model chooses several frames from a 2D video of an object or scene filmed from various angles.
2. The tech company says the process is akin to an “artist considering a subject from multiple sides to get a sense of depth, size, and shape.”
3. Next, the AI program determines the camera positions of each frame.
4. Then, it produces a 3D representation of the scene, like a sculptor chiseling a subject’s shape.
5. Neuralangelo optimizes the render to sharpen details, reminiscent of a sculptor ensuring a figure emulates real-world textures.
6. Afterward, the AI model produces a 3D object or scene ready for virtual reality applications, robotics, or digital twin development.

Ming-Yu Liu, senior director of research and co-author of the Neuralangelo research paper, explained how the model would benefit creative people. “The 3D reconstruction capabilities Neuralangelo offers will be a huge benefit to creators, helping them recreate the real world in the digital world,” he said.

You may also like: Nvidia DLSS3 Improves LOTR: Gollum

“This tool will eventually enable developers to import detailed objects — whether small statues or massive buildings — into virtual environments for video games or industrial digital twins.”

Nvidia said some AI models have been reconstructing 3D scenes like its new AI program. However, they struggle to accurately capture repetitive texture patterns, strong color variations, and homogenous colors.

In comparison, Neuralangelo uses NVIDIA Instant NeRF to capture finer details. Also, it is one of the projects that will appear at the Conference on Computer Vision and Pattern Recognition (CVPR) on June 18 to 22 in Vancouver.

What are the implications of Neuralangelo?

These scenes were created from 2D:

NVIDIA’s new AI model “Neuralangelo” reconstructs 3D scenes from 2D videos pic.twitter.com/TNUuausevO

— AI Breakfast (@AiBreakfast) June 2, 2023

More people play video games and virtual reality experiences nowadays. Their creators require lifelike realism in their graphics, particularly in their environments.

Some games use real-life locations and integrate them into their games. That is a painstaking process requiring numerous pictures and videos of these locations.

Then, artist teams must collaborate for days or months to compile these data into a working 3D environment. Fortunately, Neuralangelo could significantly facilitate that process.

You may also like: How To Use Generative Fill In Photoshop

You still need to take a video of your subject or scene. However, the AI program will turn your images and videos into a three-dimensional space.

Afterward, the artists may adjust the model to suit their video game better. As a result, the creative team saves time on world-building, allowing them to focus on other in-game assets.

You could also use Neuralangelo to produce a digital twin of a specific product. Consequently, companies could match that virtual model with its physical counterpart to confirm deliveries.

Conclusion

Nvidia announced an AI program that can turn 2D videos into 3D objects. As a result, it could become a revolutionary tool for video game developers and other industries.

The tech company has not shared more details about the project at the time of writing. It seems we must wait for the CVPR event on June 18 to 22 for more information.

Fortunately, you can learn more about the latest artificial intelligence tools at Inquirer Tech. It is also a great source for the latest digital tips and trends.