Recapping the latest NVIDIA Research: turning 2D images into 3D models and a new NVIDIA Canvas update: Digital Photography Review

We’ve covered an extensive collection of NVIDIA Research projects here on DPReview over the years. One interesting tool we didn’t cover at launch, however, is GANverse3D. First shown off in April 2021, GANverse3D turns flat 2D images into ‘realistic 3D models that can be visualized and controlled in virtual environments.’ The tool could help architects, creators, game developers and designers. With just a single photo, someone could quickly add a 3D model to a mockup or render without needing any expertise or needing to spend extra time or money.

Like Nvidia’s GauGAN2 technology, which we covered back in November 2021, GANverse3D uses the power of a generative adversarial network, or GAN. GANverse3D was trained using the GAN to synthesize additional images of an existing image from nonexistent viewpoints, like a photographer photographing an object from different angles. The different artificial images are put into a rendering framework for inverse graphics to create 3D mesh models. NVIDIA first shared information on creating 3D models from 2D images in 2019.

After being trained on multi-view images, GANverse3D can predict a 3D mesh model from a single 2D image. When using NVIDIA Omniverse on NVIDIA RTX GPUs, GANverse3D can recreate any 2D image into 3D. NVIDIA has shown off an example of its impressive AI technology using a photo of the famous AI-enabled car, KITT, from the 1980s television series ‘Knight Rider.’

As NVIDIA points out, previous models for inverse graphics have required training with 3D shapes. GANverse3D has required no training with 3D shapes or models. ‘We turned a GAN model into a very efficient data generator so we can create 3D objects from any 2D image on the web,’ said Wenzheng Chen, research scientist at NVIDIA and lead author on the project.

‘Because we trained on real images instead of the typical pipeline, which relies on synthetic data, the AI model generalizes better to real-world applications,’ added NVIDIA researcher Jun Gao, an author on the project.

By turning its GAN model into a dataset by manipulating its neural network layers, NVIDIA researchers were able to get the neural network to generate images from standard viewpoints, such as ‘photographing’ vehicles from specific elevations and camera distances. The final model was trained on 55,000 car images generated by the GAN. After being trained in these AI-generated multi-view images, the GANverse3D tech outperformed an inverse graphics network that was trained on the popular Pascal3D dataset. If you want to dig deeper into the technology and research, read the full research paper here.

As we look forward to what’s next from NVIDIA Research, the company announced last week that there’s now a free version of NVIDIA Omniverse available to millions of people using NVIDIA Studio on GeForce RTX and NVIDIA RTX GPUs. NVIDIA writes, ‘With Omniverse, NVIDIA’s real-time 3D design collaboration and virtual world simulation platform, artists, designers and creators can use leading design applications to create 3D assets and scenes from their laptop or workstation.’

At this year’s CES, NVIDIA also updated its Canvas app. Last June, we saw the first full beta release of NVIDIA Canvas, which is an AI-driven project built upon GauGAN. NVIDIA Canvas uses AI to convert sketches, doodles and brushstrokes into realistic landscape images.

The recent update includes a brand-new AI model that delivers four times the background resolution and adds five new materials for users to paint with. New materials include mud, dirt, straw, flowers and bush, joining existing materials like water, grass, snow and mountains. The NVIDIA Canvas update enables more realistic images with fewer artifacts, further bolstering the app’s pursuit of true photorealism. The app now offers up to 1K pixel resolution, and exports can be easily integrated into existing workflows, such as importing your NVIDIA Canvas work into Adobe Photoshop.

If you’d like to see all of NVIDIA’s latest news and announcements from CES 2022, you can watch a replay of the company’s ‘CES 2022 Special Address’ below. The presentation includes discussions about gaming, new GPUs, new monitors, NVIDIA Omniverse, and more. You can also read a written CES 2022 recap here.