Lyra 2.0Translation site

What is Lyra 2.0?

Lyra 2.0 is released by NVIDIA on April 16, 20263D scene generationSystem.The core breakthrough is the generation of highly coherent virtual environments spanning about 90 meters with only a single photo.The system solves the problems of “spatial oblivion” (loss of scene details) and “temporal drift” (object position offset) that occur when traditional models move over long distances. The system solves the two major problems of "spatial oblivion" (loss of scene details) and "temporal drift" (object position offset), which occur in traditional models when the camera moves over long distances.

Technically, Lyra 2.0 employs a dual strategy: firstly, it stores 3D geometric information of each frame in real time to ensure the environment is consistent when the camera reverts back to the old position; secondly, it introduces defective outputs in the training so that the model has the ability of self-correction. Its generated 3D scenes support interactive exploration and can be exported to mesh format for direct application in robot simulation training, game development and virtual scene construction.

Experiments show that Lyra 2.0 outperforms competitors such as GEN3C and CaM in terms of image quality, style consistency, and other metrics, providing an efficient tool for 3D content creation and physics AI training.

Key Features of Lyra 2.0

1. Generate 3D scenes from a single photo::
   • Lyra 2.0 generates a photo of the user. Span of about 90 meters of coherent 3D environments.
   • Generated scene support free roaming cap (a poem) View from any angleIn addition, the pixel coherence is up to 98%, with almost no visible visual imperfections.
2. long-distance coherence::
   • Solves the problem that traditional 3D generation techniques are prone to when moving the camera over long distances. image distortion cap (a poem) structural distortion Question.
   • Ensure that with long distance paths, the scene (math.) geometric consistency cap (a poem) photometric uniformity.
3. Interactive Exploration::
   • Generated 3D scene support Interactive ExplorationThe user can freely move the viewpoint in the scene and observe the objects from different angles.
   • Support for generating Coherent 3D animation, to meet the needs of dynamic scenarios.
4. Export & Compatibility::
   • The generated 3D scene can be exported as grid formatcompatible NVIDIA Isaac Sim and other physics engines for robot simulation training.
   • support sth. that relates to VR/AR Platform integration allows users to transform their surroundings into a virtual world with a single click.

Lyra 2.0 core technology

1. Dual strategy to address spatial distortion and error accumulation::
   • spatial memory: The system stores the 3D geometry information of each frame in real time, and when the camera revert to the old position, it directly calls up the historical information as a reference to avoid repeated generation and ensure the consistency of the environment.
   • Self-exposure training: Deliberately exposing the model to self-generated defective outputs during training forces it to learn to recognize and correct quality degradation, thereby reducing error accumulation.
2. Global consistency modeling::
   • adoption Large models with over 10 billion parameters, specializing in training scenarios for space structure cap (a poem) light logicThe virtual environments that are generated are highly consistent across any viewpoint.
3. Real-time rendering optimization::
   • utilization 3D Gaussian Splatting (3D Gaussian Splatting) Representation of scenes with fast rendering speeds, low memory footprint, and support for real-time editing.
   • The model parameters scale up to 120 millionCan support 30 frames per second real-time rendering, compatible with NVIDIA GPUs with A100 and higher arithmetic power.
4. Self-distillation technology::
   • Teaching a small model with a large model (video diffusion model) trains a more efficient student model through self-distillation, with faster reasoning, lower resource requirements, and no loss of generation quality to the large model.

Scenarios for Lyra 2.0

1. game development::
   • Instead of spending months manually modeling scenes, independent developers can quickly generate basic game maps from photos, dramatically increasing development efficiency.
2. digital twin::
   • City managers can quickly generate digital twin models of neighborhoods from aerial photographs, which can be used to transportation planning,emergency drill etc.
3. film and television production::
   • The director does not need to build an expensive live-action studio, and can generate all kinds of virtual scenes through the reference diagram, reducing post-production costs.
4. VR/AR content creation::
   • Users can transform their surroundings into a virtual world with a single click, further enriching the immersive experience.
5. real estates::
   • Homebuyers can get a better visualization of the listing information through the photo-generated model rooms and neighborhood roaming videos.
6. Robot simulation training::
   • Autonomous driving companies can use this tool to quickly generate diverse urban road scenarios, and the efficiency of acquiring training data can be improved. 80%.
   • Service robot manufacturers, on the other hand, are able to generate virtual home environments from a single family photo, speeding up the process of training robots in navigation and interaction capabilities.

Comparison of similar products

The release of Lyra 2.0 marks a new milestone in 3D generation technology. Its ability to generate 90-meter coherent 3D environments from a single photo, combined with core technologies such as long-range coherence and self-exposure training, makes it a promising solution for a wide range of applications in game development, digital twins, film and TV production, VR/AR, real estate and robot simulation training. Compared with similar products, Lyra 2.0 has obvious advantages in terms of generation range, coherence and open source, which is expected to promote the popularization and industrialization of 3D generation technology.