Open-Sora 2.0Translation site

What is Open-Sora 2.0

Open-Sora 2.0 is a new open source by Lucent Technologies.Video GenerationModeling. It has made significant progress in the field of video generation, enabling high-performance video generation at a low cost. The model is based on advanced deep learning architecture and training techniques, and supports a variety of visual generation tasks such as text-to-video and image-to-video, etc. The release of Open-Sora 2.0 marks a new stage in open source video generation technology, providing more developers with the opportunity to participate in the research and development of high-quality video generation, and jointly promoting the development of video generation technology.

Open-Sora 2.0 Model Performance and Parameters

• parameter scale: 11B (11 billion) parameters, large enough to handle complex video generation tasks.
• Training costs: Successfully trained a large model for commercial-grade video generation with only $200,000 (224 GPUs), a significant cost reduction compared to other models that can cost millions of dollars to train.
• performanceThe performance of HunyuanVideo and Step-Video with 30B parameter is comparable to or even surpasses the closed source model in many key indicators. It achieves excellent results in VBench evaluation and scores more than Tencent's HunyuanVideo.

Open-Sora 2.0 Technical Features and Innovations

• 3D Self-Encoder and Flow Matching Training Framework: Continuing the design idea of Open-Sora 1.2, it realizes the simultaneous training of different video lengths and resolutions through the multi-bucket training mechanism.
• 3D Total Attention Mechanism: Introducing a 3D full-attention mechanism to further improve the quality of video generation.
• MMDiT Architecture: Utilizes the latest MMDiT architecture to more accurately capture textual information in relation to video content.
• Toussaint video model FLUX initialization: Initialization with the help of the open-source graph-generated video model FLUX, which significantly reduces the training cost and achieves more efficient video generation optimization.
• High compression ratio video self-encoder: Trained a high compression ratio (4×32×32) video self-encoder to reduce inference time to less than 3 minutes for a single card and improve inference speed by a factor of 10.

Open-Sora 2.0 Optimizations

• Data filtering: Ensure high-quality data input through strict data screening to improve model training efficiency from the source.
• Resolution Optimization: Prioritize computing power into low-resolution training to efficiently learn motion information, reducing costs while ensuring that the model captures key dynamic features.
• Toussaint Video Task Prioritization Training: Compared to training high-resolution videos directly, graph-generated video models have faster convergence speeds when upgrading the resolution, thus further reducing the training cost.
• Efficient parallel training programs: Combine ColossalAI and system-level optimization to dramatically improve computational resource utilization and achieve more efficient video generation training. This includes efficient sequence parallelism and ZeroDP, fine-grained controlled Gradient Checkpointing, automatic training recovery mechanism, efficient data loading and memory management, efficient asynchronous model preservation, and operator optimization.

Open-Sora 2.0 Application Scenarios and Benefits

• Controlled range of motion: The motion amplitude can be set as required to better show the delicate movements of the characters or scenes.
• Picture quality and smoothness: Provides 720p high resolution and 24 FPS smooth video, giving the final video a stable frame rate with detailed performance.
• Supports rich scenariosOpen-Sora 2.0 produces images with excellent detail and camera work, from rural landscapes to nature scenes.
• open source ecology: Comprehensive open source model weights, inference code and distributed training of the whole process, to create a strong open source ecosystem, attracting the attention and participation of many developers.

Open-Sora 2.0 Impact and Recognition

• Academic citations: Open-Sora's academic paper citations received nearly 100 citations in six months, and it is firmly at the top of the global open source influence rankings.
• global reach: Leading all open source I2V/T2V video generation projects, becoming one of the world's most influential open source video generation project.

GitHub open source repository:https://github.com/hpcaitech/Open-Sora
Technical report:https://github.com/hpcaitech/Open-Sora-Demo/blob/main/paper/Open_Sora_2_tech_report.pdf