Google's strongest open source model, Gemma 2, is released!

Google Internet company released the second version of its open weighting model, Gemma 2, which includes three models with 2 billion, 9 billion, and 27 billion parameters. Currently, only the 9 and 27 billion parameter models are available. These models perform well in a variety of benchmarks, often outperforming other large models in the family. The technical report provides detailed insights into the architecture, training data, and innovations used to enhance model performance (e.g., knowledge refinement), and Prompt Engineering has created an excellent overview that provides insights.

Google explains:

high performance: At 27B, the Gemma 2 offers class-leading performance, even more competitive than models twice its size. 9B Gemma 2 models also offer class-leading performance, outperforming the Llama 3 8B and other open models in its class. For a detailed performance breakdown, please see the technical report.

Unrivaled efficiency and cost savings: 27B Gemma 2 models are designed to run inference efficiently at full precision on a single Google Cloud TPU host, NVIDIA A100 80GB Tensor Core GPU, or NVIDIA H100 Tensor Core GPU, dramatically reducing costs while maintaining high performance. This makes AI deployment easier and more affordable.

Ultra-fast reasoning across hardware: Gemma 2 is optimized to run at blazing speeds on a wide range of hardware, from powerful gaming laptops and high-end desktops to cloud-based setups. Try Gemma 2 with full precision in Google AI Studio, use the quantized version on a CPU using Gemma.cpp to unlock local performance, or try it out on a home PC with NVIDIA RTX or GeForce RTX via Hugging Face Transformers.

Google Internet company Gemma-2 AI model

While the 2 billion parameter model remains confidential, the 9 and 27 billion parameter models have been made available to the public, offering researchers and developers the opportunity to realize their potential. These models have been carefully designed to handle large-scale linguistic tasks with unparalleled efficiency and accuracy.

The Gemma 2 AI model has already proven itself in real-world applications, with the 9 billion parameter model outperforming the powerful Lama model with 38 billion parameters. Meanwhile, the 27 billion parameter model is on par with the 70 billion parameter version of Lama 3. Both Gemma 2 models are leading the LMS Chatbot Arena, proving their robustness and versatility.

Uncovering the Secrets of Gemma-2's Success

The technical report that accompanied the release of Gemma-2 demonstrates theInnovative technology at the heart of Gemma-2's success.distillate knowledgeconcept, which is a powerful method for training smaller but efficient models.

By employing a teacher-student modeling paradigm, Gemma-2 can leverage the knowledge of larger, more complex models to guide the training of more compact models. By using theKL dispersionAchieve alignment between the student model and the instructor model, thus ensuring consistency and accuracy throughout the pre-training and fine-tuning phases.

Overcoming Training Challenges

The development of Gemma-2 has not been a smooth ride, especially with respect to the large amount of data required for fine-tuning. Evidence of under-training of large models was observed, but the Google team skillfully mitigated this problem through knowledge refinement. This approach allowed them to overcome data limitations and unlock the full potential of the model.

The effectiveness of knowledge distillation was further highlighted by an ablation study conducted during the development process. Models trained from scratch were compared to models trained using this technique, and the distilled models consistently demonstrated significant improvements in benchmarks and perplexity. In addition, the robustness of the training technique was evident in terms of minimizing the impact of varying sliding window sizes on performance.

Accessibility and deployment

Google has made the Gemma-2 model available inGoogle AI Studiocap (a poem)Hugging Faceavailable on the Web, ensuring that researchers and developers can easily access and deploy these innovative tools. The availability of quantized versions of the models further enhances their utility, providing options for model compression and efficient deployment in a variety of scenarios.

• Gemma-2 modelAvailable in three sizes: 2 billion, 9 billion and 27 billion parameters
• 9. 27 billion parametric models have been released to the public
• The Gemma-2 model performs well in a variety of benchmarks
• Knowledge refinement plays a crucial role in training small, efficient models
• Ablation studies confirm the effectiveness of knowledge distillation in improving model performance

As the field of natural language processing continues to evolve, Google's Gemma-2 is at the forefront, pushing the limits of open weighting models. With its outstanding performance, innovative training techniques, and ease of use, Gemma-2 is expected to have a significant impact on applications ranging from chatbots to language translation.