June 3, 2024 LLM360, in collaboration with MBZUAI and Petuum, has unveiled K2-65B, a cutting-edge large language model (LLM) boasting 65 billion parameters. This model is fully reproducible, with all artifacts, including code, data, model checkpoints, and intermediate results, open-sourced and accessible to the public. This level of transparency aims to demystify the training processes used for similar models like Llama 2 70B and provides clear insights into development and performance metrics.
Collaborative Development
K2’s development was a joint effort by LLM360, MBZUAI, and Petuum, leveraging their combined expertise and resources. The model is available under the Apache 2.0 license, promoting widespread use and further development by the AI community.
Performance and Evaluation
LLM360 has conducted extensive evaluations of K2, covering general and domain-specific benchmarks in medical, mathematical, and coding knowledge. These evaluations ensure the model’s robust performance across various tasks. The LLM360 Performance and Evaluation Collection and the K2 Weights and Biases project document a detailed analysis of K2’s capabilities.
Training Process
K2 was trained using diverse datasets, including dm-math, PubMed-abstracts, and uspto, totaling 1.3 trillion tokens. This comprehensive data mix ensures K2’s broad understanding and capability across various subjects and languages. The training process involved two stages, resulting in performance comparable to that of the Llama 2 70B model.
Transparency and Reproducibility
LLM360 has made K2’s intermediate checkpoints available, allowing researchers and developers to track the model’s development and improvements over time. This fully reproducible nature facilitates transparency and further research and development. Tutorials for reproducing the pretraining and finetuning processes are also provided.
Open Research Lab
LLM360 is an open research lab dedicated to community-owned artificial general intelligence (AGI) through open-source large model research and development. The lab aims to create an open ecosystem with equitable computational resources, high-quality data, and a flowing technical knowledge base, ensuring ethical AGI development and universal access. By advancing the capabilities of large language models and fostering a collaborative environment, LLM360 empowers innovators in AI research and development.
K2 by LLM360 aims to set a new standard for LLM development with its transparency, performance, and robust development framework. Through open-source collaboration and comprehensive evaluation, K2 hopes to ensure ethical practices and broad accessibility for future innovations in AI.
