The year 2023 saw significant advancements in Large Language Models (LLMs) and open-source contributions from startups and companies to counter proprietary models like ChatGPT and Claude. Some notable open-source models and companies in 2023 included Meta’s LLama and LLamav2, TII’s Falcon 7B, 40B, and 180B, as well as Mistral’s Mistral 7B and Mixtral8x7B. However, while 7B models are cheaper and easier to deploy, they may not match the performance of larger models like 70B. Mistral 7B emerged as a strong open-source contestant, outperforming many larger models.
To address this gap, HuggingFace’s H4 team developed Zephyr 7B, aligning with user intent and surpassing even larger models. Zephyr’s performance is attributed to the adoption of four key techniques: Self-Instruct data creation & Distilled Supervised Fine-Tuning (DSFT), Feedback collection, and Distilled Direct Preference Optimization (DDPO).
Traditionally, models are fine-tuned using high-quality instruction-completion pairs, which are costly and require human supervision. Zephyr, however, leverages distillation by using a Teacher model to generate instructions and responses, resulting in DSFT.
Feedback from a superior teacher model, instead of traditional Reinforcement Learning from Human Feedback (RLHF), drives Zephyr’s alignment with user interests through UltraFeedback construction.
Finally, Zephyr utilizes Direct Preference Optimization (DPO) to maximize the model’s preference for high-scoring completions over low-scoring ones, achieving greater efficiency than RLHF.
Zephyr’s base model, Mistral-7B, was trained using the TRL library for fine-tuning and alignment, as well as Deep-Speed Zero 3 and Flash-Attention 2 for optimization. Zephyr matched the performance of 40B models with just 7B parameters and performed comparably to 70B models in chat contexts.
Zephyr’s models are publicly available on Hugging Face and can be used similarly to other language models with the given code examples. Zephyr-7B sets a new state-of-the-art for 7B parameter chat models and even outperforms larger models such as LLAMA2-CHAT-70B on MT-Bench.
For further details on Zephyr’s Training Procedure and its impact, refer to the original research paper and the HuggingFace Zephyr blog. Additionally, the Zephyr team’s contributions are referenced in Self Instruct and UltraFeedback papers.
In conclusion, Zephyr 7B demonstrates the power of distilling large language models into smaller, high-performing models, setting a new standard for 7B parameter chat models.