Chinese-LLaMA-Alpaca-3
中文羊驼大模型三期项目 (Chinese Llama-3 LLMs) developed from Meta Llama 3
What is Chinese-LLaMA-Alpaca-3?
中文羊驼大模型三期项目 (Chinese Llama-3 LLMs) developed from Meta Llama 3
Overview of Chinese-LLaMA-Alpaca-3
What it is
Chinese-LLaMA-Alpaca-3 is the third phase of the Chinese-LLaMA-Alpaca open-source large language model series, developed from Meta Llama 3. It provides Chinese Llama-3 base models and Chinese Llama-3-Instruct instruction-tuned models. These models are built upon the original Llama-3 with additional Chinese pre-training data and instruction fine-tuning to improve Chinese language understanding and instruction following.
Key features
- Uses the original Llama-3 vocabulary (128,256 tokens) without expansion, based on experiments showing its encoding efficiency is comparable to the expanded vocabulary used in Chinese LLaMA-2.
- Supports a context length of 8K tokens, extended from the 4K in Llama-2.
- Employs grouped-query attention (GQA) mechanism for improved efficiency.
- Includes a new instruction template for the Instruct version, incompatible with Llama-2-chat.
- Open-sources pre-training and instruction fine-tuning scripts, as well as instruction fine-tuning datasets (alpaca_zh_51k, stem_zh_instruction, ruozhiba_gpt4).
- Compatible with Llama-3 ecosystem tools such as transformers, llama.cpp, text-generation-webui, vLLM, and Ollama.
Use cases
- Llama-3-Chinese-8B (base model): text continuation, generating subsequent content given preceding text.
- Llama-3-Chinese-8B-Instruct (instruction model): question answering, writing, chatting, and other interactive tasks requiring instruction understanding.
Getting started
See the repository README for detailed instructions on model download, quantization, and local deployment using CPU/GPU.
License and pricing
Licensed under Apache-2.0. The project is open-source.
Based on the project's public README and repository metadata.