Chinese-LLaMA-Alpaca-3

中文羊驼大模型三期项目 (Chinese Llama-3 LLMs) developed from Meta Llama 3

What is Chinese-LLaMA-Alpaca-3?

中文羊驼大模型三期项目 (Chinese Llama-3 LLMs) developed from Meta Llama 3

Overview of Chinese-LLaMA-Alpaca-3

What it is

Chinese-LLaMA-Alpaca-3 is the third phase of the Chinese-LLaMA-Alpaca open-source large language model series, developed from Meta Llama 3. It provides Chinese Llama-3 base models and Chinese Llama-3-Instruct instruction-tuned models. These models are built upon the original Llama-3 with additional Chinese pre-training data and instruction fine-tuning to improve Chinese language understanding and instruction following.

Key features

  • Uses the original Llama-3 vocabulary (128,256 tokens) without expansion, based on experiments showing its encoding efficiency is comparable to the expanded vocabulary used in Chinese LLaMA-2.
  • Supports a context length of 8K tokens, extended from the 4K in Llama-2.
  • Employs grouped-query attention (GQA) mechanism for improved efficiency.
  • Includes a new instruction template for the Instruct version, incompatible with Llama-2-chat.
  • Open-sources pre-training and instruction fine-tuning scripts, as well as instruction fine-tuning datasets (alpaca_zh_51k, stem_zh_instruction, ruozhiba_gpt4).
  • Compatible with Llama-3 ecosystem tools such as transformers, llama.cpp, text-generation-webui, vLLM, and Ollama.

Use cases

  • Llama-3-Chinese-8B (base model): text continuation, generating subsequent content given preceding text.
  • Llama-3-Chinese-8B-Instruct (instruction model): question answering, writing, chatting, and other interactive tasks requiring instruction understanding.

Getting started

See the repository README for detailed instructions on model download, quantization, and local deployment using CPU/GPU.

License and pricing

Licensed under Apache-2.0. The project is open-source.

Based on the project's public README and repository metadata.

Add this badge to your README

Featured on YunoDB Badge styles & help

Maintain this project? Show that it is featured on YunoDB.

Markdown[![Featured on YunoDB](https://yunodb.com/badge/ymcuichinese-llama-alpaca-3.svg)](https://yunodb.com/tools/ymcuichinese-llama-alpaca-3?utm_source=github&utm_medium=badge)
HTML<a href="https://yunodb.com/tools/ymcuichinese-llama-alpaca-3?utm_source=github&utm_medium=badge"><img src="https://yunodb.com/badge/ymcuichinese-llama-alpaca-3.svg" alt="Featured on YunoDB" height="20"></a>