BitNet
Implementation of "BitNet: Scaling 1-bit Transformers for Large Language Models" in pytorch
What is BitNet?
Implementation of "BitNet: Scaling 1-bit Transformers for Large Language Models" in pytorch
Overview of BitNet
What it is
BitNet is a PyTorch implementation of the linear methods and model described in the paper “BitNet: Scaling 1-bit Transformers for Large Language Models”. The core idea is to replace standard linear projections (nn.Linear in PyTorch) with BitLinear modules, which binarize weights and use abs max quantization. The repository also includes an implementation of the newer “1.58 Bits” iteration and provides utilities for replacing linear layers in existing PyTorch or Hugging Face models.
Key features
- BitLinear layer: the main innovation from the paper, which applies layer normalization, binarization, and abs max quantization/dequantization.
- BitNetTransformer: a full transformer with multi-head attention and BitFeedForward layers, supporting residuals and skip connections.
- BitMGQA: multi-grouped query attention that uses BitLinear projections for faster decoding and longer context handling.
- BitFeedForward: a feed-forward network composed of BitLinear layers with GELU activation, configurable dropout and layer normalization.
- BitLora: an implementation of low-rank adaptation using BitLinear.
- Replacement utilities: functions to replace all nn.Linear layers in a PyTorch model or a Hugging Face model with BitLinear.
- Optimized CUDA kernel: a custom low-bit GEMM kernel for accelerated inference.
Use cases
BitNet is designed for training or fine-tuning large language models from scratch using 1-bit quantization. It can be applied to text, image, and potentially video or audio processing tasks. The replacement utilities allow users to experiment with quantizing existing transformer-based models for reduced memory and computation.
Getting started
bashpip3 install bitnetAfter installation, import the desired modules (e.g., from bitnet import BitLinear) and use them as drop-in replacements for standard linear layers. For full transformer usage, see the example scripts in the repository’s examples folder.
License and pricing
The project is licensed under MIT and is open source.
Based on the project's public README and repository metadata.