TreeLoRA
[ICML 2025] A pytorch implementation of the paper "TreeLoRA: Efficient Continual Learning via Layer-Wise LoRAs Guided by a Hierarchical Gradient-Similarity Tree
What is TreeLoRA?
[ICML 2025] A pytorch implementation of the paper "TreeLoRA: Efficient Continual Learning via Layer-Wise LoRAs Guided by a Hierarchical Gradient-Similarity Tree".
Overview of TreeLoRA
What it is
TreeLoRA is an official PyTorch implementation of an efficient continual learning method for Large Language Models (LLMs) presented at ICML 2025. It uses layer-wise LoRA adapters guided by a hierarchical gradient-similarity tree to enable continual learning without catastrophic forgetting. The method is designed for scenarios where an LLM must learn multiple tasks sequentially, selecting the appropriate LoRA adapter for each task based on gradient-similarity clustering.
Key features
- Layer-wise LoRA adapters for parameter-efficient fine-tuning - Hierarchical gradient-similarity tree to organize adapters for different tasks - Support for multiple LLM architectures (Gemma, LLaMA, Mistral, etc.)
- DeepSpeed integration for efficient distributed training - Flash attention implementation for improved performance
Use cases
- Sequential multi-task learning for LLMs where data privacy or memory constraints prevent storing all previous data - Domain adaptation of a base LLM to a series of specialized domains without rebuilding from scratch - Any scenario requiring a single model to handle a growing number of tasks while maintaining performance on earlier tasks
Getting started
- Clone the repository and install dependencies:
bash pip install -r requirements.txt2. Extract the dataset intodata/LLM-CL-Benchmarkand download a pretrained model (e.g., Llama-3.2-1B-Instruct) into./PTM/. 3. Run the training script:bash export model_name=“Llama-3.2-1B-Instruct” bash scripts/lora_based_methods/Tree_LoRA.shKey parameters include--model_name_or_path,--data_path,--dataset_name,--reg(default 0.5), and--num_train_epochs. A script./scripts/run_all_exps.shis also provided. For full details, see the repository README.
License and pricing
The license is not stated. The project is open-source (no pricing is listed).
Based on the project's public README and repository metadata.