Static-to-Dynamic-LLMEval
The official GitHub repository of the paper "Recent advances in large language model benchmarks against data contamination: From static to dynamic evaluation"
What is Static-to-Dynamic-LLMEval?
The official GitHub repository of the paper "Recent advances in large language model benchmarks against data contamination: From static to dynamic evaluation"
Overview of Static-to-Dynamic-LLMEval
What it is
This is the official GitHub repository for the paper titled “Recent advances in large language model benchmarks against data contamination: From static to dynamic evaluation.” The repository provides a survey of benchmarking methods for large language models (LLMs) that aim to reduce the risk of data contamination—where benchmark data inadvertently appears in the training data, inflating performance metrics. It covers both static benchmarks (fixed, human-curated datasets) and dynamic benchmarks (methods that generate or update test data to avoid contamination). The repository includes a table of contents listing various static and dynamic benchmarking applications and mitigation methods, along with references to related papers and resources.
Key features
- Contains a structured survey of static and dynamic LLM benchmarking methods for data contamination mitigation.
- Lists static benchmark applications in areas such as math, knowledge, coding, instruction following, reasoning, safety, language, and reading comprehension.
- Covers dynamic benchmark applications including temporal cutoff, rule-based generation (template-based, table-based, graph-based), LLM-based generation (benchmark rewriting, interactive evaluation, multi-agent evaluation), hybrid generation, and network/system approaches.
- Provides references to numerous papers and resources for each benchmark category.
- Actively maintained; the repository welcomes pull requests and issues for updates and corrections.
Use cases
This repository is intended for researchers and practitioners studying data contamination in LLM evaluation. It can be used to understand the landscape of existing static and dynamic benchmarks, identify gaps in current evaluation methods, and find references for specific benchmark types (e.g., math, coding, reasoning). The survey also proposes design principles for dynamic benchmarks, which can guide future research and standardization efforts.
Getting started
See the repository README for details on how to contribute or access the latest updates. No installation or quick-start commands are provided in the README.
License and pricing
The license is not stated in the metadata or README. The repository is listed as open source.
Based on the project's public README and repository metadata.