pairjudge

Pairwise LLM judges (A/B/tie): budget-aware multi-turn packing, position-bias correction, pseudo-label distillation. Generalized from the 4th-place (gold) solut

What is pairjudge?

Pairwise LLM judges (A/B/tie): budget-aware multi-turn packing, position-bias correction, pseudo-label distillation. Generalized from the 4th-place (gold) solution to Kaggle LMSYS Chatbot Arena.

Overview of pairjudge

What it is

pairjudge is a library that compares a prompt and two responses, returning A wins / B wins / tie probabilities, retained-content diagnostics and an optional comparison in both presentation orders. It provides a trained, versioned example model that can be loaded directly after installation. The methods originated in the 4th-place, gold-medal solution to LMSYS — Chatbot Arena Human Preference Predictions. The new 0.5B example is a small, separately evaluated model; the competition medal is not its quality certification.

Key features

  • Budget-aware multi-turn packing: balanced_v2 reserves field titles and final instructions separately, allocates the remaining content budget with configurable weights, and redistributes unused space. Nonempty fields retain content tokens when a round fits; cuts and dropped rounds are explicit.
  • Swap diagnostics: scoring (A,B) and (B,A), exchanging A/B output columns and averaging produces swap-equivariant probabilities in deterministic inference.
  • Three-class training and distillation: hard labels use cross-entropy; soft distributions use KL. Trained heads, tokenizer, all packer fields and class mapping travel together in safe model bundles.
  • CLI commands: pairjudge compare, pairjudge batch, pairjudge demo.
  • Python API: PairwiseJudge class with from_pretrained and predict methods.
  • Local demo at http://127.0.0.1:7860 after model loads, showing single-pass, swapped-and-aligned, and averaged probabilities plus content retention report.

Use cases

  • Comparing two responses to a given prompt.
  • Predicting human preferences between chatbot responses.
  • A/B testing of model outputs with position-bias correction.
  • Training a custom judge on user-provided preference data.

Getting started

bash python -m pip install 'pairjudge[judge]==0.3.0' pairjudge compare --prompt 'Why does ice float?' --a 'Its crystal structure lowers its density.' --b 'Cold objects always rise.' --swap For batch processing: bash pairjudge batch --input pairs.jsonl --output results.jsonl --swap For the local demo: bash pairjudge demo Model inference and training require Python 3.10+. The lightweight core supports Python 3.9+. First inference downloads approximately 2 GB of model weights and tokenizer files.

License and pricing

The code is MIT licensed. Example weights and their base/data provenance are Apache-2.0, with third-party notices retained in the model repository. The library is open source with no stated pricing.

Based on the project's public README and repository metadata.

Add this badge to your README

Featured on YunoDB Badge styles & help

Maintain this project? Show that it is featured on YunoDB.

Markdown[![Featured on YunoDB](https://yunodb.com/badge/daoyuanli2816pairjudge.svg)](https://yunodb.com/tools/daoyuanli2816pairjudge?utm_source=github&utm_medium=badge)
HTML<a href="https://yunodb.com/tools/daoyuanli2816pairjudge?utm_source=github&utm_medium=badge"><img src="https://yunodb.com/badge/daoyuanli2816pairjudge.svg" alt="Featured on YunoDB" height="20"></a>