pairjudge
Pairwise LLM judges (A/B/tie): budget-aware multi-turn packing, position-bias correction, pseudo-label distillation. Generalized from the 4th-place (gold) solut
What is pairjudge?
Pairwise LLM judges (A/B/tie): budget-aware multi-turn packing, position-bias correction, pseudo-label distillation. Generalized from the 4th-place (gold) solution to Kaggle LMSYS Chatbot Arena.
Overview of pairjudge
What it is
pairjudge is a library that compares a prompt and two responses, returning A wins / B wins / tie probabilities, retained-content diagnostics and an optional comparison in both presentation orders. It provides a trained, versioned example model that can be loaded directly after installation. The methods originated in the 4th-place, gold-medal solution to LMSYS — Chatbot Arena Human Preference Predictions. The new 0.5B example is a small, separately evaluated model; the competition medal is not its quality certification.
Key features
- Budget-aware multi-turn packing:
balanced_v2reserves field titles and final instructions separately, allocates the remaining content budget with configurable weights, and redistributes unused space. Nonempty fields retain content tokens when a round fits; cuts and dropped rounds are explicit. - Swap diagnostics: scoring (A,B) and (B,A), exchanging A/B output columns and averaging produces swap-equivariant probabilities in deterministic inference.
- Three-class training and distillation: hard labels use cross-entropy; soft distributions use KL. Trained heads, tokenizer, all packer fields and class mapping travel together in safe model bundles.
- CLI commands:
pairjudge compare,pairjudge batch,pairjudge demo. - Python API:
PairwiseJudgeclass withfrom_pretrainedandpredictmethods. - Local demo at
http://127.0.0.1:7860after model loads, showing single-pass, swapped-and-aligned, and averaged probabilities plus content retention report.
Use cases
- Comparing two responses to a given prompt.
- Predicting human preferences between chatbot responses.
- A/B testing of model outputs with position-bias correction.
- Training a custom judge on user-provided preference data.
Getting started
bash python -m pip install 'pairjudge[judge]==0.3.0' pairjudge compare --prompt 'Why does ice float?' --a 'Its crystal structure lowers its density.' --b 'Cold objects always rise.' --swap For batch processing: bash pairjudge batch --input pairs.jsonl --output results.jsonl --swap For the local demo: bash pairjudge demo Model inference and training require Python 3.10+. The lightweight core supports Python 3.9+. First inference downloads approximately 2 GB of model weights and tokenizer files.
License and pricing
The code is MIT licensed. Example weights and their base/data provenance are Apache-2.0, with third-party notices retained in the model repository. The library is open source with no stated pricing.
Based on the project's public README and repository metadata.