writing

This benchmark tests how well LLMs incorporate a set of 10 mandatory story elements (characters, objects, core concepts, attributes, motivations, etc.) in a sho

What is writing?

This benchmark tests how well LLMs incorporate a set of 10 mandatory story elements (characters, objects, core concepts, attributes, motivations, etc.) in a short creative story

Overview of writing

What it is

This benchmark tests how well LLMs incorporate a set of 10 mandatory story elements (characters, objects, core concepts, attributes, motivations, etc.) in a short creative story. It compares short stories written to the same constrained creative briefs. Separate evaluator models read matched stories in both orders and judge which is better. Their judgments produce relative scores for each writer model. Higher ratings mean stronger estimated performance within this comparison set. Scores are centered at zero across all writer models in the comparison set.

Key features

  • Compares short stories written to constrained creative briefs with 10 mandatory story elements.
  • Uses separate evaluator models that read matched stories in both orders and judge which is better.
  • Produces relative comparison scores centered at zero across all writer models.
  • Covers 59 writer models, 977 direct model pairings, 16,552 distinct story pairs, and 110,748 judgments.
  • Provides a leaderboard with comparison scores, estimated win chance, and 95% bootstrap intervals.
  • Includes head-to-head comparison charts and evaluator agreement matrices.
  • Stories are constrained to 600–800 words; out-of-range stories are regenerated from original prompts.

Use cases

  • Evaluating and comparing the creative writing capabilities of different LLMs.
  • Testing how well models adhere to a set of required story elements in a constrained generation task.
  • Benchmarking model performance for story generation in research or product development.

Getting started

See the repository README for details on how to run the benchmark.

License and pricing

The project is open source, but no specific license is stated in the repository.

Based on the project's public README and repository metadata.

Add this badge to your README

Featured on YunoDB Badge styles & help

Maintain this project? Show that it is featured on YunoDB.

Markdown[![Featured on YunoDB](https://yunodb.com/badge/lechmazurwriting.svg)](https://yunodb.com/tools/lechmazurwriting?utm_source=github&utm_medium=badge)
HTML<a href="https://yunodb.com/tools/lechmazurwriting?utm_source=github&utm_medium=badge"><img src="https://yunodb.com/badge/lechmazurwriting.svg" alt="Featured on YunoDB" height="20"></a>