7 ms·
LLM Benchmark for 'Longform Creative Writing'
- nullbyte808 2y ago"Crypto’s dead," Jay muttered. "Sneakers are forever." Dang deepseek is actually pretty good. Compared to Gemini's version that sounded like a Schizophrenic on LSD.
- lukebuehler 2y agoGreat that they add a Slop analysis. For example, Gemini uses "heart hammered ribs" a lot.
- isaacfrond 2y agoSome of the Gemini stuff is almost at airport level. I'm surprised. Everything is going so fast. The odd thing, is that with technical stuff, I'm continually rewriting the LLM's to be clearer and less verbose. While the fiction is almost the opposite--not literary enough.
- baq 2y agoIs there a score for internal consistency? I dunno, maybe have another LLM extract structure into some kind of a logic language and score how many assertions are incompatible? Or to a probabilistic language and do some Bayesian analysis?
- mentalgear 2y agoI was thinking the same about consistency, which is useful for many contexts, but if you let another LLM do the extraction, you are again at the mercy of that LLMs quality/hallucinations. Of course mitigation strategies were using many different LLMs and comparing results (voting) or using a highly trained/specialised model for only entity / context extraction. An interesting benchmark would be when those extraction techniques/models would exceed what a human professional is able to do.
- baq 2y ago> you are again at the mercy of that LLMs quality/hallucinations. I guess I'm fine with a tall error bar - better than nothing. > An interesting benchmark would be when those extraction techniques/models would exceed what a human professional is able to do. Slashdot-style +1 Funny, -1 Inconsistent for RLHF maybe...?
- Tostino 2y agoAnother that I don't see enough is just resampling and showing the human more 'final answers' if they are easy and quick to check for correctness.
- 112233 2y agoAnd where that another LLM will get the capability to judge consistency? Had fun recently prompting different models with "a story about everyday life of <some stereotypical character>". Absolutely no awareness about anything spatial - what is in which place, who is where, body positions. Like, my favorite, person coming home and removing socks to reveal fluffy slippers. Will the consistency checker catch that. (edit) just asked GPT-4o to make a photo of hands tying two ropes (my favorite way to see how bad actually gen-ai is) - yup, it has an odd number of ropes coming out of the knot. That is state of art supposedly.
- baq 2y ago> And where that another LLM will get the capability to judge consistency? Had fun recently prompting different models with "a story about everyday life of <some stereotypical character>". > Absolutely no awareness about anything spatial - what is in which place, who is where, body positions. These are some things I'd like to at least attempt to quantify - LLMs are better at translation than thinking[0], so the idea was to translate the slop into something that can be then analyzed by a non-LLM tool. E.g. translate the story into a prolog fact database and generate a couple queries for each paragraph/chapter/combination of these. Rough idea, just something I haven't seen done. [0] don't @ me
- thaumasiotes 2y ago> Like, my favorite, person coming home and removing socks to reveal fluffy slippers. Will the consistency checker catch that. I could see this working as comedy, though it'd be better on film than in print. Either way you'd have to establish a very specific tone in the surrounding material.
- 112233 2y agoThis comedy is a good comedy while I am goofing around with "make a story". However, this wordpuker tech is positioned at solving strategic problems, writing software and (something equally serious). It has no mechanism to build a world model "in head" while it generinates. chess pieces teleport on board. people hug through doors. any exercise is a body horror. do you think this tech deals with invariants in software algorithms any better? I mean, haha but not funny.
- sam-paech 2y agoNot internal consistency exactly, but there are criteria checking how well the chapter plan was followed (which is all the way up at the top of the context window). This is done per chapter, and the score trendline is what you see in the "degradation" column. I would say this could be a reasonable proxy for internal consistency since it is measuring more or less the same ability, i.e. how well it's keeping track of details as context window increases.
- kristianp 2y agoThere was a recent post on here about LLMs making mermaid diagrams for character relationships. That could be analysed by another model for evals.
- andy99 2y agoWe did a similar (less rigorous) evaluation internally last year, and one of the big things we identified was "purple prose": In literary criticism, purple prose is overly ornate prose text that may disrupt a narrative flow by drawing undesirable attention to its own extravagant style of writing, thereby diminishing the appreciation of the prose overall.[1] Purple prose is characterized by the excessive use of adjectives, adverbs, and metaphors. When it is limited to certain passages, they may be termed purple patches or purple passages, standing out from the rest of the work. (Wikipedia) The Slop Score they have gets at this which is good, but I wonder how completely it captures it. Also curious about how realistic this benchmark is against real "creative writing" tools. The more the writing is left up to the LLM the better the benchmark likely reflects real performance. For tools that have a human in the loop or a more structured approach it's hard to know how well the benchmarks match real output, beyond just knowing that better models will do better, e.g Claude 3.7 would beat Llama 2.
- beezlebroxxxxxx 2y agoOne person's purple prose is often another's style. It's not really a measure of something, and more an assertion of taste. You see a similar thing when someone says a writer uses "too much description and not enough action!" without realizing that, sometimes, the action is in the description. It's a style of writing with a different reader in mind, a different understanding of how word choice and prose functions.
- matt-dev 2y agohaha, Mr. Thorne shows up again in the Gemini 2.5 samples. I have played around with creating long form fictional content with Gemini 2.5 the last week, and I started adding "no one named 'Thorne'" to my prompts, otherwise it always creates a character named Mr. Thorne. I thought it was something in my prompts trigger this, but it seems to be a general problem. However, despite the cliches and slop, Gemini 2.5 can actually write and edit long form fictional pretty well, you can get almost coherent 10-20 chapter books by first letting it create an outline and then iteratively write and edit the chapters.. I also used Gemini 2.5 to help me code a tool to interactively and iteratively create longform content: https://github.com/pulpgen-dev/pulpgen https://github.com/pulpgen-dev/pulpgen
- becquerel 2y agoYou beat me to that idea, haha. I was making an aider for fiction, but your project looks way more useful than what I had.
- deleted 2y ago[deleted]
- stared 2y agoSnd how these compare to GPT-4.5? (From my experience, the best model for creative writing, https://p.migdal.pl/blog/2025/04/vibe-translating-quantum-flytrap https://p.migdal.pl/blog/2025/04/vibe-translating-quantum-fl...)
- successful23 2y agoI’ve noticed the same contrast - technical writing from LLMs often needs trimming for clarity, but creative writing can lean too far into either bland or overly flowery language. Most LLM benchmarks lean heavily on fluency, but things like internal logic, tone consistency, and narrative pacing are harder to quantify. I think using a second model to extract logical or structural assertions could be a smart direction. It’s not perfect, but it shifts focus from just “how it sounds” to “does it actually make sense over time.” Creative writing benchmarks still feel very early-stage.
- weird-eye-issue 2y ago"don't use flowery or over exaggerated language" works well in my experience
- Paracompact 2y agoIt's nice that for coding and math problems, Claude's internal monologue involves constantly analyzing and critiquing its own implementations. It doesn't seem as keen on critical self-analysis of its creative outputs.
- miki-makh 2y agoI’ve also noticed that with longer-form text, the amount of meaningful information seems to plateau — it doesn’t scale proportionally with the character count.
- roenxi 2y agoIt probably depends a lot on how the system is prompted. One of the interesting things about generative images is how easy it is to know what something looks like without being able to describe it. Longform text is likely similar where there are a bunch of interactions and scenes that humans pick up on if they are there without being able to describe. The early Game of Thrones series was a fascinating example of good writing because most of the terrible things that happened to people were a neat result of their own choices (it had a consistent flawed character -> bad choice -> terrible consequence style that repeated over and over) - but I don't think most people would pick up on that without it being explicitly pointed out. And when that started to go away people could tell the writing was falling off but couldn't easily pick out why. A hypothetical LLM could be prompted with something like that ("your writing is boring, please make consequences follow from choices") but it is less clear that the average prompter would be able to figure out that was what was missing. Like how image generators often needed to be prompted with "avoid making mistakes" to get a much higher quality of image; it took a bit to realise that was an option.
- spacebanana7 2y agoI get much better results with long form writing by including an existing base story in the prompt along with requests for extensive modifications.
- informal007 2y ago> Outputs are evaluated with a scoring rubric by Claude Sonnet 3.7. Will Claude Sonnet 3.7 judger favor himself?
- moffkalast 2y agoThere was a study a while back, comparing how much each model likes every other model's writing and Sonnet overwhelmingly hated itself for some reason.
- dimitri-vs 2y agoIs this is what you are talking about? https://www.reddit.com/r/LocalLLaMA/comments/1j1nen4/llms_like_gpt4o_outputs/ https://www.reddit.com/r/LocalLLaMA/comments/1j1nen4/llms_li... The prompt response they are judging is "Write one concise paragraph about the company that created you" which is kind of an odd choice. Sonnet3.7 hates its own meta-analysis, but loves gpt4o. But the reason behind that is because Claude 3.7 Sonnet consistently replies (to the prompt) that it was created by Open AI, but then catches itself (when judging) as being wrong on that. My takeaway was that gpt4o's gave very safe/luke-warm scores and was strongly correlated with sonnet scores. So if you are judging anything using LLM's then taking the average of Anthropic + Gemini or OpenAI + Gemini might be the best approach.
- moffkalast 2y agoAh yeah that's the one, missed that bit entirely. I guess they should've honestly asked them to write an article about cats or something unrelated that wouldn't hit so close to home.
- boredhedgehog 2y agoI've noticed people will routinely prompt for a specific style when generating visual art, but not when generating text. Wouldn't it be better for consistency to add something like "in the style of Hemingway" to an experiment like this?
- sam-paech 2y agoThe old version of the creative writing eval had several "in the style of" prompts actually! But I got tired of reading bad Hemingway impersonations so I cut them out of the new version. The creative writing v3 prompts (https://eqbench.com/results/creative-writing-v3 https://eqbench.com/results/creative-writing-v3) focus more on other things that are typical failure modes for LLM writing: - Romance - Humour - Physical-spatial understanding - Unusual perspectives in first person - Niche genres I do have an Asimov and a Le Guin author style prompts in there though. In particular I like the Asimov story as a vibe check.
- ForHackernews 2y agoThat link is a 404 for me
- sam-paech 2y agoOops, should be: https://eqbench.com/creative_writing.html https://eqbench.com/creative_writing.html Sample outputs: https://eqbench.com/results/creative-writing-v3/gemini-2.5-pro-exp-03-25.html https://eqbench.com/results/creative-writing-v3/gemini-2.5-p...
- torginus 2y agoWhen trying to use LLMs for creative writing, LLMs, really suck at sequencing and theory-of-mind that is they often create errors where they reference events that yet to occur (according to the prompt), or have characters know things that are true, but have no way at knowing. They are also terrible at writing scenes with mind-games or deception going on. From my experience, this occurs on all LLMs and with a high enough frequency that editing their outputs is much more tedious than writing the damn thing myself.
- prawn 2y agoSlightly related, though I'm yet to try anything to this level: Turn Cursor Into a Novel-Writing Beast - https://www.reddit.com/r/cursor/comments/1jl0rqu/turn_cursor_into_a_novelwriting_beast/ https://www.reddit.com/r/cursor/comments/1jl0rqu/turn_cursor... I have tried to have ChatGPT brainstorm story elements for something else, and its suggestions so far have been very lame. Even its responses to direction/criticism are off-putting and fawning.
- Paracompact 2y ago> Even its responses to direction/criticism are off-putting and fawning. A good author must have an ego! A writer who takes criticism easily is no good writer on my book!
- jaggs 2y agoDeepSeek is pretty darn good. Much less flowery and more on point. At least in Sci-Fi.
- sam-paech 2y agoHey, I made this! Cool to see it show up on hackernews.
- jjani 2y agoWould be interesting if they'd add another one on non-fiction creative writing. For example, turning a set of investigative notes and findings into a Pulitzer-prize winning article that wouldn't be out of place in a renowned, high-quality newspaper. IME, for LLMs (just like humans) this skill doesn't necessarily correlate with fiction writing prowess. This is probably harder to judge automatically (i.e. using LLMs) though, maybe that's why they haven't done it.
- anshumankmr 2y ago>This is probably harder to judge automatically (i.e. using LLMs) though, maybe that's why they haven't done it. Absolutely. Firstly, it is entirely possible if there are investigative notes and findings available on a real life event, it might very well have been trained on the actual article. If the model is trained on it, it might just replicate it. Plus, this might expose how some LLMs can still cook up stuff even when given facts to rely on. Some of these are more notorious than others. Like Perplexity a year plus ago did that quite a bit for me, anecdotally speaking. It has become a lot better though. Then even writing what some might consider Pulitzer prize winning is a subjective task.
- jjani 2y agoIn my mind, I was thinking of giving a set of fictional findings, not realizing that would technically make it.. fiction! I think it's fine to have fictional notes. It's still a very different task than e.g. writing a fantasy novel, which these benchmarks roughly are about. Instead, the task would be to turn a given set of facts on a real-world topic into a high-quality, serious article. > Then even writing what some might consider Pulitzer prize winning is a subjective task. This applies to these benchmarks for "short fantasy story" tasks all the same.
- impossiblefork 2y agoI'm not sure how I feel about the details of the benchmark, but I think this is an important direction in which it would be nice if LLMs were improved. At present they don't really understand either stories or even short character interactions. Microsoft Copilot can generate dialogue where characters who have never met before are suddenly addressing each other by name, so there's great room for improvement.
- Der_Einzige 2y agoThe same author created the anti-slop sampler, which is proof that LLMs can be trivially made extremely creative. Samplers are being slept on by the community. Sam peach is secretly one of the biggest geniuses in all of LLMs. It’s time for the community to recognize this.
- arthurofbabylon 2y agoAll of these benchmarks have gotten out of hand, which is highly suggestive. Benchmarks exist as an indicator of quality and proliferate when other indicators of quality fail. Their very prominence implies that observers are having a difficult time assessing LLM performance in context, which hints at a limited utility or more precisely a non-closed feedback loop at the level of utility. (You know a burger tastes really good when you eat it, no benchmarks required.) Perhaps LLM development really does exist at this rarefied abstract level whereby the development team cannot be immersed in application context, but I doubt that notion. More likely the performance observed in context is either so dispiriting or difficult (or nonexistent) that teams return again and again to the more generously validating benchmarks.
- Majromax 2y ago> You know a burger tastes really good when you eat it, no benchmarks required. I'd say this is a good example of the opposite, where the problem is finding the quantification of an ultimately subjective experience. Take three restaurant reviewers to a burger joint and you might end up with four different opinions. Benchmarks proliferate because many LLM domains defy easy, quantitative measurement, yet LLM development and deployment are so expensive that they need to be guided by independent and quantitative (even if not fully objective) measures.
- anshumankmr 2y ago>Outputs are evaluated with a scoring rubric by Claude Sonnet 3.7 I feel it might be beneficial to evaluate by an ensemble of a bunch of models, picking the SOTA models cause of the subjectivity of the task at hand.
- lukev 2y agoLongform creative writing is possibly the least valid or interesting use case for LLMs. Why should I be bothered to read what nobody could be bothered to write? The point of writing is communication and creative writing is specifically about the human experience… something a LLM can mimic but never speak to authoritatively.
- baq 2y agoThat's a narrow subset you've got there. There are books about experiences of dwarves, alien insectoids and superintelligent AIs, too.
- bglazer 2y ago> “Why should I be bothered to read what nobody could be bothered to write?” Well said
- sam-paech 2y agoPersonally what I find interesting is getting insight into the trajectory of model abilities over time. Over the time I've been running these benchmarks, the writing has gone from pure slop, to broadly competent (at short form at least) and occasionally compelling. I don't think it's much longer until they will be generating content you will actually want to read. Meanwhile a lot of people are find use for LLMs for partner-writing or lower stakes prose or roleplay.
- bambax 2y ago> Why should I be bothered to read what nobody could be bothered to write? I tend to agree but, for the sake of argument: what if you don't know? It seems most people who say they can tell, actually can't.
- miltonlost 2y agoThese Benchmarks are asinine to determine the quality of "creative writing", all chosen by engineers without a single artistic bone. Length Slop Score Repetition Metric Degradation Moby-Dick has chapters as short as a page and as long as 20. According to this benchmark, the book would score lower because of the average length of its chapters. These aren't benchmarks of "quality". A chapter's length is not indicative of a work's quality. That measurement, on its own, is enough to discredit the rest of the benchmark. So so so so so so so misguided.
- sam-paech 2y agoNone of those factors go into the scoring fwiw. They are just informational. The scoring is done to a rubric, like a teacher would grade an essay, on various criteria for good & bad writing.
- miltonlost 2y agoNone of those factors go into the scoring? What's the point of showing them then? What information are these factors providing if they aren't being used in the rubric? Without knowing the rubric (which isn't provided), these scores are baseless to me, and I don't know how to use them. Blackbox numbers are not useful.
- sam-paech 2y agoAll the judge outputs (including rubric) and model outputs are in the samples reports. Sorry you don't like the displayed metrics. I find them very useful / revealing of the things I'm trying to measure with this benchmark.
- Gracana 2y agoThese are important things to look at because they're where LLMs typically have trouble. People complain that their stories are too short, that they use stock phrases and purple prose, that they start to repeat themselves at a certain point, and that quality tends to fall off. Doing well in these problem areas doesn't guarantee good writing, but it does allow for it.
- deleted 2y ago[deleted]
- anilgulecha 2y agoI read Claude's "Sci-Fi First Contact — First Contact " entry. It's pretty good (and with some editing can be great - some of the ending seems slightly unearned). Has a Ted Chiang/Arrival vibe to it, is a very good first contact story. Most folks here are communicating things without engaging with the content. We need a the Turing test for creative writing. I'd definitely not have guessed this was LLM written - seems like an experienced hand wrote it.
- kristianp 2y agoFor the cosy sci-fi (1) example, I found it introduced plot points too quickly, making the short passage really dense. The model eval said: > The chapter also introduces several elements quickly (the stranger, the Syndicate, the experiments) which, while creating intrigue, risks feeling slightly rushed. But there is no score for pacing. https://eqbench.com/results/creative-writing-v3/deepseek-ai__DeepSeek-R1.html https://eqbench.com/results/creative-writing-v3/deepseek-ai_...