4 ms·
> In practice, most of what people generate is boring and easy to validate: you know within a couple minutes whether it's the right shape, whether anything stic
by kiitos 1y ago
> In practice, most of what people generate is boring and easy to validate: you know within a couple minutes whether it's the right shape, whether anything sticks out the wrong way, the way a veteran chess player can quickly pattern-match a whole chessboard. When it isn't boring, you read carefully (and expensively), or you just say "no, try again, give me something more boring", or you break your LLM generation into smaller steps that are easier to pattern match and recurse.
nope!
https://sketch.dev/blog/our-first-outage-from-llm-written-code https://sketch.dev/blog/our-first-outage-from-llm-written-co...
- tptacek 1y agoWhat do you think your "nope" here is about? (You and I are probably the only 2 people reading this long-dead thread now, and I'm very familiar with the post you just shared --- I've been using Sketch for several months and following along with their Github repo. Sketch is great.)
- kiitos 1y ago"what people generate is boring and easy to validate" mostly that -^ but i guess also transitively, that (a) you can even identify whether code is boring and simple in the first place; and (b) you're able to tell the LLM to make it "more boring" in any kind of effective way i think these kinds of conversations are revealing a really interesting dichotomy among software professionals, around the (recent and LLM-mandatory) concept of inference, and how it relates to their pre-existing mental model of software some software professionals have a mental model of programming that is just fundamentally incompatible with non-determinism, and for them a tool that's built on inference is delivering way more cost than benefit, and maybe essentially incompatible with how they see what they're doing whereas the LLM maxis never really thought of their software as deterministic or anything like that in the first place, so for them all of the non-determinism from inference is, like, same as it ever was, and delivers way more benefit than cost
- tptacek 1y agoSketch is an extraordinarily solid piece of software and they continue to use Sketch to build Sketch; meanwhile, show me someone who hasn't shipped an outage and I will show you someone who has probably never seriously worked in infra, so I don't really see what the high-order bit of the observation is. I have literally seen this exact pattern --- a refactoring PR that flipped a continue to a break --- cause an incident in human code (which is what all code was up until like 9 months ago).
- kiitos 1y agoi don't know how to explain my points any more clearly than i have, so if you don't grok the intent then i guess we're at an impasse, and i guess also to my points you would be someone in the "llm maxi" camp :)
- tptacek 1y agoI'm curious what kind of response you could have received where you'd read it as something other than "llm maxi".