12 ms·

--dangerously-skip-reading-code

200 points by fagnerbrack 5mo ago

3vo-ai 5mo ago
[flagged]
wizzwizz4 5mo ago
> There would need to be automated pull-request checks verifying not only that tests pass but that code conforms to the spec. As I understand, this is an unsolved problem.
- Ecys 5mo ago
  this is actually precisely what humans' roles will be. "is this implementation/code actually aligned with what i want to do?" humanic responsibility's focus will move entirely from implementing code to deciding whether it should be implemented or not. u probably mean unsolved as in "not yet able to be automated", and that's true. if pull-request checks verifying that tests are conforming to the spec are automated, then we'd have AGI.
  - wizzwizz4 5mo ago
    This is a task that humans are exceptionally bad at, because we are not computers. If something uses the right words in the right order such that it communicates the correct algorithm to a human, then a human is likely to say "yup, that's correct", even if an hour's study of these 15 lines reveals that a subtle punctuation choice, or a subtle mismatch between a function's name and its semantics, would reveal that it implements a different algorithm to the expected one. LLMs do not understand prose or code in the same way humans do (such that "understand" is misleading terminology), but they understand them in a way that's way closer to fuzzy natural language interpretation than pedantic programming language interpretation. (An LLM will be confused if you rename all the variables: a compiler won't even notice.) So we've built a machine that makes the kinds of mistakes that humans struggle to spot, used RLHF to optimise it for persuasiveness, and now we're expecting humans to do a good job reviewing its output. And, per Kernighan's law: > Everyone knows that debugging is twice as hard as writing a program in the first place. So if you're as clever as you can be when you write it, how will you ever debug it? And that's the ideal situation where you're the one who's written it: reading other people's code is generally harder than reading your own. So how do you expect to fare when you're reading nobody's code at all?
    - Ecys 5mo ago
      i meant on a higher, agentic level where the AI's code is infallible. and that's going to happen very soon: say: human wants to make a search engine that money for them. 1. for a task, ask several agents to make their own implementation and a super agent to evaluate each one and interrogate each agent and find the best implementation/variable names, and then explain to the human what exactly it does. or just mythos 2. the feature is something like "let videos be in search results, along with links" 3. human's job "is it worth putting videos in this search engine? will it really drive profits higher? i guess people will stay on teh search engine longer, but hmmm maybe not. maybe let's do some a/b testing and see whether it's worth implementing???" etc... this is where the developer has to start thinking like a product manager. meaning his position is abolished and the product manager can do the "coding" part directly. now this should be basic knowledge in 2026. i am just reading and writing back the same thing on HN omds.
      - wizzwizz4 5mo ago
        The AI's code is not going to be infallible any time soon. It's been "very soon" for the past 4 years, and the AI systems are still making the same kinds of mistakes, which are the mistakes you'd expect from a first-principles study of their model architectures. There's no straightforward path to modifying the systems we have now, to make them infallible.
  - phainopepla2 5mo ago
    > humanic No, thank you
  - lefra 5mo ago
    That's what formal verification is about. I did some (using PSL for hardware verification); writing the formal spec is way harder than the actual code. It will find a lot of subtle issues, and you spend a most of the time deciding if it's the spec or the code that's wrong. Having the code-writing part automated would have a negligible impact on the total project time.
- InsideOutSanta 5mo ago
  Step 1: solve the halting problem.
  - soraminazuki 5mo ago
    Yep, calling it an "unsolved" problem is a misnomer. We already have mathematical proof that it's impossible. But that aside, it's such a shame that many drinking the AI Kool-Aid aren't even aware of the theoretical limits of a computer's capabilities.
    - mjd 5mo ago
      This sort of theoretical result is not always as clear-cut as you suggest. Computers are finite machines. There is a theorem that although a machine with finite memory can add, multiplication requires unbounded memory. Somehow we muddle along and use computers for multiplication anyway. More to your point there is a whole field of people who write useful programs using languages in which every program must be accompanied by a proof that it halts on all inputs. (See for example https://lean-lang.org/ https://lean-lang.org/ or David Turner's work on Total Functional Programming from about 20 years ago.) Other examples are easy to find. The simplex algorithm for linear optimization requires exponential time in general, and the problem it solves is NP-hard, but in practice works well on problems of interest and is widely used. Or consider the dynamic programming algorithms for problems like subset-sum. Theory is important, but engineering is also important.
      - soraminazuki 5mo ago
        But it's not like these systems make theory go away, they make compromises. So the question is, what's the compromise required for an algorithm that can check the conformance of computer programs to natural language specifications that doesn't involve hoping for the best?
        wizzwizz4 5mo ago
        Natural language specifications often aren't specifications at all: interpreting them requires context that is not available to the computer, and often not even available to the specification's authors without further research / decision-making. LLMs address this problem by just making things up (and they don't do a great job of comprehending the natural language, either), which I think qualifies as "hoping for the best", but I'm not sure there is another way, unless you reframe the problem to allow the algorithm to request the information it's missing.
montroser 5mo ago
This could very well be a pattern that some teams evolve into. Specs are the new source -- they describe the architectural approach, as well as the business rules and user experience details. End to end tests are described here too. This all is what goes through PRs and review process, and the code becomes a build artifact.
- vips7L 5mo ago
  It just doesn’t work though. Anthropic couldn’t even get Claude to build a working C compiler which has a way better specification than any team can write and multiple reference implementations.
Ecys 5mo ago
very true. and we already know and agree with this. user experience/what the app actually does >>> actually implementing it. elon musk said this a looong time ago. we move from layer 1 (coding, how do we implement this?) to layer 2 thinking (what should the code do? what do we code? should we implement this? (what to code to get the most money?)) this is basic knowledge
- duskdozer 5mo ago
  Elon Musk has been saying Teslas would have fully autonomous self-driving within 1-3 years since 2013
  - vinnymac 5mo ago
    I left a similar comment elsewhere in this thread. I still remember when so many people hallucinated that we would suddenly have flying cars by 2002 at the latest. If we achieve several more major improvements on current technology, these thoughts are interesting to consider. But not before that occurs. We need the pragmatic engineer more than ever.
tyleo 5mo ago
The underlying mechanism is still the same: humans type and products come out. So something which must be true if this author is right is that whatever the new language is—the thing people are typing into markdown—must be able to express the same rigor in less words than existing source code. Otherwise the result is just legacy coding in a new programming language.
- SoftTalker 5mo ago
  > Otherwise the result is just legacy coding in a new programming language. And this is why starting with COBOL and through various implementations of CASE tools, "software through pictures" or flowcharts or UML, etc, which were supposed to let business SMEs write software without needing programmers, have all failed to achieve that goal.
  - tyleo 5mo ago
    While they failed to achieve the goal outright, I'd argue that each is a concrete step towards it. The languages we have today are more productive than the languages we had decades ago. I think it's an open question of whether we achieve the holy grail language as the submission describes. My guess is that we inch towards the submission's direction, even if we never achieve it. It won't surprise me if new languages take LLMs into account just like some languages now take the IDE experience into account.
- dpark 5mo ago
  > must be able to express the same rigor in less words than existing source code Yes but also no. Writing source means rigorously specifying the implementation itself in deep detail. Most of the time, the implementation does not need to be specified with this sort of rigor. Instead the observable behavior needs to be specified rigorously.
  - tyleo 5mo ago
    That doesn't sound right. For example, there's plenty of software with the correct observable behavior which leaks credentials. So what needs to be captured goes beyond observable behavior.
ramoz 5mo ago
> If I had to roll out such a development process today, I’d make a standardized Markdown specification the new unit of knowledge for the software project. Product owners and engineers could initially collaborate on this spec and on test cases to enforce business rules. Those should be checked into the project repositories along with the implementing code. There would need to be automated pull-request checks verifying not only that tests pass but that code conforms to the spec. This specification, and not the code that materializes it, is what the team would need to understand, review, and be held accountable for. The constant urge I have today is for some sort of spec or simpler facts to be continuously verified at any point in the development process; Something agents would need to be aware of. I agree with the blog and think it's going to become a team sport to manage these requirements. I'm going to try this out by evolving my open source tool [1] (used to review specs and code) into a bit more of a collaborative & integrated plane for product specs/facts - https://plannotator.ai/workspaces/ https://plannotator.ai/workspaces/ [1] https://github.com/backnotprop/plannotator https://github.com/backnotprop/plannotator
- jondwillis 5mo ago
  I’ve been considering this as well, and trying to get my colleagues to understand and start doing it. I use it to pretty decent effect in my vibe coded slop side projects. In the new world of mostly-AI code that is mostly not going to be properly reviewed or understood by humans, having a more and more robust manifestation and enforcement, and regeneration of the specs via the coding harness configuration combined with good old fashioned deterministic checks is one potential answer. Taken to an extreme, the code doesn’t matter, it’s just another artifact generated by the specs, made manifest through the coding harness configuration and CI. If cost didn’t matter, you could re-generate code from scratch every time the specs/config change, and treat the specs/config as the new thing that you need to understand and maintain. “Clean room code generation-compiler-thing.”
  - SpicyLemonZest 5mo ago
    > If cost didn’t matter, you could re-generate code from scratch every time the specs/config change, and treat the specs/config as the new thing that you need to understand and maintain. The critical insight is that this is not true. When people depend on your software, replacing it with an entirely different program satisfying all of your specs and configurations is a large, months-long project requiring substantial effort and coordination even after new program is written. It seems to work in vibe coded side projects because you don't have those dependencies; if you got an angry email from a CEO saying that moving a critical button ruined their monthly review cycle, and demanding 7 days notice before you move any buttons going forwards, you'd just tell them no.
debesyla 5mo ago
I found that adding "philosophy" descriptions help guide the tooling. No specs, just general vibes what's the point, because we can't make everyone happy and it's not a goal of a good tool (I believe). Technology, implementation may change, but general point of "why!?" stays.
ninalanyon 5mo ago
> Rework is almost free Is it? All the electricity and capital investment in computing hardware costs real money. Is this properly reflected in the fees that AI companies charge or is venture capital propping each one up in the hope that they will kill off the competition before they run out of (usually other people's) money?
- gessha 5mo ago
  Yeah, a lot of Claude Code users(me included) found in March if rework is free or not.
- fractaled 5mo ago
  Even ignoring the AI costs, 'rework' is going to be more expensive as soon as you have customers. For example any sort of data migration. Or UX expectations. Or public API interface. None of these can change without some thought, so one would be leaning on these specs quite a lot.
moritzwarhier 5mo ago
Entertaining flag name! React team seems to really have set a precedent with their "dangerouslySetInnerHTML" idea. Or did they borrow it somewhere? I'm just curious about that etymology, of course the idea is not universally helpful: for example, for dd CLI parameters, it would only make a mess. But when there's a flag/option that really requires you to be vigilant and undesired the input and output and all edge cases, calling it "dangerous" is quite a feat!
- wrxd 5mo ago
  I’m pretty sure this comes from Claude code’s --dangerously-skip-permissions
  - saulpw 5mo ago
    which sounds like it came from React's "dangerouslySetInnerHTML", per the comment you replied to.
    - brabel 5mo ago
      I think people used similar prefixes for a long time. For example, Haskell has had `unsafePerformIO` since the 90's... and MSFT's Hungarian notation was also similar, though it used abbreviations for things like "unsafe" (not "dangerous"). Perhaps React was the most famous case of using "dangerously" though.
      - culi 5mo ago
        "unsafe" seems quite different from the "dangerously [...]" phrasal template. I don't think it's a stretch to suppose it was inspired by React. Still waiting for this one to catch on: React.__SECRET_INTERNALS_DO_NOT_USE_OR_YOU_WILL_BE_FIRED https://github.com/reactjs/react.dev/issues/3896 https://github.com/reactjs/react.dev/issues/3896
        urbandw311er 5mo ago
hombre_fatal 5mo ago
Yeah, this has been my process for months now. I might even start my own blog to write about things I've found. 1. Always get the agent to create a plan file (spec). Whatever prompt you were going to yolo into the agent, do it in Plan Mode first so it creates a plan file. 2. Get agents to iterate on the plan file until it's complete and thorough. You want some sort of "/review-plan <file>" skill. You extend it over time so that the review output is better and better. For example, every finding should come with a recommended fix. 3. Once the plan is final, have an agent implement it. 4. Check the plan in with the impl commit. The plan is the unit of work really since it encodes intent. Impl derives from it, and bugs then become a desync from intent or intent that was omitted. It's a nicer plane to work at. From this extends more things: PRs should be plan files, not code. Impl is trivial. The hard part is the plan. The old way of deriving intent from code sucked. Why even PR code when we haven't agreed on a plan/intent? This process also makes me think about how code implementation is just a more specific specification about what the computer should do. A plan is a higher level specification. A one-line prompt into an LLM is the highest level specification. It's kinda weird to think about. Finally, this is why I don't have to read code anymore. Over time, my human review of the code unearthed fewer and fewer issues and corrections to the point where it felt unnecessary. I only read code these days so I can impose my preferences on it and get a feel for the system, but one day you realize that you can accumulate your preferences (like, use TDD and sum types) in your static prompt/instructions. And you're back to watching this thing write amazing code, often better than what you would have written unless you have maximum time + attention + energy + focus no matter how uninteresting the task, which you don't.
- geraneum 5mo ago
  > PRs should be plan files, not code. Impl is trivial. Doesn’t it bother you that the outcome each PR is different every time you/CI “run it”?
  - hombre_fatal 5mo ago
    No, because consider the pre-AI status quo where a human PR will come in like "Added tab support", maybe scribbles out some guiding ideas, maybe references some issue where we kinda hashed out some ideas of how it could work, and then we must derive all of the intentions/assumptions/decisions of the implementor from the PR's code changes. Basically zero plan. Or rather, the "internal" plan that the human implementor used while writing the code is hidden from us because it's a mix of ideas they held in their head, jotted in some notes, existed in a sequence of commits that were lost when squashed into a PR, etc. There's zero reproducibility in the implementation. So take my idea and pretend we still don't have AI yet: the main point is that we move to a pipeline where we work on a first-class plan first before we begin implementation. This gets us closer to reproducible implementation no matter who is implementing it. It just so happens that now with implementation becoming automated, we have more attention and energy freed up to focus on this plan-based model.
jmull 5mo ago
The lesson I've learned from our new AI age is how little a large number of people who've worked in software development their entire careers understand software development. I suppose all the money floating around AI helps dummify everything, as people glom on to narratives, regardless of merit, that might position them to partake. What we actually have now is the ability to bang out decent quality code really fast and cheaply. This is massive, a huge change, one which upends numerous assumptions about the business of software development. ...and it only leaves us to work through every other aspect of software development. The approach this article advocates is to essentially pretend none of this exists. Simple, but will rarely produce anything of value. This paragraph from the post gives you the gist of it: > ...we need to remove humans-in-the-loop, reduce coordination, friction, bureaucracy, and gate-keeping. We need a virtually infinite supply of requirements, engineers acting as pseudo-product designers, owning entire streams of work, with the purview to make autonomous decisions. Rework is almost free so we shouldn’t make an effort to prevent incorrect work from happening. As if the only reason we ever had POs or designers or business teams, or built consensus between multiple people, or communicated with others, or reviewed designs and code, or tested software, was because it took individual engineers too long to bang out decent code. AI has just gotten people completely lost. Or I guess just made it apparent they were lost the whole time?
- vinnymac 5mo ago
  I appreciate your insights in a sea of psychosis comments. I find it strange how many people think we have achieved the likes of Y2K flying cars 20 years ago, or the dream of having every car on the road be an electric fully self driving car by now (a promise made at least over a decade ago by several of these types). The point I’m making is that we give the spotlight to people who are making absurd claims. We have not achieved the ability to remove the human from the loop and continually produce value-able outputs. Until we do, I don’t see how any of the claims made in this article are even close to anything more than simply gate-keeping slop.
  - ninalanyon 5mo ago
    And if we do remove the human from the loop? What then, what are humans for? Do we get Keynes' idea that we only need to work a few hours a week or do we get a continuation and intensification of what we already have: a few high 'earners' and a sea of people struggling to make ends meet?
testplzignore 5mo ago
> Product owners and engineers could initially collaborate on this spec and on test cases to enforce business rules. LOL. I had to check if this was published on April 1st.
fijiol 5mo ago
[flagged]
lesscode 5mo ago
Instead of accepting 20,000 lines of slop per PR (and never-ending combinatorial complexity), maybe we should aim to think about abstractions and how to steer LLMs to generate code similar to that of a skilled human developer. Then it could actually be a maintainable artifact by humans and LLMs alike.
- crnkofe 5mo ago
  I don't get why every AI article is so hyper-focused on coding speed. If the coding is so fast doesn't it make sense to invest more time into quality, learning, documentation, testing refactoring, making a better product? I'm beginning to think that the slopcoders are evaluated by kLOCs of lines written in addition to LLM token usage and they're just maximising the measured metrics. Whether that actually ends up in production or is used by any real person is seemingly irrelevant. Likely the more bugs that are produced the more agents can be spun in parallel to simulate busywork.
  - lesscode 4mo ago
    Indeed. If the goal is to make useful (quality) software artifacts, we are going in the wrong direction.
phyzix5761 5mo ago
I wonder if with the speed of iteration with AI the industry will switch back to waterfall. Clear documentation first so the LLM can easily produce what's being asked with a round of testing before going back to the documentation stage and running it again. History does repeat itself.
- yibers 5mo ago
  We already switched
Ozzie-D 5mo ago
[flagged]
farmerbb 5mo ago
I legit can't tell if this article is satire, or not.
retinaros 5mo ago
markdown became the language I hate the most thank to LLMs and specs-driven approach. everything feels so dumb right now in agentic coding. looping blindlessly and aimlessly until it compiles then until the playwright server or whatever devtools shows that it somehow works. push the code, have a llm autoreview/autofix,push to prod, run a mythos (perfect name) to identify the bug that opus 4.7 create. loops on loops on loops of some kind of zombie processes running to a "goal" that everyone seems to mystify in talks to just hide the fact that we do nothing anymore. the bottleneck never was code. it was the gate that was keeping away the Elizabeth Holmes and SBF from software engineering and it just opened.
- abalashov 5mo ago
  A colleague and I have taken to use of the verb "meatspin", from another era in Internet shock humour, to describe what it is that coding agents actually do 99% of the time.
throwaw12 5mo ago
> my first bet would be specifications and tests You are missing another dimension how easy it would be to migrate if adding new feature hits a ceiling and LLM keeps breaking the system. Imagine all tests are passing and code is confirming the spec, but everything is denormalized because LLM thought this was a nice idea at the beginning since no one mentioned that requirement in the spec. After a while you want to add a feature which requires normalized table and LLM keeps failing, but you also have no idea how this complex system works. Don't forget that very very detailed spec is actually the code
- abalashov 5mo ago
  > Don't forget that very very detailed spec is actually the code Came here to say this, but you said it for me. If the problem were merely one of insufficient rigour or detail in specs, it would have been solved long before LLMs.
- theshrike79 5mo ago
  > Don't forget that very very detailed spec is actually the code In the age of AI this is more true than you know. Given a detailed enough spec and test suite you can effectively rewrite any application with any language in a fully automated way. I've coined that as "Duck coding" :D If it quacks like a duck, walks like a duck and looks like a duck - it's duck enough for as far as the spec is concerned. Does it matter what is inside the duck?
  - npodbielski 5mo ago
    Yes. It would be like buying a car that you have no idea about the engine and gearbox and kind of fuel it is using or if at all. It have four wheels and and it can drive you from point A to B. Sure but sometimes it happens that some particular brands and they particular model requires engine renovation after 100k km because it is so shitty design. Right now we are just starting vibe coded software, nobody knows how it will behave in 2 years or 5 or 10. My guess it won't. So we will enter age of scratch software. You build it, ship it. And after few months you will ship entirely new one. And then again. And again. And again. Because maintenance is hard and costly and writing from scratch will cost like 1k$ in tokens. And users will have problem of migrating the data if possible at all. But if migration is hard and everything changes all the time does it even matter if you are using X o Y software? Does it even matter since you can write your own software and migrate your data there? I think we saw how this ends with Chinese manufacturing. You buy some stuff from AliExpress for 2$ and throw it away in two weeks and buy a new one. So quality does not matter anymore.
zoogeny 5mo ago
>... my first bet would be specifications ... and tests ... If I had to roll out such a development process today, I’d make a standardized Markdown specification the new unit of knowledge for the software project. I've found that adopting RFC Keywords (e.g. RFC 2119 [1]; MUST, SHOULD, MAY) at least makes the LLM report satisfaction. I'd love to see a proper study on the usage of RFC keywords and their effect on compliance and effectiveness. 1. https://www.rfc-editor.org/info/rfc2119/ https://www.rfc-editor.org/info/rfc2119/
- kortex 5mo ago
  That's literally what OpenSpec does (https://openspec.dev/ https://openspec.dev/). It's quite nice. I've only exceptionally rarely seen claude do something wrong based on spec docs when it's fully spec'd out. More often it's because something wasn't nailed down and claude was forced to make assumptions. The downside is the ospx markdown specs sometimes end up too granular, focusing on the wrong or less important details, so reading the specs feels like a slog. Also at times aspects of the english language spec end up way more verbose than just giving a code example would be.
  - jonnytran 5mo ago
    Is it time for the literate programming renaissance?
DavidVoid 5mo ago
> Product owners and engineers could initially collaborate on this spec and on test cases to enforce business rules. Those should be checked into the project repositories along with the implementing code. There would need to be automated pull-request checks verifying not only that tests pass but that code conforms to the spec. This specification, and not the code that materializes it, is what the team would need to understand, review, and be held accountable for. This just sounds like typical requirements management software (IBM DOORS for example, which has been around since the 90s). It's kind of funny how AI evangelists keep re-discovering the need for work methods and systems that have existed for decades. When I worked as a software developer at a big telecom company and I had no say in what the software was supposed to do, that was up to the software design people--they were the ones responsible for designing the software and defining all the requirements--I was just responsible for implementing that behavior in code.
- irishcoffee 5mo ago
  One of my first tasks at my first job out of college required me to learn dxl (doors extension language) and implement some really intricate requirements management features. It was gratifying to build the confidence of learning a new language quickly that I had never even heard of before. DXL was also pretty awful. Opened a lot of doors for me though, no pun intended.
- bitwize 5mo ago
  Spec-driven development is basically PRIDE, the first proven commercial software methodology dating back to 1971. In fact it may be the culmination of PRIDE because PRIDE's creators realized coding wasn't the hard part; the hard part was systems analysis, determining what problem needed to be solved and what to build. Coding comes last and when you did it right, was simply a translation step. And now that step can be 100% automated. Information systems design was a solved problem in the 1970s. PRIDE turned it from an art into a proven, repeatable science. Programmers, afraid of losing their perceived importance, resisted the discipline it imposes as the mustang resists the bit, but now that they're going the way of buggy-whip makers, maybe systems design as a science will make a comeback after 50 years.
k3vinw 5mo ago
> We can stop reading LLM-generated code just like we don’t read assembly, or bytecode, or transpiled JavaScript; our high-level language source would now be another form of machine code This is too weird for me. At least with programming languages I can consult the documentation and if the programming language isn’t behaving as documented, it’s obviously a defect and if you’re savvy enough you often have open channels that accept contributions. Can we say the same for Claude or other AI solutions?
- Sevii 5mo ago
  If you run a local LLM and an open source agent harness you are pretty close to that.
  - throawayonthe 5mo ago
    can you explain how? with a compiler you can rely on the adage "it's never a compiler bug" (until it is! and then you can fix it) how can a local LLM with an open source agent harness provide the same trustworthiness?
    - zoogeny 5mo ago
      > ... then you can fix it I recall working on a project that used (MSVC) VC++ and a coworker found a bug in the compiler. We reported the issue to Microsoft and they eventually patched it. You may find yourself arguing explicitly for open source dev tools if you continue down this line. There are many commercial cases where "you can fix it" does not apply to the dev toolchain and you will find yourself reliant on a provider. At that point, the trustworthiness of "compiler provider" and "local LLM provider" is the pertinent discussion (e.g. provider vs. provider instead of LLM vs compiler).
      - skydhash 5mo ago
        > There are many commercial cases where "you can fix it" does not apply to the dev toolchain and you will find yourself reliant on a provider. That’s only on the hobbyist level. On the enterprise level, there are lots of contracts involved that requires speedy bugs correction.
humbleharbinger 5mo ago
My amazon orgs leadership has been obsessed with spec driven development while individual engineers tell me the only use they have is to placate leadership. I'm tired
- culi 5mo ago
  How does spec driven development differ from test driven development?
  - nullsex 5mo ago
    [dead]
  - invalidator 5mo ago
    TDD is done in a tight loop (minutes) while coding. For every little micro-feature/fix, you write a test for the new behavior you want, implement the minimal ugly fix to get the test to pass, then rely on the tests so you don't regress as you clean up. LLMs struggle with TDD. They want to generate a bunch of code and tests in large passes. You can instruct them to do red/green TDD, but the results aren't great. SDD starts before implementation, and formalizes intent and high-level design. LLMs eat it up. The humans can easily reinvent the worst parts of waterfall if they're not careful. They're not mutually exclusive.
    - culi 5mo ago
      In many frameworks the tests are referred to as the "spec". I guess that's where my confusion arises from. > SDD starts before implementation No different from TDD.
donbventures 5mo ago
[flagged]
dundunUp 5mo ago
[flagged]
Uptrenda 5mo ago
Does this post mark the top of the hype train or is there still more to come?
- 0xpgm 5mo ago
  Still more to come I think. Until all the major AI companies IPO starting this year.
facundo_olano 5mo ago
Author here. I'm surprised to see this surfacing now. I just wanted to clarify, since apparently the post doesn't do a good job at it, that what I discussed there is not a methodology I advocate for. The point of the post was: ok, since there are organizations mandating to maximize speed by reducing time spent on typing code (or even mandating to maximize agents usage), is there a way we can meet that requirement while still preserving the rigor somewhere else? This was a follow up to a previous article[1] and the pair tried to express what I still think today (using AI daily at work): every time I use AI for coding, to some capacity I'm sacrificing system understanding and stability in favor of programming speed. This is not necessarily always a bad tradeoff, but I think it's important to constantly remind ourselves we are making it. [1] https://olano.dev/blog/tactical-tornado/ https://olano.dev/blog/tactical-tornado/
- ignoreusernames 5mo ago
  Don’t you think that the provider of the LLM is also a dimension on these discussions about responsibility? We often talk about the tech itself (LLM driven development) but how we access it is just as important imo. It’s either locked behind a non trivial amount of hardware (for open models) or some monopolistic driven provider entity like OpenAI or anthropic. In the provider case, it’s not really the LLM that will “own” the code, it’s the provider itself and we’ll be at the mercy of whatever pricing model they shove down our throats.
- LelouBil 5mo ago
  I don't like the premise of the article, but I agree that if you accept the premise, the contents of the articles are a good way to do it.
- Uptrenda 5mo ago
  [flagged]
  - AlexCoventry 5mo ago
    He was establishing the context of The current blog post. Very unlikely that he was doing it for Google juice.
iloveoof 5mo ago
Software engineering has always worked this way, just not to ICs. “The LLMs produce non-deterministic output and generate code much faster than we can read it, so we can’t seriously expect to effectively review, understand, and approve every diff anymore. But that doesn’t necessarily mean we stop being rigorous, it could mean we should move rigor elsewhere.“ Direct reports, when delegated tasks by managers, product non-deterministic outputs much faster than team leads/managers can review, understand or approve every diff. Being a manager of software developers has always been a non-deterministic form of software engineering.
- devmor 5mo ago
  > Being a manager of software developers has always been a non-deterministic form of software engineering. Unless the manager is also a principal/architect, I don’t find this to be agreeable. It’s similar to saying that you are a non-deterministic chef when you order food from a restaurant.
- teaearlgraycold 5mo ago
  Well yes but if no humans at the company understand the code then no one is truly responsible for it.
  - Npovview 5mo ago
    what about the artifacts that were supposed to test the correctness of the code? are they passing willy nilly?
    - ekidd 5mo ago
      No amount of testing will save a large program with a dogshit architecture. Roughly, this is because tests increase coverage linearly with the number of tests, but weird interactions increase exponentially with code size. This might be fine if you're building a tiny app, or if you're building a medium-sized app that follows a strict existing architecture (like a web app consisting mostly of forms). In which case, have fun. But if you're building something slightly novel and interesting, then Claude is surprisingly bad at architecture and taste, and it tends to "fix" problems by spewing more slop. What you need instead is actual insight that leads to simplifying principles. This, in turn, allows breaking up the exponential complexity into disciplined patterns. This allows your code complexity to scale far more slowly, allowing an essentially linear number of tests to provide coverage. I actually download and try people's vibe-coded developer tools. And frankly, those tools are some of the worst software I've used in my life, worse than even Unix-vendor Motif implementations from the early 90s. Like, I'm super happy that people can vibe-code themselves simple, one-off personal tools. That's incredibly empowering. But that doesn't mean you can big, novel stuff the same way without a competent human actively in the loop.
ricardobeat 5mo ago
> We can’t leverage agents if our unit of work is still “add a new endpoint to the RESTful API” Why not? You just make every task faster. Not everything has to be an uncontrollable rocket launch. > We need a virtually infinite supply of requirements, engineers acting as pseudo-product designers, owning entire streams of work Why? To build what? You can only build as fast as you understand the business and your users.
- charcircuit 5mo ago
  >You can only build as fast as you understand the business and your users. It should be possible to go faster by having AI understand the business and users.
  - arcwhite 5mo ago
    It doesn't do that though. Understand. That's not how LLMs work.
    - charcircuit 5mo ago
      LLMs are not the only possible AI models to use and create.
      - paulryanrogers 5mo ago
        Aren't they state of the art though?
blacob 5mo ago
> Then where does the rigor go? Similar to the Thoughtworks report, my first bet would be specifications (which is not the same as prompts) and tests (which is not the same as TDD). This is what we're building for at Saldor (https://saldor.com https://saldor.com). It's a hard problem, to get a team in the habit of writing good specs. Probably because it's a hard thing to do: thinking of the behavior of your program, especially at the edges. But I agree (biased) that this is probably the way forward for writing code in the near future. I'm excited to see other people thinking about it.
- Terretta 5mo ago
  Saldor pitch is on point. I have team do this using CLAUDE.md telling Claude to do it in a set of interconnected steps, but in brief: they are to make it write every aspect of transcript somewhere: PRD, research notes, spec, dev log and debate log, break/fix/retro notes, commit log, PR, release notes, README, docs .mds... heavy emphasis on the edges in our thinking, and just as important, the edges in its ability to provide good leverage. It needs a core set of guidance on the ordering and how to write "as of" a given phase or release so context stays current, trusting the old info is in git history it can navigate for the story of how we got here. CC's /insights claims I have 10:1 md edits to code edits, and we both note this way of working is resulting in far fewer error loops per higher quality outcome. // So yes, interested in your product. Baking something more broadly battle tested in so we don't have to reinvent it makes sense.
nirui 5mo ago
> We can stop reading LLM-generated code just like we don’t read assembly, or bytecode, or transpiled JavaScript; our high-level language source would now be another form of machine code. My opinion is very close to this. Currently the reason that it's bad to not reviewing/testing the code LLMs generated is because the LLMs can sometime generate bad codes. But it's a bug that can be improved. One day you'll have LLMs generating code consistently better than what a human could write. And then you just stop needing to review them. (And that's probably also the time where most programmers/developers got fired too) Don't get surprised if anyday the LLMs starts to generate binaries directly. THAT will be impossible to read and costs more time to analyze.
- 9029 5mo ago
  Is it possible to reason or prove the correctness of an LLM?
- furyofantares 5mo ago
  > Currently the reason that it's bad to not reviewing/testing the code LLMs generated is that the LLMs can sometime generate bad codes. Sometimes? I am heavily into vibe coding and I think they almost always generate bad code. At least as soon as you're distant enough from the code to call it vibe coding. When you're still in touch with the code, have at least been recently talking to it about code rather than 100% about features, and its context is filled with good code, it can generate good code.
adelks 5mo ago
"A sufficiently precise spec is code". I've read somewhere here before. So guardrails, i.e. sufficiently precise spec and tests, will need to be as strict as the LLM is bad at getting the right context and asking back the right questions. I suppose at that point not much difference between a human engineer and it.
jelmersnoeck 5mo ago
> "I'd make a standardized Markdown specification the new unit of knowledge for the software project. ... There would need to be automated pull-request checks verifying not only that tests pass but that code conforms to the spec." Agree, this is how you make the development loop more deterministic and ultimately autonomous. It's how I've been using coding agents myself for the past few months (by building my own to support this natively [1]). If you have a spec you approve/agree on, have an agent code against it, and then have a review phase verify the implementation didn't drift from the spec (either by adding or removing features), you get to a position where you can trust the outcome. There's still a lot to be said about spec definition and what if during implementation gaps are discovered, and that's where HITL comes into play. [1] https://github.com/jelmersnoeck/forge https://github.com/jelmersnoeck/forge
immanuwell 5mo ago
it's the most honest framing I've seen, but specs as the new source of truth is exactly what we promised ourselves with UML, then WSDL, then OpenAPI. the graveyard of just make the artifact above the code authoritative is long
okandship 5mo ago
making the review artifact explicit feels like the part teams skip
CraigJPerry 5mo ago
I prefer "the bottleneck is understanding" framing. The author is nibbling at the same problem ultimately, but i don't think "hey one strategy is we could just let cognitive debt accumulate so we can go faster!" is a particularly insightful tool in the toolbox. Don't misread me, i'm not denying it can be a valid strategy. Instead i want to read about insightful strategies for optimising that system-wide bottleneck we have: understanding. Tell me about how you managed to shift to a higher level of abstraction, tell me about how and when that abstraction leaks. Tell me how you reduced the amount of information that has to flow through the system bottleneck.
stuaxo 5mo ago
In short: We will have code full of unknown bugs, that is unfixable. The solution is to replace it with more of the same but with some new specification (fix some bug add some new feature). And this will be done by using astounding amounts of compute in massive new data centres.
nlitened 5mo ago
I feel like people who program in JavaScript or whose projects pull megabytes of dependencies, don’t get a moral right to complain about this. You guys just sit and calm down this time, you already said what you could. Your app takes 20 seconds to load, pulling 50 megabytes of minified JS. Your backend is a mess of 20 Rust microservices, 300 megabytes docker image each. Nobody has actually been reading and understanding code in your org for the past 15 years. And nobody has ever been responsible, everybody has just been job hopping for a 15% total comp bump. Now the secret is out.
EFLKumo 5mo ago
> just like we don’t read assembly, or bytecode, or transpiled JavaScript This makes sense since certain higher-level code produces certain lower-level code, while LLM cannot. If the transpired JS code doesn't work we could just find out the bug in minifiers, etc. but one cannot figure out why LLM fails at one task, especially considering LLMs, even SOTA ones, could be strongly affected by even small prompt changes. Taking this into consideration, I don't think this is a sound reasoning why we don't need to review ai-generated code. > The LLMs produce non-deterministic output and generate code much faster than we can read it, so we can’t seriously expect to effectively review, understand, and approve every diff anymore. Exactly. However, this could also indicate a weaker review standard instead of just dropping review. We could also suggest an idea where devs mainly review code design or interfaces, leveraging one's *taste*, while leaving strict logic reasoning, validating and testing to other tools or approaches. It cannot pursuade me that the nature of LLM's code generation must lead to a complete cancel of the code review. Anyway, I'm not opposing this article and its thought of shift in the future is really good.
- trimethylpurine 5mo ago
  Couldn't we slowly add guardrails that eventually lead to code generation becoming more and more deterministic over time? I'm seeing in my experience that Claude has become better with every version at producing uniformity in its code output. Especially where the architecture is clear and documented. And even more so in languages with built in uniformity (Go, HTMX, SQL) where there is intentionally only one or two ways of doing things. In such environments, the output is nearly deterministic.
  - EFLKumo 5mo ago
    I once thought about this and found that n-shots makes greater influences on LLMs. In other words, in a repo with good code quality and architecture (which offers good n-shots) and on a task with clear instructions and goals, LLM's output seems reliable enough, which meets your opinion. And n-shots is always better than relying on instruction following, instruction following mentioned in the article ("specifications") as an approach facing LLM's productivity, so imo the idea you suggested is another probability against/comparing with the article as well.
QuantumNoodle 5mo ago
It's too early for me to have a firm opinion one way or another. Just a data point: this month I had a knarly bug in generated bpf code. The C language was correct but the compiler produced a bug that corrupted packets. I spent around 8 hours debugging _where_ the issue is and how to work around, never really understanding what went wrong. That knowledge came with several more days on and off looking at it--after I had mitigated the production issue. So if I extrapolate this experience to LLMs (who are not deterministic) and who will make larger systems. What we trade for velocity we will pay for with hours of debugging because we won't understand how things work. I think this is unavoidable. Another way I'm looking at it: after some time of not writing code, it will be analogous to instructing the LLM and the output being assembly--where I simply don't have the muscle to grok the output. How do I mitigate that knowledge gap? I see micro serves coming back. Today it is easy to slop up disposable scripts. Our services need to be modular so we can dispose of broken things--so they are only coupled with each other by strict APIs.

200 points by fagnerbrack 5mo ago

[flagged]

> There would need to be automated pull-request checks verifying not only that tests pass but that code conforms to the spec. As I understand, this is an unsolved problem.

Ecys 5mo ago
this is actually precisely what humans' roles will be. "is this implementation/code actually aligned with what i want to do?" humanic responsibility's focus will move entirely from implementing code to deciding whether it should be implemented or not. u probably mean unsolved as in "not yet able to be automated", and that's true. if pull-request checks verifying that tests are conforming to the spec are automated, then we'd have AGI.
- wizzwizz4 5mo ago
  This is a task that humans are exceptionally bad at, because we are not computers. If something uses the right words in the right order such that it communicates the correct algorithm to a human, then a human is likely to say "yup, that's correct", even if an hour's study of these 15 lines reveals that a subtle punctuation choice, or a subtle mismatch between a function's name and its semantics, would reveal that it implements a different algorithm to the expected one. LLMs do not understand prose or code in the same way humans do (such that "understand" is misleading terminology), but they understand them in a way that's way closer to fuzzy natural language interpretation than pedantic programming language interpretation. (An LLM will be confused if you rename all the variables: a compiler won't even notice.) So we've built a machine that makes the kinds of mistakes that humans struggle to spot, used RLHF to optimise it for persuasiveness, and now we're expecting humans to do a good job reviewing its output. And, per Kernighan's law: > Everyone knows that debugging is twice as hard as writing a program in the first place. So if you're as clever as you can be when you write it, how will you ever debug it? And that's the ideal situation where you're the one who's written it: reading other people's code is generally harder than reading your own. So how do you expect to fare when you're reading nobody's code at all?
  - Ecys 5mo ago
    i meant on a higher, agentic level where the AI's code is infallible. and that's going to happen very soon: say: human wants to make a search engine that money for them. 1. for a task, ask several agents to make their own implementation and a super agent to evaluate each one and interrogate each agent and find the best implementation/variable names, and then explain to the human what exactly it does. or just mythos 2. the feature is something like "let videos be in search results, along with links" 3. human's job "is it worth putting videos in this search engine? will it really drive profits higher? i guess people will stay on teh search engine longer, but hmmm maybe not. maybe let's do some a/b testing and see whether it's worth implementing???" etc... this is where the developer has to start thinking like a product manager. meaning his position is abolished and the product manager can do the "coding" part directly. now this should be basic knowledge in 2026. i am just reading and writing back the same thing on HN omds.
    - wizzwizz4 5mo ago
      The AI's code is not going to be infallible any time soon. It's been "very soon" for the past 4 years, and the AI systems are still making the same kinds of mistakes, which are the mistakes you'd expect from a first-principles study of their model architectures. There's no straightforward path to modifying the systems we have now, to make them infallible.
- phainopepla2 5mo ago
  > humanic No, thank you
- lefra 5mo ago
  That's what formal verification is about. I did some (using PSL for hardware verification); writing the formal spec is way harder than the actual code. It will find a lot of subtle issues, and you spend a most of the time deciding if it's the spec or the code that's wrong. Having the code-writing part automated would have a negligible impact on the total project time.
InsideOutSanta 5mo ago
Step 1: solve the halting problem.
- soraminazuki 5mo ago
  Yep, calling it an "unsolved" problem is a misnomer. We already have mathematical proof that it's impossible. But that aside, it's such a shame that many drinking the AI Kool-Aid aren't even aware of the theoretical limits of a computer's capabilities.
  - mjd 5mo ago
    This sort of theoretical result is not always as clear-cut as you suggest. Computers are finite machines. There is a theorem that although a machine with finite memory can add, multiplication requires unbounded memory. Somehow we muddle along and use computers for multiplication anyway. More to your point there is a whole field of people who write useful programs using languages in which every program must be accompanied by a proof that it halts on all inputs. (See for example https://lean-lang.org/ https://lean-lang.org/ or David Turner's work on Total Functional Programming from about 20 years ago.) Other examples are easy to find. The simplex algorithm for linear optimization requires exponential time in general, and the problem it solves is NP-hard, but in practice works well on problems of interest and is widely used. Or consider the dynamic programming algorithms for problems like subset-sum. Theory is important, but engineering is also important.
    - soraminazuki 5mo ago
      But it's not like these systems make theory go away, they make compromises. So the question is, what's the compromise required for an algorithm that can check the conformance of computer programs to natural language specifications that doesn't involve hoping for the best?
      - wizzwizz4 5mo ago
        Natural language specifications often aren't specifications at all: interpreting them requires context that is not available to the computer, and often not even available to the specification's authors without further research / decision-making. LLMs address this problem by just making things up (and they don't do a great job of comprehending the natural language, either), which I think qualifies as "hoping for the best", but I'm not sure there is another way, unless you reframe the problem to allow the algorithm to request the information it's missing.

montroser 5mo ago

This could very well be a pattern that some teams evolve into. Specs are the new source -- they describe the architectural approach, as well as the business rules and user experience details. End to end tests are described here too. This all is what goes through PRs and review process, and the code becomes a build artifact.

vips7L 5mo ago
It just doesn’t work though. Anthropic couldn’t even get Claude to build a working C compiler which has a way better specification than any team can write and multiple reference implementations.

Ecys 5mo ago

very true. and we already know and agree with this. user experience/what the app actually does >>> actually implementing it. elon musk said this a looong time ago. we move from layer 1 (coding, how do we implement this?) to layer 2 thinking (what should the code do? what do we code? should we implement this? (what to code to get the most money?)) this is basic knowledge

duskdozer 5mo ago
Elon Musk has been saying Teslas would have fully autonomous self-driving within 1-3 years since 2013
- vinnymac 5mo ago
  I left a similar comment elsewhere in this thread. I still remember when so many people hallucinated that we would suddenly have flying cars by 2002 at the latest. If we achieve several more major improvements on current technology, these thoughts are interesting to consider. But not before that occurs. We need the pragmatic engineer more than ever.

tyleo 5mo ago

The underlying mechanism is still the same: humans type and products come out. So something which must be true if this author is right is that whatever the new language is—the thing people are typing into markdown—must be able to express the same rigor in less words than existing source code. Otherwise the result is just legacy coding in a new programming language.

SoftTalker 5mo ago
> Otherwise the result is just legacy coding in a new programming language. And this is why starting with COBOL and through various implementations of CASE tools, "software through pictures" or flowcharts or UML, etc, which were supposed to let business SMEs write software without needing programmers, have all failed to achieve that goal.
- tyleo 5mo ago
  While they failed to achieve the goal outright, I'd argue that each is a concrete step towards it. The languages we have today are more productive than the languages we had decades ago. I think it's an open question of whether we achieve the holy grail language as the submission describes. My guess is that we inch towards the submission's direction, even if we never achieve it. It won't surprise me if new languages take LLMs into account just like some languages now take the IDE experience into account.
dpark 5mo ago
> must be able to express the same rigor in less words than existing source code Yes but also no. Writing source means rigorously specifying the implementation itself in deep detail. Most of the time, the implementation does not need to be specified with this sort of rigor. Instead the observable behavior needs to be specified rigorously.
- tyleo 5mo ago
  That doesn't sound right. For example, there's plenty of software with the correct observable behavior which leaks credentials. So what needs to be captured goes beyond observable behavior.

ramoz 5mo ago

> If I had to roll out such a development process today, I’d make a standardized Markdown specification the new unit of knowledge for the software project. Product owners and engineers could initially collaborate on this spec and on test cases to enforce business rules. Those should be checked into the project repositories along with the implementing code. There would need to be automated pull-request checks verifying not only that tests pass but that code conforms to the spec. This specification, and not the code that materializes it, is what the team would need to understand, review, and be held accountable for. The constant urge I have today is for some sort of spec or simpler facts to be continuously verified at any point in the development process; Something agents would need to be aware of. I agree with the blog and think it's going to become a team sport to manage these requirements. I'm going to try this out by evolving my open source tool [1] (used to review specs and code) into a bit more of a collaborative & integrated plane for product specs/facts - https://plannotator.ai/workspaces/ https://plannotator.ai/workspaces/ [1] https://github.com/backnotprop/plannotator https://github.com/backnotprop/plannotator

jondwillis 5mo ago
I’ve been considering this as well, and trying to get my colleagues to understand and start doing it. I use it to pretty decent effect in my vibe coded slop side projects. In the new world of mostly-AI code that is mostly not going to be properly reviewed or understood by humans, having a more and more robust manifestation and enforcement, and regeneration of the specs via the coding harness configuration combined with good old fashioned deterministic checks is one potential answer. Taken to an extreme, the code doesn’t matter, it’s just another artifact generated by the specs, made manifest through the coding harness configuration and CI. If cost didn’t matter, you could re-generate code from scratch every time the specs/config change, and treat the specs/config as the new thing that you need to understand and maintain. “Clean room code generation-compiler-thing.”
- SpicyLemonZest 5mo ago
  > If cost didn’t matter, you could re-generate code from scratch every time the specs/config change, and treat the specs/config as the new thing that you need to understand and maintain. The critical insight is that this is not true. When people depend on your software, replacing it with an entirely different program satisfying all of your specs and configurations is a large, months-long project requiring substantial effort and coordination even after new program is written. It seems to work in vibe coded side projects because you don't have those dependencies; if you got an angry email from a CEO saying that moving a critical button ruined their monthly review cycle, and demanding 7 days notice before you move any buttons going forwards, you'd just tell them no.

debesyla 5mo ago

I found that adding "philosophy" descriptions help guide the tooling. No specs, just general vibes what's the point, because we can't make everyone happy and it's not a goal of a good tool (I believe). Technology, implementation may change, but general point of "why!?" stays.

ninalanyon 5mo ago

> Rework is almost free Is it? All the electricity and capital investment in computing hardware costs real money. Is this properly reflected in the fees that AI companies charge or is venture capital propping each one up in the hope that they will kill off the competition before they run out of (usually other people's) money?

gessha 5mo ago
Yeah, a lot of Claude Code users(me included) found in March if rework is free or not.
fractaled 5mo ago
Even ignoring the AI costs, 'rework' is going to be more expensive as soon as you have customers. For example any sort of data migration. Or UX expectations. Or public API interface. None of these can change without some thought, so one would be leaning on these specs quite a lot.

moritzwarhier 5mo ago

Entertaining flag name! React team seems to really have set a precedent with their "dangerouslySetInnerHTML" idea. Or did they borrow it somewhere? I'm just curious about that etymology, of course the idea is not universally helpful: for example, for dd CLI parameters, it would only make a mess. But when there's a flag/option that really requires you to be vigilant and undesired the input and output and all edge cases, calling it "dangerous" is quite a feat!

wrxd 5mo ago
I’m pretty sure this comes from Claude code’s --dangerously-skip-permissions
- saulpw 5mo ago
  which sounds like it came from React's "dangerouslySetInnerHTML", per the comment you replied to.
  - brabel 5mo ago
    I think people used similar prefixes for a long time. For example, Haskell has had `unsafePerformIO` since the 90's... and MSFT's Hungarian notation was also similar, though it used abbreviations for things like "unsafe" (not "dangerous"). Perhaps React was the most famous case of using "dangerously" though.
    - culi 5mo ago
      "unsafe" seems quite different from the "dangerously [...]" phrasal template. I don't think it's a stretch to suppose it was inspired by React. Still waiting for this one to catch on: React.__SECRET_INTERNALS_DO_NOT_USE_OR_YOU_WILL_BE_FIRED https://github.com/reactjs/react.dev/issues/3896 https://github.com/reactjs/react.dev/issues/3896
      - urbandw311er 5mo ago

hombre_fatal 5mo ago

Yeah, this has been my process for months now. I might even start my own blog to write about things I've found. 1. Always get the agent to create a plan file (spec). Whatever prompt you were going to yolo into the agent, do it in Plan Mode first so it creates a plan file. 2. Get agents to iterate on the plan file until it's complete and thorough. You want some sort of "/review-plan <file>" skill. You extend it over time so that the review output is better and better. For example, every finding should come with a recommended fix. 3. Once the plan is final, have an agent implement it. 4. Check the plan in with the impl commit. The plan is the unit of work really since it encodes intent. Impl derives from it, and bugs then become a desync from intent or intent that was omitted. It's a nicer plane to work at. From this extends more things: PRs should be plan files, not code. Impl is trivial. The hard part is the plan. The old way of deriving intent from code sucked. Why even PR code when we haven't agreed on a plan/intent? This process also makes me think about how code implementation is just a more specific specification about what the computer should do. A plan is a higher level specification. A one-line prompt into an LLM is the highest level specification. It's kinda weird to think about. Finally, this is why I don't have to read code anymore. Over time, my human review of the code unearthed fewer and fewer issues and corrections to the point where it felt unnecessary. I only read code these days so I can impose my preferences on it and get a feel for the system, but one day you realize that you can accumulate your preferences (like, use TDD and sum types) in your static prompt/instructions. And you're back to watching this thing write amazing code, often better than what you would have written unless you have maximum time + attention + energy + focus no matter how uninteresting the task, which you don't.

geraneum 5mo ago
> PRs should be plan files, not code. Impl is trivial. Doesn’t it bother you that the outcome each PR is different every time you/CI “run it”?
- hombre_fatal 5mo ago
  No, because consider the pre-AI status quo where a human PR will come in like "Added tab support", maybe scribbles out some guiding ideas, maybe references some issue where we kinda hashed out some ideas of how it could work, and then we must derive all of the intentions/assumptions/decisions of the implementor from the PR's code changes. Basically zero plan. Or rather, the "internal" plan that the human implementor used while writing the code is hidden from us because it's a mix of ideas they held in their head, jotted in some notes, existed in a sequence of commits that were lost when squashed into a PR, etc. There's zero reproducibility in the implementation. So take my idea and pretend we still don't have AI yet: the main point is that we move to a pipeline where we work on a first-class plan first before we begin implementation. This gets us closer to reproducible implementation no matter who is implementing it. It just so happens that now with implementation becoming automated, we have more attention and energy freed up to focus on this plan-based model.

jmull 5mo ago

The lesson I've learned from our new AI age is how little a large number of people who've worked in software development their entire careers understand software development. I suppose all the money floating around AI helps dummify everything, as people glom on to narratives, regardless of merit, that might position them to partake. What we actually have now is the ability to bang out decent quality code really fast and cheaply. This is massive, a huge change, one which upends numerous assumptions about the business of software development. ...and it only leaves us to work through every other aspect of software development. The approach this article advocates is to essentially pretend none of this exists. Simple, but will rarely produce anything of value. This paragraph from the post gives you the gist of it: > ...we need to remove humans-in-the-loop, reduce coordination, friction, bureaucracy, and gate-keeping. We need a virtually infinite supply of requirements, engineers acting as pseudo-product designers, owning entire streams of work, with the purview to make autonomous decisions. Rework is almost free so we shouldn’t make an effort to prevent incorrect work from happening. As if the only reason we ever had POs or designers or business teams, or built consensus between multiple people, or communicated with others, or reviewed designs and code, or tested software, was because it took individual engineers too long to bang out decent code. AI has just gotten people completely lost. Or I guess just made it apparent they were lost the whole time?

vinnymac 5mo ago
I appreciate your insights in a sea of psychosis comments. I find it strange how many people think we have achieved the likes of Y2K flying cars 20 years ago, or the dream of having every car on the road be an electric fully self driving car by now (a promise made at least over a decade ago by several of these types). The point I’m making is that we give the spotlight to people who are making absurd claims. We have not achieved the ability to remove the human from the loop and continually produce value-able outputs. Until we do, I don’t see how any of the claims made in this article are even close to anything more than simply gate-keeping slop.
- ninalanyon 5mo ago
  And if we do remove the human from the loop? What then, what are humans for? Do we get Keynes' idea that we only need to work a few hours a week or do we get a continuation and intensification of what we already have: a few high 'earners' and a sea of people struggling to make ends meet?

testplzignore 5mo ago

> Product owners and engineers could initially collaborate on this spec and on test cases to enforce business rules. LOL. I had to check if this was published on April 1st.

fijiol 5mo ago

[flagged]

lesscode 5mo ago

Instead of accepting 20,000 lines of slop per PR (and never-ending combinatorial complexity), maybe we should aim to think about abstractions and how to steer LLMs to generate code similar to that of a skilled human developer. Then it could actually be a maintainable artifact by humans and LLMs alike.

crnkofe 5mo ago
I don't get why every AI article is so hyper-focused on coding speed. If the coding is so fast doesn't it make sense to invest more time into quality, learning, documentation, testing refactoring, making a better product? I'm beginning to think that the slopcoders are evaluated by kLOCs of lines written in addition to LLM token usage and they're just maximising the measured metrics. Whether that actually ends up in production or is used by any real person is seemingly irrelevant. Likely the more bugs that are produced the more agents can be spun in parallel to simulate busywork.
- lesscode 4mo ago
  Indeed. If the goal is to make useful (quality) software artifacts, we are going in the wrong direction.

phyzix5761 5mo ago

I wonder if with the speed of iteration with AI the industry will switch back to waterfall. Clear documentation first so the LLM can easily produce what's being asked with a round of testing before going back to the documentation stage and running it again. History does repeat itself.

yibers 5mo ago
We already switched

Ozzie-D 5mo ago

[flagged]

farmerbb 5mo ago

I legit can't tell if this article is satire, or not.

retinaros 5mo ago

markdown became the language I hate the most thank to LLMs and specs-driven approach. everything feels so dumb right now in agentic coding. looping blindlessly and aimlessly until it compiles then until the playwright server or whatever devtools shows that it somehow works. push the code, have a llm autoreview/autofix,push to prod, run a mythos (perfect name) to identify the bug that opus 4.7 create. loops on loops on loops of some kind of zombie processes running to a "goal" that everyone seems to mystify in talks to just hide the fact that we do nothing anymore. the bottleneck never was code. it was the gate that was keeping away the Elizabeth Holmes and SBF from software engineering and it just opened.

abalashov 5mo ago
A colleague and I have taken to use of the verb "meatspin", from another era in Internet shock humour, to describe what it is that coding agents actually do 99% of the time.

throwaw12 5mo ago

> my first bet would be specifications and tests You are missing another dimension how easy it would be to migrate if adding new feature hits a ceiling and LLM keeps breaking the system. Imagine all tests are passing and code is confirming the spec, but everything is denormalized because LLM thought this was a nice idea at the beginning since no one mentioned that requirement in the spec. After a while you want to add a feature which requires normalized table and LLM keeps failing, but you also have no idea how this complex system works. Don't forget that very very detailed spec is actually the code

abalashov 5mo ago
> Don't forget that very very detailed spec is actually the code Came here to say this, but you said it for me. If the problem were merely one of insufficient rigour or detail in specs, it would have been solved long before LLMs.
theshrike79 5mo ago
> Don't forget that very very detailed spec is actually the code In the age of AI this is more true than you know. Given a detailed enough spec and test suite you can effectively rewrite any application with any language in a fully automated way. I've coined that as "Duck coding" :D If it quacks like a duck, walks like a duck and looks like a duck - it's duck enough for as far as the spec is concerned. Does it matter what is inside the duck?
- npodbielski 5mo ago
  Yes. It would be like buying a car that you have no idea about the engine and gearbox and kind of fuel it is using or if at all. It have four wheels and and it can drive you from point A to B. Sure but sometimes it happens that some particular brands and they particular model requires engine renovation after 100k km because it is so shitty design. Right now we are just starting vibe coded software, nobody knows how it will behave in 2 years or 5 or 10. My guess it won't. So we will enter age of scratch software. You build it, ship it. And after few months you will ship entirely new one. And then again. And again. And again. Because maintenance is hard and costly and writing from scratch will cost like 1k$ in tokens. And users will have problem of migrating the data if possible at all. But if migration is hard and everything changes all the time does it even matter if you are using X o Y software? Does it even matter since you can write your own software and migrate your data there? I think we saw how this ends with Chinese manufacturing. You buy some stuff from AliExpress for 2$ and throw it away in two weeks and buy a new one. So quality does not matter anymore.

zoogeny 5mo ago

>... my first bet would be specifications ... and tests ... If I had to roll out such a development process today, I’d make a standardized Markdown specification the new unit of knowledge for the software project. I've found that adopting RFC Keywords (e.g. RFC 2119 [1]; MUST, SHOULD, MAY) at least makes the LLM report satisfaction. I'd love to see a proper study on the usage of RFC keywords and their effect on compliance and effectiveness. 1. https://www.rfc-editor.org/info/rfc2119/ https://www.rfc-editor.org/info/rfc2119/

kortex 5mo ago
That's literally what OpenSpec does (https://openspec.dev/ https://openspec.dev/). It's quite nice. I've only exceptionally rarely seen claude do something wrong based on spec docs when it's fully spec'd out. More often it's because something wasn't nailed down and claude was forced to make assumptions. The downside is the ospx markdown specs sometimes end up too granular, focusing on the wrong or less important details, so reading the specs feels like a slog. Also at times aspects of the english language spec end up way more verbose than just giving a code example would be.
- jonnytran 5mo ago
  Is it time for the literate programming renaissance?

DavidVoid 5mo ago

> Product owners and engineers could initially collaborate on this spec and on test cases to enforce business rules. Those should be checked into the project repositories along with the implementing code. There would need to be automated pull-request checks verifying not only that tests pass but that code conforms to the spec. This specification, and not the code that materializes it, is what the team would need to understand, review, and be held accountable for. This just sounds like typical requirements management software (IBM DOORS for example, which has been around since the 90s). It's kind of funny how AI evangelists keep re-discovering the need for work methods and systems that have existed for decades. When I worked as a software developer at a big telecom company and I had no say in what the software was supposed to do, that was up to the software design people--they were the ones responsible for designing the software and defining all the requirements--I was just responsible for implementing that behavior in code.

irishcoffee 5mo ago
One of my first tasks at my first job out of college required me to learn dxl (doors extension language) and implement some really intricate requirements management features. It was gratifying to build the confidence of learning a new language quickly that I had never even heard of before. DXL was also pretty awful. Opened a lot of doors for me though, no pun intended.
bitwize 5mo ago
Spec-driven development is basically PRIDE, the first proven commercial software methodology dating back to 1971. In fact it may be the culmination of PRIDE because PRIDE's creators realized coding wasn't the hard part; the hard part was systems analysis, determining what problem needed to be solved and what to build. Coding comes last and when you did it right, was simply a translation step. And now that step can be 100% automated. Information systems design was a solved problem in the 1970s. PRIDE turned it from an art into a proven, repeatable science. Programmers, afraid of losing their perceived importance, resisted the discipline it imposes as the mustang resists the bit, but now that they're going the way of buggy-whip makers, maybe systems design as a science will make a comeback after 50 years.

k3vinw 5mo ago

> We can stop reading LLM-generated code just like we don’t read assembly, or bytecode, or transpiled JavaScript; our high-level language source would now be another form of machine code This is too weird for me. At least with programming languages I can consult the documentation and if the programming language isn’t behaving as documented, it’s obviously a defect and if you’re savvy enough you often have open channels that accept contributions. Can we say the same for Claude or other AI solutions?

Sevii 5mo ago
If you run a local LLM and an open source agent harness you are pretty close to that.
- throawayonthe 5mo ago
  can you explain how? with a compiler you can rely on the adage "it's never a compiler bug" (until it is! and then you can fix it) how can a local LLM with an open source agent harness provide the same trustworthiness?
  - zoogeny 5mo ago
    > ... then you can fix it I recall working on a project that used (MSVC) VC++ and a coworker found a bug in the compiler. We reported the issue to Microsoft and they eventually patched it. You may find yourself arguing explicitly for open source dev tools if you continue down this line. There are many commercial cases where "you can fix it" does not apply to the dev toolchain and you will find yourself reliant on a provider. At that point, the trustworthiness of "compiler provider" and "local LLM provider" is the pertinent discussion (e.g. provider vs. provider instead of LLM vs compiler).
    - skydhash 5mo ago
      > There are many commercial cases where "you can fix it" does not apply to the dev toolchain and you will find yourself reliant on a provider. That’s only on the hobbyist level. On the enterprise level, there are lots of contracts involved that requires speedy bugs correction.

humbleharbinger 5mo ago

My amazon orgs leadership has been obsessed with spec driven development while individual engineers tell me the only use they have is to placate leadership. I'm tired

culi 5mo ago
How does spec driven development differ from test driven development?
- nullsex 5mo ago
  [dead]
- invalidator 5mo ago
  TDD is done in a tight loop (minutes) while coding. For every little micro-feature/fix, you write a test for the new behavior you want, implement the minimal ugly fix to get the test to pass, then rely on the tests so you don't regress as you clean up. LLMs struggle with TDD. They want to generate a bunch of code and tests in large passes. You can instruct them to do red/green TDD, but the results aren't great. SDD starts before implementation, and formalizes intent and high-level design. LLMs eat it up. The humans can easily reinvent the worst parts of waterfall if they're not careful. They're not mutually exclusive.
  - culi 5mo ago
    In many frameworks the tests are referred to as the "spec". I guess that's where my confusion arises from. > SDD starts before implementation No different from TDD.

donbventures 5mo ago

[flagged]

dundunUp 5mo ago

[flagged]

Uptrenda 5mo ago

Does this post mark the top of the hype train or is there still more to come?

0xpgm 5mo ago
Still more to come I think. Until all the major AI companies IPO starting this year.

facundo_olano 5mo ago

Author here. I'm surprised to see this surfacing now. I just wanted to clarify, since apparently the post doesn't do a good job at it, that what I discussed there is not a methodology I advocate for. The point of the post was: ok, since there are organizations mandating to maximize speed by reducing time spent on typing code (or even mandating to maximize agents usage), is there a way we can meet that requirement while still preserving the rigor somewhere else? This was a follow up to a previous article[1] and the pair tried to express what I still think today (using AI daily at work): every time I use AI for coding, to some capacity I'm sacrificing system understanding and stability in favor of programming speed. This is not necessarily always a bad tradeoff, but I think it's important to constantly remind ourselves we are making it. [1] https://olano.dev/blog/tactical-tornado/ https://olano.dev/blog/tactical-tornado/

ignoreusernames 5mo ago
Don’t you think that the provider of the LLM is also a dimension on these discussions about responsibility? We often talk about the tech itself (LLM driven development) but how we access it is just as important imo. It’s either locked behind a non trivial amount of hardware (for open models) or some monopolistic driven provider entity like OpenAI or anthropic. In the provider case, it’s not really the LLM that will “own” the code, it’s the provider itself and we’ll be at the mercy of whatever pricing model they shove down our throats.
LelouBil 5mo ago
I don't like the premise of the article, but I agree that if you accept the premise, the contents of the articles are a good way to do it.
Uptrenda 5mo ago
[flagged]
- AlexCoventry 5mo ago
  He was establishing the context of The current blog post. Very unlikely that he was doing it for Google juice.

iloveoof 5mo ago

Software engineering has always worked this way, just not to ICs. “The LLMs produce non-deterministic output and generate code much faster than we can read it, so we can’t seriously expect to effectively review, understand, and approve every diff anymore. But that doesn’t necessarily mean we stop being rigorous, it could mean we should move rigor elsewhere.“ Direct reports, when delegated tasks by managers, product non-deterministic outputs much faster than team leads/managers can review, understand or approve every diff. Being a manager of software developers has always been a non-deterministic form of software engineering.

devmor 5mo ago
> Being a manager of software developers has always been a non-deterministic form of software engineering. Unless the manager is also a principal/architect, I don’t find this to be agreeable. It’s similar to saying that you are a non-deterministic chef when you order food from a restaurant.
teaearlgraycold 5mo ago
Well yes but if no humans at the company understand the code then no one is truly responsible for it.
- Npovview 5mo ago
  what about the artifacts that were supposed to test the correctness of the code? are they passing willy nilly?
  - ekidd 5mo ago
    No amount of testing will save a large program with a dogshit architecture. Roughly, this is because tests increase coverage linearly with the number of tests, but weird interactions increase exponentially with code size. This might be fine if you're building a tiny app, or if you're building a medium-sized app that follows a strict existing architecture (like a web app consisting mostly of forms). In which case, have fun. But if you're building something slightly novel and interesting, then Claude is surprisingly bad at architecture and taste, and it tends to "fix" problems by spewing more slop. What you need instead is actual insight that leads to simplifying principles. This, in turn, allows breaking up the exponential complexity into disciplined patterns. This allows your code complexity to scale far more slowly, allowing an essentially linear number of tests to provide coverage. I actually download and try people's vibe-coded developer tools. And frankly, those tools are some of the worst software I've used in my life, worse than even Unix-vendor Motif implementations from the early 90s. Like, I'm super happy that people can vibe-code themselves simple, one-off personal tools. That's incredibly empowering. But that doesn't mean you can big, novel stuff the same way without a competent human actively in the loop.

ricardobeat 5mo ago

> We can’t leverage agents if our unit of work is still “add a new endpoint to the RESTful API” Why not? You just make every task faster. Not everything has to be an uncontrollable rocket launch. > We need a virtually infinite supply of requirements, engineers acting as pseudo-product designers, owning entire streams of work Why? To build what? You can only build as fast as you understand the business and your users.

charcircuit 5mo ago
>You can only build as fast as you understand the business and your users. It should be possible to go faster by having AI understand the business and users.
- arcwhite 5mo ago
  It doesn't do that though. Understand. That's not how LLMs work.
  - charcircuit 5mo ago
    LLMs are not the only possible AI models to use and create.
    - paulryanrogers 5mo ago
      Aren't they state of the art though?

blacob 5mo ago

> Then where does the rigor go? Similar to the Thoughtworks report, my first bet would be specifications (which is not the same as prompts) and tests (which is not the same as TDD). This is what we're building for at Saldor (https://saldor.com https://saldor.com). It's a hard problem, to get a team in the habit of writing good specs. Probably because it's a hard thing to do: thinking of the behavior of your program, especially at the edges. But I agree (biased) that this is probably the way forward for writing code in the near future. I'm excited to see other people thinking about it.

Terretta 5mo ago
Saldor pitch is on point. I have team do this using CLAUDE.md telling Claude to do it in a set of interconnected steps, but in brief: they are to make it write every aspect of transcript somewhere: PRD, research notes, spec, dev log and debate log, break/fix/retro notes, commit log, PR, release notes, README, docs .mds... heavy emphasis on the edges in our thinking, and just as important, the edges in its ability to provide good leverage. It needs a core set of guidance on the ordering and how to write "as of" a given phase or release so context stays current, trusting the old info is in git history it can navigate for the story of how we got here. CC's /insights claims I have 10:1 md edits to code edits, and we both note this way of working is resulting in far fewer error loops per higher quality outcome. // So yes, interested in your product. Baking something more broadly battle tested in so we don't have to reinvent it makes sense.

nirui 5mo ago

> We can stop reading LLM-generated code just like we don’t read assembly, or bytecode, or transpiled JavaScript; our high-level language source would now be another form of machine code. My opinion is very close to this. Currently the reason that it's bad to not reviewing/testing the code LLMs generated is because the LLMs can sometime generate bad codes. But it's a bug that can be improved. One day you'll have LLMs generating code consistently better than what a human could write. And then you just stop needing to review them. (And that's probably also the time where most programmers/developers got fired too) Don't get surprised if anyday the LLMs starts to generate binaries directly. THAT will be impossible to read and costs more time to analyze.

9029 5mo ago
Is it possible to reason or prove the correctness of an LLM?
furyofantares 5mo ago
> Currently the reason that it's bad to not reviewing/testing the code LLMs generated is that the LLMs can sometime generate bad codes. Sometimes? I am heavily into vibe coding and I think they almost always generate bad code. At least as soon as you're distant enough from the code to call it vibe coding. When you're still in touch with the code, have at least been recently talking to it about code rather than 100% about features, and its context is filled with good code, it can generate good code.

adelks 5mo ago

"A sufficiently precise spec is code". I've read somewhere here before. So guardrails, i.e. sufficiently precise spec and tests, will need to be as strict as the LLM is bad at getting the right context and asking back the right questions. I suppose at that point not much difference between a human engineer and it.

jelmersnoeck 5mo ago

> "I'd make a standardized Markdown specification the new unit of knowledge for the software project. ... There would need to be automated pull-request checks verifying not only that tests pass but that code conforms to the spec." Agree, this is how you make the development loop more deterministic and ultimately autonomous. It's how I've been using coding agents myself for the past few months (by building my own to support this natively [1]). If you have a spec you approve/agree on, have an agent code against it, and then have a review phase verify the implementation didn't drift from the spec (either by adding or removing features), you get to a position where you can trust the outcome. There's still a lot to be said about spec definition and what if during implementation gaps are discovered, and that's where HITL comes into play. [1] https://github.com/jelmersnoeck/forge https://github.com/jelmersnoeck/forge

immanuwell 5mo ago

it's the most honest framing I've seen, but specs as the new source of truth is exactly what we promised ourselves with UML, then WSDL, then OpenAPI. the graveyard of just make the artifact above the code authoritative is long

okandship 5mo ago

making the review artifact explicit feels like the part teams skip

CraigJPerry 5mo ago

I prefer "the bottleneck is understanding" framing. The author is nibbling at the same problem ultimately, but i don't think "hey one strategy is we could just let cognitive debt accumulate so we can go faster!" is a particularly insightful tool in the toolbox. Don't misread me, i'm not denying it can be a valid strategy. Instead i want to read about insightful strategies for optimising that system-wide bottleneck we have: understanding. Tell me about how you managed to shift to a higher level of abstraction, tell me about how and when that abstraction leaks. Tell me how you reduced the amount of information that has to flow through the system bottleneck.

stuaxo 5mo ago

In short: We will have code full of unknown bugs, that is unfixable. The solution is to replace it with more of the same but with some new specification (fix some bug add some new feature). And this will be done by using astounding amounts of compute in massive new data centres.

nlitened 5mo ago

I feel like people who program in JavaScript or whose projects pull megabytes of dependencies, don’t get a moral right to complain about this. You guys just sit and calm down this time, you already said what you could. Your app takes 20 seconds to load, pulling 50 megabytes of minified JS. Your backend is a mess of 20 Rust microservices, 300 megabytes docker image each. Nobody has actually been reading and understanding code in your org for the past 15 years. And nobody has ever been responsible, everybody has just been job hopping for a 15% total comp bump. Now the secret is out.

EFLKumo 5mo ago

> just like we don’t read assembly, or bytecode, or transpiled JavaScript This makes sense since certain higher-level code produces certain lower-level code, while LLM cannot. If the transpired JS code doesn't work we could just find out the bug in minifiers, etc. but one cannot figure out why LLM fails at one task, especially considering LLMs, even SOTA ones, could be strongly affected by even small prompt changes. Taking this into consideration, I don't think this is a sound reasoning why we don't need to review ai-generated code. > The LLMs produce non-deterministic output and generate code much faster than we can read it, so we can’t seriously expect to effectively review, understand, and approve every diff anymore. Exactly. However, this could also indicate a weaker review standard instead of just dropping review. We could also suggest an idea where devs mainly review code design or interfaces, leveraging one's *taste*, while leaving strict logic reasoning, validating and testing to other tools or approaches. It cannot pursuade me that the nature of LLM's code generation must lead to a complete cancel of the code review. Anyway, I'm not opposing this article and its thought of shift in the future is really good.

trimethylpurine 5mo ago
Couldn't we slowly add guardrails that eventually lead to code generation becoming more and more deterministic over time? I'm seeing in my experience that Claude has become better with every version at producing uniformity in its code output. Especially where the architecture is clear and documented. And even more so in languages with built in uniformity (Go, HTMX, SQL) where there is intentionally only one or two ways of doing things. In such environments, the output is nearly deterministic.
- EFLKumo 5mo ago
  I once thought about this and found that n-shots makes greater influences on LLMs. In other words, in a repo with good code quality and architecture (which offers good n-shots) and on a task with clear instructions and goals, LLM's output seems reliable enough, which meets your opinion. And n-shots is always better than relying on instruction following, instruction following mentioned in the article ("specifications") as an approach facing LLM's productivity, so imo the idea you suggested is another probability against/comparing with the article as well.

QuantumNoodle 5mo ago

It's too early for me to have a firm opinion one way or another. Just a data point: this month I had a knarly bug in generated bpf code. The C language was correct but the compiler produced a bug that corrupted packets. I spent around 8 hours debugging _where_ the issue is and how to work around, never really understanding what went wrong. That knowledge came with several more days on and off looking at it--after I had mitigated the production issue. So if I extrapolate this experience to LLMs (who are not deterministic) and who will make larger systems. What we trade for velocity we will pay for with hours of debugging because we won't understand how things work. I think this is unavoidable. Another way I'm looking at it: after some time of not writing code, it will be analogous to instructing the LLM and the output being assembly--where I simply don't have the muscle to grok the output. How do I mitigate that knowledge gap? I see micro serves coming back. Today it is easy to slop up disposable scripts. Our services need to be modular so we can dispose of broken things--so they are only coupled with each other by strict APIs.

-​-dangerously-skip-reading-code

-​-dangerously-skip-reading-code

--dangerously-skip-reading-code

--dangerously-skip-reading-code