7 ms·
I think when LLMs first came out people thought they could just say something like, "Make a Facebook clone". But now we're realizing we need to be more exact wi
by phyzix5761 5mo ago
I think when LLMs first came out people thought they could just say something like, "Make a Facebook clone". But now we're realizing we need to be more exact with our requirements and define things better. That has always been the bottle neck in software.
When I was working we used to get requirements that literally said things like, "Get data and give it to the user". No definition of what data is, where its stored, or in what format to return it. We would then spend a significant amount of time with the product person trying to figure out what they really wanted.
In order to get good results with LLMs we need to do something similar. Vague requirements get vague results.
- steveBK123 5mo ago> When I was working we used to get requirements that literally said things like, "Get data and give it to the user". No definition of what data is, where its stored, or in what format to return it. We would then spend a significant amount of time with the product person trying to figure out what they really wanted. This is a big HN LLM discussion divide. I am in the same no-specs work background camp, and so the idea that the humans who input that into dev teams are suddenly going to get anything out of an LLM if they directly input the same is laughable. In my career most orgs there has been no product person and we just talked directly to end users. For that kind of org, it will accelerate some parts of the SWEs job at different multipliers, but all the non-dev work to get there with discussions, discovery, iteration, rework, etc remains. If the input to your work is a 20 page specification document to accompany multi-paragraph Jira tickets with embedded acceptance criteria / test cases / etc, then yes there is a danger the person creating that input just feed it into an LLM.
- et1337 5mo agoI’ve never understood engineers who complain about vague specs… if the spec was complete, it would be code and the job would be done already! Getting a 20 page spec delivered from upon high and mechanically translating it to code without any chance to send feedback up the chain sounds like… a compiler.
- steveBK123 5mo agoYes, I don't think a job where I am programmed by a product manager would be terribly interesting. I would move on to be the product manager if I found myself in such a role. Probably why I haven't ended up in any.
- skydhash 5mo agoIn my experience, the complaints are not about the specs and their vagueness. It's more about the political game to get them detailed. If you've not encountered the kind of organizational issues where getting an answer is like pulling teeth, you're kind of lucky.
- et1337 5mo agoOh no, I’ve definitely experienced that, it’s terrible. But that situation makes me wish for more agency (for example, talking to customers directly), whereas it seems to make other engineers wish for less agency (please hand me a complete spec and I will mindlessly translate it to code). That’s what I don’t understand.
- pirates 5mo agosome of us couldn’t give a rat’s ass about the customer. One of our customers charges people for paying their own bills via certain methods, which is completely bogus and I remind everyone loudly all the time that they do this. Everyone agrees that this customer sucks to work with, and the less time spent with them the better. The people from the customer’s end suck, they’re not technical, they have in-fighting with their own teams during calls, have decades long errors with their integration that they have never fixed…the list goes on. For this customer and a few others, please give me a spec that I can implement, shove it back across the aisle, and forget about. The absolute last thing I want is to have to talk to them more.
- hirako2000 5mo agoThe demands are for functional requirements. Plenty to translate on the non functional side of things.
- shalmanese 5mo ago> I think when LLMs first came out people thought they could just say something like, "Make a Facebook clone". But now we're realizing we need to be more exact with our requirements and define things better. That has always been the bottle neck in software. This was substantially predicted by Fred Brooks in 1986 in the classic No Silver Bullets [1] essay under the sections "Expert Systems" and "Automatic Programming". In it, he lays out the core features of vibe coding and exactly the experience we are having now with it: Initial success in a few carefully chosen domains and then a reasonable but not ground breaking increase in productivity as it expands outside of those domains. [1] https://worrydream.com/refs/Brooks_1986_-_No_Silver_Bullet.pdf https://worrydream.com/refs/Brooks_1986_-_No_Silver_Bullet.p...
- steveBK123 5mo agoIt's interesting how predictable some of this is. The LLMs turn out fully formed clones of stuff for which there exists copious amounts of code openly searchable on the web doing the exact same thing. LLMs require developer-like specification, task/subtask breakdown and detail where such example code already exists. As a professional prior to LLMs, how many problems that you work on have many existing free solutions but you neglected to use that code and decided to spend days doing it yourself?
- pton_xd 5mo ago> how many problems that you work on have many existing free solutions but you neglected to use that code and decided to spend days doing it yourself? Only when the existing free solutions are licensed with something like GPL. Now I can just say, write me a C webserver library similar to mongoose and I get the functionality without the license burden.
- juvoly 5mo agoIndeed, no license burden but you get a maintenance burden instead.
- 5mo ago
- rubyfan 5mo agoWe now have product owners trying to farm out their work to an LLM. The process didn’t work before because the person writing the requirements either put out vague requirements or bad requirements because they didn’t understand the business intent (or were careless). LLMs just take the same vague or poor requirements and make them look believable until you dig in to them.
- steveBK123 5mo agoPlausible requirement generators as inputs to plausible code generators.. what could go wrong!
- eastbound 5mo agoIt’s a giant tragedy of the commons. I’ve fired remote people who pretended to work, knowing that I wouldn’t hire remote workers ever again after AI.
- mpyne 5mo ago> The process didn’t work before because the person writing the requirements either put out vague requirements or bad requirements because they didn’t understand the business intent (or were careless). You make it sound like writing good requirements is easy. If it were easy we wouldn't need all these concepts around PMF, product pivots and the like. And even before that was Peter Naur's paper "Programming as Theory Building" [1]. If you truly understand the problem you're solving with software then requirements can be easy. But usually we don't, not right away, and so we have to build up our understanding of the problem first in order to solve it. Even then, the problem we solve may not have been the problem paying users will have, so you can have "good requirements" and still have a bad business, or even the opposite where you somehow build a working business despite bad requirements, because you hit upon a customer's need quite by mistake. Nothing about any of this precludes LLMs being helpful, though nothing guarantees LLMs will be helpful either. [1]: https://cekrem.github.io/posts/programming-as-theory-building-naur/ https://cekrem.github.io/posts/programming-as-theory-buildin...
- 5mo ago
- cryo32 5mo agoIt's worse. Vague requirements still only power vague interpretations of the problem. Even if you provide good requirements, you still only have vague interpretations at your fingertips. The promise is that such things won't be a problem in the future, which is obviously not materialising. "Make a facebook clone" is the vague human promise to the end user. The reality is that it leads to so many assumptions which are insurmountable due to the vague interpretation so you have to change your requirements in the end to claim success. Thus everything turns into a mediocre compromise. There is no exceptional outcome, which is what makes a marketable product. There are just corpses everywhere. You need something better to both define requirements and implement them than this technology.
- wijej 5mo agoCan someone pull that Steve jobs quote out re. The craftsmanship between a great idea and great product? Anyone who thought that gap could be shrunk substantially lives in delululand. Hence why we haven’t seen this explosion of ‘really great’ products come out. Many will continue to parrot ‘bro but the models changed I swear’. I’m sure they did. But you’re missing the damn point.
- cryo32 5mo agoThink the AI bros are downvoting you here but I agree entirely with this perspective.
- deleted 5mo ago[deleted]
- Foobar8568 5mo agoWe arrived to that state today with Codex and Claude Code. I really don't know what people are doing wrong?
- niek_pas 5mo agoWhat's even worse is that when dealing with human software teams, a vague requirement will (at least in a well-run org) receive demands for further specification. "What do you mean by 'get data'?", etc. An LLM will just say, "Sure! Here's the fully implemented code that gets the data and give it to the user. " and be done with it.
- vidarh 5mo agoWhen the cycles are short enough, though, that is to some degree the right thing. That is, it's the right thing for things the users can then immediately see and give feedback on, because it lets them give feedback on something tangible. It's the wrong thing for important things under the hood (like durability and security requirements) that are not tangible to them.
- smokel 5mo agoChatGPT 5.5 responds: > What data should I retrieve, and where should I get it from? Please specify at least: ... And it then goes on to ask just exactly what is necessary, being all constructive about it.
- airstrike 5mo agoYou're both right. The parent was a toy example, and if asked literally to an LLM, it will definitely ask for more information. Yes, it's important to be accurate but I don't think that applies here. But the point still stands: in most contexts, the LLM will fill in the blanks with what it deems appropriate like an overconfident intern at best and a bull in a China shop at worst.
- pydry 5mo agoIME you give it very precise specifications and it still fucks it up. When we talk about "the" bottleneck being specs it just isnt the case that it's the only thing LLMs do poorly. Theyre really bad at a lot of stuff in the SDLC. They're also good at providing results which are bad but look ok if you either dont look too closely or dont know what you're looking for.
- 5mo ago
- faangguyindia 5mo agoproduct people love LLM because it doesn't ask "what does X means? how will it work?" while a programmer will ask, about all cases.
- gib444 5mo agoDo they have a point? Can't good marketing teams, backed up by World Class Product people, sell anything we build, more or less? </devil's advocate>
- deleted 5mo ago[deleted]
- layer8 5mo agoEven if that were the case, I wouldn’t want to spend my working life building software poorly fit for the purpose, that nevertheless sells due to marketing.
- bonesss 5mo agoDid everyone forget about outsourcing and how outsourcing works? The dudes in Eastern-Wherever not asking what something means is the scary part. You only find out at the end how deeply confused everyone was when making the thing. You can fix it with attention and management, but then only some projects sometimes are profitably outsourced and you still need competency.
- latexr 5mo agoCue that commitstrip comic from 2016. https://web.archive.org/web/20161211074810/http://www.commitstrip.com/en/2016/08/25/a-very-comprehensive-and-precise-spec/ https://web.archive.org/web/20161211074810/http://www.commit...
- satvikpendem 5mo agoIn what I've seen, tickets are much richer in detail now because PMs are using AI (connected to the codebase itself, like Claude Code or Codex) to fill out a template as to what and why the problem is (ie X field exists in the backend not frontend), how and where to get any data (query the backend), and what acceptance criteria is needed (frontend should have the field exposed and "submit" should push the field's data to the backend where it should show up in the databas), which is something they would not have done before, due I guess to laziness and thinking the devs can figure it out. Then devs can copy paste this Jira ticket content into the LLM agent of choice (or even use the Atlassian MCP to have the LLM read it automatically). This has significantly helped devs and made sure that requirements are very clear. Honestly, with the first step, it seems the PMs are already halfway there to implementation of the feature so I wonder if in the future they'll just do everything themselves and a few devs will be around as SDETs rather than full blown implementers.
- OptionOfT 5mo agoExcept... no one validates the generated tickets, and it's full of inaccuracies. And then someone copy pastes it into Claude and now those inaccuracies become part of the code and tests.
- satvikpendem 5mo agoThe PMs validate it, why do you think they don't read over it to make sure it fits what they want? You might say "well they're lazy, look why they didn't write enough detail to start off with" but for lots of people, reviewing something to make sure it's close to what they want and then tweaking it is much easier than writing it from scratch. It's the equivalent of writer's block and is why a common advice given to writers is to put anything they can onto the page then edit it later.
- BugsJustFindMe 5mo ago> The PMs validate it, why do you think they don't read over it to make sure it fits what they want? Hahahahahaha. Sorry, I couldn't help myself; this reads like satire. The answer is "real life experience says otherwise".
- stelonix 5mo agoYou're completely right and I thought this would be obvious. I never prompted anything remotely closely to "make a facebook clone". Instead, I make an explanation of how it should work. To give you an example: I need a python script that 1) reads /etc/hosts 2) find values of specific configured hosts (read from a .conf which) eg server1, localhost, etc 3) it'll assign a name to those configs eg if the .conf has [Env1] 192.168.0.1 production-read 192.168.0.2 production-write 192.168.0.27 amqp [Env2] 192.168.0.101 production-read 192.168.0.201 production-write 192.168.1.127 amqp Basically format: [CONFIG_NAME] <ip> <hostname> Like an usual hosts file 4) And each of those will be stored in memory 5) if in /etc/hosts it matches one of those, it sets the "current env" as the configname 5) It'll create an icon on the top-right of ubuntu 22 default gnome with 6) that icon could be the text of the current config name or if nothing matches, "custom" text would show 7) When the user clicks the "tray"/appindicator(or whatever gnome is calling them) it'll list the config names in a simple gtk/gnome 8) When the user clicks one config, we create a backup of /etc/hosts in ~/.config/backups/ named hosts-%UNIX_TIMESTAMP% 9) we then apply it to hosts file (find only the line with the hostnames to change and modify only those) And that one-shotted a simple gnome app indicator env switcher. Had to fix a few lines here and there but it mostly just worked. If you give the proper spec to the LLM, it'll do it right. You can even fake a DSL to describe what you want and it'll figure it out.
- Mossy9 5mo agoJuxt's Allium https://juxt.github.io/allium/ https://juxt.github.io/allium/ is an interesting entry in this 'pseudo DSL' space to define and store system specifications and requirements. I think it's likely that this sort of 'persistent specifications to help bots work correctly' will be a good approach when things finally cool down a bit.
- LtWorf 5mo agoThat's easily more spec than script.
- popcorncowboy 5mo ago> You're completely right I mean, no comment
- whstl 5mo agoThis was already a reality for a years. In several companies I have seen product managers joining teams and failing to even have minor requirement ready for months during “onboarding” of the PM. And then code being ready but taking months to release because DevOps is busy or QA can’t find time. The pace of release of software has been disconnected from the coding part for the longest time, and we have been quiet about it.
- tedd4u 5mo agoThe solution I've seen work is have engineers and designers that can take much of the detailed spec writing on, and have the PMs spend time with users/prospective users, partners, etc, understanding the market and users better. When you pull PMs in to all the details, often they turn into project managers, shuffling bug tickets around etc, taking time away from owning the user and the problem and shifting them too much to the solution side. Have a lead engineer own much / most of that. Every org / product is different of course.
- whstl 5mo agoI agree with you. The healthier organizations work in the way you mention.
- paulddraper 5mo agoSo the agent needs a “plan” mode where it works with the user and asks questions to define the ask.
- mike_hock 5mo ago> But now we're realizing we need to be more exact with our requirements and define things better. That's why we write programs in programming languages and not English. Because they are much more efficient at giving precise instructions than natural language.
- nnoremap 5mo agoBut horribly token inefficient.
- startages 5mo agoNow it's worst, you get an PDF export of a long ChatGPT chat history with one sentence "Can you give an estimation for this?"
- derefr 5mo ago> I think when LLMs first came out people thought they could just say something like, "Make a Facebook clone". But now we're realizing we need to be more exact with our requirements and define things better. The annoying thing is that giving an LLM vague instructions like "make a Facebook clone" does work... in certain limited cases. Those being mostly the exact things a not-very-creative "ideas person" would think to try first. Which gave the "ideas people" totally the wrong idea about what these things can do. These same "ideas people" have been contracting human software developers to "make them a Facebook clone" (and other requests of similar quality) for decades now. And every so often, the result of one of those requests would end up out there on the internet; most recently on Github. (Which is, once there's enough of them laying about, already enough to allow a coding-agent LLM trained on Github sources to spew out a gestalt reconstruction of these attempts. For better or worse.) But for the most common of these harebrained ideas (both social-media-feed websites and e-commerce marketplace websites fit here), entire frameworks or "engines" have also been developed to make shipping one of these derivative projects as easy as shipping a Wordpress.org site. You don't rewrite the code; you just use the engine. And so, if you ask an LLM to build you Facebook, it won't build you Facebook from scratch. It'll just pull in one of those frameworks. And if you're an "ideas person", you'll think the LLM just did something magical. You won't necessarily understand what a library ecosystem even is; you won't realize the LLM didn't just generate all the code that powers the site itself, spitting out something perfectly functional after just a minute.
- suzzer99 5mo ago> These same "ideas people" have been contracting human software developers to "make them a Facebook clone" (and other requests of similar quality) for decades now. I was doing an exercise boot camp when mobile apps started to hit big. The instructor came to me saying he needed an app. "Okay, what should the app do?" He had no idea, he just knew he needed an app.
- rockbruno 5mo agoRealizing? Will be very happy if that is the case, but in my view all big company execs are still balls deep into the notion that you will be able to just ask it for the facebook clone and everything sucks as a result
- yoyohello13 5mo agoEven purely from an information theory perspective it was obvious “make me a Facebook clone” was not going to work. The more you compress the information prompt the more detail you lose.