4 ms·
I'll risk sounding antagonistic and ask you this: if LLMs are not "good enough", why are they still around? Obviously "good enough" was a poor pick of words of
by notashelf 2mo ago
I'll risk sounding antagonistic and ask you this: if LLMs are not "good enough", why are they still around?
Obviously "good enough" was a poor pick of words of words on my behalf---and I'll gladly own it---but they should be "good enough" for something if a significant amount of resources keep getting allocated to them. Yes on a very personal level you look at LLM-generated code and think to yourself "wow this is garbage" but what about the people, as pointed out many times in this thread, that simply do not care? Does that not count as "good enough"?
- SCLeo 2mo ago> I'll risk sounding antagonistic and ask you this: if LLMs are not "good enough", why are they still around? To quote the parent commenter: > They seem to be able to solve the immediate problem, but not long term
- notashelf 2mo agoBut that means they're good enough for the short them. This is why I find it such an interesting question to wield, but also, it is why my wording was poor in the post.
- deleted 2mo ago[deleted]
- deathanatos 2mo agoShort term can, IME, be very short. I've seen people generate, say, a bash script with an LLM. It's generated: short term, the problem is "solved": we've generated a bash script. … but does it work? Someone comes along, reviews it, "this is garbage, and does not do what it says it purports to do". Perhaps it even gave an output: the script computed … something, but it's just GIGO. But that "check if this works" friction is the same friction that is what people try to avoid by generating it with an LLM in the first place. If you're too lazy to write the script, you're practically by definition too lazy to verify it.
- aprdm 2mo agoYou can have another agent write the tests and verify the former agent ? This is pretty basic stuff. Makes me question if people are actually trying to use AI
- habinero 2mo agoNow you have two problems lol, in that you don't know if the tests are any good or actually test the thing in question. Sooner or later you run out of turtles to put on the stack.
- ileonichwiesz 2mo agoThat’s a solved problem, you just add another agent to check if the tests are any good, and one more to oversee the test-checker, and one more…
- TheOtherHobbes 2mo agoIf this is your working environment, it sounds like quite an unusual place. I literally can't imagine generating a script with an LLM without testing it at all. Bash is one of those situations where LLMs can do really well. No human on Earth can remember all of the commands and even fewer humans can remember all of the switches for all of the commands. This kind of remembering, searching, and assembling is exactly what LLMs are good at - as long as you're not writing a gigantic build system with hundreds of moving parts, in which case you should probably be using something more streamlined anyway.
- lmm 2mo ago> I literally can't imagine generating a script with an LLM without testing it at all. Then you're extremely unimaginative as well as unusually fastidious. Certainly someone - several someones - are generating lots of scripts and not testing them, given the PRs I'm seeing.
- throwatdem12311 2mo agoBreaking down long term problems into small shorter problems we can solve one after another until the long term problem is solved is the essence of engineering.
- timacles 2mo agoNo, the "essence" of engineering is having a cohesive vision for a larger plan. No quality large scale project can exist without it. And any seasoned engineer should understand by now that no SOTA LLM can produce quality engineering at scale.
- aprdm 2mo agoLol. That's not true at all. And not how projects are done on startups or big companies. Execute fast or get nowhere, a series of small wins get you to live long term.
- dwattttt 2mo agoLong term projects that survived on short term wins are still around. But you're going to want to check how big the graveyard is of "project collapsed under tech debt from short term decisions" to see whether it's a good strategy.
- habinero 2mo agoNah. You can succeed in every single individual thing but fail at the overall project because the parts don't align into anything that makes sense. If you need examples, look at game dev. There's plenty of games that have good execution but aren't fun to play or they're a confusing mess because there wasn't good overall direction.
- throwatdem12311 2mo agoVideogames are a weird comparison because they are basically the ultimate culmination of the combination of both engineering and art. The two can crossover in weird ways but still have distinct aspects that do not really compromise the other. Just because a videogame isn’t fun doesn’t mean there wasnt a cohesive engineering vision. God of War: Ragnarok is a good example of this. Excellent technical execution and (arguably) great direction but a terribly boring game. And many games that are terribly engineered are also incredibly fun. Dark Souls is a good example - runs like ass, looks very rough even by the standards of the time when the game released, with terrible enemy AI in a combat focused game - one might even call it a confusing mess (and I laugh at the notion that it doesn’t have good direction)… despite all this it’s a classic and arguably the most influential game since its release. Elden Ring carries a lot of legacy baggage from years of iteration on the same engine as ~~Dark~~ Demons Souls - and it’s widely considered From Software’s magnum opus and a masterpiece. Yet considering the scope of the game it is a great example of how small wins added up can solve large problems when executed with the skill and judgment gained from experience. So one might still consider it a success in terms of “engineering”. Would From Software have been able to make the same game using Unreal Engine 5 built from scratch with dime a dozen “Unreal experts” brought in off the street? Not a chance. Want to know why western game development studios are suffering right now? Compare American layoff culture and race to the bottom economics to the retention rates of Japanese game studios like Capcom, Nintendo and other heavy hitters. For many of the most marvellous things humans have ever built, if you take a peak behind the curtain you’ll still probably find some amount of duct tape and popsicles sticks somewhere keeping it all together. Fun, however, is a matter of taste and you can’t engineer that.
- boron1006 2mo agoIt’s a good question and I don’t doubt they’re useful. But I mean they’re not good enough to do the things I need them to. Not enough that any difference can be chalked up to “taste”. Put it this way, if you hired a very good handyman to build you a cathedral, you wouldn’t stand looking at the smouldering wreckage of the construction site saying “oh well that’s just a matter of taste”.
- pornel 2mo agoThey're here, because even "bad" code has uses. We can have quantity over quality. We can generate throw-away programs (where the lack of maintainability doesn't matter). We can solve some problems by brute force (keep trying until tests pass). For people who can't program, even bad code can do more than no code. LLMs are also useful as a natural language interface. They're pretty useful for search (as RAG and autoresearch, not the glue-on-pizza Google search crap). They're good for filtering and classification against rules that can be specified in natural language instead of special syntax or by training dedicated models.
- AdieuToLogic 2mo ago> They're here, because even "bad" code has uses. We can have quantity over quality. We can generate throw-away programs (where the lack of maintainability doesn't matter). We can solve some problems by brute force (keep trying until tests pass). For people who can't program, even bad code can do more than no code. If this is an acceptable definition of software engineering productivity, where "quantity over quality" is prized, then hire me. Because I guarantee I can produce dozens of PRs daily, each having tens of thousands of LoC deltas (pick any language you desire), none of which having research and/or understanding underpinning them, and all easily quantified as being "brute force."
- didibus 2mo agoThere's a quote I forgot that says: "There is a Quality to Quantity". I think it was from a Chinese person, as they took manufacturing of cheap things, low quality stuff, but managed to scale it to mass quantities. I agree with that, it may not be reliable, full featured, well designed, ergonomic, etc. But quantity enables things and thus is a quality in itself. It allows use-cases that benefit from accessible cheap software. Those use-cases were not viable prior, because cost/time was prohibitive.
- frowaway4anoday 2mo agoOften attributed to Stalin in reference to the mass production focus of the Soviet war machine. https://quotees.co.uk/war/joseph-stalin-quantity-has-a-quality-all-its-own-quote/ https://quotees.co.uk/war/joseph-stalin-quantity-has-a-quali... When you treat scale as a design criteria, you get a different product
- assimpleaspossi 2mo agoOne part of the answer is marketing it to people looking for an easy way out without having to do the work or thinking at all. There's value in that as shown in everything like things for the home that purport to make your daily life easier. LLMs market themselves in much the same way
- a2ff6eeb0 2mo agoThey're "good enough", but they're not good. If you're trying to run a business, programmer skill isn't worth the money you spend, LLMs are cheaper. If you care about the quality of what you produce, they're trash. Programming is no longer skilled labor. Quality is a hobby.
- rwyinuse 2mo agoIf your business involves storing any kind of customer data or handling payments, you probably should still hire a programmer who at least reviews the results of your vibe coding. Getting hacked is bad for business.
- a2ff6eeb0 2mo agoAI is just fine at security audits these days. It's better at finding bugs than it is at writing code, overall. You need to prompt for it, but that's not skilled labor. I'm also not sure getting hacked is so bad for business. Most businesses that got hacked seem to still be around just fine.
- Brian_K_White 2mo agoConversely, if they are good enough, why do they degrade frome consuming their own output? How come humans consuming their own output gets better over time, but llms and any other form of ai consuming their own output only get worse?
- AdieuToLogic 2mo ago> Yes on a very personal level you look at LLM-generated code and think to yourself "wow this is garbage" but what about the people, as pointed out many times in this thread, that simply do not care? Does that not count as "good enough"? Aside from the "amateur" qualifier, the Booch quote below provides insight to those questions: The amateur software engineer is always in search of magic, some sensational method or tool whose application promises to render software development trivial. It is the mark of the professional software engineer to know that no such panacea exist.[0] "Code" is shorthand for "encode", which can be defined as "encoding a solution to a problem", which implies knowing what problem to solve, and thus ultimately implying understanding of same. So "good enough" depends on the stakeholders' definition of that valuation. IOW, if a person tasked with delivering a solution can instead deliver garbage and still get paid for it, then it is "good enough" for some definition of same. If the people approving a person's remuneration are not satisfied with this type of work product, then the answer as to whether it is "good enough" or not will be quickly answered. 0 - https://en.wikiquote.org/wiki/Grady_Booch#Quotes https://en.wikiquote.org/wiki/Grady_Booch#Quotes
- chongli 2mo agoThe use of “amateur” is indeed a very poor choice. An amateur is someone who seeks to master a skill or perform a craft for the love of the art, not for money. It’s the professional who is looking for the shortcut, not the amateur. The amateur is the one who hones a tool’s edge until it’s razor sharp, and cuts away all unnecessary detail until they’ve found the essence of what they’re trying to achieve. The work of an amateur is often highly impractical and uneconomical, but sublimely beautiful.
- bluefirebrand 2mo agoI think you are torturing this comparison by using different meanings of amateur and professional A professional does mean "someone who does something for money" but it also often just means "someone who is very good at something" Similarly, amateur does mean "someone who does something out of passion without wanting/needing to be paid" but also often just means "someone who is just starting out, is new at something, and is probably not good enough to be a (paid) professional doing it" You're talking about the "passionate" amateur and comparing it to the "getting paid" professional. I think many people talking about amateurs and professionals are mostly talking about "amateurs (newbs)" and "professionals (very experienced)"
- agentultra 2mo agoThe market logic isn’t going to be defensible. The market is highly irrational and prone to be influenced by those with the most capital, not the, “invisible hand.” Cars are freaking everywhere not because they are the best solution of transportation but because people with vested interest in the sale of oil and cars made it so by buying politicians, starting political and marketing campaigns, and on for generations. It took decades for the car to become the de-facto mode of transportation in North America let alone the rest of the world. It’s not good for us. It’s highly irrational to prefer cars when freaking wildfires and climate change threaten our very existence. In a similar vein, LLMs as they are, are not around because they are necessary and superior.
- KurSix 2mo ago"The market chose it" doesn't mean it was the best solution, just that it fit the incentives
- BlueTemplar 2mo agoThe current kind of car is extremely good at doing what it does. It's just that most of the benefits are short term, and for the car user (and the oil salesman), while the costs are borne by everyone else, and some of the worst ones are long term. (Some similarities to hard drugs here.)
- account42 2mo agoCars are actually the best solution to transportation once you stop ignoring inconvenient but real requirements like autonomy and isolation.
- ben_w 2mo ago> Cars are actually the best solution to transportation "the" is the error here. There's places and people for whom cars are so. There's other cases where public transport covers the people while business-owned vans cover deliveries of bulky items, which is essentially how I've lived the last 8 years, though for "bulky" I should note I've seen someone take an actual kitchen skink on public transport here in Berlin. I've also found some islands here Berlin with no bridge connection to the mainland and hence no practical use for a car at those locations; that said, my main reference for "car free" domiciles would be Sark: https://en.wikipedia.org/wiki/Sark https://en.wikipedia.org/wiki/Sark
- didibus 2mo agoI'd agree with the OP you are responding, what he means is that this isn't just about taste. It's that the code, while it works, and the text, while it has the ideas in it, is still not to the level of a human author. It's not that it isn't tasteful, in the sense of having the flair of specific author X, but it is verbose, goes around the point to get to it, etc. Taste seems to imply an art, but I don't think we're at the point of cleverness, of LLM innovating a new beautiful programming paradigm or pattern out of immaculate taste for code. We're just talking more basic, even bad devs have more terseness, more clear intent in their prose and code, etc. It's "good enough" for many use-cases, but it could be better, even before expecting the most beautifully clear code from the best devs out there, it has not reached the level of the average one in readability and clarity of expression (both code and prose). That annoys the author, he has to intervene manually to trim all the fat, and therefore not "good enough".
- samiv 2mo agoNot disagreeing with you but the "resource allocation" which I presume to hint at the fallacy that capitalism is somehow great or optiomal at allocating resources is not really true. Just look at any big company such as meta and see how much inefficiency and bad choices there is. Stock market as a measuring stick is completely irrational.
- BlueTemplar 2mo agoFacebook/Meta is a great example indeed when one remembers that they took part in a genocide, and still haven't faced consequences for it.
- nunez 2mo agoBecause if you criticize the LLMs, you get headbutted into the Dante's Inferno that is the current job market
- bigbuppo 2mo agoThere's the hope that they'll make all employees redundant and that businesses that adopted AI quickly will be able to exist as a pure money-printing entity without the liabilities of products, customers, or employees. Put another way, LLMs are just another part of enshittification.
- kingkongjaffa 2mo ago> if LLMs are not "good enough", why are they still around They are not good enough yet, the people using these tools every day, and with a high bar for quality and design 'taste' notice the rough edges. They are still around because LLMs are plenty good enough for casual consumers, but where 'taste' really starts to matter in design or in writing, the rough edges show up much more obviously and users who have been working with these tools feel it immediately.
- dwedge 2mo agoThe market desperately wants to replace wages and the idea of a machine doing the work you previously needed to do before is so enticing, and the initial results so promising, that the rest doesn't matter. Look at how many AI lay offs there already were a year or two ago - back when almost everyone would agree they weren't good enough. If good enough didn't matter then does it matter how when AI is better? And despite what you might be trying to prove, it's obvious I'm replying to an LLM (a green account at that), but it's not for you, I don't expect you to bother reading it.