8 ms·

Cheating Is All You Need

73 points by sebg 2y ago

fiddlerwoaroof 2y ago
> You get the LLM to draft some code for you that’s 80% complete/correct. > You tweak the last 20% by hand. > How much of a productivity increase is that? Well jeepers, if you’re only doing 1/5th the work, then you are… punches buttons on calculator watch… five times as productive Except, it’s probably four times as hard to verify the 80% right code as it is to just write the code yourself in the first place.
- __loam 2y ago
  It's fascinating watching people in ostensibly leadership positions in our industry continue to fundamentally misunderstand how productivity works in knowledge work.
  - ezekg 2y ago
    They never understood. AI is the new Mythical Man-Month.
    - saghm 2y ago
      The example I used to hear to demonstrate this type of logical fallacy is that even though one woman can make a baby in 9 months, having eight other women help doesn't make the baby available after a single month. I guess this might need to be updated to "one woman and eight LLMs can't make a baby in one month".
  - seabombs 2y ago
    Totally. It used to be that LoC was widely accepted as a a terrible metric to measure dev productivity by, but when talking about AI code-gen tools it's like it's all that matters.
- remram 2y ago
  The problem is that now, when your colleagues/contributors push out shitty code that you have to fix... you don't know if they even tried or just put in bad code from an LLM.
  - deleted 2y ago
    [deleted]
  - mintplant 2y ago
    Yep, this is a hell I've come to know over the past year. You get people trying to build things they really have no business building, not without at least doing their own research and learning first, but the LLM in their editor lets them spit out something that looks convincing enough to get shipped, unless someone with domain experience looks at it more closely and realizes it's all wrong. It's not the same as bodged-together code from Stack Overflow because the LLM code is better at getting past the sniff test, and magical thinking around generative AI leads people to put undue trust in it. I don't think we've collectively appreciated the negative impact this is going to have on software quality going forward.
  - rad_gruchalski 2y ago
    Yeah, but you can teach your colleagues to get better. An LLM will give you the same crap every time.
    - remram 2y ago
      You missed my point, there is no use teaching those who are cheating. And there is no use learning if you don't get called out.
      - rad_gruchalski 2y ago
        Thanks tor pointing that out. I missed that, indeed. That’s a stellar point.
      - BeetleB 2y ago
        > You missed my point, there is no use teaching those who are cheating. It's a very invalid point. Perhaps in your life you've never encountered people who cheat and later change their ways, but it's fairly common for them to change their ways.
- bongodongobob 2y ago
  Except you saved yourself all that mental effort. If it takes the same amount of time and I don't need to put 110% focus in, it's a win.
  - fiddlerwoaroof 2y ago
    Verifying code is much more mental effort than writing it because you have to reconstruct the data structure and the logic of the code in order to show that the code you’re reading is correct.
  - yoyohello13 2y ago
    You also absorbed no knowledge of the problem domain. I'm pretty positive on LLMs but with the caveat that the programmer still should be an active participant in the development of the solution.
    - bongodongobob 2y ago
      I gave it the knowledge of the domain. I never said the programmer shouldn't be an active participant either. There's no way around that.
  - phyzome 2y ago
    No, now I get to be a code reviewer instead of a writer, and that's a lot more mental energy for me.
- ezekg 2y ago
  This is pretty much it. I don't trust code produced by AI, tbh, because it almost always has logical errors, or it ignored part of the prompt so it's subtly wrong. I really like AI for prototyping and sketching, but then I usually take that prototype, maybe write some solid tests, and then write the real implementation.
  - energy123 2y ago
    If the code these LLMs produce has logical errors and bugs, I have formed the opinion that this is mostly user error. LLMs produce great code if they're used properly. By that I mean: strictly control the maximum level of complexity that you're asking the LLMs to solve. GPT-4o can only handle rudimentary coding tasks. So I only ask it rudimentary questions. And it's rarely wrong or bugged. o1-2024-12-17 is much better, it can handle "mid" complexity tasks, it's usually never wrong on those, but a step above and it will output buggy code. This is subjective and not perfect, but when you use the models more you can get a sense for where it starts falling apart and stay below that threshold. This means chunking your problem into multiple prompts, where each chunk is nearing the maximum level of complexity that the model can handle.
    - onemoresoop 2y ago
      The way many LLms are coded is to not ask questions where clarification is needed but just eagerly produce something and fill in the gaps.
- dudeinjapan 2y ago
  [dead]
- saghm 2y ago
  Yeah, this is almost literally the 80-20 rule[1]; often getting the first 80% of something done takes 20% of the work, and getting the last 20% takes 80% of the work. Debugging the last 20% is already the hard part for most things, and as you mention, it's going to be a lot harder debugging LLM-generated code. [1]: https://en.wikipedia.org/wiki/Pareto_principle https://en.wikipedia.org/wiki/Pareto_principle
  - gregw2 2y ago
    re: debugging I dont disagree but here's another spin on what you observed. Kernighan's law says you have to be twice as intelligent/skilled when debugging the code as you were when writing it. A LLM centric corollary would be -don't have the LLM write code at the boundary of its capabilities in your domain. It's a little hard these days to know where that line lies with the rate of improvements but perhaps is an intuition people are still new at with LLMs.
- deleted 2y ago
  [deleted]
- golergka 2y ago
  I've built a pet project with LLMs recently. It's rough around the edges, with bugs, incomplete functionality and docs, but I've built it around 4 times as fast as I would have done it myself. https://github.com/golergka/hn-blog-recommender https://github.com/golergka/hn-blog-recommender Basically 95% of the code is written by Aider. I still had to do QA, make high-level decisions, review code it wrote, assign tasks and tell it when to refactor stuff. I would never use this workflow to write a program where absolute correctness is top priority, like a radiation therapy controller. But from this point on, I'll never waste time building pet projects, small web app and utilities myself.
  - __loam 2y ago
    And you'll be a worse programmer for it because you won't have the reps.
    - nradov 2y ago
      Programming is a means to an end for most of us. If you want to become an expert programmer for the sake of the craft then go for it, but most would benefit more from learning more about the business side. If you're building the wrong functionality then it doesn't matter how good your code is.
      - __loam 2y ago
        It's fascinating seeing this take on Hackernews of all places. I'm probably one of the most mercenary programmers you'll meet but this take is cynical even to me. If you have a skill that you use professionally and get paid hundreds of thousands of dollars to do, maybe you should cultivate and maintain it, idk. You'll probably get paid more and work less if you know what you're doing as opposed to being yet another disposable I'll handle the business side guy.
    - golergka 2y ago
      I've been programming since 95. I have crafted beautiful handwritten codebases with much more complicated requirements. But at this point, I see code quality not as an end goal, but one of the trade offs.
- onethumb 2y ago
  I've deployed code with Claude that was >100X faster, all-in (verification, testing, etc) than doing it myself. (I've been writing code professionally and successfully for >30 years). >100X isn't typical, I'd guess typical in my case is >2X, but it's happened at large scale and saved me months of work in this single instance. (Years of work overall)
  - machinekob 2y ago
    If you wirte 100x faster code you could probably automate almost all of it away as it seems to be super trivial and already resolved problems. I also use Claude for my coding a lot, but i’m not sure about real gains if it even give me noticeable speed improvement as i’m in a loop of “waiting” and fixing bugs that LLM made still it is super useful for writing big doc strings and smaller tests maybe if i focus on some basic tasks like classical backend or frontend it’ll be more useful
- Diggsey 2y ago
  Using AI is like trying to build a product with only extremely junior devs. You can get something working fairly quickly but sooner rather than later it's going to collapse in on itself and there won't be anyone around capable of fixing it. Moving past that requires someone with more experience to figure out what they did and eventually fix it, which ends up being far more expensive than correcting things along the way. The error rate needs to come down significantly from where it's currently at, and we'd also need AIs that cover each others weaknesses - at the moment you can't just run another AI to fix the 20% of errors the first one generated because they tend to have overlapping blind spots. Having said all that, I think AI is hugely useful as a programming tool. Here are the places I've got most value: - Better IDE auto-complete. - Repetetive re-factors that are beyond what's possible to do with a regex. For example, replaing one function with another whose usage is subtly different. LLMs can crack out those changes across a codebase and it's easy to review. - Summarizing and extracting intent from dense technical content. I'm often implementing some part of a web standard, and being able to quickly ask an AI questions, and have it give references back to specific parts of the standard is super useful. - Bootstrapping tests. The hardest part of writing tests in my experience is writing the first one - there's always a bit of boilerplate that's annoying to write. The AI is pretty good at generating a smoke test and then you can use that a basis to do proper test coverage. I'm sure there are more. The point is that it's currently incredibly valuable as a tool, but it's not yet capable of being an agent.
  - onemoresoop 2y ago
    It seems like nobody cares about the code quality, it just has to work until the conpany is acquired
- yapyap 2y ago
  why dont they get another LLM to do the last 20%, are they stupid?
- johnfn 2y ago
  The thing that everyone seems to miss here is it's not surprising which code falls into the 80% or 20%. I can guess if an AI will write something bug-free with almost 100% accuracy. For the remaining 20%, whatever, it gives me a fine enough starting point.
- deleted 2y ago
  [deleted]
- vunderba 2y ago
  Agreed. To add to this: The article is using a rather poor comparison because it makes the assumption that all sections of a program take an equal amount of time to write. An LLM might be able to hammer out the unit tests, boilerplate react logic, and the express backend in 5 minutes, but if it collapses on the "20%" that you need actual assistance with, then the productivity boost is significantly less than suggested.
- taneq 2y ago
  To debug code that doesn't work, you generally have to fully understand the goal and the code. To write code that does work, you generally have to fully understand the goal and what code to use. So unless you just flat-out don't know what code to use (ie. libraries, architecture, approach etc.) it's going to be no faster and probably less satisfying to fix LLM code as it would be to just do it yourself. Where LLMs do shine in coding (in my limited experience) though is discovery. If you need to write something in a mostly unfamiliar language and ecosystem they can be great for getting off the ground. I'm still not convinced that the mental gear shifting between 'student asking questions' and 'researcher/analyst/coder mode' is worth the slightly more targeted responses vs. just looking up tutorials, but it's more useful scenario.
- edanm 2y ago
  > Except, it’s probably four times as hard to verify the 80% right code as it is to just write the code yourself in the first place. Why would that be true in general? You think it just so happens that a new tool gives an X speedup in ability but costs an exactly equivalent X slowdown in other places?
  - fiddlerwoaroof 2y ago
    No, the claim is just that it’s harder to read and verify existing code than to write new code. Depending on the level of certainty you want, verifying code is much more expensive than writing code, which is why there are so many bugs and why PR review is, imo, practically worthless. Switching the job of programming from designing and writing code to reading and verifying code isn’t a good trade-off in terms of making programming easier. And my first speei node with AI tools has done nothing to change my mind on this point.
gibsonf1 2y ago
A technology as unreliable as LLM's is definitely not the biggest thing since the WWW, and perhaps will fade out in time except for niche cases where reliability is not important and users are willing to pay the enormous amount it costs to run them. (After investor money has run out)
Mr-Frog 2y ago
Enthusiastic article for sure. Lots of the hype in this piece suggests that effective semantic search for populating limited context windows of LLMs will be big business. My thoughts: 1. Will context windows grow enough so that we can afford to be lazy and drop entire knowledge bases into our queries? 2. Will the actual models of LLMs be able to more reliably capture data in a semi-structured way, perhaps bypassing the "data moat" that the author claims?
- BOOSTERHIDROGEN 2y ago
  A major innovation in search solutions is eagerly anticipated.
- deleted 2y ago
  [deleted]
debo_ 2y ago
I got to the bottom of the post and thought, wow this sounds like Steve Yegge, I wonder what happened to that guy. So, I scrolled back up to the top, and, lo and behold... I'm positive about coding + LLMs, personally. I like it better than going without. I also think it's ok to just use a thing and be happy about it without having to predict whether it's going to be great or awful 10 years from now. It's ok to not know the future. While we are here, I'm currently using Cursor as my programming assistant of choice. I assume things are moving fast enough that there are already better alternatives. Any suggestions?
deleted 2y ago
[deleted]
trescenzi 2y ago
I'm 100% an AI skeptic. I don't think we're near AGI, well any reasonable definition of it. I don't think we're remotely near a super intelligence or the singularity or whatever. But that doesn't matter. These tools will change how work is done. What matters is what he says here: > Peeps, let’s do some really simple back-of-envelope math. Trust me, it won’t be difficult math. > You get the LLM to draft some code for you that’s 80% complete/correct. > You tweak the last 20% by hand. > How much of a productivity increase is that? Well jeepers, if you’re only doing 1/5th the work, then you are… punches buttons on calculator watch… five times as productive. The hard part is the engineering, yes, and now we can actually focus on it. 90% of the work that software engineers do isn't novel. People aren't sitting around coding complicated algorithms that require PhDs. Most of the work is doing pretty boring stuff. And even if LLMs can only write 50% of that you're still going to get a massive boost in productivity. Of course this isn't always going to be great. Debugging this code is probably going to suck, and even with the best engineering it'll likely accelerate big balls of mud. But it's going to enable a lot of code to get written by a lot more people and it's super exciting.
- eric-hu 2y ago
  > I don't think we're near AGI, well any reasonable definition of it. But wait just hear me out for a second! What if we've also sufficiently lowered the definition of "GI" at the same time...
- yoyohello13 2y ago
  > But it's going to enable a lot of code to get written by a lot more people and it's super exciting. Is that exciting though? A lot of the code we've currently got is garbage. We are already slogging through a morass of code. I could be wrong, but I don't think LLMs are going to change this trend.
  - trescenzi 2y ago
    I think it's exciting that more people will be able to build more things. I agree though that most code is garbage, and that it's likely going to get worse. The future can be both concerning and exciting.
phyzome 2y ago
That was a lot of words to describe... I think just "LLMs use context"? And the intro made me think he was mad about something, but I think he's actually excited? It's very confusing. I will admit, I skimmed. No jury would convict.
- rm445 2y ago
  It's a Steve Yegge article. His writing is enjoyable, but it's a miracle he wrote an article short enough not to skim. He's excitable and verbose. Although he's talking up his product, it's an interesting premise. The idea is that cooking relevant context and parts from a codebase down into an LLM's context window, is a useful product for coding assistants.
james_promoted 2y ago
Worth noting this is from March 2023.
deleted 2y ago
[deleted]
hn_throwaway_99 2y ago
I know how popular Steve Yegge is, and there is a bunch of stuff he's written in the past that I've enjoyed, but like, what is this? I know everyone says the Internet has ruined our attention spans, but I think it's fair to let the reader know what the point of your article is fairly early on. As far as I can tell, this is a really unnecessarily long-winded article that is basically just "make sure you have good prompt context." And yes, when you're doing RAG, you need to make sure you pull the right data for your context. Is this just a spam blog post for Sourcegraph?
- devmor 2y ago
  The semi-obvious conclusion here is that he had an idea and asked an LLM to write an article about it, in his own style.
- 9question1 2y ago
  It's an ad. The key point he works up to is `Put another way, you need a sidecar database. The data moat needs to be fast and queryable. This is a Search Problem!` And then of course this is hosted on the corporate blog of a code search platform. It's `make sure you have good prompt context` but trying to define that specifically as the thing they're selling
  - hn_throwaway_99 2y ago
    As another commenter said, this needs a March 2023 tag. RAG is now pretty standard and their are lots of vector databases that do this.
- deleted 2y ago
  [deleted]
- deleted 2y ago
  [deleted]
dannyobrien 2y ago
I think one important aspect of this is that many -- perhaps the bulk, now -- of people writing or talking on the Internet don't have much memory of a time when digital technology was underestimated -- or at least, did not get to their current positions by recognising when something was underestimated, and going with their own, overestimated beliefs. They probably got into technology because it was self-evidently a route to success, they were advised to do so, or they were as excited as anyone else by its overt promise. Their experience is dominated by becoming disillusioned with what they were sold on. A different slice of time, and a different set of opportunities -- as Yegge says, the recollection that if you hadn't poo-pooed some prototype, you'd have $130 million based on an undeniably successful model -- gives you a different instinct. I really do hear the -- utterly valid -- criticisms of the early Web and Internet when I hear people's skepticism of AI, including that it is overhyped. Overhyped or not, betting on the Internet in 1994 would have gotten you far, far, more of an insight (and an income) than observing all its early flaws and ignoring or decrying it. This is somewhat orthogonal to whether those criticisms are correct!
- __loam 2y ago
  To be fair to the skeptics, for every Amazon or Google, there were 10 other companies that were actually bad ideas or didn't survive the dot com crash. It's also hard to say how much of the take here is apocryphal in terms of AWS being a plucky demo. It seems inevitable that someone would realize that with good enough bandwidth infrastructure, concentrating utility computing in data centers and increasing utilization of those resources through multi-tenency is just a good idea that is beneficial for the capital owners and the renters. Research into distributed and shared computing was around since the 80's, Amazon was just the first to realize the business potential at a time when hardware virtualization actually supported what they were trying to do.
- MobiusHorizons 2y ago
  > This is somewhat orthogonal to whether those criticisms are correct! I think this is the most important take-away in evaluating the success of many technical businesses. At the end of the day businesses succeed or fail due to lots of factors that have nothing to do with the soundness of the enigneering or R&D that is at the core of the value proposition. This is an extremely frustrating fact for technically minded folks (I count myself). Reminds me of the quote about how stock markets are like betting on the outcome of a beauty contest. The humans who control large purchasing decisions are likely much less technical than the average audience of this blog. The fact that this is frustrating doesn't make it any less true or important to internalize.
gatkinso 2y ago
Have any of these LLM coding assistant enthusiasts actually used them? They are ok at autocomplete and writing types. They cannot yet write "80% of your code" on anything shippable. They regularly miss small but crucial details. The more 'enthusiastic' ones, like Windsurf, subtly refactor code - removing or changing functionality which will have you scratching your head. Programming is a process of building mental models. LLMs are trying to replace something that is fundamental to engineering. Maybe they'll improve, but writing the code was never the hard part. I think the more likely outcome of LLMs is the broad lowering of standards than 5x productivity for all.
- Inviz 2y ago
  I'm trying to use Cursor for full time building of applications for my personal need ( years of experience in the field). It can be rough, but you see the glimpses of the future. In the early weeks, I am constantly having to think of new ways to minimize surprise. Prompt-injection via dotfile (collection of Dos and Donts), forcing LLM to write itself jsdoc @instructions in problematic files, coming up with project overviews that states goals and responsibilities of different parts. It's all very flaky, but there's a promise. I think it is a good time to try mastering the LLM as a helper, and to figure out how to play your best game using this ultra sharp tool. My 2c is that engineer will be needed in the loop to build anything meaningful. It's really hard to shape the play-dough of the codebase having to give up so much control. I see other people who can't validate the LLM BS get themselves into the hole really quickly when using multi-file editing capabilities like the Composer feature. One thing that kinda blew me away even though it's embarrasingly predictable is "parallel edit" workflow, where LLM can edit multiple files at the same time using a shared strategy. It almost never works currently for anything non-trivial, so the agent almost never uses it unless you ask it to (Run typescript check and use parallel edit to fix the bugs). It's also dumb, using the same auto-completion model that doesnt have access to Composer context, it feels like. But this shows how we can absorb the tax of waiting for LLM to respond, is by doing multiple things at once to many files. And tests. Lots of tests. There's no way about it, LLM can spice things up really subtly and without tests on a big project it's impossible to fix. LLMS dont have the same fear of old files from previous ages, they can start hacking what's not broken. Cursor has a capable "code review" system with checkpoints and quick rollbacks. At this point in time we have to review most of what LLM is spewing out. On the upside, LLM can write decent tests, extract failing cases, etc. There's no excuse not to use tests. However extra care needs to be taken with LLM-generated tests, as to review their legitimacy in asserts. I tried to write a popular table game engine without understanding game rules personally, and i had a bunch of reference material. It was difficult when I couldnt validate the assertions. I think I failed the experiment and had to learn game rules by debugging failing tests one by one. And in many cases the culprit was the fixture generated by LLM. We can used combined workflows to make smarter models like o1-pro to validate the fixtures, that typically works on well known concepts and games
lelandbatey 2y ago
The above post is mega-ranty. I am tempted to mega-rant in return, but I'll try to keep that down and share some material that seems related but solvable by someone making such broad claims: - The parent article complains about a lot of "meh" responses by engineers. I often feel very "meh" about LLMs and don't use them very often. It's not cause I hate them, but it is because my context window that I'd need to feed into an LLM is just a bit too big to be tractable for having it help me code. What I need is for my LLM code assistant to be able to comfortably accept ~7-10 git repositories, each with a code+comment line count of ~100k lines of novel (not in the training set) code, and be able to answer my questions and write somewhat useful code. I'm working in big, old, corporate, proprietary codebases, and I need an LLM to be even a little helpful with that. Maybe that's possible and easy today, but I don't know how to do that so I don't. I'm open to suggestions. - When I talk to a co-worker, even a junior engineer, I feel like I can have sensible discussions with them even if they may not know exactly what I'm talking about. I can trust that they'll say things like "I don't really know all the details, but it sounds like...", or they'll ask follow up questions. I do not have to tell my co-workers to ask follow up questions or to tell me about the bounds of their understanding. We're human, we talk. When I try to chat with an LLM about business problems, I feel more like I'm trying to chat with a 15 year old who is desperately trying to sound knowledgeable, where they will try their best to confidently answer my questions every time. They won't ever specify "I don't know about that, but tell me if this sounds right...". Instead, they'll confidently lie to my face with insanely fabricated details. I don't want to have to navigate a conversation with the background question of "is this the fabrications of a madman?" in the back of my head all the time, so I don't. Maybe that's the wrong value calculus, and if so, I'm open to being persuaded. Please, share your thoughts.
d_burfoot 2y ago
Warren Buffett says "Be fearful when others are greedy, and greedy when others are fearful." But in the AI space, people are both greedy AND fearful!
- cadamsdotcom 2y ago
  That’s a sign of not enough granularity. Some folks use it for code and love it, some use it for code and hate it. Writing code is so broad though- are we coding device drivers? Prototyping greenfield CRUD apps? Fixing hairy bugs in 50year old codebases? All of those are very different activities where folks have different needs, but discussions rarely include this context. If you can find the right dividing line where on one side people are greedy/love the tech, and on the other they’re fearful/doubting… you’re ahead of the pack - and a bit more ready to invest.
kibwen 2y ago
> How much of a productivity increase is that? Well jeepers, if you’re only doing 1/5th the work, then you are… punches buttons on calculator watch… five times as productive. Cool, so I'm going to make five times more money? What's that you say, I'm not? So who is going to be harvesting the profits, then?
- theultdev 2y ago
  If you're 5x more productive from AI, you can do 5x less effort if you feel your previous productivity was sufficient for the current pay.
- fragmede 2y ago
  It's a pretty good time to be a contractor, as long as you negotiate getting paid per project/milestone and not by the hour. What used to take three days now takes one, and you can use the extra time to book more clients.
- efs24 2y ago
  You will get paid 5x as much… assuming you are the only developer that uses this and the market for engineers is rational. However those other developers are going to be 5x as productive too so your rate should stay the same. The company that pays you will get more code for less money and so they would make more profit if they are the only company taking advantage of this. But all companies will start to produce software at a lower cost so their profits per line of code will be squeezed to where they are now. You will spend a lot less time writing boiler plate and searching docs or stack overflow. That frees you up to work on real problems and since you’ll be able to create products with less cost per line of code you could potentially compete with your own offerings. That’s my take anyway.
itissid 2y ago
The only thing missing right now is a solid goal planning, goal validation engine and search to define rewards. Since code is one of those things you can train LLMs with using RL – not RHLF – it can actually do goal based AI planning, all the people who say "I don't trust LLM code" are missing how beside the point this is. A lot of code in this world is accepted, even if it is suboptimal and people don't realize how much money is to be made by writing it so with ML models.
sciurus 2y ago
This would benefit from a "[2023]" appended to the title to make clear it isn't new. Past discussion: https://news.ycombinator.com/item?id=35273406 https://news.ycombinator.com/item?id=35273406
jldugger 2y ago
(2023)
huqedato 2y ago
This article is just a nicely crafted marketing ad for Cody. It impressed more than the product itself (which is not so unique, there are many althernatives on the market, see Codeium for example)
ColonelPhantom 2y ago
This article really grates me the wrong way. A company selling an AI product saying the following about AI sceptics: > All you crazy MFs are completely overlooking the fact that software engineering exists as a discipline because you cannot EVER under any circumstances TRUST CODE. is straight up insulting to me, because it effectively comes down to "use my product or you're a looney". Also, while two years after this post (which should be labeled 2023) I've still barely tried to entirely offload coding to an LLM, the few times I did try have been pretty crap. I also really, really don't want to 'chat' with my codebase or editor. 'Chatting' to me feels about as slow as writing something myself, while I also don't get a good mental model 'for free'. I am a moderately happy user of AI autocomplete (specifically Supermaven), but I only ever accept suggestions that are trivially correct to me. If it's not trivial, it might be useful as a guide of where to look in actual documentation if relevant, but just accepting it will lead me down a wrong path more often than not.
simonw 2y ago
This piece is coming up on two years old now - it was written in March 2023. I think it holds up. Saying "LLMs are useful for code" is a lot less controversial today than it was back in March 2023 - there are still some hold-outs who will swear that all LLM-produced code is untrustworthy junk and no good engineer would dream of using them, but I think their numbers are continuing to drop.
spondyl 2y ago
As a bit of anecdata, I used an LLM to refactor a core part of one of my site projects about 6 months or so ago. I don't really remember how it works, probably because I didn't have to think hard about the design, I just sort of iterated it in conversation which probably meant it didn't stick. While the iteration part was mostly positive, I don't know that it was worth it in the end to be left with something that I can understand but feels a bit foreign at the same time.
cadamsdotcom 2y ago
(2023)
pilgrim0 2y ago
Starts the article giving examples of novel and unexpected ways people employed available paradigms to create value, then proceeds to sell LLMs which are completely biased by the most standard ways of doing things…
gatienboquet 2y ago
Gemini-Exp-1206 got 2M context token, what's preventing Google from fine-tuning it with internal google PR etc ? So in the end you'll get a fine-tuned google llm dev and one finetuned for review, etc etc
lukev 2y ago
One thing I'll note is that I'm inclined to trust the poll of engineers who have actually been using this stuff for real work, as opposed to a N=1 example of it performing well (outside the context of a real software project) and using that as the basis for declaring that This Is The Future. I'm not saying there's not a money volcano to be had somewhere, built on top of LLMs. But his call to action is stupid. Every other "money volcano" he cites was a clever use of existing technology to meet a previously-unseen but very real need. LLMs are the mirror image of that. They are something that's clearly, obviously useful and yet people are struggling to leverage them to the degree that _ought_ to be possible. There are no truly slam dunk applications yet, let alone money volcanos (except for the AI hype train itself.) They are the ultimate hammer in search of a nail. And it's a pretty amazing hammer, I'm not a complete LLM skeptic. But at this point we should all be searching for the best nails, not just cheerleading for the hammer.
cheevly 2y ago
And the even more jaw-dropping reality is that even this article heavily underestimates the transformative power that generative AI is about to bring to software. Using AI to generate code like this is akin to using a horse to pull an automobile.
from-nibly 2y ago
If programming as theory building is any bit true then writing code with an LLM doesn't co UK BT as writing code by hand. At least not wuite the same way. This is the problem. Managers think that the code is the product. It isn't. And LLMs can only produce the code.
seeg 2y ago
1. If you can easily write boring code with LLM, you might not have the incentive to refactor it. 2. If you let someone else solve your problems, and you don't reflect in the solution, you won't learn. 3. I rarely read the fine print, but are all LLM IDEs really the same when it comes to intellectual property of the code? Or the responsibily of bad code?

73 points by sebg 2y ago

> You get the LLM to draft some code for you that’s 80% complete/correct. > You tweak the last 20% by hand. > How much of a productivity increase is that? Well jeepers, if you’re only doing 1/5th the work, then you are… punches buttons on calculator watch… five times as productive Except, it’s probably four times as hard to verify the 80% right code as it is to just write the code yourself in the first place.

__loam 2y ago
It's fascinating watching people in ostensibly leadership positions in our industry continue to fundamentally misunderstand how productivity works in knowledge work.
- ezekg 2y ago
  They never understood. AI is the new Mythical Man-Month.
  - saghm 2y ago
    The example I used to hear to demonstrate this type of logical fallacy is that even though one woman can make a baby in 9 months, having eight other women help doesn't make the baby available after a single month. I guess this might need to be updated to "one woman and eight LLMs can't make a baby in one month".
- seabombs 2y ago
  Totally. It used to be that LoC was widely accepted as a a terrible metric to measure dev productivity by, but when talking about AI code-gen tools it's like it's all that matters.
remram 2y ago
The problem is that now, when your colleagues/contributors push out shitty code that you have to fix... you don't know if they even tried or just put in bad code from an LLM.
- deleted 2y ago
  [deleted]
- mintplant 2y ago
  Yep, this is a hell I've come to know over the past year. You get people trying to build things they really have no business building, not without at least doing their own research and learning first, but the LLM in their editor lets them spit out something that looks convincing enough to get shipped, unless someone with domain experience looks at it more closely and realizes it's all wrong. It's not the same as bodged-together code from Stack Overflow because the LLM code is better at getting past the sniff test, and magical thinking around generative AI leads people to put undue trust in it. I don't think we've collectively appreciated the negative impact this is going to have on software quality going forward.
- rad_gruchalski 2y ago
  Yeah, but you can teach your colleagues to get better. An LLM will give you the same crap every time.
  - remram 2y ago
    You missed my point, there is no use teaching those who are cheating. And there is no use learning if you don't get called out.
    - rad_gruchalski 2y ago
      Thanks tor pointing that out. I missed that, indeed. That’s a stellar point.
    - BeetleB 2y ago
      > You missed my point, there is no use teaching those who are cheating. It's a very invalid point. Perhaps in your life you've never encountered people who cheat and later change their ways, but it's fairly common for them to change their ways.
bongodongobob 2y ago
Except you saved yourself all that mental effort. If it takes the same amount of time and I don't need to put 110% focus in, it's a win.
- fiddlerwoaroof 2y ago
  Verifying code is much more mental effort than writing it because you have to reconstruct the data structure and the logic of the code in order to show that the code you’re reading is correct.
- yoyohello13 2y ago
  You also absorbed no knowledge of the problem domain. I'm pretty positive on LLMs but with the caveat that the programmer still should be an active participant in the development of the solution.
  - bongodongobob 2y ago
    I gave it the knowledge of the domain. I never said the programmer shouldn't be an active participant either. There's no way around that.
- phyzome 2y ago
  No, now I get to be a code reviewer instead of a writer, and that's a lot more mental energy for me.
ezekg 2y ago
This is pretty much it. I don't trust code produced by AI, tbh, because it almost always has logical errors, or it ignored part of the prompt so it's subtly wrong. I really like AI for prototyping and sketching, but then I usually take that prototype, maybe write some solid tests, and then write the real implementation.
- energy123 2y ago
  If the code these LLMs produce has logical errors and bugs, I have formed the opinion that this is mostly user error. LLMs produce great code if they're used properly. By that I mean: strictly control the maximum level of complexity that you're asking the LLMs to solve. GPT-4o can only handle rudimentary coding tasks. So I only ask it rudimentary questions. And it's rarely wrong or bugged. o1-2024-12-17 is much better, it can handle "mid" complexity tasks, it's usually never wrong on those, but a step above and it will output buggy code. This is subjective and not perfect, but when you use the models more you can get a sense for where it starts falling apart and stay below that threshold. This means chunking your problem into multiple prompts, where each chunk is nearing the maximum level of complexity that the model can handle.
  - onemoresoop 2y ago
    The way many LLms are coded is to not ask questions where clarification is needed but just eagerly produce something and fill in the gaps.
dudeinjapan 2y ago
[dead]
saghm 2y ago
Yeah, this is almost literally the 80-20 rule[1]; often getting the first 80% of something done takes 20% of the work, and getting the last 20% takes 80% of the work. Debugging the last 20% is already the hard part for most things, and as you mention, it's going to be a lot harder debugging LLM-generated code. [1]: https://en.wikipedia.org/wiki/Pareto_principle https://en.wikipedia.org/wiki/Pareto_principle
- gregw2 2y ago
  re: debugging I dont disagree but here's another spin on what you observed. Kernighan's law says you have to be twice as intelligent/skilled when debugging the code as you were when writing it. A LLM centric corollary would be -don't have the LLM write code at the boundary of its capabilities in your domain. It's a little hard these days to know where that line lies with the rate of improvements but perhaps is an intuition people are still new at with LLMs.
deleted 2y ago
[deleted]
golergka 2y ago
I've built a pet project with LLMs recently. It's rough around the edges, with bugs, incomplete functionality and docs, but I've built it around 4 times as fast as I would have done it myself. https://github.com/golergka/hn-blog-recommender https://github.com/golergka/hn-blog-recommender Basically 95% of the code is written by Aider. I still had to do QA, make high-level decisions, review code it wrote, assign tasks and tell it when to refactor stuff. I would never use this workflow to write a program where absolute correctness is top priority, like a radiation therapy controller. But from this point on, I'll never waste time building pet projects, small web app and utilities myself.
- __loam 2y ago
  And you'll be a worse programmer for it because you won't have the reps.
  - nradov 2y ago
    Programming is a means to an end for most of us. If you want to become an expert programmer for the sake of the craft then go for it, but most would benefit more from learning more about the business side. If you're building the wrong functionality then it doesn't matter how good your code is.
    - __loam 2y ago
      It's fascinating seeing this take on Hackernews of all places. I'm probably one of the most mercenary programmers you'll meet but this take is cynical even to me. If you have a skill that you use professionally and get paid hundreds of thousands of dollars to do, maybe you should cultivate and maintain it, idk. You'll probably get paid more and work less if you know what you're doing as opposed to being yet another disposable I'll handle the business side guy.
  - golergka 2y ago
    I've been programming since 95. I have crafted beautiful handwritten codebases with much more complicated requirements. But at this point, I see code quality not as an end goal, but one of the trade offs.
onethumb 2y ago
I've deployed code with Claude that was >100X faster, all-in (verification, testing, etc) than doing it myself. (I've been writing code professionally and successfully for >30 years). >100X isn't typical, I'd guess typical in my case is >2X, but it's happened at large scale and saved me months of work in this single instance. (Years of work overall)
- machinekob 2y ago
  If you wirte 100x faster code you could probably automate almost all of it away as it seems to be super trivial and already resolved problems. I also use Claude for my coding a lot, but i’m not sure about real gains if it even give me noticeable speed improvement as i’m in a loop of “waiting” and fixing bugs that LLM made still it is super useful for writing big doc strings and smaller tests maybe if i focus on some basic tasks like classical backend or frontend it’ll be more useful
Diggsey 2y ago
Using AI is like trying to build a product with only extremely junior devs. You can get something working fairly quickly but sooner rather than later it's going to collapse in on itself and there won't be anyone around capable of fixing it. Moving past that requires someone with more experience to figure out what they did and eventually fix it, which ends up being far more expensive than correcting things along the way. The error rate needs to come down significantly from where it's currently at, and we'd also need AIs that cover each others weaknesses - at the moment you can't just run another AI to fix the 20% of errors the first one generated because they tend to have overlapping blind spots. Having said all that, I think AI is hugely useful as a programming tool. Here are the places I've got most value: - Better IDE auto-complete. - Repetetive re-factors that are beyond what's possible to do with a regex. For example, replaing one function with another whose usage is subtly different. LLMs can crack out those changes across a codebase and it's easy to review. - Summarizing and extracting intent from dense technical content. I'm often implementing some part of a web standard, and being able to quickly ask an AI questions, and have it give references back to specific parts of the standard is super useful. - Bootstrapping tests. The hardest part of writing tests in my experience is writing the first one - there's always a bit of boilerplate that's annoying to write. The AI is pretty good at generating a smoke test and then you can use that a basis to do proper test coverage. I'm sure there are more. The point is that it's currently incredibly valuable as a tool, but it's not yet capable of being an agent.
- onemoresoop 2y ago
  It seems like nobody cares about the code quality, it just has to work until the conpany is acquired
yapyap 2y ago
why dont they get another LLM to do the last 20%, are they stupid?
johnfn 2y ago
The thing that everyone seems to miss here is it's not surprising which code falls into the 80% or 20%. I can guess if an AI will write something bug-free with almost 100% accuracy. For the remaining 20%, whatever, it gives me a fine enough starting point.
deleted 2y ago
[deleted]
vunderba 2y ago
Agreed. To add to this: The article is using a rather poor comparison because it makes the assumption that all sections of a program take an equal amount of time to write. An LLM might be able to hammer out the unit tests, boilerplate react logic, and the express backend in 5 minutes, but if it collapses on the "20%" that you need actual assistance with, then the productivity boost is significantly less than suggested.
taneq 2y ago
To debug code that doesn't work, you generally have to fully understand the goal and the code. To write code that does work, you generally have to fully understand the goal and what code to use. So unless you just flat-out don't know what code to use (ie. libraries, architecture, approach etc.) it's going to be no faster and probably less satisfying to fix LLM code as it would be to just do it yourself. Where LLMs do shine in coding (in my limited experience) though is discovery. If you need to write something in a mostly unfamiliar language and ecosystem they can be great for getting off the ground. I'm still not convinced that the mental gear shifting between 'student asking questions' and 'researcher/analyst/coder mode' is worth the slightly more targeted responses vs. just looking up tutorials, but it's more useful scenario.
edanm 2y ago
> Except, it’s probably four times as hard to verify the 80% right code as it is to just write the code yourself in the first place. Why would that be true in general? You think it just so happens that a new tool gives an X speedup in ability but costs an exactly equivalent X slowdown in other places?
- fiddlerwoaroof 2y ago
  No, the claim is just that it’s harder to read and verify existing code than to write new code. Depending on the level of certainty you want, verifying code is much more expensive than writing code, which is why there are so many bugs and why PR review is, imo, practically worthless. Switching the job of programming from designing and writing code to reading and verifying code isn’t a good trade-off in terms of making programming easier. And my first speei node with AI tools has done nothing to change my mind on this point.

gibsonf1 2y ago

A technology as unreliable as LLM's is definitely not the biggest thing since the WWW, and perhaps will fade out in time except for niche cases where reliability is not important and users are willing to pay the enormous amount it costs to run them. (After investor money has run out)

Mr-Frog 2y ago

Enthusiastic article for sure. Lots of the hype in this piece suggests that effective semantic search for populating limited context windows of LLMs will be big business. My thoughts: 1. Will context windows grow enough so that we can afford to be lazy and drop entire knowledge bases into our queries? 2. Will the actual models of LLMs be able to more reliably capture data in a semi-structured way, perhaps bypassing the "data moat" that the author claims?

BOOSTERHIDROGEN 2y ago
A major innovation in search solutions is eagerly anticipated.
deleted 2y ago
[deleted]

debo_ 2y ago

I got to the bottom of the post and thought, wow this sounds like Steve Yegge, I wonder what happened to that guy. So, I scrolled back up to the top, and, lo and behold... I'm positive about coding + LLMs, personally. I like it better than going without. I also think it's ok to just use a thing and be happy about it without having to predict whether it's going to be great or awful 10 years from now. It's ok to not know the future. While we are here, I'm currently using Cursor as my programming assistant of choice. I assume things are moving fast enough that there are already better alternatives. Any suggestions?

deleted 2y ago

[deleted]

trescenzi 2y ago

I'm 100% an AI skeptic. I don't think we're near AGI, well any reasonable definition of it. I don't think we're remotely near a super intelligence or the singularity or whatever. But that doesn't matter. These tools will change how work is done. What matters is what he says here: > Peeps, let’s do some really simple back-of-envelope math. Trust me, it won’t be difficult math. > You get the LLM to draft some code for you that’s 80% complete/correct. > You tweak the last 20% by hand. > How much of a productivity increase is that? Well jeepers, if you’re only doing 1/5th the work, then you are… punches buttons on calculator watch… five times as productive. The hard part is the engineering, yes, and now we can actually focus on it. 90% of the work that software engineers do isn't novel. People aren't sitting around coding complicated algorithms that require PhDs. Most of the work is doing pretty boring stuff. And even if LLMs can only write 50% of that you're still going to get a massive boost in productivity. Of course this isn't always going to be great. Debugging this code is probably going to suck, and even with the best engineering it'll likely accelerate big balls of mud. But it's going to enable a lot of code to get written by a lot more people and it's super exciting.

eric-hu 2y ago
> I don't think we're near AGI, well any reasonable definition of it. But wait just hear me out for a second! What if we've also sufficiently lowered the definition of "GI" at the same time...
yoyohello13 2y ago
> But it's going to enable a lot of code to get written by a lot more people and it's super exciting. Is that exciting though? A lot of the code we've currently got is garbage. We are already slogging through a morass of code. I could be wrong, but I don't think LLMs are going to change this trend.
- trescenzi 2y ago
  I think it's exciting that more people will be able to build more things. I agree though that most code is garbage, and that it's likely going to get worse. The future can be both concerning and exciting.

phyzome 2y ago

That was a lot of words to describe... I think just "LLMs use context"? And the intro made me think he was mad about something, but I think he's actually excited? It's very confusing. I will admit, I skimmed. No jury would convict.

rm445 2y ago
It's a Steve Yegge article. His writing is enjoyable, but it's a miracle he wrote an article short enough not to skim. He's excitable and verbose. Although he's talking up his product, it's an interesting premise. The idea is that cooking relevant context and parts from a codebase down into an LLM's context window, is a useful product for coding assistants.

james_promoted 2y ago

Worth noting this is from March 2023.

deleted 2y ago

[deleted]

hn_throwaway_99 2y ago

I know how popular Steve Yegge is, and there is a bunch of stuff he's written in the past that I've enjoyed, but like, what is this? I know everyone says the Internet has ruined our attention spans, but I think it's fair to let the reader know what the point of your article is fairly early on. As far as I can tell, this is a really unnecessarily long-winded article that is basically just "make sure you have good prompt context." And yes, when you're doing RAG, you need to make sure you pull the right data for your context. Is this just a spam blog post for Sourcegraph?

devmor 2y ago
The semi-obvious conclusion here is that he had an idea and asked an LLM to write an article about it, in his own style.
9question1 2y ago
It's an ad. The key point he works up to is `Put another way, you need a sidecar database. The data moat needs to be fast and queryable. This is a Search Problem!` And then of course this is hosted on the corporate blog of a code search platform. It's `make sure you have good prompt context` but trying to define that specifically as the thing they're selling
- hn_throwaway_99 2y ago
  As another commenter said, this needs a March 2023 tag. RAG is now pretty standard and their are lots of vector databases that do this.
deleted 2y ago
[deleted]
deleted 2y ago
[deleted]

dannyobrien 2y ago

I think one important aspect of this is that many -- perhaps the bulk, now -- of people writing or talking on the Internet don't have much memory of a time when digital technology was underestimated -- or at least, did not get to their current positions by recognising when something was underestimated, and going with their own, overestimated beliefs. They probably got into technology because it was self-evidently a route to success, they were advised to do so, or they were as excited as anyone else by its overt promise. Their experience is dominated by becoming disillusioned with what they were sold on. A different slice of time, and a different set of opportunities -- as Yegge says, the recollection that if you hadn't poo-pooed some prototype, you'd have $130 million based on an undeniably successful model -- gives you a different instinct. I really do hear the -- utterly valid -- criticisms of the early Web and Internet when I hear people's skepticism of AI, including that it is overhyped. Overhyped or not, betting on the Internet in 1994 would have gotten you far, far, more of an insight (and an income) than observing all its early flaws and ignoring or decrying it. This is somewhat orthogonal to whether those criticisms are correct!

__loam 2y ago
To be fair to the skeptics, for every Amazon or Google, there were 10 other companies that were actually bad ideas or didn't survive the dot com crash. It's also hard to say how much of the take here is apocryphal in terms of AWS being a plucky demo. It seems inevitable that someone would realize that with good enough bandwidth infrastructure, concentrating utility computing in data centers and increasing utilization of those resources through multi-tenency is just a good idea that is beneficial for the capital owners and the renters. Research into distributed and shared computing was around since the 80's, Amazon was just the first to realize the business potential at a time when hardware virtualization actually supported what they were trying to do.
MobiusHorizons 2y ago
> This is somewhat orthogonal to whether those criticisms are correct! I think this is the most important take-away in evaluating the success of many technical businesses. At the end of the day businesses succeed or fail due to lots of factors that have nothing to do with the soundness of the enigneering or R&D that is at the core of the value proposition. This is an extremely frustrating fact for technically minded folks (I count myself). Reminds me of the quote about how stock markets are like betting on the outcome of a beauty contest. The humans who control large purchasing decisions are likely much less technical than the average audience of this blog. The fact that this is frustrating doesn't make it any less true or important to internalize.

gatkinso 2y ago

Have any of these LLM coding assistant enthusiasts actually used them? They are ok at autocomplete and writing types. They cannot yet write "80% of your code" on anything shippable. They regularly miss small but crucial details. The more 'enthusiastic' ones, like Windsurf, subtly refactor code - removing or changing functionality which will have you scratching your head. Programming is a process of building mental models. LLMs are trying to replace something that is fundamental to engineering. Maybe they'll improve, but writing the code was never the hard part. I think the more likely outcome of LLMs is the broad lowering of standards than 5x productivity for all.

Inviz 2y ago
I'm trying to use Cursor for full time building of applications for my personal need ( years of experience in the field). It can be rough, but you see the glimpses of the future. In the early weeks, I am constantly having to think of new ways to minimize surprise. Prompt-injection via dotfile (collection of Dos and Donts), forcing LLM to write itself jsdoc @instructions in problematic files, coming up with project overviews that states goals and responsibilities of different parts. It's all very flaky, but there's a promise. I think it is a good time to try mastering the LLM as a helper, and to figure out how to play your best game using this ultra sharp tool. My 2c is that engineer will be needed in the loop to build anything meaningful. It's really hard to shape the play-dough of the codebase having to give up so much control. I see other people who can't validate the LLM BS get themselves into the hole really quickly when using multi-file editing capabilities like the Composer feature. One thing that kinda blew me away even though it's embarrasingly predictable is "parallel edit" workflow, where LLM can edit multiple files at the same time using a shared strategy. It almost never works currently for anything non-trivial, so the agent almost never uses it unless you ask it to (Run typescript check and use parallel edit to fix the bugs). It's also dumb, using the same auto-completion model that doesnt have access to Composer context, it feels like. But this shows how we can absorb the tax of waiting for LLM to respond, is by doing multiple things at once to many files. And tests. Lots of tests. There's no way about it, LLM can spice things up really subtly and without tests on a big project it's impossible to fix. LLMS dont have the same fear of old files from previous ages, they can start hacking what's not broken. Cursor has a capable "code review" system with checkpoints and quick rollbacks. At this point in time we have to review most of what LLM is spewing out. On the upside, LLM can write decent tests, extract failing cases, etc. There's no excuse not to use tests. However extra care needs to be taken with LLM-generated tests, as to review their legitimacy in asserts. I tried to write a popular table game engine without understanding game rules personally, and i had a bunch of reference material. It was difficult when I couldnt validate the assertions. I think I failed the experiment and had to learn game rules by debugging failing tests one by one. And in many cases the culprit was the fixture generated by LLM. We can used combined workflows to make smarter models like o1-pro to validate the fixtures, that typically works on well known concepts and games

lelandbatey 2y ago

The above post is mega-ranty. I am tempted to mega-rant in return, but I'll try to keep that down and share some material that seems related but solvable by someone making such broad claims: - The parent article complains about a lot of "meh" responses by engineers. I often feel very "meh" about LLMs and don't use them very often. It's not cause I hate them, but it is because my context window that I'd need to feed into an LLM is just a bit too big to be tractable for having it help me code. What I need is for my LLM code assistant to be able to comfortably accept ~7-10 git repositories, each with a code+comment line count of ~100k lines of novel (not in the training set) code, and be able to answer my questions and write somewhat useful code. I'm working in big, old, corporate, proprietary codebases, and I need an LLM to be even a little helpful with that. Maybe that's possible and easy today, but I don't know how to do that so I don't. I'm open to suggestions. - When I talk to a co-worker, even a junior engineer, I feel like I can have sensible discussions with them even if they may not know exactly what I'm talking about. I can trust that they'll say things like "I don't really know all the details, but it sounds like...", or they'll ask follow up questions. I do not have to tell my co-workers to ask follow up questions or to tell me about the bounds of their understanding. We're human, we talk. When I try to chat with an LLM about business problems, I feel more like I'm trying to chat with a 15 year old who is desperately trying to sound knowledgeable, where they will try their best to confidently answer my questions every time. They won't ever specify "I don't know about that, but tell me if this sounds right...". Instead, they'll confidently lie to my face with insanely fabricated details. I don't want to have to navigate a conversation with the background question of "is this the fabrications of a madman?" in the back of my head all the time, so I don't. Maybe that's the wrong value calculus, and if so, I'm open to being persuaded. Please, share your thoughts.

d_burfoot 2y ago

Warren Buffett says "Be fearful when others are greedy, and greedy when others are fearful." But in the AI space, people are both greedy AND fearful!

cadamsdotcom 2y ago
That’s a sign of not enough granularity. Some folks use it for code and love it, some use it for code and hate it. Writing code is so broad though- are we coding device drivers? Prototyping greenfield CRUD apps? Fixing hairy bugs in 50year old codebases? All of those are very different activities where folks have different needs, but discussions rarely include this context. If you can find the right dividing line where on one side people are greedy/love the tech, and on the other they’re fearful/doubting… you’re ahead of the pack - and a bit more ready to invest.

kibwen 2y ago

> How much of a productivity increase is that? Well jeepers, if you’re only doing 1/5th the work, then you are… punches buttons on calculator watch… five times as productive. Cool, so I'm going to make five times more money? What's that you say, I'm not? So who is going to be harvesting the profits, then?

theultdev 2y ago
If you're 5x more productive from AI, you can do 5x less effort if you feel your previous productivity was sufficient for the current pay.
fragmede 2y ago
It's a pretty good time to be a contractor, as long as you negotiate getting paid per project/milestone and not by the hour. What used to take three days now takes one, and you can use the extra time to book more clients.
efs24 2y ago
You will get paid 5x as much… assuming you are the only developer that uses this and the market for engineers is rational. However those other developers are going to be 5x as productive too so your rate should stay the same. The company that pays you will get more code for less money and so they would make more profit if they are the only company taking advantage of this. But all companies will start to produce software at a lower cost so their profits per line of code will be squeezed to where they are now. You will spend a lot less time writing boiler plate and searching docs or stack overflow. That frees you up to work on real problems and since you’ll be able to create products with less cost per line of code you could potentially compete with your own offerings. That’s my take anyway.

itissid 2y ago

The only thing missing right now is a solid goal planning, goal validation engine and search to define rewards. Since code is one of those things you can train LLMs with using RL – not RHLF – it can actually do goal based AI planning, all the people who say "I don't trust LLM code" are missing how beside the point this is. A lot of code in this world is accepted, even if it is suboptimal and people don't realize how much money is to be made by writing it so with ML models.

sciurus 2y ago

This would benefit from a "[2023]" appended to the title to make clear it isn't new. Past discussion: https://news.ycombinator.com/item?id=35273406 https://news.ycombinator.com/item?id=35273406

jldugger 2y ago

(2023)

huqedato 2y ago

This article is just a nicely crafted marketing ad for Cody. It impressed more than the product itself (which is not so unique, there are many althernatives on the market, see Codeium for example)

ColonelPhantom 2y ago

This article really grates me the wrong way. A company selling an AI product saying the following about AI sceptics: > All you crazy MFs are completely overlooking the fact that software engineering exists as a discipline because you cannot EVER under any circumstances TRUST CODE. is straight up insulting to me, because it effectively comes down to "use my product or you're a looney". Also, while two years after this post (which should be labeled 2023) I've still barely tried to entirely offload coding to an LLM, the few times I did try have been pretty crap. I also really, really don't want to 'chat' with my codebase or editor. 'Chatting' to me feels about as slow as writing something myself, while I also don't get a good mental model 'for free'. I am a moderately happy user of AI autocomplete (specifically Supermaven), but I only ever accept suggestions that are trivially correct to me. If it's not trivial, it might be useful as a guide of where to look in actual documentation if relevant, but just accepting it will lead me down a wrong path more often than not.

simonw 2y ago

This piece is coming up on two years old now - it was written in March 2023. I think it holds up. Saying "LLMs are useful for code" is a lot less controversial today than it was back in March 2023 - there are still some hold-outs who will swear that all LLM-produced code is untrustworthy junk and no good engineer would dream of using them, but I think their numbers are continuing to drop.

spondyl 2y ago

As a bit of anecdata, I used an LLM to refactor a core part of one of my site projects about 6 months or so ago. I don't really remember how it works, probably because I didn't have to think hard about the design, I just sort of iterated it in conversation which probably meant it didn't stick. While the iteration part was mostly positive, I don't know that it was worth it in the end to be left with something that I can understand but feels a bit foreign at the same time.

cadamsdotcom 2y ago

(2023)

pilgrim0 2y ago

Starts the article giving examples of novel and unexpected ways people employed available paradigms to create value, then proceeds to sell LLMs which are completely biased by the most standard ways of doing things…

gatienboquet 2y ago

Gemini-Exp-1206 got 2M context token, what's preventing Google from fine-tuning it with internal google PR etc ? So in the end you'll get a fine-tuned google llm dev and one finetuned for review, etc etc

lukev 2y ago

One thing I'll note is that I'm inclined to trust the poll of engineers who have actually been using this stuff for real work, as opposed to a N=1 example of it performing well (outside the context of a real software project) and using that as the basis for declaring that This Is The Future. I'm not saying there's not a money volcano to be had somewhere, built on top of LLMs. But his call to action is stupid. Every other "money volcano" he cites was a clever use of existing technology to meet a previously-unseen but very real need. LLMs are the mirror image of that. They are something that's clearly, obviously useful and yet people are struggling to leverage them to the degree that _ought_ to be possible. There are no truly slam dunk applications yet, let alone money volcanos (except for the AI hype train itself.) They are the ultimate hammer in search of a nail. And it's a pretty amazing hammer, I'm not a complete LLM skeptic. But at this point we should all be searching for the best nails, not just cheerleading for the hammer.

cheevly 2y ago

And the even more jaw-dropping reality is that even this article heavily underestimates the transformative power that generative AI is about to bring to software. Using AI to generate code like this is akin to using a horse to pull an automobile.

from-nibly 2y ago

If programming as theory building is any bit true then writing code with an LLM doesn't co UK BT as writing code by hand. At least not wuite the same way. This is the problem. Managers think that the code is the product. It isn't. And LLMs can only produce the code.

seeg 2y ago

1. If you can easily write boring code with LLM, you might not have the incentive to refactor it. 2. If you let someone else solve your problems, and you don't reflect in the solution, you won't learn. 3. I rarely read the fine print, but are all LLM IDEs really the same when it comes to intellectual property of the code? Or the responsibily of bad code?