8 ms·

Sorry, GenAI is NOT going to 10x computer programming

67 points by kmdupree 2y ago

codingwagie 2y ago
I'll never understand people that cannot project current progress into what we may see with further progress.
- dom96 2y ago
  I'll never understand people that cannot imagine progress hitting a wall. Where are our flying cars?
- KaiserPro 2y ago
  I've never understood why people are projecting infinite and uniform growth of x. The path isnt linear. I'm lucky to have some of the best AI integrations around, most of the time they suggest boneheadded bollocks that blocks out the proper actual autocomplete. It is good at suggesting code hints outside of the IDE.
- grumpopotamus 2y ago
  I was just thinking about how common this is, even for experts in a field. The line of thinking goes something like: 1. Observe the current state of the art. 2. Make a claim implying that we will NEVER advance beyond the current state of the art (often despite accelerating progress in the field).
- lawn 2y ago
  Didn't xkcd teach us the dangers of extrapolation years ago? https://xkcd.com/605/ https://xkcd.com/605/
rogerclark 2y ago
It already has, for better or worse. Why does anyone still take this guy seriously? It's one thing to be skeptical that AI is going to make the world a better place... it's another thing to be skeptical that it exists and actually does things.
ayhanfuat 2y ago
It must be really hard to be the person who said "Deep learning is hitting a wall" in 2022. This is just doubling down.
resters 2y ago
I think it's already surpassed 10x, and is closer to 25x or 30x. You just have to know what to ask it to do.
- vundercind 2y ago
  I should probably get around to actually using one of these things.
  - gadflyinyoureye 2y ago
    I’d love to see people going through a real world walk through using such things. Some people who do cookie cutter common tasks find that an LLM can produce their whole code base. But ask it to make a change and what happens?
    - abdullin 2y ago
      This is an example of my prompt to o1 a few days ago. First request produced refactoring suggestions. Second request (yes, I like results, implement them) produced multiple files that have just worked. —- Take a look at this code from my multi-mode (a la vim or old terminal apps) block-based content editor. I want to build on the keyboard interface and introduce a simple way to have simple commands with small popup. E.g. after doing "A" in "view" mode, show user a popup that expects H,T,I, or V. Or, after pressing "P" in view mode - show a small popup that has an text input waiting for the permission role for the page. Don't implement the changes, just think through how to extend existing code to make logic like that simple. Remember, I like simple code, I don't like spaghetti code and many small classes/files.
    - falcolas 2y ago
      IME: I will either get a repeat of something already written (this happens all the time with java getter/setter boilerplate if the class isn't the one I'm currently working in), or I get some junior level attempt at what I ask it to do. The second case is the best case, and it still requires a code review with a fine toothed comb. For example, it loves to suggest plausible looking but non-existant methods on popular libraries. It's one of those things where a language aware autocomplete will often save me more time, because it doesn't try and second guess me.
richard___ 2y ago
This guy doesnt code and is breaking his back looking for data to support his biased hypothesis
- aerhardt 2y ago
  I code, I use GenAI every day, all day and it is not even close to 10x my productivity. Maybe 1,3x?
  - tobyhinloopen 2y ago
    For me 1,2 of that 1,3 is because it’s the fastest way to get somewhat working autocomplete and inline documentation hah
carterparks 2y ago
It doesn't always 10x my development but on certain problems I can work 30x faster. In due time, this is only going to accelerate more as tooling becomes more closely integrated into dev workflows.
ccvannorman 2y ago
Just because this article isn't well formed or sourced doesnt make its claim incorrect. I program daily and I use AI for it daily. For short and simple programs AI can do it 100x faster, but it is fundamentally limited by its context size. As a program grows in complexity AI is not currently able to create or modify the codebase with success (around 2000 lines is where I found it has a barrier). I suspect it's due to exponential complexity associated with input size. Show me an AI that can do this for a 10,000 lines complex program and I'll eat my own shorts
- leblancfg 2y ago
  Here's a thought experiment. Think back on how that statement would have sounded like to past-you, 3 years ago. You would probably have dismissed it as bullshit, right? We've gone a long way since then. Both in terms of better, faster and cheaper models, but also how they're being intertwined with developer tooling. Now imagine 3 years from now.
  - falcolas 2y ago
    As you inadvertently pointed out, AI improvements are not linear. They depend on new discoveries more than they do iteration. We could be in either out of jobs or lamenting the stagnation of AI (again).
    - leblancfg 2y ago
      After an innovation phase there is an implementation phase. Depending on the usefulness of the innovation, the integration with existing systems takes time. It is calculated in years, tens of years. Think back on the 80-90s, where it took years to integrate PCs into offices and workspaces. From your comment, is sounds like you think that the implementation phase of LLMs is already over? And if so, how do you come to this conclusion?
      - falcolas 2y ago
        It's not as if we have no idea how to make use of AI in programming. We've been working on AI in one form or another since the 70's, and integrated them with our programming workflow almost as long (more recently in the form of autocomplete using natural language processing and machine learning models). It's already completely integrated into our IDEs, often with options to collate the output of multiple LLMs. What further implementations of integrating AI and programming workflows have LLMs shown to be missing?
koliber 2y ago
I am an experienced programmer, and just recently started using ChatGPT and Cursor to help me code. Some things it does like magic, and it's hard to say what n-fold improvement there is. I'll put the lower limit on 3x and an upper, on certain tasks, at 20x. The project I am currently working on is took me about 16 hours to get to an MVP. A hobby project of similar size I did a few years ago took me about 80 hours. A lot of the work is NOT coding work that an LLM can help me with. 10x over everything is overstating it. However, if I can take my star developer who is already delivering significantly more value per dollar than my average guys and 3x him, that's quite a boost.
- fwsgonzo 2y ago
  I think I agree. It's maybe 2x for me after the project has been solidly established. However, it's very useful to have the AI during the beginning, when you're writing a lot of boilerplate or integrating with APIs that you don't know super well. The AI can surprise you and connect some dots.
- handzhiev 2y ago
  And it's not just the speed. The mental energy I save from having AI do some work vs. me doing it is allowing me to feel better, have mental energy for different things, be more productive, or just have some rest.
- janalsncm 2y ago
  Internet of Bugs did a decent side-by-side of four of them. https://youtu.be/-ONQvxqLXqE https://youtu.be/-ONQvxqLXqE They can get you 95% of the way there, the question is whether fixing the last 5% will take more time than doing all of it yourself. I presume this depends on your familiarity with the domain.
- microtonal 2y ago
  The lower bound is certainly lower than 3x. I do very domain-specific work (in Rust/Python) and usually the output is useless and ignoring the garbage suggestions can have a higher mental load than not using it. It does work well for boilerplate though and since boilerplate is the most annoying code to write, I'll tolerate it. I agree that the upper bound is pretty high, with more straightforward MVPs, it can generate large swabs of code for you from a description. Also for little DSLs that I haven't memorized, generating a pyproject.toml from a bunch of constraints is so much nicer/quicker than reading the docs again every few months.
sva_ 2y ago
This guy really tries too hard to make himself relevant.
- baanist 2y ago
  I think he does a pretty good job of it. I know who he is and I read what he has to say on all the latest hype trends in AI.
manofmanysmiles 2y ago
I think this guy has not worked with enough people who aren't that good at programming. GenAI lets me spit out reams of what to me is boilerplate, but to someone more junior might legitimately take weeks. It's a tool that right now empowers people who already are extremely effective. It is not a substitute for deep knowledge, wisdom and experience.
- Ifkaluva 2y ago
  Frankly, he hasn't worked with any programmers at all. Gary Marcus is a retired psychology professor.
ratedgene 2y ago
I believe it's changed everything forever. I'm not even on the hype train. It's difficult for humans to estimate that impact, so it's easier to just deny. It's a fundamental shift on so many levels that we haven't even begun to scratch the surface and import not just to engineering but all other facets of life. The future is accelerating.
- deleted 2y ago
  [deleted]
Fripplebubby 2y ago
Do people generally really believe that GenAI would 10x programmer productivity? I find that surprising, that's not what I've read here on HN, by-and-large. 20-30% seems much more like what people are actually experiencing today. Is that controversial?
- mewpmewp2 2y ago
  It's very dependent on circumstances and what kind of environment someone is in. At certain very specific tasks it's going to have that 10x+ and more. Also we've made 1000x+ gains from the times we had punch cards etc. It will depend on the kind of projects you are working on, and the type of environment you are in. Is it a very large company or is it a side project. In right cases it's over 10x+, in worse cases it's 10%+ maybe. It may speak to however a type of movement that might happen is that there will be newer companies that will make sure they unleash a culture where AI first engineering is promoted, which might make it scale further than usual enterprise work currently can do. If a company can start small and lean they will be in a position to have a huge multiplier from AI. It's harder to get to that setup from a pre-existing large corporation stand point.
hluska 2y ago
I can only speak about what it’s done to productivity. A Gen AI is a very ambitious intern who will do anything I ask. I can keep making it refine things, or take what it has and finish it myself. There are no hurt feelings and I don’t have to worry about keeping my Gen AI motivated. It certainly doesn’t write production quality code, but that entire first paragraph was science fiction a short time ago.
yawnxyz 2y ago
Keep telling other devs to stop using AI for code!!! I don't want to lose my edge lol
eitally 2y ago
I think too many SWEs making comments like Gary has are discounting just how many software engineers are employed outside of tech, mostly working on either 1) workflow, 2) CRUD, or 3) reporting/analytics tools inside enterprise IT organizations. Between SaaS & GenAI, a very large percentage of those roles are starting to dry up.
throw4847285 2y ago
The reason GenAI has been so helpful for developers is that devs spend most of their time doing grunt work. That is, basic pattern matching: copy pasting chunks of code from one place to another and then modifying those chunks enough such that a couple code reviews will catch the things they forgot to modify. This is a miserable existence, and GenAI can alleviate it, but only because it was grunt work in the first place.
- k__ 2y ago
  Yes. I don't have the impression GenAI can replace me, and I'm not even a 10x Dev. Yet, it makes much of the daily task more bearable.
- marssaxman 2y ago
  I am so sorry that this is the experience you have. It sounds like such a drag! Of course all grunt work should be automated; that is rather the point of the industry. I suppose a reason I feel little interest in applying AI to coding may be that my experience is nothing like you describe.
- kolme 2y ago
  > copy pasting chunks of code from one place to another and then modifying those chunks That's not how you program. I've never seen anyone programing in this way unless they were just starting. Edit: if you find yourself copying chunks and modifying them, you probably need to create an abstraction. Don't mean to be rude, just honestly that is my personal experience.
hiddencost 2y ago
Why do people keep posting this stuff? He's been writing variations on the same essay for decades and he's always wrong.
- baanist 2y ago
  What is he wrong about?
  - krallistic 2y ago
    "Deep learning is hitting a wall now with just scaling" "Deep learning is only good for perception" (with language one of the areas where its not good)
    - baanist 2y ago
      I don't have any subscriptions to the latest models but what improvements have you noticed in scaling and language understanding? Last I checked people were still discussing "9.9 > 9.1" and "How many 'r's are in 'strawberry'".
  - hyperG 2y ago
    I have never read anything by him that seemed correct. He really does seem like quite a good contrarian indicator if anything.
    - baanist 2y ago
      He was right that scaling would not achieve abstract reasoning and he's been right so far on basically every new hyped development. The closed research labs like OpenAI are hoping to reduce all the gaps in their models by constantly patching out of distribution data sets but this clearly can not achieve any sort of general intelligence unless they somehow manage to obsolete themselves and the entire company by automating the out of distribution patching itself which they now perform by burning lots of cash and energy. There are people who need to write a lot of emails so for those people I'm sure OpenAI will continue to deliver some kind of value but everyone else will still have to continue thinking for themselves regardless of what Sam Altman keeps promising.
zarzavat 2y ago
There seem to be many people who are at pains to tell us what generative AI won't do in the future. Any such prediction seems extremely likely to be false without a time bound, or an argument to physical impossibility.
tobyhinloopen 2y ago
The real reason ChatGPT saves time is because it is faster to provide inaccurate answers than Google.
throwaway918299 2y ago
My average experience reviewing code written by “average developers” that start using copilot is that it ends up 0.1x them. We are going to be so screwed in 10 years.
dgoodell 2y ago
So far it has definitely saved me a lot of time googling api docs and stackoverflow for me. Anything that requires applying things in novel ways that doesn’t have lots of examples out there already seems to be completely beyond it. Also, it often comes up with very unoptimal and inefficient solutions even though it seems to be completely aware of better solutions when prodded. So basically it’s a fully competent programmer lol.
dmitrygr 2y ago
For low level work (kernels, drivers, C, assembly), things like ChatGPT and Cursor are a 0.1X multiplier. They suggest idiotic things both design-wise and implementation-wise. In this world, I’ll take my cat’s help over theirs. Maybe it’ll improve, but I won’t hold my breath.
- mahmoudimus 2y ago
  That's sort of ironic. One of the biggest value adds I've seen to my hobby work is that LLMs are REALLY good at understanding disassembly and can even do things like symbolically execute a few lines for me. I've used it to deobfuscate several malware protections and I can imagine it will get better. However, I have never tried it on kernels, drivers, etc. Just the inverse of that :)
- baanist 2y ago
  These algorithms are not capable of symbolic reasoning and abstract interpretation. The most obvious demonstration of this is that no algorithm on the market currently can solve sudoku puzzles even though they have spent billions of dollars "training" them on logic and reasoning.
rkunal 2y ago
I will believe when GenAI becomes 10x, we would be noticing a larger number of emacs packages. Or at-least few attempts to rewrite old famous softwares like Calibre. I am sure that it is being used at a large scale in web apps and cloud tech. Not aware of the same in other domains. May be in the future, everything will become a web app because genAI became so good at it.
munchausen42 2y ago
Funny to see how being anti-GenAI and anti-LLM is now the new en vogue on HN. Can't wait till that dies off as well.
- kylehotchkiss 2y ago
  People have been through a lot of hype cycles and the grandiose claims of what AI/LLMs could be / "superintellegence" etc are a complete distraction from a pretty impressive tech accomplishment. Too few people wonder if the wins we got are worth it even if we hit a wall trying to achieve "AGI". And the real world impacts with electricity and billionaires running around with no guardrails are starting to slowly come to light. It sure looks a lot like the last hype cycle where we were promised a global currency without government intervention... lol. Do you remember the part where people took out loans to buy cartoon apes? (I personally/philosophically don't believe AGI is a natural next step for LLMs as I don't believe the English language alone which training is so heavy on encapsulates all of human ability, rather it's very honed in on English speaking countries/cultures - I also don't believe humans are very capable of creating derivative products with capabilities greater than their own - we can barely make progress on what really causes mental illness[1] - how can we claim to understand our minds so well we can replicate their functionality?) [1]: https://www.science.org/content/blog-post/new-mode-schizophrenia-treatment https://www.science.org/content/blog-post/new-mode-schizophr...
hackermatic 2y ago
I hope commenters will dig into the author's citations' data, in line with HN's discussion guidelines, instead of just expressing a negative opinion about the thrust of the article. The quantifiable impact of genAI on code productivity is an important research question, and very much an open question subject to bias from all the players in the space -- especially once you factor in quality, maintainability, and bugs or revisions over time. The GitClear whitepaper that Marcus cites tries to account for some of these factors, but they're biased by selling their own code quality tools. Likewise, GitHub's whitepapers (and subsequent marketing) tend to study the perception of productivity and quality by developers, and other fuzzy factors like the suggestion acceptance rate -- but not bug rate or the durability of accepted suggestions over time. (I think perceived productivity and enjoyment of one's job are also important values, but they're not what these products are being sold on.)
janalsncm 2y ago
If you want to know how valuable GenAI is for software engineering, you don’t ask AI hype people on Twitter. You also don’t ask Gary Marcus. You ask people trying to complete software engineering tasks. I guess part of the reason we can’t agree on whether they are useful is that their usefulness depends on how familiar you are with the programming task.
rekttrader 2y ago
As a CTO of a new startup, I am not hiring contractors or employees to build software. This may not be 10x, but when the time comes and I need scale I’m likely to hire what will be my smallest team yet. One of the requirements for hiring is that they be proficient with copilot, cursor, or have rolled their own llama based code assistant. It’s like going from using manual hand tools to power drills and nail guns. Those that doubt That AI will change all of technology jobs and work in the industry are gonna find themselves doing something else.
- yodon 2y ago
  > One of the requirements for hiring is that they be proficient with copilot, cursor, or HAVE ROLLED THEIR OWN LLAMA BASED CODE ASSISTANT. [enphasis added] Isn't that a bit like "I won't hire a dev unless they've written a C++ compiler"? Very dev macho, but not necessarily the best criteria for someone working on a distributed financial transactional system.
Jimmc414 2y ago
"Gary Marcus has been coding since he was 8 years old, and was very proud when his undergrad professor said he coded an order of magnitude faster than his peers. Just as Chollet might have predicted, most of the advantage came from clarity — about task, algorithms and data structures." I'm trying to understand what sort of person would end with a 3rd person anecdote like this.
addisonj 2y ago
I want to make clear that I agree with the author that the current way in which AI is being used to make a 10x improvement in efficiency is NOT going to work, and it is for exactly reason stated: > 10x-ing requires deep conceptual understanding – exactly what GenAI lacks. Writing lines of code without that understanding can only help so much. IMHO, the real opportunity with AI, is to get developers less critical in being the only ones who translate conceptual understanding of the business problem into code and get developers more focused on the domain they should be experts in which, imho, is the large scale structure of the system and how that maps to actual computes. What I mean by that... developers, are, on average, not good at having deep conceptual understanding of the business domain... and maybe have that understanding in the computational domain. All of the 10x developers I have ever met are mostly that because they do know both the business domain (or can learn it really quick) and the computational domain so they can quickly deliver value because they understand the problem end-to-end. My internal model for thinking about this is we need developers thinking about building the right "intermediate representation" of a problem space and being able to manage the system at large, but we need domain experts, likely supported by AI assistants, in helping to use that IR to express the actual business problem. That is a super loose explanation... but if you have been in the industry awhile, you probably have felt the insanity of how long it can take to ship a fairly small feature because of a huge disconnect between the broader structure of the system (with microservices, async job systems, data pipelines, etc, which we can think of as the low-level representation of the sytem at large) and the high-level requirement of needing to store some data and process it according to some business specific rules. I have no idea if we actually can get there as an industry... it is more likely we just have more terrible code... but maybe not
norir 2y ago
The bummer of ai hype is that I think there is so much we can do to improve programming with deterministic tooling. We have seen a remarkable improvement in terms of real time feedback directly in your editing context through IDEs and editor plugins. We still don't yet have the next programming language for the ai age. I believe the biggest gains will not come from bigger models but smarter tools that can use stochastic tools like LLMs to explore the solution space and quickly verify the solution according to well defined constraints that can be expressed in the language. When I hear about some of the massive productivity gains people ascribe to ai, I also wonder where the amazement for "rails new" has gone. We already have tools that can 100x your productivity in specific areas without also hallucinating random nonsense (though Rails is maybe not the absolute best counterexample here).
xnx 2y ago
There's a big difference between "AI isn't 10x-ing computer programming" and "AI will not 10x computer programming". Am I missing the part of the article where some evidence is provided that the situation won't improve?
senko 2y ago
Gary takes credible facts there (modest gains, ai is not replacement for clear thinking), altough attacking the "10x" claim is a cheap shot because the claim was always nebulous. However, in light of Gary's other AI related writings[0], it is clear he's not saying "people, be clear headed about the promises and limits of this new technology", he's saying "genai is shit". Yes, I am attacking the character/motive instead of the argument, because there's plenty of proof about the motive. So I am kind of agreeing (facts as laid out) and vehemently disagreeing (the underlying premise his readers will get from this) with the guy at the same time. [0]: Here are a few headlines I grabbed from his blog: - How much should OpenAI’s abandoned promises be worth? - Five reasons why OpenAI’s $150B financing might fall apart - OpenAI’s slow-motion train wreck - Why California’s AI safety bill should (still) be signed into law - and why that won’t be nearly enough - Why the collapse of the Generative AI bubble may be imminent
juliendorra 2y ago
For comparison VisiCalc and other early spreadsheets like Lotus 123 and Multiplan were 80x multipliers for their users: “I would work for twenty hours,” Jackson said. “With a spreadsheet, it takes me 15 minutes.” Not sure any other computing tool ever beat that since? Source: Steven Levy reporting, Harper’s Bazaar 1984, reprinted in Wired: https://www.wired.com/2014/10/a-spreadsheet-way-of-knowledge/ https://www.wired.com/2014/10/a-spreadsheet-way-of-knowledge...
geenkeuse 2y ago
Looks like you guys are coming around.For the longest time it was the same "AI will never ever do what we do" song. Now you are all mostly singing it's praises. Good. You will be left standing when the stubborn are forced to call it quits. What is so wrong with having a computer do amazing things for you simply by asking it? That was the dream and the mission, wasn't it? Unless "obfuscation is the enemy" Most people don't want the IT guy to get into the details of how they did what they did. They just want to get on with it. Goes for the guy replacing your hard drive. Goes for the guy writing super complicated programs. IT is a commodity. A grudge purchase for most companies. But you know that, deep down. Don't you? The bar has been lowered and raised at the same time. Amazing.
m463 2y ago
My first job at college had schedules with high-level and detailed design, then coding, then integration and test. Coding was only 3-5% of the time. Also, I remember learning to race. I tried to learn to go faster through the turns. But it turns out the way to be fast was to go a little faster on the straightaway and long turns. Trying to go faster through the tight turns doesn't help much. If you go 1 mph faster for 10 feet in the slowest turn, it doesn't make as much difference as going .1 mph faster in a long 500 foot turn. so... Are we trying to haphazardly optimize for the small amount of time we code, when we should spend it elsewhere?
throwawa14223 2y ago
I'd be shocked if it wasn't a negative at the end of the day.
bitdeep 2y ago
Tools like aider or cursor composer does not help for complex code as they destroy your mental code model of the solution you working on. This tools help a bit for initial mocks, but even that, I don't like as they create code that I don't know and I need to review it all to know where things is. When you are doing complex software, you need to build a good mental code model to know where things is, especialy when you need to start to debug issues, not knowing where things is is a mess and very annoying. This days, I almost don't use this tools anymore, I just prefer basic line auto-completion.
nateless 2y ago
Completely agree. Tried spinning up a side project with just Cursor and GPT/Claude models, and we're definitely not at 'AI can build apps from just a prompt' yet. It couldn’t even create a dropdown filter like Linear’s, and the code was buggy, inefficient, and far from the full functionality I asked for. Maybe it's the niche language (Elixir), but we’re not there yet.

67 points by kmdupree 2y ago

codingwagie 2y ago

I'll never understand people that cannot project current progress into what we may see with further progress.

dom96 2y ago
I'll never understand people that cannot imagine progress hitting a wall. Where are our flying cars?
KaiserPro 2y ago
I've never understood why people are projecting infinite and uniform growth of x. The path isnt linear. I'm lucky to have some of the best AI integrations around, most of the time they suggest boneheadded bollocks that blocks out the proper actual autocomplete. It is good at suggesting code hints outside of the IDE.
grumpopotamus 2y ago
I was just thinking about how common this is, even for experts in a field. The line of thinking goes something like: 1. Observe the current state of the art. 2. Make a claim implying that we will NEVER advance beyond the current state of the art (often despite accelerating progress in the field).
lawn 2y ago
Didn't xkcd teach us the dangers of extrapolation years ago? https://xkcd.com/605/ https://xkcd.com/605/

rogerclark 2y ago

It already has, for better or worse. Why does anyone still take this guy seriously? It's one thing to be skeptical that AI is going to make the world a better place... it's another thing to be skeptical that it exists and actually does things.

ayhanfuat 2y ago

It must be really hard to be the person who said "Deep learning is hitting a wall" in 2022. This is just doubling down.

resters 2y ago

I think it's already surpassed 10x, and is closer to 25x or 30x. You just have to know what to ask it to do.

vundercind 2y ago
I should probably get around to actually using one of these things.
- gadflyinyoureye 2y ago
  I’d love to see people going through a real world walk through using such things. Some people who do cookie cutter common tasks find that an LLM can produce their whole code base. But ask it to make a change and what happens?
  - abdullin 2y ago
    This is an example of my prompt to o1 a few days ago. First request produced refactoring suggestions. Second request (yes, I like results, implement them) produced multiple files that have just worked. —- Take a look at this code from my multi-mode (a la vim or old terminal apps) block-based content editor. I want to build on the keyboard interface and introduce a simple way to have simple commands with small popup. E.g. after doing "A" in "view" mode, show user a popup that expects H,T,I, or V. Or, after pressing "P" in view mode - show a small popup that has an text input waiting for the permission role for the page. Don't implement the changes, just think through how to extend existing code to make logic like that simple. Remember, I like simple code, I don't like spaghetti code and many small classes/files.
  - falcolas 2y ago
    IME: I will either get a repeat of something already written (this happens all the time with java getter/setter boilerplate if the class isn't the one I'm currently working in), or I get some junior level attempt at what I ask it to do. The second case is the best case, and it still requires a code review with a fine toothed comb. For example, it loves to suggest plausible looking but non-existant methods on popular libraries. It's one of those things where a language aware autocomplete will often save me more time, because it doesn't try and second guess me.

richard___ 2y ago

This guy doesnt code and is breaking his back looking for data to support his biased hypothesis

aerhardt 2y ago
I code, I use GenAI every day, all day and it is not even close to 10x my productivity. Maybe 1,3x?
- tobyhinloopen 2y ago
  For me 1,2 of that 1,3 is because it’s the fastest way to get somewhat working autocomplete and inline documentation hah

carterparks 2y ago

It doesn't always 10x my development but on certain problems I can work 30x faster. In due time, this is only going to accelerate more as tooling becomes more closely integrated into dev workflows.

ccvannorman 2y ago

Just because this article isn't well formed or sourced doesnt make its claim incorrect. I program daily and I use AI for it daily. For short and simple programs AI can do it 100x faster, but it is fundamentally limited by its context size. As a program grows in complexity AI is not currently able to create or modify the codebase with success (around 2000 lines is where I found it has a barrier). I suspect it's due to exponential complexity associated with input size. Show me an AI that can do this for a 10,000 lines complex program and I'll eat my own shorts

leblancfg 2y ago
Here's a thought experiment. Think back on how that statement would have sounded like to past-you, 3 years ago. You would probably have dismissed it as bullshit, right? We've gone a long way since then. Both in terms of better, faster and cheaper models, but also how they're being intertwined with developer tooling. Now imagine 3 years from now.
- falcolas 2y ago
  As you inadvertently pointed out, AI improvements are not linear. They depend on new discoveries more than they do iteration. We could be in either out of jobs or lamenting the stagnation of AI (again).
  - leblancfg 2y ago
    After an innovation phase there is an implementation phase. Depending on the usefulness of the innovation, the integration with existing systems takes time. It is calculated in years, tens of years. Think back on the 80-90s, where it took years to integrate PCs into offices and workspaces. From your comment, is sounds like you think that the implementation phase of LLMs is already over? And if so, how do you come to this conclusion?
    - falcolas 2y ago
      It's not as if we have no idea how to make use of AI in programming. We've been working on AI in one form or another since the 70's, and integrated them with our programming workflow almost as long (more recently in the form of autocomplete using natural language processing and machine learning models). It's already completely integrated into our IDEs, often with options to collate the output of multiple LLMs. What further implementations of integrating AI and programming workflows have LLMs shown to be missing?

koliber 2y ago

I am an experienced programmer, and just recently started using ChatGPT and Cursor to help me code. Some things it does like magic, and it's hard to say what n-fold improvement there is. I'll put the lower limit on 3x and an upper, on certain tasks, at 20x. The project I am currently working on is took me about 16 hours to get to an MVP. A hobby project of similar size I did a few years ago took me about 80 hours. A lot of the work is NOT coding work that an LLM can help me with. 10x over everything is overstating it. However, if I can take my star developer who is already delivering significantly more value per dollar than my average guys and 3x him, that's quite a boost.

fwsgonzo 2y ago
I think I agree. It's maybe 2x for me after the project has been solidly established. However, it's very useful to have the AI during the beginning, when you're writing a lot of boilerplate or integrating with APIs that you don't know super well. The AI can surprise you and connect some dots.
handzhiev 2y ago
And it's not just the speed. The mental energy I save from having AI do some work vs. me doing it is allowing me to feel better, have mental energy for different things, be more productive, or just have some rest.
janalsncm 2y ago
Internet of Bugs did a decent side-by-side of four of them. https://youtu.be/-ONQvxqLXqE https://youtu.be/-ONQvxqLXqE They can get you 95% of the way there, the question is whether fixing the last 5% will take more time than doing all of it yourself. I presume this depends on your familiarity with the domain.
microtonal 2y ago
The lower bound is certainly lower than 3x. I do very domain-specific work (in Rust/Python) and usually the output is useless and ignoring the garbage suggestions can have a higher mental load than not using it. It does work well for boilerplate though and since boilerplate is the most annoying code to write, I'll tolerate it. I agree that the upper bound is pretty high, with more straightforward MVPs, it can generate large swabs of code for you from a description. Also for little DSLs that I haven't memorized, generating a pyproject.toml from a bunch of constraints is so much nicer/quicker than reading the docs again every few months.

sva_ 2y ago

This guy really tries too hard to make himself relevant.

baanist 2y ago
I think he does a pretty good job of it. I know who he is and I read what he has to say on all the latest hype trends in AI.

manofmanysmiles 2y ago

I think this guy has not worked with enough people who aren't that good at programming. GenAI lets me spit out reams of what to me is boilerplate, but to someone more junior might legitimately take weeks. It's a tool that right now empowers people who already are extremely effective. It is not a substitute for deep knowledge, wisdom and experience.

Ifkaluva 2y ago
Frankly, he hasn't worked with any programmers at all. Gary Marcus is a retired psychology professor.

ratedgene 2y ago

I believe it's changed everything forever. I'm not even on the hype train. It's difficult for humans to estimate that impact, so it's easier to just deny. It's a fundamental shift on so many levels that we haven't even begun to scratch the surface and import not just to engineering but all other facets of life. The future is accelerating.

deleted 2y ago
[deleted]

Fripplebubby 2y ago

Do people generally really believe that GenAI would 10x programmer productivity? I find that surprising, that's not what I've read here on HN, by-and-large. 20-30% seems much more like what people are actually experiencing today. Is that controversial?

mewpmewp2 2y ago
It's very dependent on circumstances and what kind of environment someone is in. At certain very specific tasks it's going to have that 10x+ and more. Also we've made 1000x+ gains from the times we had punch cards etc. It will depend on the kind of projects you are working on, and the type of environment you are in. Is it a very large company or is it a side project. In right cases it's over 10x+, in worse cases it's 10%+ maybe. It may speak to however a type of movement that might happen is that there will be newer companies that will make sure they unleash a culture where AI first engineering is promoted, which might make it scale further than usual enterprise work currently can do. If a company can start small and lean they will be in a position to have a huge multiplier from AI. It's harder to get to that setup from a pre-existing large corporation stand point.

hluska 2y ago

I can only speak about what it’s done to productivity. A Gen AI is a very ambitious intern who will do anything I ask. I can keep making it refine things, or take what it has and finish it myself. There are no hurt feelings and I don’t have to worry about keeping my Gen AI motivated. It certainly doesn’t write production quality code, but that entire first paragraph was science fiction a short time ago.

yawnxyz 2y ago

Keep telling other devs to stop using AI for code!!! I don't want to lose my edge lol

eitally 2y ago

I think too many SWEs making comments like Gary has are discounting just how many software engineers are employed outside of tech, mostly working on either 1) workflow, 2) CRUD, or 3) reporting/analytics tools inside enterprise IT organizations. Between SaaS & GenAI, a very large percentage of those roles are starting to dry up.

throw4847285 2y ago

The reason GenAI has been so helpful for developers is that devs spend most of their time doing grunt work. That is, basic pattern matching: copy pasting chunks of code from one place to another and then modifying those chunks enough such that a couple code reviews will catch the things they forgot to modify. This is a miserable existence, and GenAI can alleviate it, but only because it was grunt work in the first place.

k__ 2y ago
Yes. I don't have the impression GenAI can replace me, and I'm not even a 10x Dev. Yet, it makes much of the daily task more bearable.
marssaxman 2y ago
I am so sorry that this is the experience you have. It sounds like such a drag! Of course all grunt work should be automated; that is rather the point of the industry. I suppose a reason I feel little interest in applying AI to coding may be that my experience is nothing like you describe.
kolme 2y ago
> copy pasting chunks of code from one place to another and then modifying those chunks That's not how you program. I've never seen anyone programing in this way unless they were just starting. Edit: if you find yourself copying chunks and modifying them, you probably need to create an abstraction. Don't mean to be rude, just honestly that is my personal experience.

hiddencost 2y ago

Why do people keep posting this stuff? He's been writing variations on the same essay for decades and he's always wrong.

baanist 2y ago
What is he wrong about?
- krallistic 2y ago
  "Deep learning is hitting a wall now with just scaling" "Deep learning is only good for perception" (with language one of the areas where its not good)
  - baanist 2y ago
    I don't have any subscriptions to the latest models but what improvements have you noticed in scaling and language understanding? Last I checked people were still discussing "9.9 > 9.1" and "How many 'r's are in 'strawberry'".
- hyperG 2y ago
  I have never read anything by him that seemed correct. He really does seem like quite a good contrarian indicator if anything.
  - baanist 2y ago
    He was right that scaling would not achieve abstract reasoning and he's been right so far on basically every new hyped development. The closed research labs like OpenAI are hoping to reduce all the gaps in their models by constantly patching out of distribution data sets but this clearly can not achieve any sort of general intelligence unless they somehow manage to obsolete themselves and the entire company by automating the out of distribution patching itself which they now perform by burning lots of cash and energy. There are people who need to write a lot of emails so for those people I'm sure OpenAI will continue to deliver some kind of value but everyone else will still have to continue thinking for themselves regardless of what Sam Altman keeps promising.

zarzavat 2y ago

There seem to be many people who are at pains to tell us what generative AI won't do in the future. Any such prediction seems extremely likely to be false without a time bound, or an argument to physical impossibility.

tobyhinloopen 2y ago

The real reason ChatGPT saves time is because it is faster to provide inaccurate answers than Google.

throwaway918299 2y ago

My average experience reviewing code written by “average developers” that start using copilot is that it ends up 0.1x them. We are going to be so screwed in 10 years.

dgoodell 2y ago

So far it has definitely saved me a lot of time googling api docs and stackoverflow for me. Anything that requires applying things in novel ways that doesn’t have lots of examples out there already seems to be completely beyond it. Also, it often comes up with very unoptimal and inefficient solutions even though it seems to be completely aware of better solutions when prodded. So basically it’s a fully competent programmer lol.

dmitrygr 2y ago

For low level work (kernels, drivers, C, assembly), things like ChatGPT and Cursor are a 0.1X multiplier. They suggest idiotic things both design-wise and implementation-wise. In this world, I’ll take my cat’s help over theirs. Maybe it’ll improve, but I won’t hold my breath.

mahmoudimus 2y ago
That's sort of ironic. One of the biggest value adds I've seen to my hobby work is that LLMs are REALLY good at understanding disassembly and can even do things like symbolically execute a few lines for me. I've used it to deobfuscate several malware protections and I can imagine it will get better. However, I have never tried it on kernels, drivers, etc. Just the inverse of that :)
baanist 2y ago
These algorithms are not capable of symbolic reasoning and abstract interpretation. The most obvious demonstration of this is that no algorithm on the market currently can solve sudoku puzzles even though they have spent billions of dollars "training" them on logic and reasoning.

rkunal 2y ago

I will believe when GenAI becomes 10x, we would be noticing a larger number of emacs packages. Or at-least few attempts to rewrite old famous softwares like Calibre. I am sure that it is being used at a large scale in web apps and cloud tech. Not aware of the same in other domains. May be in the future, everything will become a web app because genAI became so good at it.

munchausen42 2y ago

Funny to see how being anti-GenAI and anti-LLM is now the new en vogue on HN. Can't wait till that dies off as well.

kylehotchkiss 2y ago
People have been through a lot of hype cycles and the grandiose claims of what AI/LLMs could be / "superintellegence" etc are a complete distraction from a pretty impressive tech accomplishment. Too few people wonder if the wins we got are worth it even if we hit a wall trying to achieve "AGI". And the real world impacts with electricity and billionaires running around with no guardrails are starting to slowly come to light. It sure looks a lot like the last hype cycle where we were promised a global currency without government intervention... lol. Do you remember the part where people took out loans to buy cartoon apes? (I personally/philosophically don't believe AGI is a natural next step for LLMs as I don't believe the English language alone which training is so heavy on encapsulates all of human ability, rather it's very honed in on English speaking countries/cultures - I also don't believe humans are very capable of creating derivative products with capabilities greater than their own - we can barely make progress on what really causes mental illness[1] - how can we claim to understand our minds so well we can replicate their functionality?) [1]: https://www.science.org/content/blog-post/new-mode-schizophrenia-treatment https://www.science.org/content/blog-post/new-mode-schizophr...

hackermatic 2y ago

I hope commenters will dig into the author's citations' data, in line with HN's discussion guidelines, instead of just expressing a negative opinion about the thrust of the article. The quantifiable impact of genAI on code productivity is an important research question, and very much an open question subject to bias from all the players in the space -- especially once you factor in quality, maintainability, and bugs or revisions over time. The GitClear whitepaper that Marcus cites tries to account for some of these factors, but they're biased by selling their own code quality tools. Likewise, GitHub's whitepapers (and subsequent marketing) tend to study the perception of productivity and quality by developers, and other fuzzy factors like the suggestion acceptance rate -- but not bug rate or the durability of accepted suggestions over time. (I think perceived productivity and enjoyment of one's job are also important values, but they're not what these products are being sold on.)

janalsncm 2y ago

If you want to know how valuable GenAI is for software engineering, you don’t ask AI hype people on Twitter. You also don’t ask Gary Marcus. You ask people trying to complete software engineering tasks. I guess part of the reason we can’t agree on whether they are useful is that their usefulness depends on how familiar you are with the programming task.

rekttrader 2y ago

As a CTO of a new startup, I am not hiring contractors or employees to build software. This may not be 10x, but when the time comes and I need scale I’m likely to hire what will be my smallest team yet. One of the requirements for hiring is that they be proficient with copilot, cursor, or have rolled their own llama based code assistant. It’s like going from using manual hand tools to power drills and nail guns. Those that doubt That AI will change all of technology jobs and work in the industry are gonna find themselves doing something else.

yodon 2y ago
> One of the requirements for hiring is that they be proficient with copilot, cursor, or HAVE ROLLED THEIR OWN LLAMA BASED CODE ASSISTANT. [enphasis added] Isn't that a bit like "I won't hire a dev unless they've written a C++ compiler"? Very dev macho, but not necessarily the best criteria for someone working on a distributed financial transactional system.

Jimmc414 2y ago

"Gary Marcus has been coding since he was 8 years old, and was very proud when his undergrad professor said he coded an order of magnitude faster than his peers. Just as Chollet might have predicted, most of the advantage came from clarity — about task, algorithms and data structures." I'm trying to understand what sort of person would end with a 3rd person anecdote like this.

addisonj 2y ago

I want to make clear that I agree with the author that the current way in which AI is being used to make a 10x improvement in efficiency is NOT going to work, and it is for exactly reason stated: > 10x-ing requires deep conceptual understanding – exactly what GenAI lacks. Writing lines of code without that understanding can only help so much. IMHO, the real opportunity with AI, is to get developers less critical in being the only ones who translate conceptual understanding of the business problem into code and get developers more focused on the domain they should be experts in which, imho, is the large scale structure of the system and how that maps to actual computes. What I mean by that... developers, are, on average, not good at having deep conceptual understanding of the business domain... and maybe have that understanding in the computational domain. All of the 10x developers I have ever met are mostly that because they do know both the business domain (or can learn it really quick) and the computational domain so they can quickly deliver value because they understand the problem end-to-end. My internal model for thinking about this is we need developers thinking about building the right "intermediate representation" of a problem space and being able to manage the system at large, but we need domain experts, likely supported by AI assistants, in helping to use that IR to express the actual business problem. That is a super loose explanation... but if you have been in the industry awhile, you probably have felt the insanity of how long it can take to ship a fairly small feature because of a huge disconnect between the broader structure of the system (with microservices, async job systems, data pipelines, etc, which we can think of as the low-level representation of the sytem at large) and the high-level requirement of needing to store some data and process it according to some business specific rules. I have no idea if we actually can get there as an industry... it is more likely we just have more terrible code... but maybe not

norir 2y ago

The bummer of ai hype is that I think there is so much we can do to improve programming with deterministic tooling. We have seen a remarkable improvement in terms of real time feedback directly in your editing context through IDEs and editor plugins. We still don't yet have the next programming language for the ai age. I believe the biggest gains will not come from bigger models but smarter tools that can use stochastic tools like LLMs to explore the solution space and quickly verify the solution according to well defined constraints that can be expressed in the language. When I hear about some of the massive productivity gains people ascribe to ai, I also wonder where the amazement for "rails new" has gone. We already have tools that can 100x your productivity in specific areas without also hallucinating random nonsense (though Rails is maybe not the absolute best counterexample here).

xnx 2y ago

There's a big difference between "AI isn't 10x-ing computer programming" and "AI will not 10x computer programming". Am I missing the part of the article where some evidence is provided that the situation won't improve?

senko 2y ago

Gary takes credible facts there (modest gains, ai is not replacement for clear thinking), altough attacking the "10x" claim is a cheap shot because the claim was always nebulous. However, in light of Gary's other AI related writings[0], it is clear he's not saying "people, be clear headed about the promises and limits of this new technology", he's saying "genai is shit". Yes, I am attacking the character/motive instead of the argument, because there's plenty of proof about the motive. So I am kind of agreeing (facts as laid out) and vehemently disagreeing (the underlying premise his readers will get from this) with the guy at the same time. [0]: Here are a few headlines I grabbed from his blog: - How much should OpenAI’s abandoned promises be worth? - Five reasons why OpenAI’s $150B financing might fall apart - OpenAI’s slow-motion train wreck - Why California’s AI safety bill should (still) be signed into law - and why that won’t be nearly enough - Why the collapse of the Generative AI bubble may be imminent

juliendorra 2y ago

For comparison VisiCalc and other early spreadsheets like Lotus 123 and Multiplan were 80x multipliers for their users: “I would work for twenty hours,” Jackson said. “With a spreadsheet, it takes me 15 minutes.” Not sure any other computing tool ever beat that since? Source: Steven Levy reporting, Harper’s Bazaar 1984, reprinted in Wired: https://www.wired.com/2014/10/a-spreadsheet-way-of-knowledge/ https://www.wired.com/2014/10/a-spreadsheet-way-of-knowledge...

geenkeuse 2y ago

Looks like you guys are coming around.For the longest time it was the same "AI will never ever do what we do" song. Now you are all mostly singing it's praises. Good. You will be left standing when the stubborn are forced to call it quits. What is so wrong with having a computer do amazing things for you simply by asking it? That was the dream and the mission, wasn't it? Unless "obfuscation is the enemy" Most people don't want the IT guy to get into the details of how they did what they did. They just want to get on with it. Goes for the guy replacing your hard drive. Goes for the guy writing super complicated programs. IT is a commodity. A grudge purchase for most companies. But you know that, deep down. Don't you? The bar has been lowered and raised at the same time. Amazing.

m463 2y ago

My first job at college had schedules with high-level and detailed design, then coding, then integration and test. Coding was only 3-5% of the time. Also, I remember learning to race. I tried to learn to go faster through the turns. But it turns out the way to be fast was to go a little faster on the straightaway and long turns. Trying to go faster through the tight turns doesn't help much. If you go 1 mph faster for 10 feet in the slowest turn, it doesn't make as much difference as going .1 mph faster in a long 500 foot turn. so... Are we trying to haphazardly optimize for the small amount of time we code, when we should spend it elsewhere?

throwawa14223 2y ago

I'd be shocked if it wasn't a negative at the end of the day.

bitdeep 2y ago

Tools like aider or cursor composer does not help for complex code as they destroy your mental code model of the solution you working on. This tools help a bit for initial mocks, but even that, I don't like as they create code that I don't know and I need to review it all to know where things is. When you are doing complex software, you need to build a good mental code model to know where things is, especialy when you need to start to debug issues, not knowing where things is is a mess and very annoying. This days, I almost don't use this tools anymore, I just prefer basic line auto-completion.

nateless 2y ago

Completely agree. Tried spinning up a side project with just Cursor and GPT/Claude models, and we're definitely not at 'AI can build apps from just a prompt' yet. It couldn’t even create a dropdown filter like Linear’s, and the code was buggy, inefficient, and far from the full functionality I asked for. Maybe it's the niche language (Elixir), but we’re not there yet.