15 ms·

> Nothing because I’m a senior and LLM’s never provide code that pass my sniff test, and it remains a waste of time. I am constantly surprised how prevalent th

> Nothing because I’m a senior and LLM’s never provide code that pass my sniff test, and it remains a waste of time. I am constantly surprised how prevalent this attitude is. ChatGPT was only just released in 2022. Is there some expectation that these things won't improve? > LLM’s never provide code that pass my sniff test This is ego speaking.

palata 2y ago
> Is there some expectation that these things won't improve? Most of the noise I hear about LLMs is about expectations that things will improve. It's always like that: people extrapolate and get money from VCs for that. The thing is: nobody can know. So when someone extrapolates and says it's likely to happen as they predict, they shouldn't be surprised to get answers that say "I don't have the same beliefs".
tomcar288 2y ago
when people's entire livehoods are threatened, you're going to see some defensive reactions.
eschaton 2y ago
They shouldn’t be expected to improve in accuracy because of what they are and how they work. Contrary to what the average HackerNews seems to believe, LLMs don’t “think,” they just predict. And there’s nothing in them that will constrain their token prediction in a way that improves accuracy.
- rybosworld 2y ago
  > Contrary to what the average HackerNews seems to believe, LLMs don’t “think,” they just predict. Anecdotally, I can't recall ever seeing someone on HackerNews accuse LLM's of thinking. This site is probably one of the most educated corners of the internet on the topic. > They shouldn’t be expected to improve in accuracy because of what they are and how they work. > And there’s nothing in them that will constrain their token prediction in a way that improves accuracy. These are both incorrect. LLM's are already quite better today than they were in 2022.
  - reshlo 2y ago
    > I can’t recall ever seeing someone on HackerNews accuse LLMs of thinking. There are definitely people here who think LLMs are conscious. https://news.ycombinator.com/item?id=41959163 https://news.ycombinator.com/item?id=41959163
- BlackSwanMan 2y ago
  [dead]
- cozzyd 2y ago
  If anything, they may regress due to being trained on lower-quality input.
nicoburns 2y ago
> > LLM’s never provide code that pass my sniff test > This is ego speaking. That's been my experience of LLM-generated code that people have submitted to open source projects I work on. It's all been crap. Some of it didn't even compile. Some of it changed comments that were previously correct to say something untrue. I've yet to see a single PR that implemented something useful.
- constantcrying 2y ago
  Terrible software engineers sent you terrible code. A good software engineer will obviously send you good quality code, whether he used AI for that is impossible to tell on your end.
- mh- 2y ago
  Isn't this a kind of survivor bias? You wouldn't know if you approved (undisclosed) LLM-generated code that was good..
  - talldayo 2y ago
    Good LLM-generated code comes from good developers that can provide detailed requests, filter through wrong/misleading/cruft results, and test a final working copy. If your personal bar for PRs is high enough then it won't matter if you use AI or not because you won't contribute shitty results. I think the problem has become people with no bar for quality submitting PRs that ruin a repo, and then getting mad when it's not merged because ChatGPT reviewed the patch already.
- cies 2y ago
  > LLM-generated code that people have submitted to open source projects I work on Are you sure it was people? Maybe the AI learned how to make PRs, or is learning how to do so by using your project as a test bed.
sitzkrieg 2y ago
ego? LLMs goof on basic math and cant even generate code for many non public things. theyre not useful to me whatsoever
- agilob 2y ago
  LLMs aren't supposed to do basic math, but be chat agents. Wolfram Alpha can't do chat.
  - simianparrot 2y ago
    Math is a major part of programming. In fact programming without math is impossible. And if you go all the way down to bare metal it’s all math. We are shifting bits through incredibly complex abstractions.
    - constantcrying 2y ago
      Has never been true in my experience, almost all programming can be done without any mathematics. Of course very specific things do require mathematics, but they are uncommon and often solved by specialists. Btw. I come from a math background and later went into programming.
    - agilob 2y ago
      No, math is major part of writing good code, but when was the last time you've seen somebody put effort into writing O(n) algorithm? 99% of programming is "import sort from sort; sort.sortThisReallyQuick". Programming is mostly writing code that just compiles and eventually gives correct results (and has bugs). You can do a lot of programming just buy copy-pasting results from stackoverflow. https://en.wikipedia.org/wiki/Npm_left-pad_incident https://en.wikipedia.org/wiki/Npm_left-pad_incident https://old.reddit.com/r/web_design/comments/35prfv/designer_applies_for_js_job_fails_at_fizzbuzz/ https://old.reddit.com/r/web_design/comments/35prfv/designer... https://www.youtube.com/watch?v=GC-0tCy4P1U https://www.youtube.com/watch?v=GC-0tCy4P1U
      - 2y ago
IshKebab 2y ago
> Is there some expectation that these things won't improve? I definitely expect them to improve. But I also think the point at which they can actually replace a senior programmer is pretty much the exact point at which they can replace any knowledge worker, at which point western society (possibly all society) is in way deeper shit than just me being out of a job. > This is ego speaking. It definitely isn't. LLMs are useful for coding now, but they can't really do the whole job without help - at least not for anything non-trivial.
- dingnuts 2y ago
  > LLMs are useful for coding now *sort of, sometimes, with simple enough problems with sufficiently little context, for code that can be easily tested, and for which sufficient examples exist in the training data. I mean hey, two years after being promised AGI was literally here, LLMs are almost as useful as traditional static analysis tools! I guess you could have them generate comments for you based on the code as long as you're happy to proofread and correct them when they're wrong. Remember when CPUs were obsolete after three years? GPT has shown zero improvement in its ability to generate novel content since it was first released as GPT2 almost ten years ago! I would know because I spent countless hours playing with that model.
  - margalabargala 2y ago
    Firstly, GPT-2 was released in 2019. Five years is not "almost ten years". Secondly, LLMs are objectively useful for coding now. That's not the same thing as saying they are replacements for SWEs. They're a tool, like syntax highlighting or real-time compiler error visibility or even context-aware keyword autocompletion. Some individuals don't find those things useful, and prefer to develop in a plain text editor that does not have those features, and that's fine. But all of those features, and LLMs are now on that list, are broadly useful in the sense that they generally improve productivity across the industry. They already right now save enormous amounts of developer time, and to ignore that because you are not one of the people whose time is currently being saved, indicates that you may not be keeping up with understanding the technology of your field. There's an important difference between a tool being useful for generating novel content, and a tool being useful. I can think of a lot of useful tools that are not useful for generating novel content.
gspencley 2y ago
>> > LLM’s never provide code that pass my sniff test > This is ego speaking. Consider this, 100% of AI training data is human-generated content. Generally speaking, we apply the 90/10 rule to human generated content: 90% of (books, movies, tv shows, software applications, products available on Amazon) is not very good. 10% shines. In software development, I would say it's more like 99 to 1 after working in the industry professionally for over 25 years. How do I divorce this from my personal ego? It's easy to apply objective criteria: - Is the intent of code easy to understand? - Are the "moving pieces" isolated, such that you can change the implementation of one with minimal risk of altering the others by mistake? - Is the solution in code a simple one relative to alternatives? The majority of human produced code does not pass the above sniff test. Most of my job, as a Principal on a platform team, is cleaning up other peoples' messes and training them how to make less of a mess in the future. If the majority of human generated content fails to follow basic engineering practices that are employed in other engineering disciplines (i.e: it never ceases to amaze me how much of an uphill battle it is just to get some SWEs just to break down their work into small, single responsibility, easily testable and reusable "modules") then we can't logically expect any better from LLMs because this is what they're being trained on. And we are VERY far off from LLMs that can weigh the merits of different approaches within the context of the overall business requirements and choose which one makes the most sense for the problem at hand, as opposed to just "what's the most common answer to this question?" LLMs today are a type of magic trick. You give it a whole bunch of 1s and 0s so that you can input some new 1s and 0s and it can use some fancy proability maths to predict "based on the previous 1s and 0s, what are the statistically most likely next 1s and 0s to follow from the input?" That is useful, and the result can be shockingly impressive depending on what you're trying to do. But the limitations are so limited that the prospect of replacing an entire high-skilled profession with that magic trick is kind of a joke.
- m_ke 2y ago
  Your customers don't care how your code smells, as long as it solves their problem and doesn't cost an arm and a leg. A ton of huge business full of Sr Principal Architect SCRUM masters are about to get disrupted by 80 line ChatGPT wrappers hacked together by a few kids in their dorm room.
groby_b 2y ago
> Is there some expectation that these things won't improve? Sure. But the expectation is quantitative improvement - qualitative improvement has not happened, and is unlikely to happen without major research breakthroughs. LLMs are useful. They still need a lot of supervision & hand holding, and they'll continue to for a long while And no, it's not "ego speaking". It's long experience. There is fundamentally no reason to believe LLMs will take a leap to "works reliably in subtle circumstances, and will elicit requirements as necessary". (Sure, if you think SWE work is typing keys and make some code, any code, appear, then LLM are a threat)
akira2501 2y ago
> Is there some expectation that these things won't improve? Yes. The current technology is at a dead end. The costs for training and for scaling the network are not sustainable. This has been obvious since 2022 and is related to the way in which OpenAI created their product. There is no path described for moving from the current dead end technology to anything that could remotely be described as "AGI." > This is ego speaking. This is ignorance manifest.
simianparrot 2y ago
At my job I review a lot of code, and I write code as well. The only type of developer an LLM’s output comes close to is a fresh junior usually straight out of university in their first real development job, with little practical experience in a professional code-shipping landscape. And the majority of those juniors improve drastically within a few weeks or months, with handholding only at the very start and then less and less guidance. This is because I teach them to reason about their choices and approaches, to question assumptions, and thus they learn quickly that programming rarely has one solution to a problem, and that the context matters so much in determining the way forward. A human junior developer can learn from this tutoring and rarely regress over time. But the LLM’s all by design cannot and do not rewire their understanding of the problem space over time, nor do they remember examples and lessons from previous iterations to build upon. I have to handhold them forever, and they never learn. Even when they use significant parts of the existing codebase as their context window they’re still blind to the whole reality and history of the code. Now just to be clear, I do use LLM’s at my job. Just not to code. I use them to parse documents and assist users with otherwise repetitive manual tasks. I use their strength as language models to convert visual tokens parsed by an OCR to grasp the sentence structure and convert that into text segments which can be used more readily by users. At that they are incredible, even something smaller like llama 7b.
- Derbasti 2y ago
  That's a good observation. How would you teach an LLM the processes and conduct in your company? I suppose you'd need to replace code reviews and tutoring with prompt engineering. And hope that the next software update to the LLM won't invalidate your prompts.
  - sigmarule 2y ago
    It's not very different from documentation, except it's not used for learning but rather immediate application, i.e. it's something you must include with each prompt/interaction (and likely need a way to determine which are the relevant bits to reduce token count, depending on the size of the main prompt.) In fact, this is probably one of the more important aspects of adapting LLMs for real-world use within real team/development settings that people don't do. If you provide clear and comprehensive descriptions of codebase patterns, pitfalls, practices, etc the latest models are good at adhering to them. It sounds difficult and open-ended, as this requires content beyond the scope of typical (or even sensible) internal documentation, but much of this content is captured in internal docs, discussions, tickets, PR comments, and git commit histories, and guess what's pretty great at extracting high-level insights from these types of inputs?
code_for_monkey 2y ago
I agree with you tbh, and it also just misses something huge that doesnt get brought up: its not about your sniff test, its about your bosses sniff test. Are you making 300k a year? Thats 300 thousand reasons to replace you for a short term boost in profit, companies love doing that.
- simianparrot 2y ago
  I'm in a leadership role and one of the primary parties responsible for hiring. So code passing my sniff test is kind of important.
  - fragmede 2y ago
    To bring it back to the question that spawned the ask: One thing to future proof a career as a SWE is to get into business and leadership/management as well as learning how to program. If you're indispensable to the business for other reasons, they'll keep you around.
    - code_for_monkey 2y ago
      in that case the conversation is still kind of missing the forest for the trees, yes your career will be safer if you get into management but thats not going to play for everyone. Mathematically, it cant, not every engineer can go be a manager. What Im interested in is the stories of the people who arent going to make it.
gosub100 2y ago
The reason it's so good at "rewrite this C program in Python" is because it was trained on a huge corpus of code at GitHub. There is no such corpus of examples of more abstract commands, thus a limited amount by which it can improve.
Bjorkbat 2y ago
> I am constantly surprised how prevalent this attitude is. ChatGPT was only just released in 2022. Is there some expectation that these things won't improve? I mean, in a way, yeah. Last 10 years were basically one hype-cycle after another filled with lofty predictions that never quite panned out. Besides the fact that many of these predictions kind of fell short, there's also the perception that progress on these various things kind of ground to a halt once the interest faded. 3D printers are interesting. Sure, they have gotten incrementally better after the hype cycle died out, but otherwise their place in society hasn't changed, nor will it likely ever change. It has its utility for prototyping and as a fun hobbyist machine for making plastic toys, but otherwise I remember people saying that we'd be able to just 3D print whatever we needed rather than relying on factories. Same story with VR. We've made a lot of progress since the first Oculus came out, but otherwise their role in society hasn't changed much since then. The latest VR headsets are still as useless and still as bad for gaming. The metaverse will probably never happen. With AI, I don't want to be overly dismissive, but at the same time there's a growing consensus that pre-training scaling laws are plateauing, and AI "reasoning" approaches always seemed kind of goofy to me. I wouldn't be surprised if generative AI reaches a kind of equilibrium where it incrementally improves but improves in a way where it gets continuously better at being a junior developer but never quite matures beyond that. The world's smartest beginner if you will. Which is still pretty significant mind you, it's just that I'm not sure how much this significance will be felt. It's not like one's skillset needs to adjust that much in order to use Cursor or Claude, especially as they get better over time. Even if it made developers 50% more productive, I feel like the impact of this will be balanced-out to a degree by declining interest in programming as a career (feel like coding bootcamp hype has been dead for a while now), a lack of enough young people to replace those that are aging out, the fact that a significant number of developers are, frankly, bad at their job and gave up trying to learn new things a long time ago, etc etc. I think it really only matters in the end if we actually manage to achieve AGI, once that happens though it'll probably be the end of work and the economy as we know it, so who cares? I think the other thing to keep in mind is that the history of programming is filled with attempts to basically replace programmers. Prior to generative AI, I remember a lot of noise over low-code / no-code tools, but they were just the latest chapter in the evolution of low-code / no-code. Kind of surprised that even now in Anno Domini 2024 one can make a living developing small-business websites due to the limitations of the latest batch of website builders.
cies 2y ago
> ChatGPT was only just released in 2022. Bitcoin was released in what year? I still cannot use it for payments. No-code solutions exist since when? And still programmers work... I dont think all hyped techs are fads. For instance: we use SaaS now instead of installing software locally. This transition took the world by storm. But those tech that needs lots of ads, and lots of zealots, and make incredible promises: they usually are fads.
- bigstrat2003 2y ago
  > For instance: we use SaaS now instead of installing software locally. If anything, I feel like this argument works against you. If people are willing to replace locally installed software with shitty web "apps" that barely compare, why do you think they won't be willing to replace good programmers with LLMs doing a bad job simply because it's trendy?
  - achierius 2y ago
    SaaS has generally been better than the local apps it replaced, particularly when you factor in 'portability'. I love local apps but it's undeniable that developers having to split their attention between platforms lowered quality by a lot. You're probably remembering the exemplars, not the bulk which half-worked and looked bad too
    - cies 2y ago
      Just look at the job market: how many programmers work in SaaS now compared to 2000?
- gabriel-uribe 2y ago
  Cryptocurrency more broadly can be used to send vast amounts of money, anywhere in the world for near-zero fees regardless of sovereignty. And Bitcoin is still faster and cheaper than shipping gold around.
jcranmer 2y ago
This is at least the third time in my life that we've seen a loudly-heralded purported the-end-of-programming technology. The previous two times both ended up being damp squibs that barely mention footnotes in the history of computing. Why do we expect that LLMs are going to buck this trend? It's not for accuracy--the previous attempts, when demonstrating their proof-of-concepts, actually reliably worked, whereas with "modern LLMs", virtually every demonstration manages to include "well, okay, the output has a bug here."
- simianparrot 2y ago
  I do seem to vaguely remember a time when there was a fair amount of noise proclaiming "visual programming is making dedicated programmers obsolete." I think the implication was that now everybody's boss could just make the software themselves or something. LLM's as a product feel practically similar, because _even if_ they could write code that worked in large enough quantities to constitute any decently complex application, the person telling them what problem to solve has to understand the problem space since the LLM's can't reason. Given that neither of those things are true, it's not much different from visual programming tools, practically speaking.
  - nickd2001 2y ago
    This is a feeling I have too. However, compared to Visual Programming it's perhaps harder to dismiss? Visual programming - its pretty obvious you can't effectively or even at all step through a UML diagram with a debugger to find a problem. The code that gets generated from diagrams, with visual programming, is obviously cr*p. So the illusion doesn't last long. Whereas AI - it kind of looks OK, you can indeed debug it, its not necessarily more complex or worse than the hideously over-complex systems human teams create. Especially if the human team is mismanaged e:g original devs left, some got burnt out and became unproductive, others had to cut corners and make tech debt to hit unrealistic deadlines, other bits get outsourced to another country where the devs themselves are great but perhaps there's a language barrier or simply geographical distance means requirements and domain understanding got lost somewhere in the mix. So, I suppose I sit on the fence, AI-generated code may be terrible but is it worse than what we were making anyway? ;). In the future there are probably going to be companies run by "unwise people" that generate massive amounts of their codebase then find themselves in a hole when no-one working at the company understands the code at all. (whereas in the past perhaps they could hire back a dev they laid off on large contractor rates of pay to save the day). Seems inevitable one day the news will be of some high profile company failure and/or tanking of stock caused by a company basically not knowing what it was doing due to AI generated code.
deathanatos 2y ago
> This is ego speaking. No, it really isn't. Repeatedly, the case is that people are trying to pass off GPT's work as good without actually verifying the output. I keep seeing "look at this wonderful script GPT made for me to do X", and it does not pass code review, and is generally extremely low quality. In one example, a bash script was generated to count number SLoC changed by author; it was extremely convoluted, and after I simplified it, I noticed that the output of the simplified version differed, because the original was omitted changes that were only a single line. In another example it took several back & forths during a review to ask "where are you getting this code? / why do you think this code works, when nothing in the docs supports that?" and after several back and forths, it was admitted that GPT wrote it. The dev who wrote it would have been far better served RTFM, than a several cycle long review that ended up with most of GPT's hallucinations being stripped from the PR. Those who think LLM's output is good have not reviewed the output strenuously enough. > Is there some expectation that these things won't improve? Because randomized token generation inherently lacks actual reasoning about the behavior of the code. My code generator does not.
- AlphaSite 2y ago
  I think fundamentally if all you do is glue together popular OSS libraries in well understood way, then yes. You may be replaced. But really you probably could be replaced by a Wordpress plugin at that point. The moment you have some weird library that 4 people in the world know (which happens more than you’d expect) or hell even something without a lot of OSS code what exactly is an LLM going to do? How is it supposed to predict code that’s not derived from its training set? My experience thus far is that it starts hallucinating and it’s not really gotten any better at it. I’ll continue using it to generate sed and awk commands, but I’ve yet to find a way to make my life easier with the “hard bits” I want help with.
  - valval 2y ago
    You ask what the LLM is going to do. It’s going to swallow the entire code base in context and allow any developer to join those 4 people in generating production grade code.
  - 2y ago
surgical_fire 2y ago
> I am constantly surprised how prevalent this attitude is. ChatGPT was only just released in 2022. Is there some expectation that these things won't improve? Is there any expectations that things will? Is there more untapped great quality data that LLMs can ingest? Will a larger model perform meaningfully better? Will it solve the pervasive issue of generating plausibly sounding bullshit? I used LLMs for a while, I found them largely useless for my job. They were helpful for things I don't really need help with, and they were mostly damaging for things I actually needed. > This is ego speaking. Or maybe it was an accurate assessment for his use case, and your wishful thinking makes you think it was his ego speaking.
- rybosworld 2y ago
  > Is there any expectations that things will? Seems like an odd question. The answer is obviously yes: There is a very pervasive expectation that LLM's will continue to improve, and it seems odd to suggest otherwise. There is hundreds of billions of dollars being spent on AI training and that number is increasing each year. > Is there more untapped great quality data that LLMs can ingest? Why wouldn't there be? AI's are currently trained on the internet but that's obviously not the only source of data. > Will a larger model perform meaningfully better? The answer to this, is also yes. It is well established that, all else being equal, a bigger model is better than a smaller model, assuming that the smaller model hasn't already captured all of the available information.
  - barrell 2y ago
    If all of this is true, then I would expect LLMs today to be in a whole different league of LLMs from two years ago. Personally I find them less helpful and worse at programming and creative writing related tasks. YMMV but count me in the camp that I think there’s better odds that LLMs are at or near their potential vs in their nascent stages.
  - tobias3 2y ago
    We recently had a few submissions about this topic. Most recently Ilyas talk. Further improvement will be a research type problem. This trend was clear for a while already, but is reaching the mainstream now. The billions of dollar spend goes into scaling existing technology. If it doesn't scale anymore and becomes a resarch problem again, rational companies will not continue to invest in this area (at least without the usual research arrangements).
deegles 2y ago
LLMs as they currently exist will never yield a true, actually-sentient AI. maybe they will get better in some ways, but it's like asking if a bird will ever fly to the moon. Something else is needed.
- mlboss 2y ago
  A bird can fly to moon if it keeps on improving every month.
  - VeejayRampay 2y ago
    it literally cannot though, unless it becomes some other form of life that doesn't need oxygen, that's the whole thing with this analogy, it's ironically suited to the discourse
    - lloeki 2y ago
      Ha I didn't even count oxygen in the mix; it could hold its breath maybe? j/k, what's for sure is that I don't see how a biological entity could ever achieve escape velocity from self-powered flight, because, well, physics. That, or they "keep improving every month" til they become evolved enough to build rockets, at which point the entire point of them being birds becomes moot.
    - simianparrot 2y ago
      It would need to: * Survive without oxygen * Propel itself without using wings * Store enough energy for the long trip and upkeep of the below * Insulation that can handle the radiation and temperature fluctuations * Have a form of skin that can withstand the pressure change from earth to space and then the moon * Have eyes that don't pop while in space It would need to become something literally extraterrestrial that has not evolved in the 3.7b+ years prior. I wouldn't say it's impossible, but if evolution ever got there that creature would be so far removed from a bird that I don't think we'd recognize it :p
bdangubic 2y ago
[flagged]
shakezooola 2y ago
>This is ego speaking. Very much so. These things are moving so quickly and agentic systems are already writing complete codebases. Give it a few years. No matter how 1337 you think you are, they are very likely to surpass you in 5-10 years.
- achierius 2y ago
  > agentic systems are already writing complete codebases Examples?
jazz9k 2y ago
Have you ever used LLMs to generate code? It's not good enough yet. In addition to this, most companies aren't willing to give away all off their proprietary IP and knowledge through 3rd party servers. It will be awhile before engineering jobs are at risk.
talldayo 2y ago
> This is ego speaking. I suspect you haven't seen code review at a 500+ seat company.
deleted 2y ago
[deleted]
goalonetwo 2y ago
100% and even if they might not "replace" a senior or staff SWE, they can make their job significantly easier which means that instead of requiring 5 Senior or Staff you will only need two. LLM WILL change the job market dynamics in the coming years. Engineers have been vastly overpaid over the last 10 years. There is no reason to not see a reversal to the mean here. Getting a 500k offer from a FAANG because you studied leetcode for a couple weeks is not going to fly anymore.
- danenania 2y ago
  “which means that instead of requiring 5 Senior or Staff you will only need two” What company ever feels they have “enough” engineers? There’s always a massive backlog of work to do. So unless you are just maintaining some frozen legacy system, why would you cut your team size down rather than doubling or tripling your output for the same cost? Especially considering all your competitors have a similar productivity boost available. If your reaction is to cut the team rather than make your product better at a much faster rate (or build more products), you will likely be outcompeted by others willing to make that investment.
tobyhinloopen 2y ago
LLMs are great for simple, common, boring things. As soon as you have something less common, it will give you widely incorrect garbage that does not make any sense. Even worse, it APPEARS correct, but it won’t work or will do something else completely. And I use 4o and o1 every day. Mostly for boilerplate and boring stuff. I have colleagues that submit ChatGPT generated code and I’ll immediately recognize it because it is just very bad. The colleague would have tested it, so the code does work, but it is always bad, weird or otherwise unusual. Functional, but not nice. ChatGPT can give very complicated solutions to things that can be solved with a one-liner.
globnomulous 2y ago
> This is ego speaking. It absolutely isn't. I have yet to find an area where LLM-generated code solves the kinds of problems I work on more reliably, effectively, of efficiently than I do. I'm also not interested in spending my mental energy on code reviews for an uncomphrehending token-prediction golem, let alone finding or fixing bugs in the code it blindly generates. That's a waste of my time and a special kind of personal hell.
- epolanski 2y ago
  Comments like these make me wonder whether we live in the same worlds. I'm a cursor user e.g. and Tab completion is by far the most powerful auto complete I've ever used. There's scenarios where you can do some major refactors by simply asking (extract this table to its own component while using best react practices to avoid double renders) and it does so istantly. Refactor the tests for it? Again. Semi instant. Meanwhile monocole wielding "senior" is proudly copy pasting, creating files and fixing indentation as I move to the next task. I don't expect LLMs to do the hard work, but to speed me up. And people ignoring LLMs are simply slower and less productive. It's a speed multiplier right now, not a substitute. If you complain that you don't like the code, you're not understanding the tools nor you can use them, end of story.
  - mplanchard 2y ago
    Sounds like you’re working in react. The models are orders of magnitude worse at less common frameworks and/or languages. React is so boilerplate-heavy that it’s almost a perfect match for LLMs. That said, every large react codebase I’ve worked in inevitably became really complex and had a ton of little quirks and tweaks to deal with unexpected re-renders and such. I avoid the FE when I can these days, but I’d be fairly surprised if LLMs generated anything other than typical performance-footgun-heavy react
    - phist_mcgee 2y ago
      Not in my experience, claude seems great at writing idiomatic and performant react. You can even highlight code and ask it to spot performance issues and it will tell you where you can improve rendering efficiency etc.
tucnak 2y ago
[flagged]
quantadev 2y ago
Even since ChatGPT 3.5 AI coding has been astoundingly good. Like you, I'm totally baffled by developers who say it's not. And I'm not just unskilled and unable to judge. I have 35yrs experience! I've been shocked and impressed from day one, how good AI is. I've even seen Github Copilot do thing that almost resemble mind-reading insofar as predicting what I'm about to type next. It predicts some things it should NOT be able to predict unless it's seeing into the future or into parallel universes or something! And I'm only half joking when I speculate that!
- whtsthmttrmn 2y ago
  This just shows that time spent is not the same as wisdom.
  - deleted 2y ago
    [deleted]
  - quantadev 2y ago
    Even ChatGPT 3.5 had "wisdom", so you're definitely wrong.
JeremyNT 2y ago
Yeah I feel like this is just the beginning. I'm in my 40s with a pretty interesting background and I feel like maybe I'll make it to retirement. There are still mainframe programmers after all. Maintaining legacy stuff will still have a place. But I think we'll be the last generation where programmers will be this prevalent and the job market will severely contract. No/low code solutions backed by llms are going to eat away at a lot of what we do, and for traditional programming the tooling we use is going to improve rapidly and greatly reduce the number of developers needed.

palata 2y ago

> Is there some expectation that these things won't improve? Most of the noise I hear about LLMs is about expectations that things will improve. It's always like that: people extrapolate and get money from VCs for that. The thing is: nobody can know. So when someone extrapolates and says it's likely to happen as they predict, they shouldn't be surprised to get answers that say "I don't have the same beliefs".

tomcar288 2y ago

when people's entire livehoods are threatened, you're going to see some defensive reactions.

eschaton 2y ago

They shouldn’t be expected to improve in accuracy because of what they are and how they work. Contrary to what the average HackerNews seems to believe, LLMs don’t “think,” they just predict. And there’s nothing in them that will constrain their token prediction in a way that improves accuracy.

rybosworld 2y ago
> Contrary to what the average HackerNews seems to believe, LLMs don’t “think,” they just predict. Anecdotally, I can't recall ever seeing someone on HackerNews accuse LLM's of thinking. This site is probably one of the most educated corners of the internet on the topic. > They shouldn’t be expected to improve in accuracy because of what they are and how they work. > And there’s nothing in them that will constrain their token prediction in a way that improves accuracy. These are both incorrect. LLM's are already quite better today than they were in 2022.
- reshlo 2y ago
  > I can’t recall ever seeing someone on HackerNews accuse LLMs of thinking. There are definitely people here who think LLMs are conscious. https://news.ycombinator.com/item?id=41959163 https://news.ycombinator.com/item?id=41959163
BlackSwanMan 2y ago
[dead]
cozzyd 2y ago
If anything, they may regress due to being trained on lower-quality input.

nicoburns 2y ago

> > LLM’s never provide code that pass my sniff test > This is ego speaking. That's been my experience of LLM-generated code that people have submitted to open source projects I work on. It's all been crap. Some of it didn't even compile. Some of it changed comments that were previously correct to say something untrue. I've yet to see a single PR that implemented something useful.

constantcrying 2y ago
Terrible software engineers sent you terrible code. A good software engineer will obviously send you good quality code, whether he used AI for that is impossible to tell on your end.
mh- 2y ago
Isn't this a kind of survivor bias? You wouldn't know if you approved (undisclosed) LLM-generated code that was good..
- talldayo 2y ago
  Good LLM-generated code comes from good developers that can provide detailed requests, filter through wrong/misleading/cruft results, and test a final working copy. If your personal bar for PRs is high enough then it won't matter if you use AI or not because you won't contribute shitty results. I think the problem has become people with no bar for quality submitting PRs that ruin a repo, and then getting mad when it's not merged because ChatGPT reviewed the patch already.
cies 2y ago
> LLM-generated code that people have submitted to open source projects I work on Are you sure it was people? Maybe the AI learned how to make PRs, or is learning how to do so by using your project as a test bed.

sitzkrieg 2y ago

ego? LLMs goof on basic math and cant even generate code for many non public things. theyre not useful to me whatsoever

agilob 2y ago
LLMs aren't supposed to do basic math, but be chat agents. Wolfram Alpha can't do chat.
- simianparrot 2y ago
  Math is a major part of programming. In fact programming without math is impossible. And if you go all the way down to bare metal it’s all math. We are shifting bits through incredibly complex abstractions.
  - constantcrying 2y ago
    Has never been true in my experience, almost all programming can be done without any mathematics. Of course very specific things do require mathematics, but they are uncommon and often solved by specialists. Btw. I come from a math background and later went into programming.
  - agilob 2y ago
    No, math is major part of writing good code, but when was the last time you've seen somebody put effort into writing O(n) algorithm? 99% of programming is "import sort from sort; sort.sortThisReallyQuick". Programming is mostly writing code that just compiles and eventually gives correct results (and has bugs). You can do a lot of programming just buy copy-pasting results from stackoverflow. https://en.wikipedia.org/wiki/Npm_left-pad_incident https://en.wikipedia.org/wiki/Npm_left-pad_incident https://old.reddit.com/r/web_design/comments/35prfv/designer_applies_for_js_job_fails_at_fizzbuzz/ https://old.reddit.com/r/web_design/comments/35prfv/designer... https://www.youtube.com/watch?v=GC-0tCy4P1U https://www.youtube.com/watch?v=GC-0tCy4P1U
    - 2y ago

IshKebab 2y ago

> Is there some expectation that these things won't improve? I definitely expect them to improve. But I also think the point at which they can actually replace a senior programmer is pretty much the exact point at which they can replace any knowledge worker, at which point western society (possibly all society) is in way deeper shit than just me being out of a job. > This is ego speaking. It definitely isn't. LLMs are useful for coding now, but they can't really do the whole job without help - at least not for anything non-trivial.

dingnuts 2y ago
> LLMs are useful for coding now *sort of, sometimes, with simple enough problems with sufficiently little context, for code that can be easily tested, and for which sufficient examples exist in the training data. I mean hey, two years after being promised AGI was literally here, LLMs are almost as useful as traditional static analysis tools! I guess you could have them generate comments for you based on the code as long as you're happy to proofread and correct them when they're wrong. Remember when CPUs were obsolete after three years? GPT has shown zero improvement in its ability to generate novel content since it was first released as GPT2 almost ten years ago! I would know because I spent countless hours playing with that model.
- margalabargala 2y ago
  Firstly, GPT-2 was released in 2019. Five years is not "almost ten years". Secondly, LLMs are objectively useful for coding now. That's not the same thing as saying they are replacements for SWEs. They're a tool, like syntax highlighting or real-time compiler error visibility or even context-aware keyword autocompletion. Some individuals don't find those things useful, and prefer to develop in a plain text editor that does not have those features, and that's fine. But all of those features, and LLMs are now on that list, are broadly useful in the sense that they generally improve productivity across the industry. They already right now save enormous amounts of developer time, and to ignore that because you are not one of the people whose time is currently being saved, indicates that you may not be keeping up with understanding the technology of your field. There's an important difference between a tool being useful for generating novel content, and a tool being useful. I can think of a lot of useful tools that are not useful for generating novel content.

gspencley 2y ago

>> > LLM’s never provide code that pass my sniff test > This is ego speaking. Consider this, 100% of AI training data is human-generated content. Generally speaking, we apply the 90/10 rule to human generated content: 90% of (books, movies, tv shows, software applications, products available on Amazon) is not very good. 10% shines. In software development, I would say it's more like 99 to 1 after working in the industry professionally for over 25 years. How do I divorce this from my personal ego? It's easy to apply objective criteria: - Is the intent of code easy to understand? - Are the "moving pieces" isolated, such that you can change the implementation of one with minimal risk of altering the others by mistake? - Is the solution in code a simple one relative to alternatives? The majority of human produced code does not pass the above sniff test. Most of my job, as a Principal on a platform team, is cleaning up other peoples' messes and training them how to make less of a mess in the future. If the majority of human generated content fails to follow basic engineering practices that are employed in other engineering disciplines (i.e: it never ceases to amaze me how much of an uphill battle it is just to get some SWEs just to break down their work into small, single responsibility, easily testable and reusable "modules") then we can't logically expect any better from LLMs because this is what they're being trained on. And we are VERY far off from LLMs that can weigh the merits of different approaches within the context of the overall business requirements and choose which one makes the most sense for the problem at hand, as opposed to just "what's the most common answer to this question?" LLMs today are a type of magic trick. You give it a whole bunch of 1s and 0s so that you can input some new 1s and 0s and it can use some fancy proability maths to predict "based on the previous 1s and 0s, what are the statistically most likely next 1s and 0s to follow from the input?" That is useful, and the result can be shockingly impressive depending on what you're trying to do. But the limitations are so limited that the prospect of replacing an entire high-skilled profession with that magic trick is kind of a joke.

m_ke 2y ago
Your customers don't care how your code smells, as long as it solves their problem and doesn't cost an arm and a leg. A ton of huge business full of Sr Principal Architect SCRUM masters are about to get disrupted by 80 line ChatGPT wrappers hacked together by a few kids in their dorm room.

groby_b 2y ago

> Is there some expectation that these things won't improve? Sure. But the expectation is quantitative improvement - qualitative improvement has not happened, and is unlikely to happen without major research breakthroughs. LLMs are useful. They still need a lot of supervision & hand holding, and they'll continue to for a long while And no, it's not "ego speaking". It's long experience. There is fundamentally no reason to believe LLMs will take a leap to "works reliably in subtle circumstances, and will elicit requirements as necessary". (Sure, if you think SWE work is typing keys and make some code, any code, appear, then LLM are a threat)

akira2501 2y ago

> Is there some expectation that these things won't improve? Yes. The current technology is at a dead end. The costs for training and for scaling the network are not sustainable. This has been obvious since 2022 and is related to the way in which OpenAI created their product. There is no path described for moving from the current dead end technology to anything that could remotely be described as "AGI." > This is ego speaking. This is ignorance manifest.

simianparrot 2y ago

At my job I review a lot of code, and I write code as well. The only type of developer an LLM’s output comes close to is a fresh junior usually straight out of university in their first real development job, with little practical experience in a professional code-shipping landscape. And the majority of those juniors improve drastically within a few weeks or months, with handholding only at the very start and then less and less guidance. This is because I teach them to reason about their choices and approaches, to question assumptions, and thus they learn quickly that programming rarely has one solution to a problem, and that the context matters so much in determining the way forward. A human junior developer can learn from this tutoring and rarely regress over time. But the LLM’s all by design cannot and do not rewire their understanding of the problem space over time, nor do they remember examples and lessons from previous iterations to build upon. I have to handhold them forever, and they never learn. Even when they use significant parts of the existing codebase as their context window they’re still blind to the whole reality and history of the code. Now just to be clear, I do use LLM’s at my job. Just not to code. I use them to parse documents and assist users with otherwise repetitive manual tasks. I use their strength as language models to convert visual tokens parsed by an OCR to grasp the sentence structure and convert that into text segments which can be used more readily by users. At that they are incredible, even something smaller like llama 7b.

Derbasti 2y ago
That's a good observation. How would you teach an LLM the processes and conduct in your company? I suppose you'd need to replace code reviews and tutoring with prompt engineering. And hope that the next software update to the LLM won't invalidate your prompts.
- sigmarule 2y ago
  It's not very different from documentation, except it's not used for learning but rather immediate application, i.e. it's something you must include with each prompt/interaction (and likely need a way to determine which are the relevant bits to reduce token count, depending on the size of the main prompt.) In fact, this is probably one of the more important aspects of adapting LLMs for real-world use within real team/development settings that people don't do. If you provide clear and comprehensive descriptions of codebase patterns, pitfalls, practices, etc the latest models are good at adhering to them. It sounds difficult and open-ended, as this requires content beyond the scope of typical (or even sensible) internal documentation, but much of this content is captured in internal docs, discussions, tickets, PR comments, and git commit histories, and guess what's pretty great at extracting high-level insights from these types of inputs?

code_for_monkey 2y ago

I agree with you tbh, and it also just misses something huge that doesnt get brought up: its not about your sniff test, its about your bosses sniff test. Are you making 300k a year? Thats 300 thousand reasons to replace you for a short term boost in profit, companies love doing that.

simianparrot 2y ago
I'm in a leadership role and one of the primary parties responsible for hiring. So code passing my sniff test is kind of important.
- fragmede 2y ago
  To bring it back to the question that spawned the ask: One thing to future proof a career as a SWE is to get into business and leadership/management as well as learning how to program. If you're indispensable to the business for other reasons, they'll keep you around.
  - code_for_monkey 2y ago
    in that case the conversation is still kind of missing the forest for the trees, yes your career will be safer if you get into management but thats not going to play for everyone. Mathematically, it cant, not every engineer can go be a manager. What Im interested in is the stories of the people who arent going to make it.

gosub100 2y ago

The reason it's so good at "rewrite this C program in Python" is because it was trained on a huge corpus of code at GitHub. There is no such corpus of examples of more abstract commands, thus a limited amount by which it can improve.

Bjorkbat 2y ago

> I am constantly surprised how prevalent this attitude is. ChatGPT was only just released in 2022. Is there some expectation that these things won't improve? I mean, in a way, yeah. Last 10 years were basically one hype-cycle after another filled with lofty predictions that never quite panned out. Besides the fact that many of these predictions kind of fell short, there's also the perception that progress on these various things kind of ground to a halt once the interest faded. 3D printers are interesting. Sure, they have gotten incrementally better after the hype cycle died out, but otherwise their place in society hasn't changed, nor will it likely ever change. It has its utility for prototyping and as a fun hobbyist machine for making plastic toys, but otherwise I remember people saying that we'd be able to just 3D print whatever we needed rather than relying on factories. Same story with VR. We've made a lot of progress since the first Oculus came out, but otherwise their role in society hasn't changed much since then. The latest VR headsets are still as useless and still as bad for gaming. The metaverse will probably never happen. With AI, I don't want to be overly dismissive, but at the same time there's a growing consensus that pre-training scaling laws are plateauing, and AI "reasoning" approaches always seemed kind of goofy to me. I wouldn't be surprised if generative AI reaches a kind of equilibrium where it incrementally improves but improves in a way where it gets continuously better at being a junior developer but never quite matures beyond that. The world's smartest beginner if you will. Which is still pretty significant mind you, it's just that I'm not sure how much this significance will be felt. It's not like one's skillset needs to adjust that much in order to use Cursor or Claude, especially as they get better over time. Even if it made developers 50% more productive, I feel like the impact of this will be balanced-out to a degree by declining interest in programming as a career (feel like coding bootcamp hype has been dead for a while now), a lack of enough young people to replace those that are aging out, the fact that a significant number of developers are, frankly, bad at their job and gave up trying to learn new things a long time ago, etc etc. I think it really only matters in the end if we actually manage to achieve AGI, once that happens though it'll probably be the end of work and the economy as we know it, so who cares? I think the other thing to keep in mind is that the history of programming is filled with attempts to basically replace programmers. Prior to generative AI, I remember a lot of noise over low-code / no-code tools, but they were just the latest chapter in the evolution of low-code / no-code. Kind of surprised that even now in Anno Domini 2024 one can make a living developing small-business websites due to the limitations of the latest batch of website builders.

cies 2y ago

> ChatGPT was only just released in 2022. Bitcoin was released in what year? I still cannot use it for payments. No-code solutions exist since when? And still programmers work... I dont think all hyped techs are fads. For instance: we use SaaS now instead of installing software locally. This transition took the world by storm. But those tech that needs lots of ads, and lots of zealots, and make incredible promises: they usually are fads.

bigstrat2003 2y ago
> For instance: we use SaaS now instead of installing software locally. If anything, I feel like this argument works against you. If people are willing to replace locally installed software with shitty web "apps" that barely compare, why do you think they won't be willing to replace good programmers with LLMs doing a bad job simply because it's trendy?
- achierius 2y ago
  SaaS has generally been better than the local apps it replaced, particularly when you factor in 'portability'. I love local apps but it's undeniable that developers having to split their attention between platforms lowered quality by a lot. You're probably remembering the exemplars, not the bulk which half-worked and looked bad too
  - cies 2y ago
    Just look at the job market: how many programmers work in SaaS now compared to 2000?
gabriel-uribe 2y ago
Cryptocurrency more broadly can be used to send vast amounts of money, anywhere in the world for near-zero fees regardless of sovereignty. And Bitcoin is still faster and cheaper than shipping gold around.

jcranmer 2y ago

This is at least the third time in my life that we've seen a loudly-heralded purported the-end-of-programming technology. The previous two times both ended up being damp squibs that barely mention footnotes in the history of computing. Why do we expect that LLMs are going to buck this trend? It's not for accuracy--the previous attempts, when demonstrating their proof-of-concepts, actually reliably worked, whereas with "modern LLMs", virtually every demonstration manages to include "well, okay, the output has a bug here."

simianparrot 2y ago
I do seem to vaguely remember a time when there was a fair amount of noise proclaiming "visual programming is making dedicated programmers obsolete." I think the implication was that now everybody's boss could just make the software themselves or something. LLM's as a product feel practically similar, because _even if_ they could write code that worked in large enough quantities to constitute any decently complex application, the person telling them what problem to solve has to understand the problem space since the LLM's can't reason. Given that neither of those things are true, it's not much different from visual programming tools, practically speaking.
- nickd2001 2y ago
  This is a feeling I have too. However, compared to Visual Programming it's perhaps harder to dismiss? Visual programming - its pretty obvious you can't effectively or even at all step through a UML diagram with a debugger to find a problem. The code that gets generated from diagrams, with visual programming, is obviously cr*p. So the illusion doesn't last long. Whereas AI - it kind of looks OK, you can indeed debug it, its not necessarily more complex or worse than the hideously over-complex systems human teams create. Especially if the human team is mismanaged e:g original devs left, some got burnt out and became unproductive, others had to cut corners and make tech debt to hit unrealistic deadlines, other bits get outsourced to another country where the devs themselves are great but perhaps there's a language barrier or simply geographical distance means requirements and domain understanding got lost somewhere in the mix. So, I suppose I sit on the fence, AI-generated code may be terrible but is it worse than what we were making anyway? ;). In the future there are probably going to be companies run by "unwise people" that generate massive amounts of their codebase then find themselves in a hole when no-one working at the company understands the code at all. (whereas in the past perhaps they could hire back a dev they laid off on large contractor rates of pay to save the day). Seems inevitable one day the news will be of some high profile company failure and/or tanking of stock caused by a company basically not knowing what it was doing due to AI generated code.

deathanatos 2y ago

> This is ego speaking. No, it really isn't. Repeatedly, the case is that people are trying to pass off GPT's work as good without actually verifying the output. I keep seeing "look at this wonderful script GPT made for me to do X", and it does not pass code review, and is generally extremely low quality. In one example, a bash script was generated to count number SLoC changed by author; it was extremely convoluted, and after I simplified it, I noticed that the output of the simplified version differed, because the original was omitted changes that were only a single line. In another example it took several back & forths during a review to ask "where are you getting this code? / why do you think this code works, when nothing in the docs supports that?" and after several back and forths, it was admitted that GPT wrote it. The dev who wrote it would have been far better served RTFM, than a several cycle long review that ended up with most of GPT's hallucinations being stripped from the PR. Those who think LLM's output is good have not reviewed the output strenuously enough. > Is there some expectation that these things won't improve? Because randomized token generation inherently lacks actual reasoning about the behavior of the code. My code generator does not.

AlphaSite 2y ago
I think fundamentally if all you do is glue together popular OSS libraries in well understood way, then yes. You may be replaced. But really you probably could be replaced by a Wordpress plugin at that point. The moment you have some weird library that 4 people in the world know (which happens more than you’d expect) or hell even something without a lot of OSS code what exactly is an LLM going to do? How is it supposed to predict code that’s not derived from its training set? My experience thus far is that it starts hallucinating and it’s not really gotten any better at it. I’ll continue using it to generate sed and awk commands, but I’ve yet to find a way to make my life easier with the “hard bits” I want help with.
- valval 2y ago
  You ask what the LLM is going to do. It’s going to swallow the entire code base in context and allow any developer to join those 4 people in generating production grade code.
- 2y ago

surgical_fire 2y ago

> I am constantly surprised how prevalent this attitude is. ChatGPT was only just released in 2022. Is there some expectation that these things won't improve? Is there any expectations that things will? Is there more untapped great quality data that LLMs can ingest? Will a larger model perform meaningfully better? Will it solve the pervasive issue of generating plausibly sounding bullshit? I used LLMs for a while, I found them largely useless for my job. They were helpful for things I don't really need help with, and they were mostly damaging for things I actually needed. > This is ego speaking. Or maybe it was an accurate assessment for his use case, and your wishful thinking makes you think it was his ego speaking.

rybosworld 2y ago
> Is there any expectations that things will? Seems like an odd question. The answer is obviously yes: There is a very pervasive expectation that LLM's will continue to improve, and it seems odd to suggest otherwise. There is hundreds of billions of dollars being spent on AI training and that number is increasing each year. > Is there more untapped great quality data that LLMs can ingest? Why wouldn't there be? AI's are currently trained on the internet but that's obviously not the only source of data. > Will a larger model perform meaningfully better? The answer to this, is also yes. It is well established that, all else being equal, a bigger model is better than a smaller model, assuming that the smaller model hasn't already captured all of the available information.
- barrell 2y ago
  If all of this is true, then I would expect LLMs today to be in a whole different league of LLMs from two years ago. Personally I find them less helpful and worse at programming and creative writing related tasks. YMMV but count me in the camp that I think there’s better odds that LLMs are at or near their potential vs in their nascent stages.
- tobias3 2y ago
  We recently had a few submissions about this topic. Most recently Ilyas talk. Further improvement will be a research type problem. This trend was clear for a while already, but is reaching the mainstream now. The billions of dollar spend goes into scaling existing technology. If it doesn't scale anymore and becomes a resarch problem again, rational companies will not continue to invest in this area (at least without the usual research arrangements).

deegles 2y ago

LLMs as they currently exist will never yield a true, actually-sentient AI. maybe they will get better in some ways, but it's like asking if a bird will ever fly to the moon. Something else is needed.

mlboss 2y ago
A bird can fly to moon if it keeps on improving every month.
- VeejayRampay 2y ago
  it literally cannot though, unless it becomes some other form of life that doesn't need oxygen, that's the whole thing with this analogy, it's ironically suited to the discourse
  - lloeki 2y ago
    Ha I didn't even count oxygen in the mix; it could hold its breath maybe? j/k, what's for sure is that I don't see how a biological entity could ever achieve escape velocity from self-powered flight, because, well, physics. That, or they "keep improving every month" til they become evolved enough to build rockets, at which point the entire point of them being birds becomes moot.
  - simianparrot 2y ago
    It would need to: * Survive without oxygen * Propel itself without using wings * Store enough energy for the long trip and upkeep of the below * Insulation that can handle the radiation and temperature fluctuations * Have a form of skin that can withstand the pressure change from earth to space and then the moon * Have eyes that don't pop while in space It would need to become something literally extraterrestrial that has not evolved in the 3.7b+ years prior. I wouldn't say it's impossible, but if evolution ever got there that creature would be so far removed from a bird that I don't think we'd recognize it :p

bdangubic 2y ago

[flagged]

shakezooola 2y ago

>This is ego speaking. Very much so. These things are moving so quickly and agentic systems are already writing complete codebases. Give it a few years. No matter how 1337 you think you are, they are very likely to surpass you in 5-10 years.

achierius 2y ago
> agentic systems are already writing complete codebases Examples?

jazz9k 2y ago

Have you ever used LLMs to generate code? It's not good enough yet. In addition to this, most companies aren't willing to give away all off their proprietary IP and knowledge through 3rd party servers. It will be awhile before engineering jobs are at risk.

talldayo 2y ago

> This is ego speaking. I suspect you haven't seen code review at a 500+ seat company.

deleted 2y ago

[deleted]

goalonetwo 2y ago

100% and even if they might not "replace" a senior or staff SWE, they can make their job significantly easier which means that instead of requiring 5 Senior or Staff you will only need two. LLM WILL change the job market dynamics in the coming years. Engineers have been vastly overpaid over the last 10 years. There is no reason to not see a reversal to the mean here. Getting a 500k offer from a FAANG because you studied leetcode for a couple weeks is not going to fly anymore.

danenania 2y ago
“which means that instead of requiring 5 Senior or Staff you will only need two” What company ever feels they have “enough” engineers? There’s always a massive backlog of work to do. So unless you are just maintaining some frozen legacy system, why would you cut your team size down rather than doubling or tripling your output for the same cost? Especially considering all your competitors have a similar productivity boost available. If your reaction is to cut the team rather than make your product better at a much faster rate (or build more products), you will likely be outcompeted by others willing to make that investment.

tobyhinloopen 2y ago

LLMs are great for simple, common, boring things. As soon as you have something less common, it will give you widely incorrect garbage that does not make any sense. Even worse, it APPEARS correct, but it won’t work or will do something else completely. And I use 4o and o1 every day. Mostly for boilerplate and boring stuff. I have colleagues that submit ChatGPT generated code and I’ll immediately recognize it because it is just very bad. The colleague would have tested it, so the code does work, but it is always bad, weird or otherwise unusual. Functional, but not nice. ChatGPT can give very complicated solutions to things that can be solved with a one-liner.

globnomulous 2y ago

> This is ego speaking. It absolutely isn't. I have yet to find an area where LLM-generated code solves the kinds of problems I work on more reliably, effectively, of efficiently than I do. I'm also not interested in spending my mental energy on code reviews for an uncomphrehending token-prediction golem, let alone finding or fixing bugs in the code it blindly generates. That's a waste of my time and a special kind of personal hell.

epolanski 2y ago
Comments like these make me wonder whether we live in the same worlds. I'm a cursor user e.g. and Tab completion is by far the most powerful auto complete I've ever used. There's scenarios where you can do some major refactors by simply asking (extract this table to its own component while using best react practices to avoid double renders) and it does so istantly. Refactor the tests for it? Again. Semi instant. Meanwhile monocole wielding "senior" is proudly copy pasting, creating files and fixing indentation as I move to the next task. I don't expect LLMs to do the hard work, but to speed me up. And people ignoring LLMs are simply slower and less productive. It's a speed multiplier right now, not a substitute. If you complain that you don't like the code, you're not understanding the tools nor you can use them, end of story.
- mplanchard 2y ago
  Sounds like you’re working in react. The models are orders of magnitude worse at less common frameworks and/or languages. React is so boilerplate-heavy that it’s almost a perfect match for LLMs. That said, every large react codebase I’ve worked in inevitably became really complex and had a ton of little quirks and tweaks to deal with unexpected re-renders and such. I avoid the FE when I can these days, but I’d be fairly surprised if LLMs generated anything other than typical performance-footgun-heavy react
  - phist_mcgee 2y ago
    Not in my experience, claude seems great at writing idiomatic and performant react. You can even highlight code and ask it to spot performance issues and it will tell you where you can improve rendering efficiency etc.

tucnak 2y ago

[flagged]

quantadev 2y ago

Even since ChatGPT 3.5 AI coding has been astoundingly good. Like you, I'm totally baffled by developers who say it's not. And I'm not just unskilled and unable to judge. I have 35yrs experience! I've been shocked and impressed from day one, how good AI is. I've even seen Github Copilot do thing that almost resemble mind-reading insofar as predicting what I'm about to type next. It predicts some things it should NOT be able to predict unless it's seeing into the future or into parallel universes or something! And I'm only half joking when I speculate that!

whtsthmttrmn 2y ago
This just shows that time spent is not the same as wisdom.
- deleted 2y ago
  [deleted]
- quantadev 2y ago
  Even ChatGPT 3.5 had "wisdom", so you're definitely wrong.

JeremyNT 2y ago

Yeah I feel like this is just the beginning. I'm in my 40s with a pretty interesting background and I feel like maybe I'll make it to retirement. There are still mainframe programmers after all. Maintaining legacy stuff will still have a place. But I think we'll be the last generation where programmers will be this prevalent and the job market will severely contract. No/low code solutions backed by llms are going to eat away at a lot of what we do, and for traditional programming the tooling we use is going to improve rapidly and greatly reduce the number of developers needed.