26 ms·

I am sympathetic to the reasoning as to why LLMs should not be used to help some programmers right now. But I get a little frustrated seeing many of these kinds

• libraryofbabel 1y ago
  Yeah. The empty “it’s just a statistical model” critique (or the dressed-up “stochastic parrots” version of it) is almost a sign at this point that the person using it formed their opinions about AI back when ChatGPT first came out, and hasn’t really bothered to engage with it much since then. If in 2022 I’d tried to convince AI skeptics that in three years we might have tools on the level of Claude Code, I’m sure I’d have heard everyone say it would be impossible because “it’s just a statistical model.” But it turned out that there was a lot more potential in the architecture for encoding structured knowledge, complex reasoning, etc., despite that architecture being probabilistic. (Don’t bet against the Bitter Lesson.) LLMs have a lot of problems, hallucination still being one of them. I’d be the first to advocate for a skeptical hype-free approach to deploying them in software engineering. But at this point we need careful informed engagement with where the models are at now rather than cherry-picked examples and rants.
  • vidarh 1y ago
    People repeating the "stochastic parrot" meme in all kinds of variations if anything appear to be more like stochastic parrots than the typical LLM is.
  • seba_dos1 1y ago
    Unless what you work on is very simple and mostly mindless, using tools like Claude Code is the exact opposite of how to make the current SotA LLMs useful for coding. The models can help and boost your productivity, but it doesn't happen by letting them do more stuff autonomously. Quite the contrary. And when what you usually work on actually is very simple and mostly mindless, you'd probably benefit more from doing it yourself, so you can progress beyond the junior stuff one day.
    • structural 1y ago
      Where it really has value is if what you work on is like 33% extremely difficult and 66% boilerplate/tedious. Being able to offload the tedious bits can make more senior engineers 2-3x more productive without the coordination effort of "find a junior engineer to do this, schedule their time, assign the work, follow up on it". (The problem of course is you still need these junior engineers to exist in order to have the next generatino of senior engineers, so we must now also think about what our junior folks should be doing to be valuable AND learn).
      • deleted 1y ago
        [deleted]
    • anuramat 1y ago
      "real programmers use ed", 2025 edition: you're forever stuck with "junior stuff" if you let a language model handle language god forbid I don't have to read 10k lines of logs to fix a typo
  • Jensson 1y ago
    > If in 2022 I’d tried to convince AI skeptics that in three years we might have tools on the level of Claude Code, I’m sure I’d have heard everyone say it would be impossible because “it’s just a statistical model.” We already had these coding models in 2022, they were already pattern matching engines with variables back then, all your imagination needed to do to go from that to Claude code today is to give it more code examples and make it bigger. They still can't replace even a junior engineers ability to navigate tasks over even short periods of time, just like back then they need constant handholding to get anything done. So I don't see what changed except the model being larger with more examples and therefore you can get larger chunks of coherent code out of them.
• raincole 1y ago
  While the normal distribution meme is notoriously overused, I think it fits the scenario here. LLMs know so much (when you just use ChatGPT for the first time like it's an Oracle machine) -> LLMs don't know anything (when you understand how machine learning works) -> LLMs know so much (when you actually think about what 'know' means)
• bwfan123 1y ago
  Humans build theories of how things work. llms dont. Theories are deterministic symbolic representation of the chaotic worlds of meaning . Take the turing machine for example as a theory of computation in general, euclidean geometry as a theory for space, and newtonian mechanics as a theory for motion. A theory gives 100% correct predictions. Although the theory itself may not model the world accurately. Such feedback between the theory, and its application in the world causes iterations to the theory. From newtonian mechanics to relativity etc. Long story short, the LLM is a long way away from any of this. And to be fair to LLMs, the average human is not creating theories, it takes some genius to create them (newton, turing, etc). Understanding something == knowing the theory of it.
  • hodgehog11 1y ago
    > Humans build theories of how things work. llms dont. Theories are deterministic symbolic representation of the chaotic worlds of meaning What made you believe this is true? Like it or not, yes, they do (at least to the best extent of our definitions of what you've said). There is a big body of literature exploring this question, and the general consensus is that all performant deep learning models adopt an internal representation that can be extracted as a symbolic representation.
    • bwfan123 1y ago
      > What made you believe this is true? I am yet to see a theory coming of the LLM that is sufficiently interesting. My comment was answering your question of what does it mean to "understanding something". My answer to that is: understanding something is knowing the theory of it. Now, that begs the question of what is a theory. And to answer that, a theory comprises of building block symbols and a set of rules to combine them. for example, building blocks for space (and geometry) could be points, lines, etc. The key point in all of this is symbolism as abstractions to represent things in some world.
      • hodgehog11 1y ago
        The "sufficiently interesting" part is the most important qualifier here. My response was talking about theories and representations that we already know, either instinctively from near-birth, or from learned experience. We have not seen anything unique from LLMs because they do not appear to have an internal understanding (in the same sense that I was talking about) that is as broad as an adult human. But that doesn't mean it lacks any understanding. > The key point in all of this is symbolism as abstractions to represent things in some world. The difficulty is understanding how to extract this information from the model, since the output of the LLM is actually a very poor representation of its internal state.
• efilife 1y ago
  > it cannot "logically reason" like a human does Reason? Maybe. But there's one limitation that we currently have no idea how to overcome; LLMs don't know how much they know. If they tell you they don't something it may be a lie. If they tell you they do, this may be a lie too. I, a human, certainly know what I know and what I don't and can recall from where I know the information
  • vidarh 1y ago
    I have never met a human who has a good grasp of what they know and don't know. They may have a better graps of it than an LLM, but humans are awfully bad at understanding the limits of our own knowledge, and will argue very strongly in favour of knowing more than we demonstrably do in all kinds of contexts.
    • ModernMech 1y ago
      LLMs are not humans, they are ostensibly tools. Tools are supposed to come with a list of things they can do. LLMs don’t and are therefore bad at being tools, so we anthropomorphize them. But they are also not good at being people, so LLMs are left in this weird in-between state where half the people say they’re useful and half the people say they cause more problems than they solve.
      • vidarh 1y ago
        They are fantastic tools to me. The provide abilities that no other tools we have can provide. That they also have flaws does not remove those benefits.
        • ModernMech 1y ago
          There's no question they have capabilities that no other tool has. But a good tool goes beyond just doing something, there are some generally agreed upon principles of tool design that make something good versus just useful. For example, I think a hammer is a good tool because every time I swing it at a nail, it provides the force necessary to drive it into the wood. It's reliable. Sure, sometimes a hammer breaks, but my baseline expectation in using one is that every time I swing the hammer it will behave the same way. Something more complicated, like a Rust compiler is also a good tool in the same way. It's vastly more intricate than the hammer, yet it still has the good tool property of being reliable; every time I press compile, if the program is wrong, then the compiler tells me that. If it's right, then the compiler passes every time. It doesn't lie, it doesn't guess, it doesn't rate limit, it doesn't charge a subscription, it doesn't silently update causing code to fail, it informs when changes are breaking and what they are, it allows me to pick my version and doesn't silently deprecate me without recourse, etc. There are of course ecosystems out there where building a project is more like a delicate dance, or a courtship ritual, and those ecosystems are pain in the ass to deal with. I'm talking XKCD #1987, or NodeJS circa 2014, or just the entire rationale behind Docker. People exit their careers to not have to deal with such technology, because it's like working at the DMV or living in Kafka's nightmares. LLMs are more in that direction, and no one is going to like where we end up if we make them our entire stack, as seems to be the intent by the powers that be. There's a difference between what LLMs are and what they're being sold as. For what they are, they can be useful and may one day they will be turned into good tools if some of the major flaws are fixed. On the other hand, we are in the process of totally upending the way our industry works on the basis of what these things will be, which they are selling as essentially an oracle. "The smartest person who know in your pocket", "a simultaneous expert PhD, MD, JD", "Smarter than all humans combined". But there's a giant gulf between what they're selling and what it is, and that gulf is what makes LLMs a poor tool.
• Barrin92 1y ago
  >Humans are statistical models too in an appropriate sense. No, we aren't and I'm getting tired of this question begging and completely wrong statement. Human beings are capable of what Kant in fancy words called "transcendental apperception", we're already bringing our faculties to bear on experience without which the world would make no sense to us. What that means in practical terms for programming problems of this kind is that, we can say "I don't know", which the LLM can't, because there's no "I", in the LLM, no unified subject that can distinguish what it knows and what it doesn't, what's within its domain of knowledge or outside. >If you take the time to define "know", "think", and "deduct", you will find it difficult to argue current LLMs do not do these things No, only if you don't spend the time to think about what knowledge is you'd make such a statement. What enables knowledge, which is not raw data but synthesized, structured cognition, is the faculties of the mind a priori categories we bring to bear on data. That's why these systems are about as useless as a monkey with a typewriter when you try to have them work on manual memory management in C, because that's less of a task in auto completion and requires you to have in your mind a working model of the machine.
  • lblume 1y ago
    The position of Kant does not align with the direction modern neuroscience is heading towards. Current evidence seems to prefer decentralized theories of consciousness like Dennett's multiple drafts model[1], suggesting there is no central point where everything comes together to form conscious experience, but instead that it itself is constituted by collaborative processes that have multiple realizations. [1]: https://en.wikipedia.org/wiki/Multiple_drafts_model https://en.wikipedia.org/wiki/Multiple_drafts_model
    • Barrin92 1y ago
      >Current evidence seems to prefer decentralized theories of consciousness like Dennett There is no such thing as consciousness in Dennett's theory, his position is that it doesn't exist, he is a Eliminativist. This is of course an absurd position with no evidence for it as people like Chalmers have pointed out (including in that Wikipedia article), and it might be the most comical and ideological position in the last 200 years.
• bilbo-b-baggins 1y ago
  Actually internally we do know what’s going on these days. Anthropic put out a white paper detailing how Claude can’t math but many math examples are out there so Claude can fake it. I wish you’d stop magic LLMs some kind of magic thing they aren’t.
  • hodgehog11 1y ago
    I work in the theory of deep learning, so I can say with some authority that while we know a really good number of things, and are able to probe the internals much better than most of the public realises, when it comes to philosophical questions that compare their nature with humans, we have a long, long way to go. The biggest problem is that we're still trying to work out what it is we even want to know that will tell us whether we have achieved AGI or not. Linear probes and autoencoders have been useful, but we're quickly reaching the limits of those techniques. And don't even get me started on approaches to theory that operate by cherry-picked examples. Anthropic's contributions have been beneficial to the field, but are far from conclusive.