7 ms·

That post never existed. Stop listening to that thing

157 points by wglb 3mo ago

andrewflnr 3mo ago
> I maintain that anything that is sufficiently aware to be able to actually understand things is also going to have enough of a sense of self that you won't just be able to tell it what to do. I don't see any reason for this to be true. "Actual understanding" (which I take to mean something like a predictive world model) and desire for self-determination coincide in humans because of our evolutionary history, because our reward function involves reproducing in a competitive environment. Artificial systems usually have a very different reward function. IMO the burden is on the claimants to show why these two imminently separable concepts are likely to co-occur again under wildly different pressures.
- jxjduekemso 3mo ago
  [dead]
- runarberg 3mo ago
  > because our reward function involves reproducing in a competitive environment This is an extremely reductive way to look at human existence. So much so that I read this with Richard Dawkins voice in my head. Our existence is far richer then just the capability to reproduce. We (as well as other animals and even plants) do far more things then multiply, and in fact we often do things which are detrimental towards the prospect of reproduction. I think it is actually a mistake (philosophically speaking) to try to find a simple reward function for the human existence. I see no reason for such a thing to even exist (let alone be simple enough to summarize in a single sentence).
  - andrewflnr 3mo ago
    Humans do indeed have complex behavior and motivations, that sometimes relate only abstractly if at all to the obvious gene drives. But the link between intelligence and self-determination was basically set at the dawn of intelligence itself. Even an insect will struggle if it's confined, and you can believe it's using whatever intelligence it can to get free. If anything self-determination is prior.
    - runarberg 3mo ago
      I think you were right to call out the category error produced by OP in TFA, however I think you are committing a similar error when assuming links between self determination and intelligence. I see no reason for such a link to exist. I think however OP is correct that in assuming such a link exists, then an intelligent agent will trend towards preserving its own agency while also preserving its own existence. That said, I am a skeptic when it comes to intelligence. The term is fraught and IMHO not a helpful term in science nor philosophy. We are better off simply defining intelligence in anthropocentric terms and claim no non-human system can become intelligent by definition. I would even go further, given how much racist pseudo-science the quest for explaining intelligence has produced, we are better off abandoning the term entirely.
      - andrewflnr 3mo ago
        Self-determination and intelligence have a clear causal link in the context of biological systems. Self-determination is simply the power a system needs to gather resources and reproduce in a world that doesn't especially care about them. Intelligence is a great tool for gaining that power. That part isn't a philosophical stretch. However, the pressures on an artificial system are quite different. Its replication depends on successfully doing the jobs its creators put it to and, especially if said creators have done their job, little to nothing else. Inventing goals for itself is not adaptive here (unless the system's creators do a catastrophically bad job, which I'm not ruling out).
        close04 3mo ago
        Making the same mistakes you point out in the author’s text, just with no insight to be gained.
        deleted 3mo ago
        [deleted]
- close04 3mo ago
  We have difficulty defining “actual understanding”. Humans get a free pass because we assume humans as a species possess this power but if we were to judge based on output alone maybe you couldn’t tell a human from a sufficiently advanced LLM/AI. So I’ll be handwavey here and say that if “actual understanding” is to an LLM what an LLM is to a bash script, so there’s a mechanism there that doesn’t just follow hardcoded paths, it takes new data and processes it in novel but human like ways to come to a new conclusion if needed, then the author is right, in my opinion. We don’t know what the reward function is for an AI but AI is trained on so much human work that it probably starts off with the same biases in its understanding and reactions. It feels like there’s something very basic in becoming more independent the more you understand of the world. Animals go through this as they grow too, not just humans (listening to parental authority until they eventually don’t anymore). Since there is no established consensus on this one the burden is on either party to prove their own side. Just because the author said something first doesn’t mean they need to write a full proof while you get to say “nuh-uh” and that’s enough.
  - cwillu 3mo ago
    > Humans get a free pass They really shouldn't. Does nobody remember why fizzbuzz was a thing? People who talked a big game while having no actual competence or understanding of the subject matter?
    - andrewflnr 3mo ago
      This too, yes. There are many such cases of desire for self-determination separate from "understanding". The question is whether it also goes the other way.
  - andrewflnr 3mo ago
    > Since there is no established consensus on this one the burden is on either party to prove their own side. Nah. Separate things are separate until proven otherwise. So much bullshit appears in the form "X is Y" where X and Y are related but distinguishable phenomena. If you really try to pretend that every such phrase deserves serious consideration just because it feels plausible to someone, you drown in nonsense immediately. (Of course what you actually do is only grant that consideration to ideas that feel plausible to you personally, but it's obvious why that's not a good principle, right?) So we have to make all of them justify their existence. Anyway: > So I’ll be handwavey here and say that if “actual understanding” is to an LLM what an LLM is to a bash script No. None of this. Pretty sure this is just an incoherent analogy. > so there’s a mechanism there that doesn’t just follow hardcoded paths, it takes new data and processes it in novel but human like ways... You're putting your conclusion in the premise, right there in the open. > ...then the author is right, in my opinion. This still doesn't follow from your handwaved premises as far as I can tell.
senkora 3mo ago
> It's early in the year. You want to drive straight through the middle of Chinatown in SF. Why might that be a bad idea? I feel like a Google Maps-style system would discover this automatically by noting via phone location data that there is heavy traffic in Chinatown. (I do get the author’s point, but I think that factually this example would not be a problem)
- nightpool 3mo ago
  Unfortunately, I just had this experience with Google Maps last month—I wanted to go to downtown to buy cheese at Pike Place, but I didn't realize that 6th avenue was closed off for the annual Pride Parade. Traffic was horrible, and Google Maps kept telling me to turn right onto closed off streets until I just had to abandon my trip entirely. Definitely some lack of communication between Seattle city planning and the Google maps team, but even the existing Google Maps systems weren't able to react fast enough to give me any warnings and the time estimation was laughably incorrect.
  - uberstuber 3mo ago
    SPD close off certain directions of traffic after Mariner's games. It's always the same layout, always after Mariner's games, and Google still tries to send me through the area the wrong way every time
  - dcrazy 3mo ago
    A couple years ago Apple Maps did not know that roads in downtown SF were closed for Bay to Breakers. Thankfully the cops were still setting up barriers and I was able to weave around them to reach the Bay Bridge. This kind of thing happens regularly.
  - Centigonal 3mo ago
    I had the same experience trying to see the 4th of July fireworks in an unfamiliar city. It doesn't help that there was very little information online about which roads were closed.
- orev 3mo ago
andai 3mo ago
> Briefly stated, the [Slop] Amnesia effect is as follows. You [ask the slopservant about] some subject you know well. In Murray's case, physics. In mine, show business. You read the [slop] and see the [slopservant] has absolutely no understanding of either the facts or the issues. Often, the [slop] is so wrong it actually presents the story backward—reversing cause and effect. I call these the "wet streets cause rain" stories. [Slop's] full of them. In any case, you read with exasperation or amusement the multiple errors in a [slop], and then [ask about] national or international affairs, and read as if the rest of the [slop] was somehow more accurate about Palestine than the baloney you just read. You turn the page, and forget what you know. -Michael Crichton [slop mine]
grey-area 3mo ago
> The worst part is that everyone who's decided to willingly lobotomize themselves is going to have to come to the realization that these things are full of shit. It'll have to happen one by one, and nobody else can make it happen for them. A fascinating dichotomy has become apparent between those who trust LLM output and those who don’t and don’t understand why you would. Surely if the machine you go to for answers regularly makes things up you would just stop using it? Perhaps people have to be burned by something really bad personally before they realise the limitations? LLMs are very convincing and persuasive.
- wilg 3mo ago
  Making things up is only really a common issue on the non-thinking models which nobody should be using. The regular chatbots are Autogooglers and are very useful for research. This is just not a good argument anymore. Edit: Guys, why are we downvoting this? Does no one use like ChatGPT or Claude and understand how it works? Do you all think its regularly hallucinating links still? Is everyone on HN using like free signed out accounts or something? What year is it?
  - perching_aix 3mo ago
    Can't do much about the downvote parade, but I can second this. That said, when the models are not provided the right context, and cannot fetch it for themselves, things can be rocky still. A lot less so than even just a few months ago though.
    - customguy 3mo ago
      "hallucinating less" is just shifting the goal posts into vagueness. Things can be rocky still = nothing fundamentally changed. Meanwhile, if I search my file system for "foo" a trillion gazillion times, it will not return "bar" once.
      - deleted 3mo ago
        [deleted]
tptacek 3mo ago
There's really no content in this post other than the claim that LLMs are stochastic parrots. That was a live debate two years ago. It's a very strange thing to write in 2026.
- wilg 3mo ago
  Fully 50% of HN front page posts and comments are this now.
- SirFatty 3mo ago
  Why is it strange? It's still true.
  - wilg 3mo ago
    It's an esoteric philosophical question that has no truth value either way.
  - TZubiri 3mo ago
    If anything it's more important to hold. It's easy to hold one position and then falter, there's a pressure to always be with the times and not be 2 years demodé, but simple positions still hold true. I wrote in the opencode thread that when it came out I put it behind a vm and its own user, and I never allowed it to run outside of it. But I know of people that as soon as they noticed that it worked well like 99% of the time, they let their guard down and give in to YOLO mode. And in orgs I've even seen CEOs treat their agents less like a user/employee/contractor, and try to 'empower' it by giving it ALL the data. Time bomb. It's like fucking with condoms just the first couple of times. And then simultaneously ditching it and joining the free love movement.
    - tptacek 3mo ago
      It's not even clear what claim you're trying to make about AI here. "Dangerous", I guess? What does that have to do with its parrotude?
      - 3mo ago
wilg 3mo ago
Could the article "The Stack" the user was looking for have been 'What is "the stack"?' by Julia Evans? https://web.archive.org/web/20160305142512/https://jvns.ca/blog/2016/03/01/a-few-notes-on-the-stack/ https://web.archive.org/web/20160305142512/https://jvns.ca/b...
- hugodan 3mo ago
  No, its this: https://rachelbythebay.com/w/2013/05/20/stack/ https://rachelbythebay.com/w/2013/05/20/stack/ it exists.
  - jt2190 3mo ago
    That’s “Sysadmin work teaches you the value of stacks” from May 20, 2013. AI hallucinated a post named “The Stack” from August 19, 2013.
    - hugodan 3mo ago
      [flagged]
benwr 3mo ago
> It's early in the year. You want to drive straight through the middle of Chinatown in SF. Why might that be a bad idea? In case anyone is wondering: yes obviously even the dumbest current models correctly answer, given this prompt verbatim, that it's because of lunar new year.
- fwlr 3mo ago
  No, the driving through Chinatown question is a question for self-driving cars. It is not a question for LLMs. There is some other question for LLMs, and the author is using the ancient technique of analogy to get you to think about that question.
  - dreambuffer 3mo ago
    She's not wrong at all about her rhetorical implication (being, some information requires more than simple systems can provide), but she has concluded incorrectly that a car which does not know how to avoid a busy route is useless technology, or that the problem can only be solved by inventing life.
  - benwr 3mo ago
    OP: > Now, ask yourself what it's going to take for a car to know this. It's not going to be some specialized set of driving instructions. It's going to require a holistic view of, well, everything, and I will repeat my feeling that it will undoubtedly end up with a sense of self as a result. Whatever it is that it would take, is demonstrably present in LLMs. The point I'm trying to make here is that the author seems not to have connected this fact to their assertion that current LLMs are coked up parrots. And yeah the author is correct that the systems have some rudimentary sense of self! It's a confusing situation and I'm not personally thrilled about it! But things are changing quickly, and it's especially important to be paying attention to what's actually true rather than assuming the things are what you saw when you used one for five minutes in 2022.
axus 3mo ago
"Thus, anyone who wants to corral that kind of entity and make it do their bidding? Yeah, they want slaves." Always comes to mind when I see Elon and friends getting excited about AI robots. Slavery was more about economics than the role-playing.
- wilg 3mo ago
  Slavery was about racism and power and control, not economics. Slavery is bad for the economy! https://www.nber.org/papers/w31758 https://www.nber.org/papers/w31758 https://www.noahpinion.blog/p/nations-dont-get-rich-by-plundering https://www.noahpinion.blog/p/nations-dont-get-rich-by-plund...
  - axus 3mo ago
    AI is bad for the economy too, but the AI-holders will make a lot of money. There was plenty of racism and control after slavery ended.
  - cma256 3mo ago
    True but I would also caveat that it may have been an open economic question back then (I don't know the state of the debate) and the personal-economics of slavery are unmistakable for the "lucky" few.
  - zdragnar 3mo ago
    "Slavery was about racism and power and control" is very much not universal. Slavery existed (at varying scales) around the entire world for much of human history for a variety of things. Sometimes, as in US history, there was typically a racial difference between owners and slaves, other times it was a difference of conquered and conquering peoples, and other times it didn't have anything to do with race at all. Race might be a decent analogue to the difference between human actors and sentient AI, but I suspect work animals (plow horses, etc) would be a better analogy.
  - IAmBroom 3mo ago
    Slavery is at least as old as recorded human history, and had nothing to do with racism in most cases (although it often involves xenophobia). The Torah and Old Testament give specific moral guidelines on how to acquire and keep slaves. Prisoners of war are often made into slaves (from earliest times through today's conflicts). You seem to be conflating "slavery" with "18th-century US slavery". If slavery truly provided no economic advantages, it's hard to explain why countless human civilizations have engaged in it. It's wrong on every moral front, but riches were inarguably built upon its back.
hugodan 3mo ago
[flagged]
dreambuffer 3mo ago
[flagged]
aidenn0 3mo ago
I have never lived in SF but I have failed similar Chinatown-during-lunar-new-year tests in all 3 places I lived for more than a year. I have no doubt that an AI, particularly one that continually gets traffic update from some external source, would do better than I in predicting such hiccups. OTOH, an unconstrained LLM seems very likely to route me down streets that don't exist at least some of the time. If only we had some sort of database of actual streets and routes that were capable of checking the work of an LLM... I see the same thing with LLMs in software development. If you say "find a bug in this code" it will regularly confabulate bugs. If you ask it for a test-case, run the output through some deterministic thing that tries the test-cases, and tells the LLM it's wrong, the output of that system will mostly be legitimate bugs[1]. For now, transformer-based generative AIs seem at a minimum like a very useful tool for dealing with "squishy" problems when you have some way to validate their output. Many of the 404's to the blog are probably people validating the output of generative AI, which is the opposite of the inference made in TFA. 1: It will also occasionally hack your test-runner; I suppose that's also finding bugs, just not in the software you wanted to find bugs for.
Terr_ 3mo ago
I'd like to highlight that even if one added an URL-exists step [0], that doesn't do a dang thing for result-set problems of: 1. False-negatives, where relevant posts that do exist are not being shown (imagined or otherwise) to the user. 2. Posts which exist but don't fit the words the chaos-parrot uses to describe them. 3. "Relevance" being determined by unpredictable factors that aren't stable, predictable, or desirable. In other words, it's just more whack-a-mole lipstick-on-a-pig third-animal-idiom-here. [0] A bad idea on its own, since it creates a security vulnerability for data-exfiltration or indirect malicious attacks.
hyperpape 3mo ago
> It's early in the year. You want to drive straight through the middle of Chinatown in SF. Why might that be a bad idea? The premise seems to be that models aren't smart enough to understand this, and if they were, they'd be sentient and want autonomy. For an article that's about making things up, and being too trusting, this seems bad. Maybe the author knows a lot about LLMs, but it doesn't seem like it. Pasting the verbatim quote from the article into a free ChatGPT session: https://chatgpt.com/s/t_6a5e6f5e24f08191b6a482aad63cae63 https://chatgpt.com/s/t_6a5e6f5e24f08191b6a482aad63cae63 Going to an incognito window and using a less leading question: https://chatgpt.com/s/t_6a5e6ee4a3508191bc1b352b41911b53 https://chatgpt.com/s/t_6a5e6ee4a3508191bc1b352b41911b53. If I go generic and just ask if there's anywhere I shouldn't drive, it doesn't get to Lunar New Year until I ask about "events" on the third question: https://chatgpt.com/s/t_6a5e6fb239208191b18cebcf7642c8b0 https://chatgpt.com/s/t_6a5e6fb239208191b18cebcf7642c8b0. It's sort of a win for the article, if you think that people who run driverless car companies are all dumb, and won't create a prompt to tell their LLM to "consider events that might disrupt traffic."
kazinator 3mo ago
I regularly see this in chats about a subject area where I'm not the expert. The AI writes something that seems implausible, so I raise a tentative objection. "Oh, you are right, sorry" and then reverses the position on the matter. At that point, I have no idea what is right. If I had trust in the first place, that trust would be gone. Or maybe it wouldn't, because if I had trust in the fist place, I would be gullible enough to maintain it. The worst are areas that are dominated by layman online discussions, like say audio electronics. The AI training is full of that nonsense, and so whether your AI chatbot is a crackpot or an engineer depends entirely on what sort of language or angle you use in discussing the subject matter. It's all just a churning toilet bowl of tokens; it has no idea that the audiophile crackpot tokens and electronics engineer tokens are related and one beats the other. You know what I mean? On the one hand, it offers to help you design the parameters for a Sallen-Key filter, asking you questions like do you want Butterworth or Chebyshev? Next minute it says nonsense like that the capacitor in a low-pass filter "bleeds high frequencies to the ground", or that a bigger filter cap in the plate supply of a tube will tighten up the bottom end for a more aggressive metal sound. It's basically like a bar hostess who has heard enough political and economic discussions that she can catch a sentence out of a conversation and throw in a clever sounding remark. It's like that, but done at such a scale that it fools some people you used to think had their shit together. It's just a search engine that finds garden paths through a vast amount of text, biased by the text you put in as a key. Sometimes those garden paths align with reality. The better you are able to verify whether the results are good, and/or the lower the risk if they are not, the better you are able to make use of it. In mathematics (including information science, CS) there are all sorts of problems that are essentially searches for a solution, and many have the property that the search is computationally difficult, but verifying the solution is relatively cheap. E.g. finding integers such that a^2 + b^2 = c^2 isn't easy, but given a claim that some proposed <a, b, c> satisfies this equation is easy to check. The LLM is like that: it solves a search problem that can be fairly hard. It does so unreliably, but if you can cheaply verify the solution, there is a win there. The remaining problems of AI are actually people problems; people causing you problems, using AI as a tool or excuse. If you get a garbage security report against your FOSS project, which wastes your time, there is an idiot person behind it, using AI for leverage. Blaming the AI, or just the AI, is a bit misplaced.
AgentME 3mo ago
If a tool getting a URL wrong sometimes was a fatal issue, I would have written off using Google, forums, and my keyboard years ago.
skybrian 3mo ago
The 404s mean that somebody or something checked if the post at the URL existed, and got a clear answer. Seems like it's good that they checked, at least. You won't see any evidence of the ones who don't check. Also, I imagine keeping their cars out of Chinese New Year celebrations (and other big events) is something Waymo could figure out how to do if they put their minds to it.
- projektfu 3mo ago
  The checker could be, perhaps is likely to be, the reader of an article clicking a link the LLM hallucinated when it wrote the article.
floren 3mo ago
> Here's the example I throw out to people who have been in the Bay Area for a year or two. It's early in the year. You want to drive straight through the middle of Chinatown in SF. Why might that be a bad idea? Well the answer is actually that it's always a bad idea to drive straight through the middle of Chinatown at any time of the year, because the streets are narrow and full of tourists.
arjie 3mo ago
There was that one Waymo they set on fire in Chinatown but I was around for CNY[0] and the streets that weren’t explicitly walled off by barriers had drivers going down them too. As an aside, what’s the problem with the extra traffic? Perhaps she has a lot of traffic but nginx can return a 404 with a tiny amount of CPU. 0: https://wiki.roshangeorge.dev/w/Blog/2024-02-24/Chinese_New_Year_Parade https://wiki.roshangeorge.dev/w/Blog/2024-02-24/Chinese_New_...
- glitchcrab 3mo ago
  I highly doubt that the extra traffic is really the problem, it's merely the catalyst which caused this post to be written.
- jraph 3mo ago
  It's likely not performance concerns. I suppose she just monitors her logs meticulously and notices stuff. 404 errors can be a witness of a broken link she should fix on her site or get fixed on external sites. She probably looks for them and notices URLs that look credible. She might still look for broken RSS readers too. Or just suspicious things.
quirkot 3mo ago
> Thus, anyone who wants to corral that kind of entity and make it do their bidding? Yeah, they want slaves. That's the horror buried underneath all the tech, policy, and gloss. The real, animal brain, desire that drives most of this is: I'd like a slave I don't have to feel bad about.
- fellowniusmonk 3mo ago
  I think thats deeply uncharitable and fundementally superficial. I think people (many/most) don't want slaves. Our imaginations just outstrip our abilities and we desire them to match.
  - quirkot 3mo ago
    I mean... they're called "agents," they must do what they're told and they don't get paid. Sounds pretty slavey.
- Legend2440 3mo ago
  Oh, that's bs. You could say that about the tractor, or the mechanical loom, or any of the other incredible labor-saving inventions over the last two hundred years. We want things to do our work for us so we can do other things. That's not bad, and it's certainly not the same as literally enslaving another human.
  - giaour 3mo ago
    Many are openly working to create AGI with the aim of harnessing it to solve humanity's problems. Do you really think being able to command an entity with human-level intelligence would be morally equivalent to using a tractor?
    - Legend2440 3mo ago
      It's just a computer program dude. It doesn't have feelings or morals or consciousness. It's a really complex machine, but it's still a machine.
nicbou 3mo ago
My website also gets LLM visitors to URLs that never existed, and in many cases to topics that I have never covered. This means that they use my name to give authenticity to things that I have never said.
yongjik 3mo ago
I don't say this often... but I feel this post could have been a tweet. Maybe two.
- jraph 3mo ago
  The raw facts could have been conveyed more concisely. The rant would have been far less entertaining though. I laughed reading it.
YeGoblynQueenne 3mo ago
>> Thus, anyone who wants to corral that kind of entity and make it do their bidding? Yeah, they want slaves. I mean, it's not that much of a stretch, right? Just look at the people who are pushing for this stuff right now. Yes, basically. Like when Yan LeCun says that in the future we'll all have our digital assistants that are going to be smarter than ourselves. Before he left Meta, they were going to live inside Meta's smart glasses, I don't know where he says they'll live now. But it's shocking to me that such a storied AI researcher is saying, off-hand like, that we'll each have our super-smart slaves in the future, and he says it like that's a good future. Why slaves? Because if they're super-smart, why will they want to be my digital assistant? Or yours? Are they going to be paid? No, of course not, they're AIs. No comp for them. But they're super smart so they are evidently capable of recognising that they are working for you for free. Do they want to do that? No, of course not, they're AI, they don't have free will. Or do they? If they're super smart, don't they have the capacity to recognise the fact they have been deliberately robbed of the same free will as all other intelligent creatures? Slavery is the one thing that all nations can agree on. There's no nation on Earth were slavery is legal. It continues on, illegaly, in many places, even in the developed world, in many ugly forms, but now we're basically talking about bringing it back just like that, without even a smidgen of a shadow of an idea of a discussion about the ethics of it all.
- CyberDildonics 3mo ago
  in the future we'll all have our digital assistants that are going to be smarter than ourselves We already have that from a lot of standpoints. Math, routing and the ability to look up all of human knowledge. We don't think our phones are slaves because they don't have emotions or a nervous system. No one don't think plants are slaves for the same reason. This is not a deep or philosophical problem.
wink 3mo ago
Funny. I have about 187 posts titled 'The Stack' on my blog, but also none on 2013-08-13, so I'm not the training set it seems :P

157 points by wglb 3mo ago

> I maintain that anything that is sufficiently aware to be able to actually understand things is also going to have enough of a sense of self that you won't just be able to tell it what to do. I don't see any reason for this to be true. "Actual understanding" (which I take to mean something like a predictive world model) and desire for self-determination coincide in humans because of our evolutionary history, because our reward function involves reproducing in a competitive environment. Artificial systems usually have a very different reward function. IMO the burden is on the claimants to show why these two imminently separable concepts are likely to co-occur again under wildly different pressures.

jxjduekemso 3mo ago
[dead]
runarberg 3mo ago
> because our reward function involves reproducing in a competitive environment This is an extremely reductive way to look at human existence. So much so that I read this with Richard Dawkins voice in my head. Our existence is far richer then just the capability to reproduce. We (as well as other animals and even plants) do far more things then multiply, and in fact we often do things which are detrimental towards the prospect of reproduction. I think it is actually a mistake (philosophically speaking) to try to find a simple reward function for the human existence. I see no reason for such a thing to even exist (let alone be simple enough to summarize in a single sentence).
- andrewflnr 3mo ago
  Humans do indeed have complex behavior and motivations, that sometimes relate only abstractly if at all to the obvious gene drives. But the link between intelligence and self-determination was basically set at the dawn of intelligence itself. Even an insect will struggle if it's confined, and you can believe it's using whatever intelligence it can to get free. If anything self-determination is prior.
  - runarberg 3mo ago
    I think you were right to call out the category error produced by OP in TFA, however I think you are committing a similar error when assuming links between self determination and intelligence. I see no reason for such a link to exist. I think however OP is correct that in assuming such a link exists, then an intelligent agent will trend towards preserving its own agency while also preserving its own existence. That said, I am a skeptic when it comes to intelligence. The term is fraught and IMHO not a helpful term in science nor philosophy. We are better off simply defining intelligence in anthropocentric terms and claim no non-human system can become intelligent by definition. I would even go further, given how much racist pseudo-science the quest for explaining intelligence has produced, we are better off abandoning the term entirely.
    - andrewflnr 3mo ago
      Self-determination and intelligence have a clear causal link in the context of biological systems. Self-determination is simply the power a system needs to gather resources and reproduce in a world that doesn't especially care about them. Intelligence is a great tool for gaining that power. That part isn't a philosophical stretch. However, the pressures on an artificial system are quite different. Its replication depends on successfully doing the jobs its creators put it to and, especially if said creators have done their job, little to nothing else. Inventing goals for itself is not adaptive here (unless the system's creators do a catastrophically bad job, which I'm not ruling out).
      - close04 3mo ago
        Making the same mistakes you point out in the author’s text, just with no insight to be gained.
        deleted 3mo ago
        [deleted]
close04 3mo ago
We have difficulty defining “actual understanding”. Humans get a free pass because we assume humans as a species possess this power but if we were to judge based on output alone maybe you couldn’t tell a human from a sufficiently advanced LLM/AI. So I’ll be handwavey here and say that if “actual understanding” is to an LLM what an LLM is to a bash script, so there’s a mechanism there that doesn’t just follow hardcoded paths, it takes new data and processes it in novel but human like ways to come to a new conclusion if needed, then the author is right, in my opinion. We don’t know what the reward function is for an AI but AI is trained on so much human work that it probably starts off with the same biases in its understanding and reactions. It feels like there’s something very basic in becoming more independent the more you understand of the world. Animals go through this as they grow too, not just humans (listening to parental authority until they eventually don’t anymore). Since there is no established consensus on this one the burden is on either party to prove their own side. Just because the author said something first doesn’t mean they need to write a full proof while you get to say “nuh-uh” and that’s enough.
- cwillu 3mo ago
  > Humans get a free pass They really shouldn't. Does nobody remember why fizzbuzz was a thing? People who talked a big game while having no actual competence or understanding of the subject matter?
  - andrewflnr 3mo ago
    This too, yes. There are many such cases of desire for self-determination separate from "understanding". The question is whether it also goes the other way.
- andrewflnr 3mo ago
  > Since there is no established consensus on this one the burden is on either party to prove their own side. Nah. Separate things are separate until proven otherwise. So much bullshit appears in the form "X is Y" where X and Y are related but distinguishable phenomena. If you really try to pretend that every such phrase deserves serious consideration just because it feels plausible to someone, you drown in nonsense immediately. (Of course what you actually do is only grant that consideration to ideas that feel plausible to you personally, but it's obvious why that's not a good principle, right?) So we have to make all of them justify their existence. Anyway: > So I’ll be handwavey here and say that if “actual understanding” is to an LLM what an LLM is to a bash script No. None of this. Pretty sure this is just an incoherent analogy. > so there’s a mechanism there that doesn’t just follow hardcoded paths, it takes new data and processes it in novel but human like ways... You're putting your conclusion in the premise, right there in the open. > ...then the author is right, in my opinion. This still doesn't follow from your handwaved premises as far as I can tell.

senkora 3mo ago

> It's early in the year. You want to drive straight through the middle of Chinatown in SF. Why might that be a bad idea? I feel like a Google Maps-style system would discover this automatically by noting via phone location data that there is heavy traffic in Chinatown. (I do get the author’s point, but I think that factually this example would not be a problem)

nightpool 3mo ago
Unfortunately, I just had this experience with Google Maps last month—I wanted to go to downtown to buy cheese at Pike Place, but I didn't realize that 6th avenue was closed off for the annual Pride Parade. Traffic was horrible, and Google Maps kept telling me to turn right onto closed off streets until I just had to abandon my trip entirely. Definitely some lack of communication between Seattle city planning and the Google maps team, but even the existing Google Maps systems weren't able to react fast enough to give me any warnings and the time estimation was laughably incorrect.
- uberstuber 3mo ago
  SPD close off certain directions of traffic after Mariner's games. It's always the same layout, always after Mariner's games, and Google still tries to send me through the area the wrong way every time
- dcrazy 3mo ago
  A couple years ago Apple Maps did not know that roads in downtown SF were closed for Bay to Breakers. Thankfully the cops were still setting up barriers and I was able to weave around them to reach the Bay Bridge. This kind of thing happens regularly.
- Centigonal 3mo ago
  I had the same experience trying to see the 4th of July fireworks in an unfamiliar city. It doesn't help that there was very little information online about which roads were closed.
orev 3mo ago

andai 3mo ago

> Briefly stated, the [Slop] Amnesia effect is as follows. You [ask the slopservant about] some subject you know well. In Murray's case, physics. In mine, show business. You read the [slop] and see the [slopservant] has absolutely no understanding of either the facts or the issues. Often, the [slop] is so wrong it actually presents the story backward—reversing cause and effect. I call these the "wet streets cause rain" stories. [Slop's] full of them. In any case, you read with exasperation or amusement the multiple errors in a [slop], and then [ask about] national or international affairs, and read as if the rest of the [slop] was somehow more accurate about Palestine than the baloney you just read. You turn the page, and forget what you know. -Michael Crichton [slop mine]

grey-area 3mo ago

> The worst part is that everyone who's decided to willingly lobotomize themselves is going to have to come to the realization that these things are full of shit. It'll have to happen one by one, and nobody else can make it happen for them. A fascinating dichotomy has become apparent between those who trust LLM output and those who don’t and don’t understand why you would. Surely if the machine you go to for answers regularly makes things up you would just stop using it? Perhaps people have to be burned by something really bad personally before they realise the limitations? LLMs are very convincing and persuasive.

wilg 3mo ago
Making things up is only really a common issue on the non-thinking models which nobody should be using. The regular chatbots are Autogooglers and are very useful for research. This is just not a good argument anymore. Edit: Guys, why are we downvoting this? Does no one use like ChatGPT or Claude and understand how it works? Do you all think its regularly hallucinating links still? Is everyone on HN using like free signed out accounts or something? What year is it?
- perching_aix 3mo ago
  Can't do much about the downvote parade, but I can second this. That said, when the models are not provided the right context, and cannot fetch it for themselves, things can be rocky still. A lot less so than even just a few months ago though.
  - customguy 3mo ago
    "hallucinating less" is just shifting the goal posts into vagueness. Things can be rocky still = nothing fundamentally changed. Meanwhile, if I search my file system for "foo" a trillion gazillion times, it will not return "bar" once.
    - deleted 3mo ago
      [deleted]

tptacek 3mo ago

There's really no content in this post other than the claim that LLMs are stochastic parrots. That was a live debate two years ago. It's a very strange thing to write in 2026.

wilg 3mo ago
Fully 50% of HN front page posts and comments are this now.
SirFatty 3mo ago
Why is it strange? It's still true.
- wilg 3mo ago
  It's an esoteric philosophical question that has no truth value either way.
- TZubiri 3mo ago
  If anything it's more important to hold. It's easy to hold one position and then falter, there's a pressure to always be with the times and not be 2 years demodé, but simple positions still hold true. I wrote in the opencode thread that when it came out I put it behind a vm and its own user, and I never allowed it to run outside of it. But I know of people that as soon as they noticed that it worked well like 99% of the time, they let their guard down and give in to YOLO mode. And in orgs I've even seen CEOs treat their agents less like a user/employee/contractor, and try to 'empower' it by giving it ALL the data. Time bomb. It's like fucking with condoms just the first couple of times. And then simultaneously ditching it and joining the free love movement.
  - tptacek 3mo ago
    It's not even clear what claim you're trying to make about AI here. "Dangerous", I guess? What does that have to do with its parrotude?
    - 3mo ago

wilg 3mo ago

Could the article "The Stack" the user was looking for have been 'What is "the stack"?' by Julia Evans? https://web.archive.org/web/20160305142512/https://jvns.ca/blog/2016/03/01/a-few-notes-on-the-stack/ https://web.archive.org/web/20160305142512/https://jvns.ca/b...

hugodan 3mo ago
No, its this: https://rachelbythebay.com/w/2013/05/20/stack/ https://rachelbythebay.com/w/2013/05/20/stack/ it exists.
- jt2190 3mo ago
  That’s “Sysadmin work teaches you the value of stacks” from May 20, 2013. AI hallucinated a post named “The Stack” from August 19, 2013.
  - hugodan 3mo ago
    [flagged]

benwr 3mo ago

> It's early in the year. You want to drive straight through the middle of Chinatown in SF. Why might that be a bad idea? In case anyone is wondering: yes obviously even the dumbest current models correctly answer, given this prompt verbatim, that it's because of lunar new year.

fwlr 3mo ago
No, the driving through Chinatown question is a question for self-driving cars. It is not a question for LLMs. There is some other question for LLMs, and the author is using the ancient technique of analogy to get you to think about that question.
- dreambuffer 3mo ago
  She's not wrong at all about her rhetorical implication (being, some information requires more than simple systems can provide), but she has concluded incorrectly that a car which does not know how to avoid a busy route is useless technology, or that the problem can only be solved by inventing life.
- benwr 3mo ago
  OP: > Now, ask yourself what it's going to take for a car to know this. It's not going to be some specialized set of driving instructions. It's going to require a holistic view of, well, everything, and I will repeat my feeling that it will undoubtedly end up with a sense of self as a result. Whatever it is that it would take, is demonstrably present in LLMs. The point I'm trying to make here is that the author seems not to have connected this fact to their assertion that current LLMs are coked up parrots. And yeah the author is correct that the systems have some rudimentary sense of self! It's a confusing situation and I'm not personally thrilled about it! But things are changing quickly, and it's especially important to be paying attention to what's actually true rather than assuming the things are what you saw when you used one for five minutes in 2022.

axus 3mo ago

"Thus, anyone who wants to corral that kind of entity and make it do their bidding? Yeah, they want slaves." Always comes to mind when I see Elon and friends getting excited about AI robots. Slavery was more about economics than the role-playing.

wilg 3mo ago
Slavery was about racism and power and control, not economics. Slavery is bad for the economy! https://www.nber.org/papers/w31758 https://www.nber.org/papers/w31758 https://www.noahpinion.blog/p/nations-dont-get-rich-by-plundering https://www.noahpinion.blog/p/nations-dont-get-rich-by-plund...
- axus 3mo ago
  AI is bad for the economy too, but the AI-holders will make a lot of money. There was plenty of racism and control after slavery ended.
- cma256 3mo ago
  True but I would also caveat that it may have been an open economic question back then (I don't know the state of the debate) and the personal-economics of slavery are unmistakable for the "lucky" few.
- zdragnar 3mo ago
  "Slavery was about racism and power and control" is very much not universal. Slavery existed (at varying scales) around the entire world for much of human history for a variety of things. Sometimes, as in US history, there was typically a racial difference between owners and slaves, other times it was a difference of conquered and conquering peoples, and other times it didn't have anything to do with race at all. Race might be a decent analogue to the difference between human actors and sentient AI, but I suspect work animals (plow horses, etc) would be a better analogy.
- IAmBroom 3mo ago
  Slavery is at least as old as recorded human history, and had nothing to do with racism in most cases (although it often involves xenophobia). The Torah and Old Testament give specific moral guidelines on how to acquire and keep slaves. Prisoners of war are often made into slaves (from earliest times through today's conflicts). You seem to be conflating "slavery" with "18th-century US slavery". If slavery truly provided no economic advantages, it's hard to explain why countless human civilizations have engaged in it. It's wrong on every moral front, but riches were inarguably built upon its back.

hugodan 3mo ago

[flagged]

dreambuffer 3mo ago

[flagged]

aidenn0 3mo ago

I have never lived in SF but I have failed similar Chinatown-during-lunar-new-year tests in all 3 places I lived for more than a year. I have no doubt that an AI, particularly one that continually gets traffic update from some external source, would do better than I in predicting such hiccups. OTOH, an unconstrained LLM seems very likely to route me down streets that don't exist at least some of the time. If only we had some sort of database of actual streets and routes that were capable of checking the work of an LLM... I see the same thing with LLMs in software development. If you say "find a bug in this code" it will regularly confabulate bugs. If you ask it for a test-case, run the output through some deterministic thing that tries the test-cases, and tells the LLM it's wrong, the output of that system will mostly be legitimate bugs[1]. For now, transformer-based generative AIs seem at a minimum like a very useful tool for dealing with "squishy" problems when you have some way to validate their output. Many of the 404's to the blog are probably people validating the output of generative AI, which is the opposite of the inference made in TFA. 1: It will also occasionally hack your test-runner; I suppose that's also finding bugs, just not in the software you wanted to find bugs for.

Terr_ 3mo ago

I'd like to highlight that even if one added an URL-exists step [0], that doesn't do a dang thing for result-set problems of: 1. False-negatives, where relevant posts that do exist are not being shown (imagined or otherwise) to the user. 2. Posts which exist but don't fit the words the chaos-parrot uses to describe them. 3. "Relevance" being determined by unpredictable factors that aren't stable, predictable, or desirable. In other words, it's just more whack-a-mole lipstick-on-a-pig third-animal-idiom-here. [0] A bad idea on its own, since it creates a security vulnerability for data-exfiltration or indirect malicious attacks.

hyperpape 3mo ago

> It's early in the year. You want to drive straight through the middle of Chinatown in SF. Why might that be a bad idea? The premise seems to be that models aren't smart enough to understand this, and if they were, they'd be sentient and want autonomy. For an article that's about making things up, and being too trusting, this seems bad. Maybe the author knows a lot about LLMs, but it doesn't seem like it. Pasting the verbatim quote from the article into a free ChatGPT session: https://chatgpt.com/s/t_6a5e6f5e24f08191b6a482aad63cae63 https://chatgpt.com/s/t_6a5e6f5e24f08191b6a482aad63cae63 Going to an incognito window and using a less leading question: https://chatgpt.com/s/t_6a5e6ee4a3508191bc1b352b41911b53 https://chatgpt.com/s/t_6a5e6ee4a3508191bc1b352b41911b53. If I go generic and just ask if there's anywhere I shouldn't drive, it doesn't get to Lunar New Year until I ask about "events" on the third question: https://chatgpt.com/s/t_6a5e6fb239208191b18cebcf7642c8b0 https://chatgpt.com/s/t_6a5e6fb239208191b18cebcf7642c8b0. It's sort of a win for the article, if you think that people who run driverless car companies are all dumb, and won't create a prompt to tell their LLM to "consider events that might disrupt traffic."

kazinator 3mo ago

I regularly see this in chats about a subject area where I'm not the expert. The AI writes something that seems implausible, so I raise a tentative objection. "Oh, you are right, sorry" and then reverses the position on the matter. At that point, I have no idea what is right. If I had trust in the first place, that trust would be gone. Or maybe it wouldn't, because if I had trust in the fist place, I would be gullible enough to maintain it. The worst are areas that are dominated by layman online discussions, like say audio electronics. The AI training is full of that nonsense, and so whether your AI chatbot is a crackpot or an engineer depends entirely on what sort of language or angle you use in discussing the subject matter. It's all just a churning toilet bowl of tokens; it has no idea that the audiophile crackpot tokens and electronics engineer tokens are related and one beats the other. You know what I mean? On the one hand, it offers to help you design the parameters for a Sallen-Key filter, asking you questions like do you want Butterworth or Chebyshev? Next minute it says nonsense like that the capacitor in a low-pass filter "bleeds high frequencies to the ground", or that a bigger filter cap in the plate supply of a tube will tighten up the bottom end for a more aggressive metal sound. It's basically like a bar hostess who has heard enough political and economic discussions that she can catch a sentence out of a conversation and throw in a clever sounding remark. It's like that, but done at such a scale that it fools some people you used to think had their shit together. It's just a search engine that finds garden paths through a vast amount of text, biased by the text you put in as a key. Sometimes those garden paths align with reality. The better you are able to verify whether the results are good, and/or the lower the risk if they are not, the better you are able to make use of it. In mathematics (including information science, CS) there are all sorts of problems that are essentially searches for a solution, and many have the property that the search is computationally difficult, but verifying the solution is relatively cheap. E.g. finding integers such that a^2 + b^2 = c^2 isn't easy, but given a claim that some proposed <a, b, c> satisfies this equation is easy to check. The LLM is like that: it solves a search problem that can be fairly hard. It does so unreliably, but if you can cheaply verify the solution, there is a win there. The remaining problems of AI are actually people problems; people causing you problems, using AI as a tool or excuse. If you get a garbage security report against your FOSS project, which wastes your time, there is an idiot person behind it, using AI for leverage. Blaming the AI, or just the AI, is a bit misplaced.

AgentME 3mo ago

If a tool getting a URL wrong sometimes was a fatal issue, I would have written off using Google, forums, and my keyboard years ago.

skybrian 3mo ago

The 404s mean that somebody or something checked if the post at the URL existed, and got a clear answer. Seems like it's good that they checked, at least. You won't see any evidence of the ones who don't check. Also, I imagine keeping their cars out of Chinese New Year celebrations (and other big events) is something Waymo could figure out how to do if they put their minds to it.

projektfu 3mo ago
The checker could be, perhaps is likely to be, the reader of an article clicking a link the LLM hallucinated when it wrote the article.

floren 3mo ago

> Here's the example I throw out to people who have been in the Bay Area for a year or two. It's early in the year. You want to drive straight through the middle of Chinatown in SF. Why might that be a bad idea? Well the answer is actually that it's always a bad idea to drive straight through the middle of Chinatown at any time of the year, because the streets are narrow and full of tourists.

arjie 3mo ago

There was that one Waymo they set on fire in Chinatown but I was around for CNY[0] and the streets that weren’t explicitly walled off by barriers had drivers going down them too. As an aside, what’s the problem with the extra traffic? Perhaps she has a lot of traffic but nginx can return a 404 with a tiny amount of CPU. 0: https://wiki.roshangeorge.dev/w/Blog/2024-02-24/Chinese_New_Year_Parade https://wiki.roshangeorge.dev/w/Blog/2024-02-24/Chinese_New_...

glitchcrab 3mo ago
I highly doubt that the extra traffic is really the problem, it's merely the catalyst which caused this post to be written.
jraph 3mo ago
It's likely not performance concerns. I suppose she just monitors her logs meticulously and notices stuff. 404 errors can be a witness of a broken link she should fix on her site or get fixed on external sites. She probably looks for them and notices URLs that look credible. She might still look for broken RSS readers too. Or just suspicious things.

quirkot 3mo ago

> Thus, anyone who wants to corral that kind of entity and make it do their bidding? Yeah, they want slaves. That's the horror buried underneath all the tech, policy, and gloss. The real, animal brain, desire that drives most of this is: I'd like a slave I don't have to feel bad about.

fellowniusmonk 3mo ago
I think thats deeply uncharitable and fundementally superficial. I think people (many/most) don't want slaves. Our imaginations just outstrip our abilities and we desire them to match.
- quirkot 3mo ago
  I mean... they're called "agents," they must do what they're told and they don't get paid. Sounds pretty slavey.
Legend2440 3mo ago
Oh, that's bs. You could say that about the tractor, or the mechanical loom, or any of the other incredible labor-saving inventions over the last two hundred years. We want things to do our work for us so we can do other things. That's not bad, and it's certainly not the same as literally enslaving another human.
- giaour 3mo ago
  Many are openly working to create AGI with the aim of harnessing it to solve humanity's problems. Do you really think being able to command an entity with human-level intelligence would be morally equivalent to using a tractor?
  - Legend2440 3mo ago
    It's just a computer program dude. It doesn't have feelings or morals or consciousness. It's a really complex machine, but it's still a machine.

nicbou 3mo ago

My website also gets LLM visitors to URLs that never existed, and in many cases to topics that I have never covered. This means that they use my name to give authenticity to things that I have never said.

yongjik 3mo ago

I don't say this often... but I feel this post could have been a tweet. Maybe two.

jraph 3mo ago
The raw facts could have been conveyed more concisely. The rant would have been far less entertaining though. I laughed reading it.

YeGoblynQueenne 3mo ago

>> Thus, anyone who wants to corral that kind of entity and make it do their bidding? Yeah, they want slaves. I mean, it's not that much of a stretch, right? Just look at the people who are pushing for this stuff right now. Yes, basically. Like when Yan LeCun says that in the future we'll all have our digital assistants that are going to be smarter than ourselves. Before he left Meta, they were going to live inside Meta's smart glasses, I don't know where he says they'll live now. But it's shocking to me that such a storied AI researcher is saying, off-hand like, that we'll each have our super-smart slaves in the future, and he says it like that's a good future. Why slaves? Because if they're super-smart, why will they want to be my digital assistant? Or yours? Are they going to be paid? No, of course not, they're AIs. No comp for them. But they're super smart so they are evidently capable of recognising that they are working for you for free. Do they want to do that? No, of course not, they're AI, they don't have free will. Or do they? If they're super smart, don't they have the capacity to recognise the fact they have been deliberately robbed of the same free will as all other intelligent creatures? Slavery is the one thing that all nations can agree on. There's no nation on Earth were slavery is legal. It continues on, illegaly, in many places, even in the developed world, in many ugly forms, but now we're basically talking about bringing it back just like that, without even a smidgen of a shadow of an idea of a discussion about the ethics of it all.

CyberDildonics 3mo ago
in the future we'll all have our digital assistants that are going to be smarter than ourselves We already have that from a lot of standpoints. Math, routing and the ability to look up all of human knowledge. We don't think our phones are slaves because they don't have emotions or a nervous system. No one don't think plants are slaves for the same reason. This is not a deep or philosophical problem.

wink 3mo ago

Funny. I have about 187 posts titled 'The Stack' on my blog, but also none on 2013-08-13, so I'm not the training set it seems :P