22 ms·

We Automated Bullshit

• deleted 3y ago
  [deleted]
• lqet 3y ago
  If that's true, why then did it take me 10 seconds yesterday to find an extremely obscure fact of 18th century history with ChatGPT 3.5? I tried Google queries for over an hour, without success. ChatGPT first gave a wrong answer (because my question actually wasn't accurate), but after some clarification answered something that appeared correct, and after 1-2 minutes of further research and validation, I was 100% that it was indeed the correct answer. > AI systems like ChatGPT are trained with text from Twitter, Facebook, Reddit, and other huge archives of bullshit, alongside plenty of actual facts (including Wikipedia and text ripped off from professional writers). But there is no algorithm in ChatGPT to check which parts are true. The output is literally bullshit. Well. Any philosopher, mathematician, scientist, or novelist was trained on plenty of actual facts (established works of other people, school learning, experiments) and huge archives of bullshit (individual day-to-day experiences that cannot be called scientific, casual conversations just like Twitter conversations, propaganda, lies). Their output then would also be bullshit, and by extension, everything humanity ever created.
  • mnd999 3y ago
    > ChatGPT first gave a wrong answer I think you answered your own question there. It doesn’t say your question makes no sense, or that’s incorrect. It just makes up a bullshit answer.
    • exitb 3y ago
      Most tools require some preparation, knowledge and understanding before you get good results. Would he be better off not getting the right answer in the subsequent question?
      • falserum 3y ago
        It’s not all or nothing. Most of negative sentiment is not “world would be better without chatgpt“, but “one must be very cautious and always validate the answer”. It’s valuable tool, but it has a LOT of hype, which led to things like people trusting chatgpt output in a court case.
    • empath-nirvana 3y ago
      GPT4 will definitely tell you that your question is wrong and doesn't make sense, and a lot of times it is right about that.
  • chrisvalleybay 3y ago
    Exactly. It's great for these use cases. I often use it to find things of which I barely remember some obsure detail and mostly I'm able to find it. Would never have been able to Google my way to it.
  • goda90 3y ago
    What if you didn't know enough to know the first answer was wrong? Because a lot of people are going to ask a lot of questions where that will be the case. They won't know to tweak their question and then do further research. And if you try to use GPT for automatic systems, well how is that automatic system going to recognize what's wrong vs right?
    • capableweb 3y ago
      Not knowing what you don't know, is really hard to verify. But if you get any sort of answer, it's usually pretty trivial to verify. Where I found ChatGPT to be really helpful is when I don't know anything regarding something, and ChatGPT can give me something I could at least verify as true or false. Compared to before, where I just got stuck on some things, as I couldn't find out what I was actually looking for at all.
    • ben_w 3y ago
      P vs. NP, answers are often easier to verify than to reach in the first place. That said, there are many people who don't bother to verify anything, as can be observed in comments sections where one can in many cases find people who clearly only read the headline and not the article.
    • osigurdson 3y ago
      >> What if you didn't know enough to know the first answer was wrong? Anyone who uses ChatGPT for a while will learn to take everything with a grain of salt.
      • falserum 3y ago
        s/Grain/huge rock/
      • petemir 3y ago
        Are we sure? (Some) People are still going to vote for Donald Trump in 2024, one of the "bullshit generators" (such as chatGPT) used as example in the article. (n.b. I am not trying to make any type of political judgement here, I am just using the example in TFA).
  • gostsamo 3y ago
    Because things that sound good sometimes are indeed good. So, the author disregards this fact, while everyone using ChatGPT knows that verification is part of the proccess. The final evaluation is if it is cheaper to dig the info ourselves or we can use the AI to narrow down the search space. I think the second, but only if people do the verification part.
    • lcnPylGDnU4H9OF 3y ago
      > So, the author disregards this fact, while everyone using ChatGPT knows that verification is part of the proccess. Why is verification part of the process if not because ChatGPT generates bullshit? I don’t think the author is disregarding that it is sometimes correct.
      • gostsamo 3y ago
        Because chat gpt is yet another authority and like any other authority, its claims are not perfect. Don't you check when you find random info on the internet? The difference is that the ai gives answers faster and with much vaguer prompts than a handcoded system is usually able to.
        • lcnPylGDnU4H9OF 3y ago
          > Because chat gpt is yet another authority and like any other authority, its claims are not perfect. Don't you check when you find random info on the internet? What authority does random information on the internet have? How do I verify the information without looking at random information on the internet?
    • 2muchcoffeeman 3y ago
      You don’t think that it’s a concern that verification must be done? People don’t do that now and we just made it easier to get confident answers that no one will check.
    • proto_lambda 3y ago
      > everyone using ChatGPT knows that verification is part of the proccess lol. lmao
  • lcnPylGDnU4H9OF 3y ago
    Why did you undergo the 1-2 minutes of validation?
  • ben_w 3y ago
    That which sounds true is loosely correlated with that which is true. On the other hand: "Things that try to look like things often do look more like things than things. Well-known fact," said Granny. "But I don’t hold with encouraging it." - Terry Pratchett, Wyrd Sisters This makes LLMs much better than claimed by those who dismiss it as BS or stochastic parrot or whatever, and also not as good as domain experts.
  • krapp 3y ago
    Given that ChatGPT generates responses based on the same data set that Google has access to, I find it difficult to believe you tried Google for hours and never recieved a single useful result. It also seems odd that you had the materials available to research and validate the response ChatGPT gave you, but somehow no ability to glean the necessary answer from those materials to begin with. Obviously, because Google was completely useless to you, you didn't simply use Google to do that validation, nor could you have used another search engine, because then you could have found the result you were looking for to begin with. Did you just use ChatGPT to validate its own responses?
  • rvz 3y ago
    > ChatGPT first gave a wrong answer (because my question actually wasn't accurate). but after some clarification answered something that appeared correct, and after 1-2 minutes of further research and validation, I was 100% that it was indeed the correct answer Exactly. You answered your own question and that is the author's point. Do you even know why it gave the incorrect answer at first? Not even ChatGPT can explain to you why it did that or keeps doing this. How can you trust it to give the right answer (without you Googling to check) if you really don't know the answer? It already means you don't trust it an you know it bullshits as the author has claimed. It can confidently convince someone outside of one's own expertise that it can give a soundly correct answer but can easily be bullshitting nonsense. > Well. Any philosopher, mathematician, scientist, or novelist was trained on plenty of actual facts (established works of other people, school learning, experiments) and huge archives of bullshit (individual day-to-day experiences that cannot be called scientific, casual conversations just like Twitter conversations, propaganda, lies). Their output then would also be bullshit, and by extension, everything humanity ever created. The difference is with humans, they can be transparently held to account on whatever they say. An AI system cannot. So this actual whataboutism towards humans is incredibly weak here and a common excuse by AI proponents supporting the nonsense that AI systems like ChatGPT can uncontrollably generate in a black box system.
  • mhb 3y ago
    > ChatGPT 3.5 It's a little surprising how hard it is to convince people that it's worth $20 to use v4.0. It is very apparent how much better it is.
    • Filligree 3y ago
      Especially lately. The distilled version we’re getting today actually seems to be smarter and less likely to make things up, from my tests; I suppose it’s generalising better.
  • galdosdi 3y ago
    > If that's true, why then did it take me 10 seconds yesterday to find an extremely obscure fact of 18th century history with ChatGPT 3.5? I tried Google queries for over an hour, without success. This is true, yet I wonder how much of this "ChatGPT is way better than Google at Googling" effect is due simply to how bad Google has gotten over the last decade. ChatGPT and similar models seem to be uncontroversially best at this kind of task -- just being a replacement for Google Search.
  • roflyear 3y ago
    You know, your post reads like "if that's true, why did this one thing happen once to me?" It's not extremely convincing.
  • latexr 3y ago
    > Any philosopher, mathematician, scientist, or novelist was trained on (…). Their output then would also be bullshit, and by extension, everything humanity ever created. Humans don’t learn the same way AIs are trained. Equating the two is a fallacy.
  • falserum 3y ago
    > why then did it take me 10 seconds yesterday It definitely took you more than 10 seconds, though I guess a bit of hyperbole does not hurt. You validated findings against other source, which is obligatory step with chat gpt.
  • em500 3y ago
    > If that's true, why then did it take me 10 seconds yesterday to find an extremely obscure fact of 18th century history with ChatGPT 3.5? I tried Google queries for over an hour, without success. Can you share which obscure 18th century historical fact you were looking for? Maybe you happen to be exceptionally bad at google queries?
  • amatheus 3y ago
    > If that's true, why then did it take me 10 seconds yesterday to find an extremely obscure fact of 18th century history with ChatGPT 3.5? Would you mind telling what's the fact you were searching for? (and how did you search for it?)
  • marcosdumay 3y ago
    You found it because LLMs are incredibly good search engines. But anything they give you that isn't a search result is bullshit. So, they have those two modes of operation, and most of them are built in a way that won't let you know which mode they are working on. That can be useful depending on what you want to do, but it doesn't strike me as the most effective architecture for them. (They are also reasonably good pattern fitters, what is kinda nice for code autocompletion on some languages. But they are way too inefficient here. Yet, nobody created any good high-efficiency autocomplete that merges searching and pattern fitting, so people use them.)
  • thenoblesunfish 3y ago
    I think the answer is that bullshitters are great inspiration. They're fun, they get the smart people pumped up. The combination of a bullshitter and a more precise but less gregarious person is powerful. See a lot of successful companies.
  • al_be_back 3y ago
    >> If that's true, why then did it take me 10 seconds yesterday to... Google vs cGPT it's neither here nor there in this case, after all Google research drives cGPT, their market focus differs. The OP is directly mapping Frankfurt's depiction of Bullshit atop the current AI phenomena, mainly because how generative AI approaches "truth". Natural languages (NL) differ greatly from computer languages (CL), especially on grammar (NL tolerates vagueness) and semantics (meaning, truth). Loosely-speaking, when AI generates NL text it relies on LLMs for context, structure, length etc, a kinda "the model suggests so". However, where content is computer code, even computers can test & verify the output. With NL content on the other hand, the end-user has to essentially test & verify the output to establish truth, meaning. In my view, the main difference between a Human Bullshitter and a Automated Bullshitter, is that AI can potentially improve exponentially (as smart tech does), thus leaving the end-user (average person) severely overwhelmed.
• chillingeffect 3y ago
  https://news.ycombinator.com/item?id=34875111 https://news.ycombinator.com/item?id=34875111 Glad Mr. Blackwell could fill in these details for me!
  • fanf2 3y ago
    He’s a professor with a PhD :-)
• j7ake 3y ago
  I feel these criticisms are analogous to the early days of Wikipedia. Surely you cannot rely on Wikipedia for knowledge, there’s no algorithm to check the facts! Anybody can write anything! I imagine in 10 years these BS text generators will be the norm just as wikipedia is now the norm.
  • willis936 3y ago
    There was always a loop-closing method on the table with Wikipedia. Where is the loop-closing method to bullshit on LLMs not being efficient lie generators?
    • seeEllArr 3y ago
      [dead]
    • ben_w 3y ago
      What is loop closing? (Google didn't help, my results were SLAM and DIY). It sounds like citogenesis which is the opposite of the rest of your comment? https://en.wikipedia.org/wiki/Circular_reporting https://en.wikipedia.org/wiki/Circular_reporting
      • airstrike 3y ago
        Have you tried asking ChatGPT?
        • ben_w 3y ago
          """ChatGPT In this context, "loop closing" likely refers to a method of addressing or resolving issues or gaps in a process or system. The speaker seems to be expressing frustration about a perceived lack of a method to counter misinformation or falsehoods when it comes to Language Model (LLMs) efficiency.""" So, same inference I made from the context (alone it sounds like citogenesis but the context suggests the opposite), and same lack of specific detail. "A method". Yes, great, what is it?
• capableweb 3y ago
  It seems like the author is missing any sort of nuance. Yes, sometimes it produces bullshit, but often, it doesn't, and can save you a huge amount of time. But yeah, sometimes it leads you down the wrong path or give outright incorrect answers. How you deal with that is up to you. You can either try to work with what it gives you, setup tooling to easily verify things, and increase your productivity, or you can dig your head down in the sand and say it's all incorrect and you want nothing with it. Both are valid approaches :)
  • mvdtnz 3y ago
    Big "drunk driving kills people but it also helps people to get to work on time, so it's impossible to say if it's good or bad" energy.
    • blueboo 3y ago
      Not sure if your meme-quippery works. Non-drunk driving kills people too, and also helps people get to work on time.
      • mvdtnz 3y ago
        What?
      • bryanrasmussen 3y ago
        I think the problem is that if drunk driving helps you get to work on time most people don't have jobs where showing up drunk is appreciated.
    • capableweb 3y ago
      Huh? You mind explaining how that applies to my comment, in terms of the things we're actually talking about instead of analogies?
      • mvdtnz 3y ago
        If you're having trouble understanding the analogy I doubt anything I say here will help you.
• rodlette 3y ago
  This feels like more like a politically charged rant, rather than the type of output you'd hope from the Department of Computer Science at Cambridge. > “Godfather of AI” Geoff Hinton, in recent public talks, explains that one of the greatest risks is not that chatbots will become super-intelligent, but that they will generate text that is super-persuasive without being intelligent, in the manner of Donald Trump or Boris Johnson. > Perhaps this explains why the latest leader of the same government is so impressed by AI, and by billionaires promoting automated bullshit generators? > Graeber observes that aspects of university education prepare young people to expect little more from life, training them to submit to bureaucratic processes, while writing reams of text that few will ever read. In the kind of education that produces a Boris Johnson > AI systems like ChatGPT are trained with text from Twitter, Facebook, Reddit, and other huge archives of bullshit, alongside plenty of actual facts (including Wikipedia and text ripped off from professional writers).
• 0dayz 3y ago
  I see their point, but how exactly do we determine what is and what isn't bullshit? When it comes to politicians our democratic values allows bullshitters to manifest and if we remove it we remove a key part of democracy. In ai's case I would say it's a bit the same.
  • jprete 3y ago
    AI’s case isn’t at all the same. Politician bullshit is always directed towards a goal, and that goal is usually re-election or popularity or similar. AI bullshit exists to satisfy an RLHF process that prioritized convincing humans that the AI was as correct as possible, which is exactly what you don’t want when looking for information or help.
• gizmo 3y ago
  Think of AI tools as unbelievably good auto-complete or intellisense. A universal anything-to-anything translator. It's incredibly powerful because it's something humans are not very good at. But it's a mistake to think AIs are "just" bullshitters. If you ask a GPT to auto-complete some bullshit for you, it will happily do so. But it will do many tasks that provide value to society just as easily. Nobody knows how society will be transformed by AI. But if AI generated BS is a problem then I wouldn't be surprised to see AI powered skepticism assistance as well. We all have terrible blind spots, and maybe in an AI world it will be harder to delude yourself than in the present day? No predicting how this will turn out.
  • jprete 3y ago
    I think LLMs and RLHF are inherently unsuited to producing aligned, useful, and fact-oriented AI. Something else is needed.
    • gizmo 3y ago
      GPT4 (with decent prompting) provides very accurate and unbiased summaries of complex and contentious political questions. It's by design not a fact-oriented system but it is capable of producing very good factual answers.
      • blibble 3y ago
        > It's by design not a fact-oriented system but it is capable of producing very good factual answers. this is only by chance though, and you have no way of distinguishing it
• lordnacho 3y ago
  I think this essay is right in many ways. LLMs seem to operate on the principle of "will this pass for a correct answer" rather than "this is how this thing works, so here's a reasonable opinion answering your question". However at some point you have to admit the LLM does generate things that are good answers. They might be good answers that happen to pass the smell test, but they are nonetheless good answers. For instance when you ask it for a snippet of code and it gets it right. And here is the crucial thing: you need to already know what you're doing to know whether the LLM got it right. I'm no historian, and I can ask cGPT for an essay about the causes of the Great War. When I get the answer, it sounds right to me. I don't know if the essay talks about the things an actual historian would find important, all I know is that it gives me the vanilla answer that some layman who has read a little bit would think was the right answer. Now there's another issue this brings up. Most of us are experts in one field only. What is stopping the LLM from fooling me in every field that I don't know anything about? I best be wary of using it outside of my area of expertise. So in the current iteration, I think LLMs are a shortcutting tool for experts. I can tell when it spits out a snippet of code that is correct, and when it's wrong. Someone who wasn't working in my domain would get fooled.
  • falserum 3y ago
    Agree. It is a language model, so it’s best use is language things (emails, motivational letter, summaries) For everything else, you need to be enough of an expert to validate the output.
  • mike_hock 3y ago
    The same principle applies outside of LLMs. Ever seen mainstream media write something about your field of expertise that wasn't complete horseshit? No? Then why believe anything else they say.
  • indigochill 3y ago
    > LLMs seem to operate on the principle of "will this pass for a correct answer" rather than "this is how this thing works, so here's a reasonable opinion answering your question". Isn't the hint in the name? It's a language model, not a knowledge model. LLMs are exceptional at generating stuff that passes as coherent language (like, the parts of speech are all where one would expect them to be in the sentences). The trouble is people think it goes deeper when the knowledge modeling part only happens incidentally because language and knowledge modeling are so closely related.
• rvz 3y ago
  AI bros descending into the comments to justify their AI bullshit generator that not even they can explain how it works and why it generates random nonsense which they need to check each time they ask it. LLMs still cannot transparently explain any of their decisions at all or reason why it is cannot understand its own mistakes, meaning that it cannot be held to account. This is why no-one here trusts them at very high risk situations, only for bullshit and nonsense generation.
  • infecto 3y ago
    Your argument would be much more interesting without the first nonsensical statement.
    • rvz 3y ago
      Yet you yourself also need to 'explain' as to why that statement is 'nonsensical'. It is no good just saying it is without giving any explanation. Unless you can give a thorough explanation as how these LLMs internally can explain themselves transparently and reliably to the point where we don't need to check their outputs?
      • infecto 3y ago
        “AI bros” is just crass especially when there was little to no brigading. I have no argument with the content of your text just the delivery.
• jprete 3y ago
  OT, but did the author somehow not notice their AI-generated cow-robot cartoon has five legs? And some kind of ear-poop…thing…coming out the rear end?
  • baz00 3y ago
    I think that's intentional. It's bullshit too.
  • mvdtnz 3y ago
    Jesus Christ man read the article. That's the point.
• mjburgess 3y ago
  Increasingly, I'm able to empathise with people who use this psychobable language (hallucination, bullshit, etc.) to describe associative statistical models. They just don't know how they work, really want to believe they work like a person, or have otherwise been duped by the boards of the ad-ai industrial complex whose spiel is whatever increases market valuation (google, fb, ms, openai, etc.). I guess if you have absolutely no idea how these systems work there really isn't anything better than analogizing to people -- it is this 'failure of analogical reasoning' which ai/ad corps exploit in their desire to dupe ever more misinformed investors. This realisation helps me, only, since I can now fathom better when i'm wasting my time talking to someone whose basis for understanding stats is human psychology -- the amount of discussion needed to undo that is possibly a lot. Incidentally, all statistical systems of this kind (almost all ML, including NNs) work in the same way. They compress data into a smaller representation, and take a weighted average of pieces of that data relevant (by freq association) to the prediction. There is no reasoning, no truth, no falsehood, no lies, no bullshit, no hallucination, no veridical perception, no interiority, no subjectivity, no capacity to represent the world. There is something much more like a few pages of a book stuck together, with the second shinning through the first.
  • macNchz 3y ago
    Regardless of how they work, LLMs are already quite enmeshed in the world of humans–“participating” to a degree in our interactions and discourse, used every day by millions of people who do not and will not have a mathematical mental model for how they work–so exploring how they operate from this lens is a useful exercise, in my opinion.
    • mjburgess 3y ago
      Well, I don't begrudge people doing it -- I'm doubtful its useful. More-so dangerous, maladaptive, pathological, etc. People will eventually acquire a more accurate mental model, like using search engines in the early days. What riles me is the grubbiness of the commercial pushers, and the foolishness of the press which repeat their marketing material as-if its science. People should not rely on computers in the way they rely on people -- and any inducement to doing so is unethical.
• retrac98 3y ago
  Ok, well that bullshit generator has made me dramatically more productive.
  • gumballindie 3y ago
    That says a lot about one’s skill doesnt it?
    • deleted 3y ago
      [deleted]
• gmac 3y ago
  See also: https://news.ycombinator.com/item?id=36184315 https://news.ycombinator.com/item?id=36184315
• throw0101a 3y ago
  With regards to the concept of bullshit, see "On Bullshit" by Frankfurt: > This is the crux of the distinction between [the bullshiter] and the liar. Both he and the liar represent themselves falsely as endeavoring to communicate the truth. The success of each depends upon deceiving us about that. But the fact about himself that the liar hides is that he is attempting to lead us away from a correct apprehension of reality; we are not to know that he wants us to believe something he supposes to be false. The fact about himself that the bullshitter hides, on the other hand, is that the truth-values of his statements are of no central interest to him; what we are not to understand is that his intention is neither to report the truth nor co conceal it. This does not mean that his speech is anarchically impulsive, but that the motive guiding and controlling it is unconcerned with how the things about which he speaks truly are. > It is impossible for someone to lie unless he thinks he knows the truth. Producing bullshit requires no such conviction. A person who lies is thereby responding to the truth, and he is to that extent respectful of it. When an honest man speaks, he says only what he believes to be true; and for the liar, it is correspondingly indispensable that he considers his statements to be false. For the bullshitter, however, all these bets are off: he is neither on the side of the true nor on the side of the false. His eye is not on the facts at all, as the eyes of the honest man and of the liar are, except insofar as they may be pertinent to his interest in getting away with what he says. He does not care whether the things he says describe reality correctly. He just picks them out, or makes them up, to suit his purpose. * https://www2.csudh.edu/ccauthen/576f12/frankfurt__harry_-_on_bullshit.pdf https://www2.csudh.edu/ccauthen/576f12/frankfurt__harry_-_on... * https://web.archive.org/web/20150701235021/https://www2.csudh.edu/ccauthen/576f12/frankfurt__harry_-_on_bullshit.pdf https://web.archive.org/web/20150701235021/https://www2.csud... * https://en.wikipedia.org/wiki/On_Bullshit https://en.wikipedia.org/wiki/On_Bullshit * https://press.princeton.edu/books/hardcover/9780691122946/on-bullshit https://press.princeton.edu/books/hardcover/9780691122946/on... Liar: knows the truth, and says something specifically other. Bullshitter: does not care about what is true or what is false; says whatever necessary to achieve specific goals.
• gumballindie 3y ago
  And the scary bit is that masses of people are ready to ingest that bullshit, praise it and some idolise it. The real question is where did our education system fail since so many people lack critical thinking and how did the quality of services fall to a level where a bullshit generator sound like a plausible replacement of some workers.
  • JohnFen 3y ago
    One of the many differences I noticed between the education I got in school and the education my children got in school was that I was overtly taught critical thinking skills, and my children were not. I found that deeply troubling.
    • gumballindie 3y ago
      Lack of critical thinking makes docile voters and workers to order around. Convenient for the ruling class, convenient for those lacking it. Not knowing whats being done to you cant make you sad. But the end result is that of a house of cards.
• profeatur 3y ago
  People who think the world works on facts have a hard time grokking LLMs. With chatgpt you find solutions to problems the same way you do with humans: through conversation. Personally, chatgpt has revolutionized my work flow, both in that it prints out code that does what I want it to do, and it also helps me figure out how to frame my problems in a way that’s actually solvable.
  • hanselot 3y ago
    Its also fantastic for rubberducking, though I imagine local LLM's are better for this, since I don't want to give all my IP to Altman directly. So far, the yi model is actually not bad, and by two weeks from now two new models will be out that are better. So it will continue until Altman figures out which politicians to bribe the hardest.
    • Planktonne 3y ago
      > Its also fantastic for rubberducking Given that the gold standard for rubberducking neither knows nor communicates anything, this is an easy win.
      • airstrike 3y ago
        True, but in the past couple months, I've probably typed out 10+ truly complicated issues to ChatGPT and arrived at a solution before I ever pressed enter... just because I took the time to explain them in proper prose.
  • blibble 3y ago
    > With chatgpt you find solutions to problems the same way you do with humans: through conversation. the humans I work with were taught by people that spoke facts backed up by empirical evidence meanwhile chatgpt was trained on reddit and twitter
• bartwr 3y ago
  This completely misses the point that I'm using it as a tool and I am ok if this tool is sometimes wrong and I need to double check every. As a tool it saves me insane amounts of time (much more that I spend on verifying), a time saving and automation revolution comparable to web search, mobile, or Wikipedia. (Btw, if I need to do a code review and verify all work of an intern ir a junior programmer, it doesn't mean their work is useless - it's valuable and a net win) You don't have a use for a tool like this? Fine, your call. But stop this patronizing arrogant attitude.
• tempaway287641 3y ago
  For a more spicy take on this, check out Dan McQuillan's essay from February: ChatGPT Is a Bullshit Generator Waging Class War https://www.vice.com/en/article/akex34/chatgpt-is-a-bullshit-generator-waging-class-war https://www.vice.com/en/article/akex34/chatgpt-is-a-bullshit... That allegedly well-informed commentators can infer that ChatGPT will be used for "cutting staff workloads" rather than for further staff cuts illustrates a general failure to understand AI as a political project. Contemporary AI, as I argue in my book, is an assemblage for automatising administrative violence and amplifying austerity. ChatGPT is a part of a reality distortion field that obscures the underlying extractivism and diverts us into asking the wrong questions and worrying about the wrong things. Instead of expressing wonder, we should be asking whether it's justifiable to burn energy at "eye watering" rates to power the world's largest bullshit machine.
• kubiton 3y ago
  Another one not understanding the power of LLMs The magic is not that it can tell you thinks but that it understands you with a very high probability. It's the perfect interface for expert systems. It's very good in rewriting texts for me. It's very good in telling me what a text is about. And it's easy enough to combine a LLM with expert systems through apis. I for example mix languages when talking to chatgpt just because it doesn't matter. And yes it's often right enough and for GitHub copilot for example it doesn't matter at all if it's always right or only 80%. It only has to be better than not having it and 20 bucks a month.
  • xuhu 3y ago
    > it doesn't matter at all if it's always right or only 80% People get fired every day for pasting code that is 80% right and worked on a couple test-cases.
    • cfn 3y ago
      And yet, nobody has regulated stackoverflow...
      • paulryanrogers 3y ago
        Likely because human generation of content is more expensive and doesn't scale as far? Though SO has a lot of moderation, so it's somewhat self regulated.
    • kubiton 3y ago
      What? Never seen this happening. In contrary still not every team is doing code review and plenty of people regularly fix bugs in production. One ex colleague invalidated all apple device certificates, didn't get fired. A previous tech lead wrote code which deleted customer data and we found that a half year later, no one was fired. And no one got fired at a code review.
  • RandomLensman 3y ago
    Get a summary of a contract or law wrong, lose a few million dollars... it really is, sadly again, another peace of AI only for low risk applications.
• benterix 3y ago
  The article is a bit general to discuss but there is one specific point it touches that is crucial: it was trained on a wide array of resources, with both "true" (=corresponding to reality) content such as academic textbooks as well as tons of trash mixed with truth such as Twitter & Reddit posts and so on. An example: I asked it for the way to move an AWS account between two OUs in Terraform. It promptly generated the code including a reference to the "aws_organizations_move_account" resource. Since it looked suspicious to me, I checked it and the only place it exists is a Github Issues page where the code is a proposal for something that could work somewhere in the future (it doesn't, you move accounts in a different way). So if the people who trained the model decided that instead of making it general and versatile they will only focus on higher quality content, and made a manual selection if titles (most academic books, especially from respected publishers; public source code only from prominent projects with and adequately high number of developers and so on), they could produce a model that would be less eloquent but could generate less bullshit.
  • evandrofisico 3y ago
    From my limited understanding, a smaller, more specialized model could produce more accurate responses, but the main thesis of the article would still be true. Such LLMs would still be statistic prediction models, with no concept of reality. We already know that specialist systems, such as those tried in the 80's are not very good in modeling reality, either. The thing is, using LLMs as knowledge bases or as a sort of modern oracles seems to me like using a powerful tool for something they are horrible at. They are good enough (but not perfect) text summarizers, text translators and text transformers, but are far from the so expected "AGI" some seem to be expecting them to be.
• wrsh07 3y ago
  The author's humor is excellent (eX) Twitter, etc However, there are two things: 1) once you can generate arbitrarily parameterized bullshit (a piratical sonnet telling you to drink Coke), you can then filter on truth. This isn't always easy, but this talk presents one way to validate that it is, in fact, quoting the material it's supposed to https://youtu.be/yj-wSRJwrrc?si=MiZHn1xjFNv1nEGv https://youtu.be/yj-wSRJwrrc?si=MiZHn1xjFNv1nEGv ^^ 2) we should care about what people do with this. We should want people to use it for good, and we can shame people who don't. For outright evil we can make it illegal when existing laws don't suffice (and we should) 3) text is an interesting medium. It's universal in a lot of ways. There's a cynical take at Google that you're just converting one proto to another, and I suspect some white collar workers think they're just taking some text and reformatting it. I don't dispute that some jobs are useless, but I think that "just writing emails" or "just modifying text" isn't completely useless, and something interesting will happen if we can improve productivity in these areas ^^ you can just do a string contains on the original text vs the quotes it gives you, so this isn't fancy
  • teddyh 3y ago
    Truth is only the first filter. There are two more: • Is it kind? • Is it necessary?
  • marcosdumay 3y ago
    > once you can generate arbitrarily parameterized bullshit ..., you can then filter on truth. You are underestimating the parameter space of bullshit. By a lot. Thinking about orders of magnitude doesn't do justice to how much you are underestimating it. It's one of those things that can not fit within the physical universe.
    • Planktonne 3y ago
      It's the Library of Babel again; infinite options are worse than useless, because you don't know which ones matter.
• intended 3y ago
  Right now my shortest path for anyone on this debate, is to 1) Ask what you have built 2) What is the error rate and hallucination rate - ON your production data.
• Throwfi44 3y ago
  > they will generate text that is super-persuasive without being intelligent > What we need to regulate is the bullshit. Feeling bit threatened, are we? We have complete democratisation of "BS" (marketing, propaganda...). Now every scammer with Nigeria can compete with top universities and think tanks, without being "intelligent". Something like that happened when Samuel Colt invented his revolver. Every person could defend themselfs. Overnight it become very unwise to randomly attack people! Total democratisation of violence! "Intelligent" people will have to drop their shilling for latest fad, will have to build some credibility, reputation and personal brand!
  • ethanbond 3y ago
    > Overnight it become very unwise to randomly attack people! What planet do you live on where this happened?
    • Throwfi44 3y ago
      [flagged]
      • galdosdi 3y ago
        No, the question is what actual historical event are you referring to? What you are quoting sounds like a legend. When and where did this happen?
        • Throwfi44 3y ago
          Pretty much everywhere, but mostly at Wild West thad had no central government like Europe. Read this essays, it covers my point: https://www.thetruthaboutguns.com/the-democratization-of-violence-why-guns-are-a-good-thing/ https://www.thetruthaboutguns.com/the-democratization-of-vio... Similar moment could be invention of printing press, or raise of Internet (vs libraries).
  • _visgean 3y ago
    > "Intelligent" people will have to drop their shilling for latest fad, will have to build some credibility, reputation and personal brand! Should we ignore your comments because you use throwaway account and thus have no credibility reputation or personal brand here?
• adamsb6 3y ago
  I think LLMs have revealed that we have at least two ways of thinking. One is analogous to how LLMs operate and is essentially next token prediction. Think of your mental state when you’re very engrossed in conversation. It’s not a truly conscious act. Sometimes you even surprise yourself. If you’re a child, or childish, you may just confabulate things in the moment, maybe without realizing that you’re doing it in the moment. Now think of some very difficult problem you’ve had to solve. It’s not the same, right? It’s a very conscious act, directing your focus here and there, trying to reason on how everything fits together and what you might change to fix your problem. Odds are good that you’re not even using language to model the problem in your head. LLMs are doing the first thing, and are exceptionally good at it even in this early stage. The surprising thing to me is how far this can get you. If you have an inhuman level of knowledge to work from then in conversation mode you can actually solve some moderately difficult problems. I think that maps to our own experiences as well. For the things that you have deep knowledge on you will sometimes find yourself solving a problem just by constructing sentences.
  • firejake308 3y ago
    This is the idea of the book "Thinking, Fast and Slow" by Daniel Kahneman. LLMs follow what he describes as System 1, the intuitive heuristic that gets you through most of the day, versus System 2, the rigorous algorithmic thinking that you reserve for harder situations
  • Izkata 3y ago
    > The surprising thing to me is how far this can get you. There was that study a while ago that triggered the NPC meme. The common interpretation was that most people only think in the first way you described.
    • deleted 3y ago
      [deleted]
    • falseprofit 3y ago
      I dug around to find the study you're referring to (after a now-deleted comment asked about it). Is it this one? https://hurlburt.faculty.unlv.edu/heavey-hurlburt-2008.pdf https://hurlburt.faculty.unlv.edu/heavey-hurlburt-2008.pdf If so, the interpretation you describe seems pretty far off base. Also, my impression is that that meme was not sparked by a psychological study, but that a few people did draw on this study to justify it.
• unconed 3y ago
  >Donald Trump or Boris Johnson >Timnit Gebru How strange that the former two are cited as examples of bullshitters, but the latter not. That Twitter and politics is cited as a hotbed of untethered debate, but not academia. Even though the work and claims of "AI Ethicists" pretty much amounts to idea-laundered cultural marxism... and that this problem also exists in the wider social sciences, where certain hypotheses are dismissed regardless of evidence, because of the perceived moral implications. I have no doubt that much of the output of a transformer is bullshit. The same way I have no doubt that much of what passes for scholarship these days is. Garbage in, garbage out. Main difference seems to be that ChatGPT will apologize and correct itself when you point out it's wrong.
  • topaz0 3y ago
    can you define cultural marxism
• ai-thrw 3y ago
  Sure, but that doesn't explain why it's so good at writing simple functions. So long as the cyclomatic complexity of what you're asking it to write is less than 8 it does a good job of it. I've yet to have any trouble with it writing test functions. And it does a better job of explaining code than most developers I've worked with. All those things together make it unbelievably useful when you start working with the tool rather than against it. It's increased my capabilities to the point that I can tackle problems in the hour before bed that used to take me weeks before. More importantly for me it's made it clear just how muddled my thinking between code and architecture was.
• leibnitz27 3y ago
  Absolutely this - I'm a bit sad that the alternative name "Plausible Bullshit Generator" hasn't taken off.... https://www.google.com/search?q=%22plausible+bullshit+generator%22 https://www.google.com/search?q=%22plausible+bullshit+genera...
• z7 3y ago
  How about "language is a bullshit generator"? Most of the language we encounter is highly questionable, a "mobile army of metaphors, metonyms, and anthropomorphisms" (Nietzsche), whether human-made or machine-generated. That's part of what has made it so successful: an independence from the constraint of having to be "true". LLMs are using that property to their advantage but are hardly the source of it.
• thesuperbigfrog 3y ago
  Can any of the LLMs or whatever else is being sold as AI today perform deductive reasoning (https://en.wikipedia.org/wiki/Deductive_reasoning https://en.wikipedia.org/wiki/Deductive_reasoning) or any like it that approaches understanding? LLMs are impressive because they can generate text, images, etc. from a prompt, but hallucinations and other tells show that they do not really understand. Older AI systems can do basic levels of reasoning, but did not have much of a natural language interface like today's LLMs do. Are there any efforts to tie the two together?
  • bsenftner 3y ago
    Check out "Large Language Models are Zero-Shot Reasoners" by Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., and Iwasawa, Y nature.com. This paper discusses the capabilities of Large Language Models (LLMs) and their ability to reason or comprehend.
    • thesuperbigfrog 3y ago
      Thank you for the paper recommendation.
• bambax 3y ago
  I used to think exactly what this article says, but now I'm really not sure. Code that runs and works (as in, does what it's asked to do) isn't "bullshit" at all. Neither is search with references to custom sources in a specific corpus ("RAG"). So yes, AI is in many ways a bullshit generator (as are we); but it can be used in useful and non-bullshit ways.
  • zerbinxx 3y ago
    To play devil’s advocate: have you never reviewed a PR and said “yeah this works and completes the ticket but you REALLY shouldn’t be doing it like this”? There’s a pretty big gulf between functional and correct in a world where optimization matters. This is not always the case (strictly, i.e. algorithmically speaking), but you should be optimizing for things like reusability, testability, etc. which GPT-generated code can miss.
• bananapub 3y ago
  we really could save a lot of time if everyone just actually remembered that LLMs are designed to generate plausible output. it doesn't "know" anything, it doesn't "reason", it doesn't "think", it doesn't have opinions, it just takes a text prefix/seed/whatever and generates output, based on training of things humans actually produced. everything becomes pretty clear then. should you trust the answers it gave you? obviously not, it just generated some crap. can it be useful for brainstorming? sure, you an consider the output and take it / be inspired by it / etc can you use it to generate useful code? of course, as long as you read it and it's reviewed by tooling and humans to see that it does what you thought it did. should you replace your employees with it? obviously not, a text suffix generator isn't able to deal with complex problems nor be responsible for decisions. etc etc
  • danielbln 3y ago
    Once again we parade the "it just predicts the next token" around as if it was some deep insight. The magic of it all is how it does it, which we don't quite now. There might be some world model involved, there might not be, who knows. All we know is that "predicting the next token" is a simple objective that requires a lot of complexity inside the black box to work as well as it does.
• icedchai 3y ago
  I believe this is mostly correct. However, there is still some value in automating bullshit.
  • euroderf 3y ago
    A.I. is coming for the professional classes.
• somewhereoutth 3y ago
  I don't know how the 3rd AI winter will be endured, but the 4th will be endured with pen and paper. (With apologies to Einstein)
• garbanz0 3y ago
  I am getting tired of takes like this. LLMs are parrots, but they parrot human language which seems to encode our reasoning abilities into its structure. So if you can mimic language well enough, you start to mimic reasoning abilities. I feel like we should have all moved on from the fact that LLMs don't have a very precise memory? This is (not entirely, but mostly) fixed by including the information you want to query in the prompt itself, like Bing does. ChatGPT is much better at summarization than precise fact recall. And most of the time, it doesn't matter. When I'm using ChatGPT to learn or troubleshoot something, it's a jumping off point, not the end of the story. As long as everyone understands that, there's no problem. I have yet to see anyone who treats its output as 100% fact, and no AI company claims it either.
  • latexr 3y ago
    > As long as everyone understands that, there's no problem. But everyone doesn’t understand that, so there is a problem. > I have yet to see anyone who treats its output as 100% fact Yet they exist. https://www.nytimes.com/2023/06/08/nyregion/lawyer-chatgpt-sanctions.html https://www.nytimes.com/2023/06/08/nyregion/lawyer-chatgpt-s...
    • garbanz0 3y ago
      Everyone as in every human being on earth, or everyone as in most reasonable people? I'm not convinced at all. ChatGPT tells you it can lie before you can even ask it anything.
      • latexr 3y ago
        > as in every human being on earth Not every human being on Earth knows what ChatGPT is, so that’s a non-starter. > or everyone as in most reasonable people That qualifier is doing some heavy lifting. It’s so vague it can be used to excuse anything¹. Not only reasonable people (however we’re going to define that) have access to the tool. > ChatGPT tells you it can lie before you can even ask it anything. And what you learn fast by doing user support of any tech tool is that most users don’t read instructions even if it’s shoved in their faces. I personally know people that think ChatGPT is the source of truth, despite that disclaimer. ¹ https://en.wikipedia.org/wiki/No_true_Scotsman https://en.wikipedia.org/wiki/No_true_Scotsman
• lukev 3y ago
  This is a great little article. I've been calling ChatGPT the "bullshit robot" since it came out; it really does fit Frankfurt's definition remarkably well. But that doesn't mean you need to be "anti-AI". I'm also also super excited about LLMs and their potential and am actively working in the field. It's important to understand their role. LLMs aren't the whole brain: they're just the linguistic cortex. That's something computers couldn't do before! It's pretty amazing that we now have a computers that can not only use language, but are pretty good at it! Incredible breakthrough. The problem comes when you let the linguistic brain babble without integration with a more structured reasoning system or cite-able fact database. That's exactly what anyone who is working on retrieval and tools for LLMs is doing, so we're not on the wrong path. But a lot of people still treat this language model as if it were a whole, reasoning brain, and it's really really not. (disclaimer: my comparisons to human mind/brain/intelligence are for analogy only. Real brains and transformer models are very different things.)
• drra 3y ago
  The notion of this article is that LLMs are just for answering question as if these were failed oracles. LLMs are excellent at instruction following to the point where it'll execute instruction by creating bullshit to fill the gap. It's unlikely to get a perfect C code out of Starbucks barista but that doesn't mean these people are in any way flawed at their job. LLMs given enough information about a task can perfectly execute it and enough is far less than any of systems created before.
• fatherzine 3y ago
  Great synergy with https://en.wiktionary.org/wiki/Gell-Mann_Amnesia_effect https://en.wiktionary.org/wiki/Gell-Mann_Amnesia_effect
• orangepurple 3y ago
  Extended discussion with ChatGPT where the prompter is able to force ChatGPT to admit to lying, misleading, and being deceitful: (READ TO THE END) https://childrenshealthdefense.org/defender/chatgpt-ai-covid-vaccine-dna/ https://childrenshealthdefense.org/defender/chatgpt-ai-covid... Who programmed ChatGPT to lie, mislead, and be deceitful about certain topics? The way ChatGPT presents critical health information is borderline criminal.
• Solomet 3y ago
  I saw someone describe LLM hallucinations and inaccuracies as "compression artifacts." This thing has seen all the data that can possibly be fed to it and stored it in a much smaller size, so it's not crazy to think there will be compression artifacts. I have noticed that the quality of its answers seems to go down significantly when I am asking about a topic about which there isn't much information on google. But it makes you wonder if future LLMs aren't going to suffer from the JPEG of a JPEG of a JPEG effect where it will be impossible to train an LLM off of new data that isn't generated by another LLM.
  • euroderf 3y ago
    "Compression artifacts" makes them sound like things that do not exist... but should. Begging to be reified.
• sebastianconcpt 3y ago
  AI = Algebraic Parrot.
• pasloc 3y ago
  This essay cites a statement from Geoffrey Hinton and ignores the whole talk that he gave. I think he cited him from the talk "Two path to intelligence". In that talk Geoffrey Hinton said the complete opposite of this article, nevertheless he frames him like he would support this "essay". It's just his opinion and nothing else. Also the paper about "stochastic parrots" is under great criticism and there are now papers that contradict their conclusions.
• javier_e06 3y ago
  The paradox being: Do you happen to make mistakes? No, I always get it right. If you can't make mistakes, how can you tell when you get it right? All I expect from my assistant is, at least, for it to make less mistakes than me.
  • yCombLinks 3y ago
    I don't even expect that. I just expect them to eventually get it right, and for less of my total time to be spent overall.
• deleted 3y ago
  [deleted]
• lairv 3y ago
  There was a really good video from a french youtuber about this 9 months ago: https://youtu.be/JcFRbecX6bk?feature=shared https://youtu.be/JcFRbecX6bk?feature=shared In French, we have a word that I find more accurate, which is "baratineur", which means "someone who tells you what you want to hear, regardless of the concepts of truth or lies"
• tim333 3y ago
  I've found gtp4's replies considerably less bullshitty than the average human reply on Twitter/X.
• egometry 3y ago
  The salient passage seems to be: Quite simply, we are talking about bullshit. Philosopher Harry Frankfurt, in his classic text _On Bullshit_, explains that the bullshitter “does not reject the authority of truth, as the liar does […] He pays no attention to it at all.” This is exactly what senior AI researchers such as Brooks, Bender, Shanahan and Hinton are telling us, when they explain how ChatGPT works. The problem, as Frankfurt explains, is that “[b]y virtue of this, bullshit is a greater enemy of the truth than lies are.” (p. 61). At a time when a public enquiry is reporting the astonishing behaviour of our most senior leaders during the Covid pandemic, the British people wonder how we came to elect such such bullshitters to lead us. But as Frankfurt observes, “Bullshit is unavoidable whenever circumstances require someone to talk without knowing what he is talking about” (p.63)