3 ms·

The way to understand it is when you catch yourself almost falling asleep at night while reading something. You lose the ability to understand anything, even th

The way to understand it is when you catch yourself almost falling asleep at night while reading something. You lose the ability to understand anything, even though you are still reading and the words are still English. LLM is great at generating that sort of thing. When you lose concentration, or didn't want to pay attention, that document can be generated by LLM to fool you. That's a lot of documents. All the powerpoints that young investment bankers and strategy consultants make. Every restaurant menu. Brochures. The middle 100 pages of a pop science book from the airport. When it comes to writing something actually importantant, LLM can only help you a little bit. Well, it's a lot in historical context. But it can do things like bullet point ideas so you don't forget things, or move a few things around while preserving grammatical structure. All it can do is bring you the ingredients. That's why we get these odd AI-generated articles nowadays, it's a bunch of chopped onions and tomatoes, very nicely done up but with no intention other than to trick you into thinking it's a meal.

bobson381 1y ago
You would like this article - https://www.theintrinsicperspective.com/p/curious-george-and-the-case-of-the https://www.theintrinsicperspective.com/p/curious-george-and... The author orders some Curious George stickers and gets a surprising array of kind of scary "closely related" stickers with em.... a fuzzy interpreter. The plus c when you integrate. Those missing harmonics and back-channel data that get inaudibly cut out of MP3s. The goo between the prickles. We don't know how to say what it is, but we sure can tell when it's missing!
- xdavidliu 1y ago
  > The plus c when you integrate. Those missing harmonics and back-channel data that get inaudibly cut out of MP3s. The goo between the prickles come again?
  - bobson381 1y ago
    In between rules in formal systems there's lots of space. all models are wrong, some are useful. I just really like thinking about how much is there that we don't even know about - and the more structured a system for understanding something is, the more fragile it becomes.
    - xdavidliu 1y ago
      ok but what does "goo between the prickles" refer to?
      - bithive123 1y ago
        “See, there are basically two kinds of Philosophy - one's called prickly, the other one is called goo. Prickly people are precise, rigorous, logical - they like everything chopped up and clear. Goo people like it vague, big picture, random, imprecise, incomplete and irrational. Prickly people believe in particles, goo people believe in waves. They always argue with each other but what they don't realize is neither one of them can take their position without their opposition being there. You wouldn't know you are advocating prickles unless someone else was advocating goo. You wouldn't even know what prickles was and what goo was. Life is not prickles or goo, its gooey-prickles or prickly-goo.” ― Alan Watts
HarHarVeryFunny 1y ago
Maybe because LLMs are literally designed to generate boring content. They are trained to output the best possible prediction - i.e most predictable version - of how the input will continue (i.e the least surprising/interesting continuation). Humans use surprise (= prediction failure) as a learning signal and to focus our attention, so not surprising that we dose off and lose interest when something is highly predictable. Good human authors know how to introduce plot twists and suspense, to keep the reader engaged and guessing. Imagine an entire novel that was instead written by an LLM, where the only plot twists are the predictable ones it copied from it's training material. The same goes for any LLM generated prose, whether advertising copy or training manual - it's going to be a snooze fest because it's so predictable - very low signal to noise ratio.
- deleted 1y ago
  [deleted]
- nullc 1y ago
  Base models are less boring, a lot less-- so I think it's less a product of most likely prediction, I think it's more of a product of reinforcement which biases towards text that uses a lot of words but says little, since saying nothing is at least not saying something identifiably hallucinatory.
  - HarHarVeryFunny 1y ago
    RL isn't in general going to bias the model to use a lot of words, unless your RL training had the goal of favoring long over short responses. There are now multiple levels of RL being used to post-train these models, from RLHF (use RL to bias the model to generate outputs matching human feedback preferences) to RL used to improve reasoning by generating reasoning steps that lead to verified correct conclusions (in areas like math and programming where correctness can be verified). RLHF (not RL in general) may lead to longer more verbose outputs to the extent that human testers indicated longer responses as their preference. Maybe testers are easily bullshitted and like something that is longer and sounds like a more comprehensive authoritative answer ? There is also the fact that an LLM is, unless prompted otherwise, is trying to predict Mr. Average (entire training set), who is more likely to waffle on than an expert who will cut to the chase and just give the facts, which they have a firm grip of. You can of course prompt the model to behave like an expert, or any given role, or to be more concise, which may or may not result in better output. It's a bit like asking the model to summarize when it's not really summarizing but instead predicting what a summary would look like (form vs function).
couscouspie 1y ago
> The way to understand it is when you catch yourself almost falling asleep at night while reading something. You lose the ability to understand anything, even though you are still reading and the words are still English. That's btw a perfect analogy what ADHD is when consuming anything, anytime.
anonzzzies 1y ago
I find LLMs great for the type of client/business partner that keep asking for docs but where I know they will never read them. This happens a lot especially when the 'tech team' is just all management mba's and the actual tech is outsourced: we have a client with a tech team like that; we are in meeting with 15+ people from their side who are from the tech team: they know nothing about their tech. They spend the meeting flagging areas where they 'need documents' and we provide them next time. We make a lot of money from them so we do it and AI makes this easier than before: basically chuck the meeting in an llm which has as 'memory' a bunch of our actual technical stuff (which is publicly available) and then generate those docs with some prompts like 'exhaustive' and 'thorough'.

bobson381 1y ago

You would like this article - https://www.theintrinsicperspective.com/p/curious-george-and-the-case-of-the https://www.theintrinsicperspective.com/p/curious-george-and... The author orders some Curious George stickers and gets a surprising array of kind of scary "closely related" stickers with em.... a fuzzy interpreter. The plus c when you integrate. Those missing harmonics and back-channel data that get inaudibly cut out of MP3s. The goo between the prickles. We don't know how to say what it is, but we sure can tell when it's missing!

xdavidliu 1y ago
> The plus c when you integrate. Those missing harmonics and back-channel data that get inaudibly cut out of MP3s. The goo between the prickles come again?
- bobson381 1y ago
  In between rules in formal systems there's lots of space. all models are wrong, some are useful. I just really like thinking about how much is there that we don't even know about - and the more structured a system for understanding something is, the more fragile it becomes.
  - xdavidliu 1y ago
    ok but what does "goo between the prickles" refer to?
    - bithive123 1y ago
      “See, there are basically two kinds of Philosophy - one's called prickly, the other one is called goo. Prickly people are precise, rigorous, logical - they like everything chopped up and clear. Goo people like it vague, big picture, random, imprecise, incomplete and irrational. Prickly people believe in particles, goo people believe in waves. They always argue with each other but what they don't realize is neither one of them can take their position without their opposition being there. You wouldn't know you are advocating prickles unless someone else was advocating goo. You wouldn't even know what prickles was and what goo was. Life is not prickles or goo, its gooey-prickles or prickly-goo.” ― Alan Watts

HarHarVeryFunny 1y ago

Maybe because LLMs are literally designed to generate boring content. They are trained to output the best possible prediction - i.e most predictable version - of how the input will continue (i.e the least surprising/interesting continuation). Humans use surprise (= prediction failure) as a learning signal and to focus our attention, so not surprising that we dose off and lose interest when something is highly predictable. Good human authors know how to introduce plot twists and suspense, to keep the reader engaged and guessing. Imagine an entire novel that was instead written by an LLM, where the only plot twists are the predictable ones it copied from it's training material. The same goes for any LLM generated prose, whether advertising copy or training manual - it's going to be a snooze fest because it's so predictable - very low signal to noise ratio.

deleted 1y ago
[deleted]
nullc 1y ago
Base models are less boring, a lot less-- so I think it's less a product of most likely prediction, I think it's more of a product of reinforcement which biases towards text that uses a lot of words but says little, since saying nothing is at least not saying something identifiably hallucinatory.
- HarHarVeryFunny 1y ago
  RL isn't in general going to bias the model to use a lot of words, unless your RL training had the goal of favoring long over short responses. There are now multiple levels of RL being used to post-train these models, from RLHF (use RL to bias the model to generate outputs matching human feedback preferences) to RL used to improve reasoning by generating reasoning steps that lead to verified correct conclusions (in areas like math and programming where correctness can be verified). RLHF (not RL in general) may lead to longer more verbose outputs to the extent that human testers indicated longer responses as their preference. Maybe testers are easily bullshitted and like something that is longer and sounds like a more comprehensive authoritative answer ? There is also the fact that an LLM is, unless prompted otherwise, is trying to predict Mr. Average (entire training set), who is more likely to waffle on than an expert who will cut to the chase and just give the facts, which they have a firm grip of. You can of course prompt the model to behave like an expert, or any given role, or to be more concise, which may or may not result in better output. It's a bit like asking the model to summarize when it's not really summarizing but instead predicting what a summary would look like (form vs function).

couscouspie 1y ago

> The way to understand it is when you catch yourself almost falling asleep at night while reading something. You lose the ability to understand anything, even though you are still reading and the words are still English. That's btw a perfect analogy what ADHD is when consuming anything, anytime.

anonzzzies 1y ago

I find LLMs great for the type of client/business partner that keep asking for docs but where I know they will never read them. This happens a lot especially when the 'tech team' is just all management mba's and the actual tech is outsourced: we have a client with a tech team like that; we are in meeting with 15+ people from their side who are from the tech team: they know nothing about their tech. They spend the meeting flagging areas where they 'need documents' and we provide them next time. We make a lot of money from them so we do it and AI makes this easier than before: basically chuck the meeting in an llm which has as 'memory' a bunch of our actual technical stuff (which is publicly available) and then generate those docs with some prompts like 'exhaustive' and 'thorough'.