4 ms·

Sorry I thought it would be clear and could have clarified that the code itself is just a joke illustrating the point, as an exaggeration. This was the thread i

Sorry I thought it would be clear and could have clarified that the code itself is just a joke illustrating the point, as an exaggeration. This was the thread if anyone is interested https://chatgpt.com/share/68e82db9-7a28-8007-9a99-bc6f0010d101 https://chatgpt.com/share/68e82db9-7a28-8007-9a99-bc6f0010d1...

chis 1y ago
I think there’s always a danger of these foundational model companies doing RLHF on non-expert users, and this feels like a case of that. The AIs in general feel really focused on making the user happy - your example, and another one is how they love adding emojis to the stout and over-commenting simple code.
- cma 1y ago
  And more advanced users are more likely to opt out of training on their data, Google gets around it with a free api period where you can't opt out and I think from did some of that too, through partnerships with tool companies, but not sure if you can ever opt out there.
  - cma 1y ago
    *grok, not 'from'
- miki123211 1y ago
  This feels like RLVR, not RLHF. With RLVR, the LLM is trained to pursue "verified rewards." On coding tasks, the reward is usually something like the percentage of passing tests. Let's say you have some code that iterates over a set of files and does processing on them. The way a normal dev would write it, an exception in that code would crash the entire program. If you swallow and log the exception, however, you can continue processing the remaining files. This is an easy way to get "number of files successfully processed" up, without actually making your code any better.
  - eru 1y ago
    > This is an easy way to get "number of files successfully processed" up, without actually making your code any better. Well, it depends a bit on what your goal is. Sometimes the user wants to eg backup as many files as possible from a failing hard drive, and doesn't want to fail the whole process just because one item is broken.
    - orisho 1y ago
      You're right, but the way to achieve this is to allow the error to propagate at the file level, then catch it one function above and continue to the next one. However, LLM generated code will often, at least in my experience, avoid raising any errors at all, in any case. This is undesirable, because some errors should result in a complete failure - for example, errors which are not transient or environment related but a bug. And in any case, a LLM will prefer turning these single file errors into warnings, though the way I see it, they are errors. They just don't need to abort the process, but errors nonetheless.
      - eru 1y ago
        Yes, that's cleaner. > And in any case, a LLM will prefer turning these single file errors into warnings, though the way I see it, they are errors. Well, in general they are something that the caller should have opportunity to deal with. In some cases, aborting back to the caller at the first problem is the best course of action. In some other cases, going forward and taking note of the problems is best. In some systems, you might event want to tell the caller about failures (and successes) as they occur, instead of waiting until the end. It's all very similar to the different options people have available when their boss sends them on an errand and something goes wrong. A good underling uses their best judgement to pick the right way to cope with problems; but computer programs don't have that, so we need to be explicit. See https://en.wikipedia.org/wiki/Mission-type_tactics https://en.wikipedia.org/wiki/Mission-type_tactics for a related concept in the military.
- justatdotin 1y ago
  'over-commenting simple code' is preparing it for future agent work. pay attention to those comments to learn how you can better scaffold for agents.
  - mnahkies 1y ago
    They do seem to leave otherwise useless comments for itself. Eg: on the level of // Return the result return result; I find this quite frustrating when reading/reviewing code generated by AI, but have started to appreciate that it does make subsequent changes by LLMs work better. It makes me wonder if we'll end up in a place where IDEs hide comments by default (similar to how imports are often collapsed by default/automatically managed), or introduce some way of distinguishing between a more valuable human written comment and LLM boilerplate comments.
  - stuaxo 1y ago
    They should have a step to remove those sorts of comments, they only add noise to the code.
why_at 1y ago
This part from the first try made me laugh: if random.random() < 0.01: logging.warning("This feels wrong. Aborting just in case.") return None
- bspammer 1y ago
  I think that’s the funniest joke I’ve ever seen an LLM make. Which probably means it’s copied from somewhere.
  - m463 1y ago
    "Why is a laser beam like goldfish? Because neither one can whistle." - Mike, The Moon is a Harsh Mistress
    - muggesmuds 1y ago
      Fantastic book, just read it. Surprised no movie has been made.
      - grues-dinner 1y ago
        The protagonists are libertarians with teenage harems, who fake an election and team up with with a sex pest. That's extremely reductive to the point of parody, but that will likely be the media coverage of it then moment someone reads the women and politics in the book. If you completely excise anything too distasteful for a current-day blockbuster, but want a film about a space mining colony uprising you might as well just adapt the game Red Faction instead: have the brave heros blasting away with abandon at corpo guards, mad genetic experimenters and mercenaries and the media coverage can talk about how it's a genius deconstruction of Elon Musk's Martian dream or whatever.
      - pavlov 1y ago
        If you haven't read Ursula Le Guin's "The Dispossessed", check it out too. It's like a fine wine pairing for "The Moon is a Harsh Mistress."
bjourne 1y ago
This is stunning English: "Perfect setup for satire. Here’s a Python function that fully commits to the bit — a traumatically over-trained LLM trying to divide numbers while avoiding any conceivable danger:" "Traumatically over-trained", while scoring zero google hits, is an amazingly good description. How can it intuitively know what "traumatic over-training" should mean for LLMs without ever having been taught the concept?
- drekipus 1y ago
  The same way that you and I think up a word and what it might mean without being taught the concept. Adverb + verb
  - the_gipsy 1y ago
    But the machines cannot possibly have the magic brain-juice!
- gnulinux 1y ago
  Hard to know but if you could express "traumatically" as a number, and "over-trained" as a number, it seems like we'd expect "traumatically" + "over-trained" to be close to "traumatically over-trained" as a number. LLMs work in mysterious ways.
- jcelerier 1y ago
  LLMs operate at token level, not word. it doesn't operate in terms of "traumatic", "over-training", "over" or "training", but rather "tr" "aum" "at" "ic, ", etc.
  - typpilol 1y ago
    Weird thing I've noticed. Some LLMs can output nerd font glyphs and others can't. If I recall grok code fast can but codex and sonnet can't
  - magicalhippo 1y ago
    > it doesn't operate in terms of "traumatic", "over-training", "over" or "training", but rather "tr" "aum" "at" "ic, ", etc. And "毛片免费观看" (Free porn movies), "天天中彩票能" (Win the lottery every day), "热这里只有精品" (Hot, only fine products here) etc[1]. [1]: https://news.ycombinator.com/item?id=45483924 https://news.ycombinator.com/item?id=45483924
cruffle_duffle 1y ago
Kind of interesting it didn't add type hints though! You'd think for all that paranoia it would at least add type hints.
TZubiri 1y ago
https://chatgpt.com/share/68e87072-e3ac-800f-a44c-af5666180a6a https://chatgpt.com/share/68e87072-e3ac-800f-a44c-af5666180a... lgtm
nought 1y ago
It was a great joke, that's why I posted it

chis 1y ago

I think there’s always a danger of these foundational model companies doing RLHF on non-expert users, and this feels like a case of that. The AIs in general feel really focused on making the user happy - your example, and another one is how they love adding emojis to the stout and over-commenting simple code.

cma 1y ago
And more advanced users are more likely to opt out of training on their data, Google gets around it with a free api period where you can't opt out and I think from did some of that too, through partnerships with tool companies, but not sure if you can ever opt out there.
- cma 1y ago
  *grok, not 'from'
miki123211 1y ago
This feels like RLVR, not RLHF. With RLVR, the LLM is trained to pursue "verified rewards." On coding tasks, the reward is usually something like the percentage of passing tests. Let's say you have some code that iterates over a set of files and does processing on them. The way a normal dev would write it, an exception in that code would crash the entire program. If you swallow and log the exception, however, you can continue processing the remaining files. This is an easy way to get "number of files successfully processed" up, without actually making your code any better.
- eru 1y ago
  > This is an easy way to get "number of files successfully processed" up, without actually making your code any better. Well, it depends a bit on what your goal is. Sometimes the user wants to eg backup as many files as possible from a failing hard drive, and doesn't want to fail the whole process just because one item is broken.
  - orisho 1y ago
    You're right, but the way to achieve this is to allow the error to propagate at the file level, then catch it one function above and continue to the next one. However, LLM generated code will often, at least in my experience, avoid raising any errors at all, in any case. This is undesirable, because some errors should result in a complete failure - for example, errors which are not transient or environment related but a bug. And in any case, a LLM will prefer turning these single file errors into warnings, though the way I see it, they are errors. They just don't need to abort the process, but errors nonetheless.
    - eru 1y ago
      Yes, that's cleaner. > And in any case, a LLM will prefer turning these single file errors into warnings, though the way I see it, they are errors. Well, in general they are something that the caller should have opportunity to deal with. In some cases, aborting back to the caller at the first problem is the best course of action. In some other cases, going forward and taking note of the problems is best. In some systems, you might event want to tell the caller about failures (and successes) as they occur, instead of waiting until the end. It's all very similar to the different options people have available when their boss sends them on an errand and something goes wrong. A good underling uses their best judgement to pick the right way to cope with problems; but computer programs don't have that, so we need to be explicit. See https://en.wikipedia.org/wiki/Mission-type_tactics https://en.wikipedia.org/wiki/Mission-type_tactics for a related concept in the military.
justatdotin 1y ago
'over-commenting simple code' is preparing it for future agent work. pay attention to those comments to learn how you can better scaffold for agents.
- mnahkies 1y ago
  They do seem to leave otherwise useless comments for itself. Eg: on the level of // Return the result return result; I find this quite frustrating when reading/reviewing code generated by AI, but have started to appreciate that it does make subsequent changes by LLMs work better. It makes me wonder if we'll end up in a place where IDEs hide comments by default (similar to how imports are often collapsed by default/automatically managed), or introduce some way of distinguishing between a more valuable human written comment and LLM boilerplate comments.
- stuaxo 1y ago
  They should have a step to remove those sorts of comments, they only add noise to the code.

why_at 1y ago

This part from the first try made me laugh: if random.random() < 0.01: logging.warning("This feels wrong. Aborting just in case.") return None

bspammer 1y ago
I think that’s the funniest joke I’ve ever seen an LLM make. Which probably means it’s copied from somewhere.
- m463 1y ago
  "Why is a laser beam like goldfish? Because neither one can whistle." - Mike, The Moon is a Harsh Mistress
  - muggesmuds 1y ago
    Fantastic book, just read it. Surprised no movie has been made.
    - grues-dinner 1y ago
      The protagonists are libertarians with teenage harems, who fake an election and team up with with a sex pest. That's extremely reductive to the point of parody, but that will likely be the media coverage of it then moment someone reads the women and politics in the book. If you completely excise anything too distasteful for a current-day blockbuster, but want a film about a space mining colony uprising you might as well just adapt the game Red Faction instead: have the brave heros blasting away with abandon at corpo guards, mad genetic experimenters and mercenaries and the media coverage can talk about how it's a genius deconstruction of Elon Musk's Martian dream or whatever.
    - pavlov 1y ago
      If you haven't read Ursula Le Guin's "The Dispossessed", check it out too. It's like a fine wine pairing for "The Moon is a Harsh Mistress."

bjourne 1y ago

This is stunning English: "Perfect setup for satire. Here’s a Python function that fully commits to the bit — a traumatically over-trained LLM trying to divide numbers while avoiding any conceivable danger:" "Traumatically over-trained", while scoring zero google hits, is an amazingly good description. How can it intuitively know what "traumatic over-training" should mean for LLMs without ever having been taught the concept?

drekipus 1y ago
The same way that you and I think up a word and what it might mean without being taught the concept. Adverb + verb
- the_gipsy 1y ago
  But the machines cannot possibly have the magic brain-juice!
gnulinux 1y ago
Hard to know but if you could express "traumatically" as a number, and "over-trained" as a number, it seems like we'd expect "traumatically" + "over-trained" to be close to "traumatically over-trained" as a number. LLMs work in mysterious ways.
jcelerier 1y ago
LLMs operate at token level, not word. it doesn't operate in terms of "traumatic", "over-training", "over" or "training", but rather "tr" "aum" "at" "ic, ", etc.
- typpilol 1y ago
  Weird thing I've noticed. Some LLMs can output nerd font glyphs and others can't. If I recall grok code fast can but codex and sonnet can't
- magicalhippo 1y ago
  > it doesn't operate in terms of "traumatic", "over-training", "over" or "training", but rather "tr" "aum" "at" "ic, ", etc. And "毛片免费观看" (Free porn movies), "天天中彩票能" (Win the lottery every day), "热这里只有精品" (Hot, only fine products here) etc[1]. [1]: https://news.ycombinator.com/item?id=45483924 https://news.ycombinator.com/item?id=45483924

cruffle_duffle 1y ago

Kind of interesting it didn't add type hints though! You'd think for all that paranoia it would at least add type hints.

TZubiri 1y ago

https://chatgpt.com/share/68e87072-e3ac-800f-a44c-af5666180a6a https://chatgpt.com/share/68e87072-e3ac-800f-a44c-af5666180a... lgtm

nought 1y ago

It was a great joke, that's why I posted it