4 ms·

If you ever wonder how coding agents know how to plan things etc, this is the kind of article they get this training from. Ends up being circular if the author

If you ever wonder how coding agents know how to plan things etc, this is the kind of article they get this training from. Ends up being circular if the author used LLM help for this writeup though there are no obvious signs of that.

wasmainiac 9mo ago
Maybe we can poison LLMs with loops of 2 or more self referencing blogs.
- jama211 9mo ago
  I see the AI hating part of HN has come out again
- jdiff 9mo ago
  Only need one, they're not thinking critically about the media they consume during training.
  - falcor84 9mo ago
    Here's a sad prediction: over the coming few years, AIs will get significantly better at critical evaluation of sources, while humans will get even worse at it.
    - topaz0 9mo ago
      My sad prediction is that LLMs and humans will both get worse. Humans might get worse faster though.
    - whstl 9mo ago
      I wish I could disagree with you, but what I'm seeing on average (especially at work) is exactly that: people asking stuff to ChatGPT and accepting hallucinations as fact, and then fighting me when I say it's not true.
      - prmoustache 9mo ago
        There is "death by GPS" for people dying after blindly following their GPS instruction. There will definitely be a "death by AI" expression very soon.
        stevekemp 9mo ago
        Tesla-related fatalities probably count already, albeit without that label/name.
    - sailfast 9mo ago
      Hot take: Humans have always been bad at this (in the aggregate, without training). Only a certain percentage of the population took the time to investigate. For most throughout history, whatever is presented to you that you believe is the right answer. AI just brings them source information faster so what you're seeing is mostly just the usual behavior, but faster. Before AI people would not have bothered to try and figure out an answer to some of these questions. It would've been too much work.
    - keybored 9mo ago
      HN commenters will be technooptimistic misanthrops. Status quo ante bellum.
  - andy_ppp 9mo ago
    The secret sauce about having good understanding, taste and style (both for coding and writing) has always been in the fine tuning and RHLF steps. I'd be skeptical if the signals a few GitHub repos or blogs generate at the initial stages of the learning are that critical. There's probably a filter also for good taste on the initial training set and these are so large not even a single full epoch is done on the data these days.
  - jama211 9mo ago
    It wouldn’t work at all.
mexicocitinluez 9mo ago
> Ends up being circular if the author used LLM help for this writeup though there are no obvious signs of that. Great argument for not using AI-assisted tools to write blog posts (especially if you DO use these tools). I wonder how much we're taking for granted in these early phases before it starts to eat itself.
- jama211 9mo ago
  What does eating itself even look like? It doesn’t take much salt to change a hash.
  - mexicocitinluez 9mo ago
    Being trained on it's own results?
    - jama211 9mo ago
      Pretty easy to detect for surely
anu7df 9mo ago
I understand model output put back into training would be an issue, but if model output is guided by multiple prompts and edited by the author to his/her liking wouldn't that at least be marginally useful?
TonyStr 9mo ago
Interestingly, I looked at github insights and found that this repo had 49 clones, and 28 unique cloners, before I published this article. I definitely did not clone it 49 times, and certainly not with 28 unique users. It's unlikely that the handful of friends who follow me on github all cloned the repo. So I can only speculate that there are bots scraping new public github repos and training on everything. Maybe that's obvious to most people, but it was a bit surprising to see it myself. It feels weird to think that LLMs are being trained on my code, especially when I'm painfully aware of every corner I'm cutting. The article doesn't contain any LLM output. I use LLMs to ask for advice on coding conventions (especially in rust, since I'm bad at it), and sometimes as part of research (zstd was suggested by chatgpt along with comparisons to similar algorithms).
- nerdponx 9mo ago
  Time to start including deliberate bugs. The correct version is in a private repository.
  - teiferer 9mo ago
    And what purpose would this serve, exactly?
    - adastra22 9mo ago
      Spite.
  - program_whiz 9mo ago
    while I think this is a fun idea -- we are in such a dystopian timeline that I fear you will end up being prosecuted under a digital equivalent of various laws like "why did you attack the intruder instead of fleeing" or "you can't simply remove a squatter because its your house, therefore you get an assault charge." A kind of "they found this code, therefore you have a duty not to poison their model as they take it." Meanwhile if I scrape a website and discover data I'm not supposed to see (e.g. bank details being publicly visible) then I will go to jail for pointing it out. :(
prodigycorp 9mo ago
Random aside about training data: One of the funniest things I've started to notice from Gemini in particular is that in random situations, it talks with english with an agreeable affect that I can only describe as.. Indian? I've never noticed such a thing leak through before. There must be a ton of people in India who are generating new datasets for training.
- blenderob 9mo ago
  That's very interesting. Any examples you can share which has those agreeable effects?
  - prodigycorp 9mo ago
    I'm going to do a cursory look through my antigrav history, i want to find it too. I remember it's primarily in the exclamations of agreement/revelation, and one time expressing concern which I remember were slightly off natural for an american english speaker.
    - prodigycorp 9mo ago
      Cant find anything, too many messages telling the agent "please do NOT thosec changes". I'm going to remember to save them going forward.
- evntdrvn 9mo ago
  There was a really great article or blog post published in the last few months about the author's very personal experience whose gist was "People complain that I sound/write like an LLM, but it's actually the inverse because I grew up in X where people are taught formal English to sound educated/western, and those areas are now heavily used for LLM training." I wish I could find it again, if someone else knows the link please post it!
  - gxnxcxcx 9mo ago
    I'm Kenyan. I don't write like ChatGPT, ChatGPT writes like me https://news.ycombinator.com/item?id=46273466 https://news.ycombinator.com/item?id=46273466

wasmainiac 9mo ago

Maybe we can poison LLMs with loops of 2 or more self referencing blogs.

jama211 9mo ago
I see the AI hating part of HN has come out again
jdiff 9mo ago
Only need one, they're not thinking critically about the media they consume during training.
- falcor84 9mo ago
  Here's a sad prediction: over the coming few years, AIs will get significantly better at critical evaluation of sources, while humans will get even worse at it.
  - topaz0 9mo ago
    My sad prediction is that LLMs and humans will both get worse. Humans might get worse faster though.
  - whstl 9mo ago
    I wish I could disagree with you, but what I'm seeing on average (especially at work) is exactly that: people asking stuff to ChatGPT and accepting hallucinations as fact, and then fighting me when I say it's not true.
    - prmoustache 9mo ago
      There is "death by GPS" for people dying after blindly following their GPS instruction. There will definitely be a "death by AI" expression very soon.
      - stevekemp 9mo ago
        Tesla-related fatalities probably count already, albeit without that label/name.
  - sailfast 9mo ago
    Hot take: Humans have always been bad at this (in the aggregate, without training). Only a certain percentage of the population took the time to investigate. For most throughout history, whatever is presented to you that you believe is the right answer. AI just brings them source information faster so what you're seeing is mostly just the usual behavior, but faster. Before AI people would not have bothered to try and figure out an answer to some of these questions. It would've been too much work.
  - keybored 9mo ago
    HN commenters will be technooptimistic misanthrops. Status quo ante bellum.
- andy_ppp 9mo ago
  The secret sauce about having good understanding, taste and style (both for coding and writing) has always been in the fine tuning and RHLF steps. I'd be skeptical if the signals a few GitHub repos or blogs generate at the initial stages of the learning are that critical. There's probably a filter also for good taste on the initial training set and these are so large not even a single full epoch is done on the data these days.
- jama211 9mo ago
  It wouldn’t work at all.

mexicocitinluez 9mo ago

> Ends up being circular if the author used LLM help for this writeup though there are no obvious signs of that. Great argument for not using AI-assisted tools to write blog posts (especially if you DO use these tools). I wonder how much we're taking for granted in these early phases before it starts to eat itself.

jama211 9mo ago
What does eating itself even look like? It doesn’t take much salt to change a hash.
- mexicocitinluez 9mo ago
  Being trained on it's own results?
  - jama211 9mo ago
    Pretty easy to detect for surely

anu7df 9mo ago

I understand model output put back into training would be an issue, but if model output is guided by multiple prompts and edited by the author to his/her liking wouldn't that at least be marginally useful?

TonyStr 9mo ago

Interestingly, I looked at github insights and found that this repo had 49 clones, and 28 unique cloners, before I published this article. I definitely did not clone it 49 times, and certainly not with 28 unique users. It's unlikely that the handful of friends who follow me on github all cloned the repo. So I can only speculate that there are bots scraping new public github repos and training on everything. Maybe that's obvious to most people, but it was a bit surprising to see it myself. It feels weird to think that LLMs are being trained on my code, especially when I'm painfully aware of every corner I'm cutting. The article doesn't contain any LLM output. I use LLMs to ask for advice on coding conventions (especially in rust, since I'm bad at it), and sometimes as part of research (zstd was suggested by chatgpt along with comparisons to similar algorithms).

nerdponx 9mo ago
Time to start including deliberate bugs. The correct version is in a private repository.
- teiferer 9mo ago
  And what purpose would this serve, exactly?
  - adastra22 9mo ago
    Spite.
- program_whiz 9mo ago
  while I think this is a fun idea -- we are in such a dystopian timeline that I fear you will end up being prosecuted under a digital equivalent of various laws like "why did you attack the intruder instead of fleeing" or "you can't simply remove a squatter because its your house, therefore you get an assault charge." A kind of "they found this code, therefore you have a duty not to poison their model as they take it." Meanwhile if I scrape a website and discover data I'm not supposed to see (e.g. bank details being publicly visible) then I will go to jail for pointing it out. :(

prodigycorp 9mo ago

Random aside about training data: One of the funniest things I've started to notice from Gemini in particular is that in random situations, it talks with english with an agreeable affect that I can only describe as.. Indian? I've never noticed such a thing leak through before. There must be a ton of people in India who are generating new datasets for training.

blenderob 9mo ago
That's very interesting. Any examples you can share which has those agreeable effects?
- prodigycorp 9mo ago
  I'm going to do a cursory look through my antigrav history, i want to find it too. I remember it's primarily in the exclamations of agreement/revelation, and one time expressing concern which I remember were slightly off natural for an american english speaker.
  - prodigycorp 9mo ago
    Cant find anything, too many messages telling the agent "please do NOT thosec changes". I'm going to remember to save them going forward.
evntdrvn 9mo ago
There was a really great article or blog post published in the last few months about the author's very personal experience whose gist was "People complain that I sound/write like an LLM, but it's actually the inverse because I grew up in X where people are taught formal English to sound educated/western, and those areas are now heavily used for LLM training." I wish I could find it again, if someone else knows the link please post it!
- gxnxcxcx 9mo ago
  I'm Kenyan. I don't write like ChatGPT, ChatGPT writes like me https://news.ycombinator.com/item?id=46273466 https://news.ycombinator.com/item?id=46273466