3 ms·

Can someone with more experience developing AI tools explain what these bots are mostly doing? Are they collecting data for training, or are they for the more r

Can someone with more experience developing AI tools explain what these bots are mostly doing? Are they collecting data for training, or are they for the more recent search functionality? Or are they enhancing responses with links?

xena 2y ago
AI expert here. It's probably for collecting training data and the crawlers are probably very unsupervised. I'd guess that they're literally the most simplistic crawler code you can imagine combined with parallelism across machines. The good news is that it's easy to disrupt these crawlers with some easy hacks. Tactical reverse slowloris is probably gonna make a comeback.
- yetihehe 2y ago
  Zipbombing rogue AI's will be the new hotness.
  - xena 2y ago
    Unironically I tried that. Limited success.
- biophysboy 2y ago
  If its for training data, why are they straining FOSS so much? Is there thousands of actors repeatedly making training data all the time? I thought it was a sort of one-off thing w/ the big tech players.
  - xena 2y ago
    Git forges are some of the worst case for this. The scrapers click on every link on every page. If you do this to a git forge, it gets very O(scary) very fast because you have to look at data that is not frequently looked at and will NOT be cached. Most of how git forges are fast is through caching. The thing about AI scrapers is that they don't just do this once. They do this every day in case every file in a glibc commit from 15 years ago changed. It's absolutely maddening and I don't know why AI companies do this, but if this is not handled then the git forge falls over and nobody can use it. Anubis is a solution that should not have to exist, but the problem it solves is catastrophically bad, so it needs to exist.
    - biophysboy 2y ago
      That's very strange to me that they do it everyday. I thought training runs took months. Do they throw away the vast majority of their training attempts (e.g. one had suboptimal hyperparameters, etc)?
      - xena 2y ago
        Training attempts != training data

xena 2y ago

AI expert here. It's probably for collecting training data and the crawlers are probably very unsupervised. I'd guess that they're literally the most simplistic crawler code you can imagine combined with parallelism across machines. The good news is that it's easy to disrupt these crawlers with some easy hacks. Tactical reverse slowloris is probably gonna make a comeback.

yetihehe 2y ago
Zipbombing rogue AI's will be the new hotness.
- xena 2y ago
  Unironically I tried that. Limited success.
biophysboy 2y ago
If its for training data, why are they straining FOSS so much? Is there thousands of actors repeatedly making training data all the time? I thought it was a sort of one-off thing w/ the big tech players.
- xena 2y ago
  Git forges are some of the worst case for this. The scrapers click on every link on every page. If you do this to a git forge, it gets very O(scary) very fast because you have to look at data that is not frequently looked at and will NOT be cached. Most of how git forges are fast is through caching. The thing about AI scrapers is that they don't just do this once. They do this every day in case every file in a glibc commit from 15 years ago changed. It's absolutely maddening and I don't know why AI companies do this, but if this is not handled then the git forge falls over and nobody can use it. Anubis is a solution that should not have to exist, but the problem it solves is catastrophically bad, so it needs to exist.
  - biophysboy 2y ago
    That's very strange to me that they do it everyday. I thought training runs took months. Do they throw away the vast majority of their training attempts (e.g. one had suboptimal hyperparameters, etc)?
    - xena 2y ago
      Training attempts != training data