5 ms·

Isn't this just poor, sloppy crawler implementation? You shouldn't need to fetch a repo more than once to add it to a training set.

diggan 2y ago
Sounds like it me. Why build a crawler that fetches one HTML page per commit in a repository instead of doing a bare-clone and then just get the data from there? Surely would contain even more data too, compared to the HTML page.
csydas 2y ago
You are correct it's poor and sloppy, but it's not "just" that. It's a lack of concern over the effects of their poor/sloppy crawler implementation. The poor implementation is not really relevant, it's companies deciding they own the internet and can take whatever they want, let everyone else deal with the consequences. The companies do not care what the impact of their ai non-sense is..
- Forbo 2y ago
  A capitalist externalizing costs?! Why, no, never! /s
- fuzztester 2y ago
  https://news.ycombinator.com/item?id=43429532 https://news.ycombinator.com/item?id=43429532
Havoc 2y ago
It’s people that don’t care if they ruin things for everyone else. Crawlers have existed forever in search engine space and mostly behave. This sort of no rate limit, fake user agent, 100s of IPs approach used by AI teams is obviously intentionally not caring who it fucks over. More malicious than sloppy implementation
- mistrial9 2y ago
  it is an ecosystem of social roles, not just "people" .. casting the decision into individual choices is not the right filter to understand this situation..
  - wadadadad 2y ago
    I'm not sure I'm following what you mean by 'social roles'. Which roles are you referring to here? I'll disagree that it's not at least individual malicious choice, though. Someone decided that they needed to fake/change user agents (as one example), and implemented it. Most likely it was more than one person- some manager(s)/teams probably also either suggested or agreed to this choice. I would like to think at some point in this decision making process, someone would have considered 'is it ethical to change user agents to get around bans? Is it ethical to ignore robots.txt?' and decided not to proceed, but apparently that's not happening here...
mrweasel 2y ago
From reading Drew Devaults angry post from earlier this week, my take is that not only is it poorly implemented crawlers, it's also that it's cheaper to scrape, rather than keep copies on hand. Effectively these companies are outsourcing the storage of "their" training data to everyone on the internet. Ideally a site would get scraped once, and then the scraper would check if content has changed, e.g. etag, while also learning how frequently content changes. So rather than just hammer some poor personal git repo over and over, it would learn that Monday is a good time to check if something changed and then back off for a week.
- QuadrupleA 2y ago
  That seems crazy - millions of $ on GPUs but they can't afford some cheap storage? And direct network scraping seems super high latency. Although I guess a massive pretaining run might cycle through the corpus very slowly. Dunno, sounds fishy.
  - danaris 2y ago
    It's not that they can't afford storage. It's that not doing so means they can increase their profit numbers just a skoshe more. And at least as long as they haven't IPOed, that number's the only thing that matters. Everything getting in the way of increasing it is just an obstacle to be removed.
  - dalke 2y ago
    I see ChatGPT's bots pull down all of my Python wheels every couple of weeks. Wheels that haven't changed in years, with a "Last-Modified" and "ETag" that haven't changed. The only thing that makes sense to me is it's cheaper them to re-pull and re-analyze the data than to develop a cache.
    - fuzztester 2y ago
      Or, could it be, just possibly be (gasp), that some of the devs at these "hotshot" AI companies are just ignorant or lazy or pressured enough, so as to not do such normal checks? Wouldn't be surprised if so.
blibble 2y ago
they're probably using their own garbage AI to write the crawler the result? a mixed up version of 5000 plagiarised "baby's first webcrawler" github projects
Sharlin 2y ago
The brute-force solution to the first hard problem in computer science is to not have a cache at all.
- mycall 2y ago
  Yet in these cases mentioned in the article, if they had an HTTP static cache version of each page, using git hooks to refresh them, the bots would be negligable to their services. That is assuming the bots use HTTP 80/443 instead of git 9418
superkuh 2y ago
And poor, sloppy, website implementation. If your professional website can't handle 20k hits it's ... well, poor. Because my home connection hosted on my desktop PC website tanked 20k hits from alibaba bot (among a few more thousand of normal traffic) yesterday without missing a beat. It is literally the point of public websites to answer HTTP requests. If yours can't you're doing something wrong.

diggan 2y ago

Sounds like it me. Why build a crawler that fetches one HTML page per commit in a repository instead of doing a bare-clone and then just get the data from there? Surely would contain even more data too, compared to the HTML page.

csydas 2y ago

You are correct it's poor and sloppy, but it's not "just" that. It's a lack of concern over the effects of their poor/sloppy crawler implementation. The poor implementation is not really relevant, it's companies deciding they own the internet and can take whatever they want, let everyone else deal with the consequences. The companies do not care what the impact of their ai non-sense is..

Forbo 2y ago
A capitalist externalizing costs?! Why, no, never! /s
fuzztester 2y ago
https://news.ycombinator.com/item?id=43429532 https://news.ycombinator.com/item?id=43429532

Havoc 2y ago

It’s people that don’t care if they ruin things for everyone else. Crawlers have existed forever in search engine space and mostly behave. This sort of no rate limit, fake user agent, 100s of IPs approach used by AI teams is obviously intentionally not caring who it fucks over. More malicious than sloppy implementation

mistrial9 2y ago
it is an ecosystem of social roles, not just "people" .. casting the decision into individual choices is not the right filter to understand this situation..
- wadadadad 2y ago
  I'm not sure I'm following what you mean by 'social roles'. Which roles are you referring to here? I'll disagree that it's not at least individual malicious choice, though. Someone decided that they needed to fake/change user agents (as one example), and implemented it. Most likely it was more than one person- some manager(s)/teams probably also either suggested or agreed to this choice. I would like to think at some point in this decision making process, someone would have considered 'is it ethical to change user agents to get around bans? Is it ethical to ignore robots.txt?' and decided not to proceed, but apparently that's not happening here...

mrweasel 2y ago

From reading Drew Devaults angry post from earlier this week, my take is that not only is it poorly implemented crawlers, it's also that it's cheaper to scrape, rather than keep copies on hand. Effectively these companies are outsourcing the storage of "their" training data to everyone on the internet. Ideally a site would get scraped once, and then the scraper would check if content has changed, e.g. etag, while also learning how frequently content changes. So rather than just hammer some poor personal git repo over and over, it would learn that Monday is a good time to check if something changed and then back off for a week.

QuadrupleA 2y ago
That seems crazy - millions of $ on GPUs but they can't afford some cheap storage? And direct network scraping seems super high latency. Although I guess a massive pretaining run might cycle through the corpus very slowly. Dunno, sounds fishy.
- danaris 2y ago
  It's not that they can't afford storage. It's that not doing so means they can increase their profit numbers just a skoshe more. And at least as long as they haven't IPOed, that number's the only thing that matters. Everything getting in the way of increasing it is just an obstacle to be removed.
- dalke 2y ago
  I see ChatGPT's bots pull down all of my Python wheels every couple of weeks. Wheels that haven't changed in years, with a "Last-Modified" and "ETag" that haven't changed. The only thing that makes sense to me is it's cheaper them to re-pull and re-analyze the data than to develop a cache.
  - fuzztester 2y ago
    Or, could it be, just possibly be (gasp), that some of the devs at these "hotshot" AI companies are just ignorant or lazy or pressured enough, so as to not do such normal checks? Wouldn't be surprised if so.

blibble 2y ago

they're probably using their own garbage AI to write the crawler the result? a mixed up version of 5000 plagiarised "baby's first webcrawler" github projects

Sharlin 2y ago

The brute-force solution to the first hard problem in computer science is to not have a cache at all.

mycall 2y ago
Yet in these cases mentioned in the article, if they had an HTTP static cache version of each page, using git hooks to refresh them, the bots would be negligable to their services. That is assuming the bots use HTTP 80/443 instead of git 9418

superkuh 2y ago

And poor, sloppy, website implementation. If your professional website can't handle 20k hits it's ... well, poor. Because my home connection hosted on my desktop PC website tanked 20k hits from alibaba bot (among a few more thousand of normal traffic) yesterday without missing a beat. It is literally the point of public websites to answer HTTP requests. If yours can't you're doing something wrong.