3 ms·
That’s so many scrapers. There must be a ton of companies with very large document collections at this point, and it really sucks that they don’t at least do us
by n1xis10t 10mo ago
That’s so many scrapers. There must be a ton of companies with very large document collections at this point, and it really sucks that they don’t at least do us the courtesy of indexing them and making them available for keyword search, but instead only do AI.
It’s kind of crazy how much scraping goes on and how little search engine development goes on. I guess search engines aren’t fashionable. Reminds me of this article about search engines disappearing mysteriously: https://archive.org/details/search-timeline https://archive.org/details/search-timeline
I try to share that article as much as possible, it’s interesting.
- kstrauser 10mo agoSo! Much! Scraping! They were downloading every commit multiple times, and fetching every file as seen at each of those commits, and trying to download archives of all the code, and hitting `/me/my-repo/blame` endpoints as their IP's first-ever request to my server, and other unlikely stuff. My scraper dudes, it's a git repo. You can fetch the whole freaking thing if you wanna look at it. Of course, that would require work and context-aware processing on their end, and it's easier for them to shift the expense onto my little server and make me pay for their misbehavior.
- n1xis10t 10mo agoCrazy
- miki123211 10mo agoBut there is a lot of search engine development going on, it's just that the results of the new search engines are fed straight into AI instead of displayed in the legacy 10-links-per-page view.
- PeterStuer 10mo agoOr some anti-ddos/bot companies using ultra cheap scraping services to annoy you enough to get you into their "free" anti bot protection, so they can charge the few real ai scrapers for access to your site.
- throw10920 10mo agoIs there any evidence that this has actually happened?
- zhengyi13 10mo agoEven if there isn't (yet?), there's probably someone who's honestly thinking this is potentially a viable business model and at least napkin-mathing it out.
- kstrauser 10mo agoMy napkin mathing is that their ROI would be negative. That's a lot of compute and bandwidth they'd have to pay for even if they were just throwing away the results.
- throw10920 10mo agoSo, it hasn't happened, and you're just making stuff up.
- mrweasel 10mo ago> There must be a ton of companies with very large document collections at this point See, I don't think there is, I don't think they want that expense. It's basically the Linus Torvalds philosophy of data storage, if it's on the Internet, I don't need a backup. While I have absolutely no proof of this, I'd guess that many AI companies just crawl the Internet constantly, never saving any of the data. We're seeing some of these scrapers go to great length attempting to circumvent any and all forms of caching, they aren't interested in having a two week old copy of anything.
- n1xis10t 10mo agoCould be. Can you train a model without saving things though?
- kelvinjps10 10mo agoWhere did Linus Torvalds expressed this philosophy I have never seen it
- lelanthran 10mo ago> Where did Linus Torvalds expressed this philosophy I have never seen it https://www.goodreads.com/quotes/574706-only-wimps-use-tape-backup-real-men-just-upload-their https://www.goodreads.com/quotes/574706-only-wimps-use-tape-...
- rurban 10mo agoJust block all the big hosters IP ranges, when they ignore robots.txt. For fun add long timeouts and huge content sizes. No private individual will browse from there, and all scrapers will do.