3 ms·
The more interesting thing for me is: the crawler didn't detect it on its own that it racked up 10TB from one site in one day. If I would design a crawler, I'd
by mschuster91 2y ago
The more interesting thing for me is: the crawler didn't detect it on its own that it racked up 10TB from one site in one day.
If I would design a crawler, I'd keep at least some basic form of tracking, if only to check for people deliberately trolling me by delivering me an infinite chain of garbage.
- bastawhiz 2y agoI've been running guthib.mattbasta.workers.dev for years, and in the past few months it's hit the free limit for requests every day. Some AI company is filling their corpus with exactly that: infinite garbage.