4 ms·
Where on Earth do people get the belief that: - It's the SOTA companies doing it? - Scrapers are doing it for training data? Those are two assumptions I see
by TeMPOraL 1mo ago
Where on Earth do people get the belief that:
- It's the SOTA companies doing it?
- Scrapers are doing it for training data?
Those are two assumptions I see in posts and threads around Anubis, that are taken at faith, and never once substantiated.
- inigyou 1mo agoWhat's your alternative hypothesis?
- mschuster91 1mo ago> It's the SOTA companies doing it? There are more than just the American top dogs (OAI, Anthropic, SpaceX, Facebook)... especially the Chinese government with all its infinite cash resources and next to zero ethical constraints. I don't trust the US top dogs at all, but I think the fear of discovery alone would lead them to not use "residential proxy" services. Non-US/EU entities however... who cares?
- figglestar 1mo agoWhy would they directly use a proxy service? I'd just launder the data scraping through some third party company that I could slough off if it ever turned into a news story. Not that anything would happen to them if they directly used these services anyway.
- shakna 1mo agoBecause Anthropic already admitted it? [0] [0] https://www.ft.com/content/07611b74-3d69-4579-9089-f2fc2af61baa?syn-25a6b1a6=1 https://www.ft.com/content/07611b74-3d69-4579-9089-f2fc2af61...
- inigyou 1mo agoIt says "accused of"
- shakna 1mo ago... It also has a quote from Anthropic, admitting they backed off their scraper after a robots.txt update.
- inigyou 1mo agoNobody who gives a shit about their software actually working obeys robots.txt. most robots.txt block everything except googlebot, which is an insane policy and you don't have to follow it.
- shakna 1mo agoWhich isn't all that relevant a discussion here. Anthropic saying they did so in this case, means that they admit they were the scraper flooding the domain.