3 ms·
Anything they can, judging by the fact that they're hitting random endpoints instead of using those offered to developers. Similar thing happened to readthedocs
by lovethevoid 2y ago
Anything they can, judging by the fact that they're hitting random endpoints instead of using those offered to developers. Similar thing happened to readthedocs[1] causing a surge of costs that nobody wants to answer for.
In the readthedocs situation there was one case that was a bugged crawler causing it to try and scrape the same HTML files repeatedly to the tune of 75TB, could also be happening here with OSM (partially).
[1] https://about.readthedocs.com/blog/2024/07/ai-crawlers-abuse/ https://about.readthedocs.com/blog/2024/07/ai-crawlers-abuse...