3 ms·
> Google started locking down the API almost immediately after we published our study Isn't this ironic, given how google bots scour the web relentlessly and h
by m463 3y ago
> Google started locking down the API almost immediately after we published our study
Isn't this ironic, given how google bots scour the web relentlessly and hammer sites almost to death?
- dotandgtfo 3y agoThis is one of the most important parts of the EUs upcoming digital services act in my opinion. Platforms have to share data with (vetted) researchers, public interest groups and journalists.
- vasco 3y agoFor aggregated data and stats like this I think it could be fully publicly available.
- hotstickyballs 3y agoVetted always means people with the time, resources and desire to navigate through the vetting process, which makes them biased.
- lillecarl 3y agoI would argue it's better than nothing, and what are they going to be biased towards?
- fsckboy 3y agoAre you talking about Europe? they're certainly going to be biased against Google and any US tech giant. I'm biased against Google, but I'm honest about it. I don't ask "what could I possibly be biased about?"
- weeblewobble 3y agoYou might say the same thing about doing research in general
- LeonM 3y ago> google bots scour the web relentlessly and hammer sites almost to death I have been hosting sites and online services for a long time now and never had this problem, or heard of this issue ever before. If your site can't even handle a crawler, you need to seriously question your hosting provider, or your architecture.
- sbarre 3y agoPerhaps stop and reconsider such a dismissive opinion given that "you've never had this issue before" then? Or go read up a bit more on how crawlers work in 2023. If your site is very popular and the content changes frequently, you can find yourself getting crawled a higher frequency than you might want, particularly since Google can crawl your site at a high rate of concurrency, hitting many pages at once, which might not be great for your backend services if you're not used to that level of simultaneous traffic. "Hammered to death" is probably hyperbole but I have worked with several clients who had to use Google's Search Console tooling[0] to rate-limit how often Googlebot crawled their site because it was indeed too much. 0: https://developers.google.com/search/docs/crawling-indexing/reduce-crawl-rate https://developers.google.com/search/docs/crawling-indexing/...
- pas 3y agoif your site is popular and you have a problem with crawlers use robots.txt (in particular the Crawl-delay stanza) also for less friendly crawlers a rate limiter is needed anyway :( (of course the existence of such tools doesn't give carte blanche to any crawler to overload sites ... but let's say they implement some sensing, based on response times, that means a significant load is probably needed to increase response times, which definitely can raise some eyebrows, and with autoscaling can cost a lot of money to site operators)
- holoduke 3y agoI have a website thats get crawled at least 50 times per second. Is that a real deal? No not really. The site is probably doing 10.000 requests per second. I mean a popular site is indexed a lot. Your webserver should be designed for it. What tech are you using if I may ask?
- LocalH 3y ago"Rules for thee, but not for me"