4 ms·
90% of traffic being crawlers seems (on the face) just absolutely batshit insane.
by bombcar 3y ago
90% of traffic being crawlers seems (on the face) just absolutely batshit insane.
- Retr0id 3y agoAdds a lot of weight to the dead internet theory.
- syndicatedjelly 3y agocan you elaborate?
- pmontra 3y agoI googled this: https://en.wikipedia.org/wiki/Dead_Internet_theory https://en.wikipedia.org/wiki/Dead_Internet_theory Actually everything we discussed here is the result of genuine human activities.
- syndicatedjelly 3y agoI still think that humans are very good at identifying other humans, particularly through long-form speech and writing. Sentient and non-sentient beings alike are very good at identifying members of their own species. I wonder if there's some sort of "time" threshold for how long an AI can speak/write before it is identifiable as an AI to a human. Some sort of Moore's law, but for AI recognizability
- geraldhh 3y agotbh alot of this "theory" is common sense for internet natives. "most of everything is shit" comes to mind, but "most of email being spam" and "most of web-traffic being porn" are well known.
- amenhotep 3y agoSeems like a natural consequence of having "millions of pages", if you think about it? You might have a lot of users, but they're only looking at what they want to look at. The crawlers are hitting every single link, revisiting all the links they've seen before, their traffic scales differently.
- mckn1ght 3y agoI think you’re right. At first I thought “crawlers are actually creating large amounts of spam requests” but this is just the way a searchable web functions. The crawlers are just building the index of the internet.
- 0xDEAFBEAD 3y agoMaybe Google needs to implement an API where you can notify it when a page on your site has changed. That should cut down on redundant crawls a lot, eh?
- rented_mule 3y agoWe very much wanted this! We had people that were ex-Google and ex-Bing who reached out to former colleagues, but nothing came of it. You'd think it would be in their interest, too. The best explanation I can come up with is that a failure to notify them of a change makes them look bad when their search results are out of date. Especially if the failures are malicious, fitting in with the general theme of the article.
- tech234a 3y agoIn 2021 Bing, Yandex, Seznam.cz, and (later, in 2023) Naver ended up implementing a standard where you can notify one search engine of a page update and the other participating search engines are also notified [1, 2, 3]. [1]: https://www.indexnow.org/ https://www.indexnow.org/ [2]: https://www.bing.com/indexnow https://www.bing.com/indexnow [3]: https://blogs.bing.com/webmaster/october-2021/IndexNow-Instantly-Index-your-web-content-in-Search-Engines https://blogs.bing.com/webmaster/october-2021/IndexNow-Insta...
- 0xDEAFBEAD 3y ago>The best explanation I can come up with is that a failure to notify them of a change makes them look bad when their search results are out of date. Especially if the failures are malicious, fitting in with the general theme of the article. Should be easy to crosscheck the reliability of update notifications by doing a little bit of polling too.
- sidewndr46 3y agoThe number I came up with last time I looked into this was about 60% of page requests are by bots on any normal website.
- jgalt212 3y agoWe have some very long tail content and experienced this in 2023 after all the VC funded LLM start-ups tried to scrape every page ever.