8 ms·
About 10 years ago, I was working on a site that served several hundred million non-crawler hits a month. Many of our millions of pages had their content change
by rented_mule 3y ago
About 10 years ago, I was working on a site that served several hundred million non-crawler hits a month. Many of our millions of pages had their content change multiple times a day. Because of the popularity and frequent changes, the crawlers hit us constantly... crawlers accounted for ~90% of our traffic - billions of hits per month. Bing was ~70% of the crawler traffic and Google was ~25% of it. We noticed it because Bing quickly became very aggressive about crawling, exposing some of our scaling limits as they doubled our already significant traffic in a few short months.
I was working on the system that picked ads to show on our pages (we had our own internal ad system, doing targeting based on our own data). This was the most computationally intensive part of serving our pages and the ads were embedded directly in the HTML of the page. When we realized that 90% of our ad pick infrastructure was dedicated to feeding the crawlers, we immediately thought of turning ads off for them (we never billed advertisers for them anyway). But hiding the ads seemed to go directly against the spirit of Google's policy of showing their crawlers the same content.
Among other things, we ended up disabling almost all targeting and showing crawlers random ads that roughly fit the page. This dropped our ad pick infra costs by nearly 80%, saving 6-figures a month. It also let us take a step back to decide where we could make long term investments in our infra rather than being overwhelmed with quick fixes to keep the crawlers fed.
This kind of thing is what people are missing when they wonder why a company needs more than a few engineers - after all, someone could duplicate the core functionality of the product in 100 lines of code. At sufficient scale, it takes real engineering just to handle the traffic from the crawlers so they can send you more users. There are an untold number of other things like this that have to be handled at scale, but that are hard to imagine if you haven't worked at similar scale.
- bombcar 3y ago90% of traffic being crawlers seems (on the face) just absolutely batshit insane.
- Retr0id 3y agoAdds a lot of weight to the dead internet theory.
- syndicatedjelly 3y agocan you elaborate?
- pmontra 3y agoI googled this: https://en.wikipedia.org/wiki/Dead_Internet_theory https://en.wikipedia.org/wiki/Dead_Internet_theory Actually everything we discussed here is the result of genuine human activities.
- syndicatedjelly 3y agoI still think that humans are very good at identifying other humans, particularly through long-form speech and writing. Sentient and non-sentient beings alike are very good at identifying members of their own species. I wonder if there's some sort of "time" threshold for how long an AI can speak/write before it is identifiable as an AI to a human. Some sort of Moore's law, but for AI recognizability
- geraldhh 3y agotbh alot of this "theory" is common sense for internet natives. "most of everything is shit" comes to mind, but "most of email being spam" and "most of web-traffic being porn" are well known.
- amenhotep 3y agoSeems like a natural consequence of having "millions of pages", if you think about it? You might have a lot of users, but they're only looking at what they want to look at. The crawlers are hitting every single link, revisiting all the links they've seen before, their traffic scales differently.
- mckn1ght 3y agoI think you’re right. At first I thought “crawlers are actually creating large amounts of spam requests” but this is just the way a searchable web functions. The crawlers are just building the index of the internet.
- 0xDEAFBEAD 3y agoMaybe Google needs to implement an API where you can notify it when a page on your site has changed. That should cut down on redundant crawls a lot, eh?
- rented_mule 3y agoWe very much wanted this! We had people that were ex-Google and ex-Bing who reached out to former colleagues, but nothing came of it. You'd think it would be in their interest, too. The best explanation I can come up with is that a failure to notify them of a change makes them look bad when their search results are out of date. Especially if the failures are malicious, fitting in with the general theme of the article.
- tech234a 3y agoIn 2021 Bing, Yandex, Seznam.cz, and (later, in 2023) Naver ended up implementing a standard where you can notify one search engine of a page update and the other participating search engines are also notified [1, 2, 3]. [1]: https://www.indexnow.org/ https://www.indexnow.org/ [2]: https://www.bing.com/indexnow https://www.bing.com/indexnow [3]: https://blogs.bing.com/webmaster/october-2021/IndexNow-Instantly-Index-your-web-content-in-Search-Engines https://blogs.bing.com/webmaster/october-2021/IndexNow-Insta...
- sidewndr46 3y agoThe number I came up with last time I looked into this was about 60% of page requests are by bots on any normal website.
- jgalt212 3y agoWe have some very long tail content and experienced this in 2023 after all the VC funded LLM start-ups tried to scrape every page ever.
- bboygravity 3y agoOr you could also NOT serve targetted ads?
- CaptainOfCoit 3y ago> Among other things, we ended up disabling almost all targeting and showing crawlers random ads that roughly fit the page Seems like that's exactly what they did...
- driverdan 3y agoFor everyone, not just crawlers.
- CaptainOfCoit 3y ago[flagged]
- tempestn 3y agoThey're serving first party targeted ads based on only their own data. If you're going to complain about that, it's close to saying that websites shouldn't be able to make money from advertising at all.
- rented_mule 3y agoVery much this. It's a site/app that has probably been used by 80-90% of adults living in America over the last decade. It would not exist if these ads weren't targeted. I know because we knew (past tense because I'm no longer there) exactly how much targeting increased click-through-rate and how that affected revenue. On top of that, they were ads for doing more of what the user was doing right then, tailored to tastes we'd seen them exhibit over time. Our goal was that the ads should be relevant enough that they served as an exploration mechanism within the site/app. We didn't always do as well as we hoped there, but it was a lot better than what you see on most of the internet. And far less intrusive because they weren't random (i.e., un-targeted). I have run ad blockers plus used whole house DNS ad blocking as long as I've been aware of them, but I was fine working on these ads because it felt to me like ads done right. If we can't even allow for ads done right, then vast swaths of the internet have to be pay-walled or disappear. One consequence of that... only the rich get to use most of the internet. That's already too true as it is, I don't want to see it go further.
- ixaxaar 3y agoWhy not just block bing and save 70% straight away? Nobody uses bing anyway.
- keithalewis 3y agoNobody goes there anymore, it's too crowded.
- nucleative 3y agoLove finding a Yogi Berra quote in the wild
- fuzztester 3y agoNobody quotes Yogi Berra anymore, he's too popular.
- usr1106 3y agoI have never used Bing. I use duckduckgo though and they buy their results from Bing. At least they did in the past, I don't follow them closely enough to necessarily notice every possible change.
- Supermancho 3y agoSpecific Google searches are often useless, so I switch to Bing at work and home as needed.
- NicoJuicy 3y agoCrawler hints and index now could reduce that traffic these days - https://developers.cloudflare.com/cache/advanced-configuration/crawler-hints/ https://developers.cloudflare.com/cache/advanced-configurati... - https://www.indexnow.org/ https://www.indexnow.org/
- araes 3y agoThis seems very cannibalistic of their own business. That means somebody running Google or Microsoft (or really any web ads) only has a 10% chance to start with of getting served to an actual human (if they're not trying to block each other constantly). And on the other side, that means every customer or ad placer, has to try and filter all the bots so people with actual credit cards and money will see the Google, TEMU, or FB ads (or others). In some ways, almost feels like Microsoft is griefing online search by burying it under massive robot crawls. Like an ad DDOS.