4 ms·
Ask HN: Scaling a targeted web crawler beyond 500M pages/day
I've been reading up on crawler architecture. The two most useful sources I've found are the blog post "Crawling a billion web pages in just over 24 hours, in 2025" and the Mercator paper ("Mercator: A Scalable, Extensible Web Crawler").
Both of these, and most other material I've come across, focus on crawling the broad open web rather than a targeted set of domains. For product prices it's the latter. Mercator calls out DNS resolution as a major bottleneck, for example, but when you're only hitting a few hundred domains that isn't really a concern.
The other gap is that both assume static HTML. For our use case we need a headless browser, and we also have to deal with Cloudflare and similar anti-bot systems.
For product prices specifically, a lot of sites publish price feeds which simplifies things, but plenty don't, and getting good coverage still requires scraping. Our current system does about 500M pages/day and we're looking to improve its performance.
Does anyone here have experience in this space, or know of articles/blog posts on scaling targeted (rather than broad) crawlers with headless browsers? Any pointers appreciated.
- deleted 5mo ago[deleted]
- 4lx87 5mo agoI'm curious, how do you deal with Cloudflare and similar anti-bot systems? Just keep shopping the job around to different proxies?
- fragmede 5mo agoCloudflare reads this forum. By answering your question here, they burn that workaround. Why would someone do that? (No one bring up Warframe)
- faangguyindia 5mo agoit's fairly simple, you use browser profiles and you visit multiple website like a normal guy using residential proxyy network and cloudflare cannot detect you this way. the older your browser profile is, the less often cloudflare bans.
- faangguyindia 5mo agoIf you want to access data from websites which prevent it, you gotta use a headless browser with Residential Proxy Network Like Bright Data (formerly Luminati).
- nicbou 5mo agoOur industry's understanding of consent is terrifying
- jeong_jeong 5mo agoIt’s called hacker news, bro
- fragmede 5mo agohave you already incorporated common crawl into your index?
- ccgreg 5mo agoCommon Crawl is a sample of the web, so it's not that directly helpful for someone wanting to make a product price dataset.
- ddactic 5mo ago[dead]