7 ms·
Crawling a quarter billion webpages in 40 hours (2012)
- marginalia_nu 3y agoArchive link: https://archive.is/yUWjh https://archive.is/yUWjh Also kinda wish the author paid any sort of attention to the fact that doing this incorrectly may create a flood of DNS queries. At least have the decency to set up a bind cache or something.
- swyx 3y agowhat is a bind cache? always assume most of us have terrible knowledge of networking
- marginalia_nu 3y agoI mean running bind[1] locally configured to act as a DNS cache. The operating system does some DNS caching as well, but it's not really tuned for crawling, and as a result it's very easy to end up spamming innocent DNS servers with an insane amount of lookup requests. [1] https://www.isc.org/bind/ https://www.isc.org/bind/
- swyx 3y agook but in my understanding isnt DNS also cached on the nearest wifi/ISP routers? the whole DNS system is just layer after layer of caches right? i.e. does caching on local machine actually matter? (real question, i dont know)
- marginalia_nu 3y agoYeah sure, but most of those caching layers (including possibly on the ISP level) aren't really configured for the DNS flood a crawling operation may result in. If you're going to do way more DNS lookups than are expected from your connection, it's a good custom to provide your own caching layer that's scaled accordingly. Not doing so probably won't break anything, but it risks degrading the DNS performance of other people using the same resolvers.
- swyx 3y agogotcha. thanks for indulging my curiosity! hopefully others will learn good dns hygiene from this as well.
- bottom999mottob 3y agoJust use 500 android phones running Tmux they said. It'll be easy they said.
- jasonjayr 3y agoImagine a Beowulf cluster of android phones running tmux!
- marginalia_nu 3y agoBe careful with that reference, it's an antique.
- 0xdeadbeefbabe 3y agoA Beowulf cluster running screen
- CaptainNegative 3y agoNetcraft now confirms: Beowulf cluster memes are dead.
- ddorian43 3y agoWith async in any language of choice this should need just 1 server. 250M/40hours=1.7K requests/second. You can probably do it on a single core in a low level language like Rust/Java.
- marginalia_nu 3y agoThreads isn't the only bottleneck in crawling. Assuming you're crawling at a civilized rate of about 1 request/second, you're only so many network hiccups away from consuming the entire ephemeral port range with connections in TIME_WAIT or CLOSE_WAIT.
- sethammons 3y agoCrank up the ulimit. And what is this 1 req/second nonsense? 1 req/sec/domain maybe. I have to agree, my first thought is "why this is not a single node in Go?"
- marginalia_nu 3y agoOh yeah, I mean 1 req/sec per domain of course. It's very easy to end up with tens of thousands of not-quite-closed connections while crawling, even with SO_LINGER(0), proper closure, tweaking the TCP settings, and doing all things by the book. It's a different situation from e.g. having a bunch of incoming connections to a node, the traffic patterns are not at all similar.
- merek 3y agoGreat to see this again. This was the article that introduced me to Redis (and more broadly the NoSQL rollercoaster) all those years ago!
- luckystarr 3y agoHow to get the top million sites list today? Alexa has shifted focus recently.
- jonatron 3y agoThere's a list here: https://radar.cloudflare.com/domains https://radar.cloudflare.com/domains
- gnfargbl 3y agoCheck out Tranco [1], which uses Cisco Umbrella, Majestic and also now a list sourced from Farsight passive DNS [2]. They're "working on" adding Chrome UX and Cloudflare Radar. There's also a list from Netcraft [3]. [1] https://tranco-list.eu/ https://tranco-list.eu/ [2] https://www.domaintools.com/resources/blog/mirror-mirror-on-the-wall-whos-the-fairest-website-of-them-all/ https://www.domaintools.com/resources/blog/mirror-mirror-on-... [3] https://trends.netcraft.com/topsites https://trends.netcraft.com/topsites
- zX41ZdbW 3y agoHere is an example of how to obtain a list of the top six million domains from Tranco and analyze their content with ClickHouse: https://github.com/ClickHouse/ClickHouse/issues/18842 https://github.com/ClickHouse/ClickHouse/issues/18842
- spyder 3y agoCommonCrawl / Open PageRank ? https://www.domcop.com/top-10-million-websites https://www.domcop.com/top-10-million-websites
- swyx 3y agoa bit of context - posted this because i found it via https://twitter.com/willdepue/status/1669074758208749572 https://twitter.com/willdepue/status/1669074758208749572 if they replicate it in 2023 it would be pretty interesting to me. i can think of a few times a year i need a good scraper. but also thought it a good look into 2012 Michael Nielsen, and into thinking about performance.
- m4r71n 3y agoA note of caution: never scrape the web from your local/residential network. Few months back I wanted to verify a set of around 200k URLs from a large data set that included a set of URL references for each object, and naively wrote a simple Python script that would use ten concurrent threads to ping each URL and record the HTTP status code that came back. I let this run for some time and was happy with the results, only to find out later that a large CDN provider has identified me as a spammy client with their client reputation score and blocked my IP address on all of the websites that they serve. Changing your IP address with AT&T is a pain (even though the claim is that your IP is dynamic, in practice it hardly ever changes) so I opted to contact the CDN vendor by filling out a form and luckily my ban was lifted in a day or two. Nevertheless, it was annoying that suddenly a quarter of the websites I normally visit were not accessible to me since the CDN covers a large swath of the Internet.
- ricardo81 3y agoIt makes very little difference what IP you scrape from, unless you're from a very dodgy subnet. The major content providers tend to go on a whitelist only based approach, you're either a human-like visitor or facing their anti-scraping methodologies.
- remram 3y agoI think the emphasis is on "never scrape from YOUR local/residential network".
- marginalia_nu 3y agoI run a search engine crawler from my residential network. I get this too sometimes, but a lot of the time the IP shit-listing is temporary. It also seems to happen more often if you don't use a high enough crawl delay, ignore robots.txt, do deep crawls ignoring HTTP 429 errors and so on. You know, overall being a bad bot. Overall, it's not as bad as it seems. I doubt anyone would accidentally damage their IP reputation doing otherwise above-board stuff.
- delfinom 3y ago>(even though the claim is that your IP is dynamic, in practice it hardly ever changes) Every ISP just uses DHCP for router IPs. It's dynamic, you just have to let the lease time expire to renew it. Or, have your own configurable router instead of the ISPs so that you can actually send a dhcp release command though they don't all support this part. Changing MAC Address will work otherwise.
- chaostheory 3y agoWhat makes this post more interesting is that Reddit might now be ushering in a new era of the crawler arms race
- krishadi 3y agoI've been recently reading up on multithreading and multiprocessing in python. You mention that you've taken a multi threaded approach since the processes are i/o bound. Is this the same as running the script with asyncio as async/await?
- krishadi 3y agoOops, I meant to ask the author but realised that the author is not the same as the op. Hah!
- Jorge1o1 3y agoAt a 10,000 foot view (pedants will take offense) you should look to use multiprocessing for tasks which are CPU bound, and asyncio/threads (but really asyncio if you can) for problems which are IO-bound. This is a massive simplification but most useful for a beginner. Additionally, asyncio is not the same as multithreading, because typically asyncio is powered by a single-threaded event loop and use of a mechanism like select/kqueue/IOCP/epoll.
- deleted 3y ago[deleted]
- polote 3y agoFor all the people that say this is easy. Try it ! That's not easy at all, I've tried it and spend a few weeks to get similar performance. Receiving thousands of request is not similar to making thousands of requests, you can saturate your network, saturated with latency of random websites, get site that never timeout, parse multi megabytes malformed html, get infinite redirections. My fastest implementation in python was actually using threads and was much faster than any async variant
- trevioustrouble 3y ago[flagged]
- KomoD 3y agoNo point being a douchebag.
- trevioustrouble 3y ago[flagged]
- bryanrasmussen 3y agodon't want to release the source code because of moral reservations about how people might use it. Ok upstanding citizen.
- EGreg 3y agoJust like AI people
- dspillett 3y agoNothing to stop you writing a similar project and releasing it if you feel so strongly that it should be out there, instead of being incensed by not being give then sweat off someone else's brow.
- trevioustrouble 3y ago[flagged]
- onion2k 3y agoTo save you a click: Use 20 machines.
- sethammons 3y ago250M / 40hrs / 60min / 60s ~= 1,737 rps. That over 20 machines is ~87 rps per machine. Depending on a few factors, I rough out my backend Go stuff to handle between 1-5k rps per machine before we have real numbers.
- onion2k 3y agoYou didn't write your Go stuff 11 years ago though...
- sethammons 3y agoWe started with Go version 1.2 which was released over ten years ago. Pretty darn close to 11 years. https://go.dev/doc/go1.2 https://go.dev/doc/go1.2
- haggy102 3y agoWow crazy how time flies. I did not realize Go has been in production for over 10 years now.
- samwillis 3y agoThis is a good overview, but being from 2012 it's missing comment on one now common area, using a real browser for crawling/scraping. You will often now hear recommendations to use a real browser, in headless mode, for crawling. There are two reasons for this: - SPAs with front end only rendering are hard to scrape with a traditional http library. - Anti bot/scraping technology fingerprints browsers, looks at request patterns, and browser behaviour to try and detect and block bots. Using a "real browser" is often advised as a way around these issues. However from my experience you should avoid headless browser crawling until it is absolutely necessary, I have found: - Headless browser scraping is between 10x and 100x more resource intensive, even if you carefully block requests and cache resources. - Most SPAs now have some level of server side rendering, and often that includes having a handy JSON in the returned document that contains the data you actually want. - Advanced browser fingerprinting is vanishingly rare. At most I have seen detection of user agent strings and comparing them to http headers and the order of them. If you make your http lib look like a current browser you are %99.9 of the way there.
- londons_explore 3y agoHeadless browsers are expensive, but if you needed them, I suspect there are easy wins to be had on the performance front. For example, you can probably skip all layout/rendering work and simply return faked values whenever javascript tries to read back a canvas it just drew into, or tries to read the computed width of some element. The vast majority of those things won't prevent the page loading.
- marginalia_nu 3y agoDepends on what you want with the data I guess. From a search engine point of view, the SPA case isn't that relevant, since SPAs can't reliably be linked to and in general tend to not be very stable and it's overall difficult to figure out how to enumerate and traverse their views. I think a good middle ground might be to do a first pass with a "stupid" crawler, and then re-visit the sites where you were blocked or that just contained a bunch of javascript with a headless browser.
- 3y ago
- spacebacon 3y agoHow to spend $580 in 40 hours. More can be done for much less in 2012.
- ricardo81 3y agoYes. I used lib curl's multi interface on one $40/m server around that time. Indeed at any scale the rate limiting becomes the main bottleneck, mainly because a lot of sites are concentrated on certain hosts. Speed isn't the problem and multiple servers aren't really needed.