6 ms·

Crawling a billion web pages in just over 24 hours, in 2025

199 points by pseudolus 8mo ago

finnlab 8mo ago
Nice work, but I feel like it's not required to use AWS for this. There are small hosting companies with specialized servers (50gbit shared medium for under 10$), you could probably do this under 100$ with some optimization.
- varispeed 8mo ago
  This. AWS is like a cash furnace, only really usable for VC backed efforts with more money than sense.
- nurettin 8mo ago
  I did some crawling on hetzner back in the day. They monitor traffic and make sure you don't automate publically available data retrieval. They send you an email telling you that they are concerned because you got the ip blacklisted. Funny thing is: They own the blacklist that they refer to.
  - qingcharles 8mo ago
    This. I tried to run a very slow DHT scraper I was writing on a Hetzner server and within minutes they were on my ass. I don't want to make an enemy of them so I killed it immediately, but they are clearly very sensitive to anything outside of "normal".
  - jeroenhd 8mo ago
    If Hetzner actually puts their own customers on their blacklist then that list becomes more trustworthy. They were right to blacklist you, they were right to complain to you, and they were right not to assume malice and kick you off their platform/shut down your server.
    - nurettin 8mo ago
      Yes I wasn't banned or anything, they aren't barbarians. Also, explain your opinion don't just put it out there. This is not a football match.
      - jeroenhd 8mo ago
        I felt like the arguments behind my opinion were implicit. Consistently not showing bias makes for trustworthy lists. Nobody (well, nobody reasonable I'd argue) will trust an IP blacklist from a major hosting company that actively excludes their own customers. As for banning customers that get blacklisted, this does actually happen, especially on the more affordable cloud hosts that get plagued with massive scraper and bot loads. Anything from suspended, inaccessible servers that require manual intervention to network caps or CPU load caps. This is a rather extreme measure, but not one uncommon or even unacceptable depending on the exact IP blacklist you managed to trigger.
        nurettin 8mo ago
        Maybe you didn't get the nuance here: Hetzner says (not exact wording) "according to this public blacklist you seem to be performing something they consider bad". What they don't mention is that they operate the list.
ph4rsikal 8mo ago
When I read this, I realize how small Google makes the Internet.
thefounder 8mo ago
Well the most important part seems to be glossed over and that’s the IP addresses. Many websites simply block /want to block anything that’s not google and is not a “real user”.
handfuloflight 8mo ago
There was a time when being able to do this meant you were on the path to becoming a (m)(b)illionaire. Still is, I think.
throwaway77385 8mo ago
> spinning disks have been replaced by NVMe solid state drives with near-RAM I/O bandwidth Am I missing something here? Even Optane is an order of magnitude slower than RAM. Yes, under ideal conditions, SSDs can have very fast linear reads, but IOPS / latency have barely improved in recent years. And that's what really makes a difference. Of course, compared to spinning disks, they are much faster, but the comparison to RAM seems wrong. In fact, for applications like AI, even using system RAM is often considered too slow, simply because of the distance to the GPU, so VRAM needs to be used. That's how latency-sensitive some applications have become.
- fluoridation 8mo ago
  >for applications like AI, even using system RAM is often considered too slow, simply because of the distance to the GPU That's not why. It's because RAM has a narrower bus than VRAM. If it was a matter of distance it'd just have greater latency, but that would still give you tons of bandwidth to play with.
  - dist-epoch 8mo ago
    You could be charitable and say the bus is narrow because it has to travel a long distance and this makes it hard to have a lot of traces.
    - fluoridation 8mo ago
      It's not. It's narrow even between the CPU and RAM. That's just the way x86 is designed. Nvidia and AMD by contrast have the luxury of being able to rearchitect their single-board computers each generation as long as they honor the PCIe interface. It is also true that having a 384-bit memory bus shared with the video card would necessitate a redesigned PCIe slot as well as an outrageous number of traces on the motherboard, though.
      - dist-epoch 8mo ago
        ThreadRipper has 8 memory channels versus 2 for a desktop AMD CPU. It's not an x86 limitation.
bndr 8mo ago
I run a small startup called SEOJuice, where I need to crawl a lot of pages all the time, and I can say that the biggest issue with crawling is the blocking part and how much you need to invest to circumvent Cloudflare and similar, just to get access to any website. The bandwith and storage are the smallest cost factor. Even though, in my case, users add their own domains, it's still took me quite a bit of time to reach 99% chance to crawl a website — with a mix of residential proxies, captcha solvers, rotating user-agents, stealth chrome binaries, otherwise I would get 403 immediately with no HTML being served.
- mrweasel 8mo ago
  Can't your users just whitelist your IPs?
  - bndr 8mo ago
    They're mostly non-technical/marketing people, but yes that would be a solution. I try to solve the issue "behind the scenes" so for them it "just works", but that means building all of these extra measures.
    - cassepipe 8mo ago
      Would it make sense to advertise to the more technical minded a discount if they set up an IP whitelist with a tutorial you could provide ? A discount in exchange for reduced costs to you ?
      - dolmen 8mo ago
        The right solution is to be registered at Cloudflare, but then getting the customer reach the guy who handles Cloudflare settings (a few clicks) is the hard part.
  - dewey 8mo ago
    I'm in a similar boat and getting customers to whitelist IPs is always a big ask. In the best case they call their "tech guy", in the worst case it's a department far away and it has to go through 3 layers of reviews for someone to adapt some Cloudflare / Akamai rules. And then you better make sure your IP is stable and a cloud provider isn't changing any IP assignments in the future, where you'll then have to contact all your clients again with that ask.
dangoodmanUT 8mo ago
> because redis began to hit 120 ops/sec and I’d read that any more would cause issues Suspicious. I don’t think I’ve ever read anything that says redis taps out below tens of thousands of ops…
sunpolice 8mo ago
I was able to get 35k req/sec on a single node with Rust (custom http stack + custom html parser, custom queue, custom kv database) with obsessive optimization. It's possible to scrape Bing size index (say 100B docs) each month with only 10 nodes, under 15k$. Thought about making it public but probably no one would use it.
- charlesdenault 8mo ago
  please do
  - mamsouuu 8mo ago
    Yes! Please do!
gethly 8mo ago
> I also truncated page content to 250KB before passing it to the parser. WTF did I just read?
- tengada1 8mo ago
  It's just HTML, presumably not requesting JS libraries. So 250K is a large amount.
  - gethly 8mo ago
    Exactly - how can a html page need to be trimmed to 250 KB??? That is insane. Something is not right with this article.
    - iggldiggl 8mo ago
      A transcript for a half-hour radio comedy show with some formatting takes up about 60 kB. The English Wikipedia page for Monty Python is about 130 kB in pure UTF-8 text and the actual HTML page takes up around around 660 kB (plus/minus, depending on which Wikipedia theme exactly you use). So large, text-heavy pages don't seem too unlikely to exceed 250 kB, especially if they also include some amount of formatting that's more substantial than just a minimal bunch of <p> tags.
snowhale 8mo ago
[dead]
- overfeed 8mo ago
  > this scale per-domain politeness queuing also becomes a genuine headache Not really a headache - if you've ever implemented resource-based, server-side rate limiting (per-endpoint, with client-ID and/or IP buckets), that's all the logic that's required, adapted for the client side. One could wrap rate-limiting libraries designed for server-side usage and call it a day. I hate how people who a bad at parallelizing their user-agents across the internet are causing needless pain and giving scrapers a bad name. They are also causing blowback on the more well-behaved scrapers.
corv 8mo ago
Python is obviously too slow for web-scale
T3RMINATED 8mo ago
[dead]
lovelearning 8mo ago
As an experiment, it's interesting. If anyone actually needs such a dataset, look into CommonCrawl first. I feel using something that already exists will be more cooperative and considerate than everyone overloading every website with their spider. https://commoncrawl.org/overview https://commoncrawl.org/overview
umairnadeem123 8mo ago
[dead]
mudkipdev 8mo ago
Does AWS actually allow you to crawl like this? I've been interested in a similar project but the cloud providers I typically use seem to ban it in their terms of service

199 points by pseudolus 8mo ago

Nice work, but I feel like it's not required to use AWS for this. There are small hosting companies with specialized servers (50gbit shared medium for under 10$), you could probably do this under 100$ with some optimization.

varispeed 8mo ago
This. AWS is like a cash furnace, only really usable for VC backed efforts with more money than sense.
nurettin 8mo ago
I did some crawling on hetzner back in the day. They monitor traffic and make sure you don't automate publically available data retrieval. They send you an email telling you that they are concerned because you got the ip blacklisted. Funny thing is: They own the blacklist that they refer to.
- qingcharles 8mo ago
  This. I tried to run a very slow DHT scraper I was writing on a Hetzner server and within minutes they were on my ass. I don't want to make an enemy of them so I killed it immediately, but they are clearly very sensitive to anything outside of "normal".
- jeroenhd 8mo ago
  If Hetzner actually puts their own customers on their blacklist then that list becomes more trustworthy. They were right to blacklist you, they were right to complain to you, and they were right not to assume malice and kick you off their platform/shut down your server.
  - nurettin 8mo ago
    Yes I wasn't banned or anything, they aren't barbarians. Also, explain your opinion don't just put it out there. This is not a football match.
    - jeroenhd 8mo ago
      I felt like the arguments behind my opinion were implicit. Consistently not showing bias makes for trustworthy lists. Nobody (well, nobody reasonable I'd argue) will trust an IP blacklist from a major hosting company that actively excludes their own customers. As for banning customers that get blacklisted, this does actually happen, especially on the more affordable cloud hosts that get plagued with massive scraper and bot loads. Anything from suspended, inaccessible servers that require manual intervention to network caps or CPU load caps. This is a rather extreme measure, but not one uncommon or even unacceptable depending on the exact IP blacklist you managed to trigger.
      - nurettin 8mo ago
        Maybe you didn't get the nuance here: Hetzner says (not exact wording) "according to this public blacklist you seem to be performing something they consider bad". What they don't mention is that they operate the list.

ph4rsikal 8mo ago

When I read this, I realize how small Google makes the Internet.

thefounder 8mo ago

Well the most important part seems to be glossed over and that’s the IP addresses. Many websites simply block /want to block anything that’s not google and is not a “real user”.

handfuloflight 8mo ago

There was a time when being able to do this meant you were on the path to becoming a (m)(b)illionaire. Still is, I think.

throwaway77385 8mo ago

> spinning disks have been replaced by NVMe solid state drives with near-RAM I/O bandwidth Am I missing something here? Even Optane is an order of magnitude slower than RAM. Yes, under ideal conditions, SSDs can have very fast linear reads, but IOPS / latency have barely improved in recent years. And that's what really makes a difference. Of course, compared to spinning disks, they are much faster, but the comparison to RAM seems wrong. In fact, for applications like AI, even using system RAM is often considered too slow, simply because of the distance to the GPU, so VRAM needs to be used. That's how latency-sensitive some applications have become.

fluoridation 8mo ago
>for applications like AI, even using system RAM is often considered too slow, simply because of the distance to the GPU That's not why. It's because RAM has a narrower bus than VRAM. If it was a matter of distance it'd just have greater latency, but that would still give you tons of bandwidth to play with.
- dist-epoch 8mo ago
  You could be charitable and say the bus is narrow because it has to travel a long distance and this makes it hard to have a lot of traces.
  - fluoridation 8mo ago
    It's not. It's narrow even between the CPU and RAM. That's just the way x86 is designed. Nvidia and AMD by contrast have the luxury of being able to rearchitect their single-board computers each generation as long as they honor the PCIe interface. It is also true that having a 384-bit memory bus shared with the video card would necessitate a redesigned PCIe slot as well as an outrageous number of traces on the motherboard, though.
    - dist-epoch 8mo ago
      ThreadRipper has 8 memory channels versus 2 for a desktop AMD CPU. It's not an x86 limitation.

bndr 8mo ago

I run a small startup called SEOJuice, where I need to crawl a lot of pages all the time, and I can say that the biggest issue with crawling is the blocking part and how much you need to invest to circumvent Cloudflare and similar, just to get access to any website. The bandwith and storage are the smallest cost factor. Even though, in my case, users add their own domains, it's still took me quite a bit of time to reach 99% chance to crawl a website — with a mix of residential proxies, captcha solvers, rotating user-agents, stealth chrome binaries, otherwise I would get 403 immediately with no HTML being served.

mrweasel 8mo ago
Can't your users just whitelist your IPs?
- bndr 8mo ago
  They're mostly non-technical/marketing people, but yes that would be a solution. I try to solve the issue "behind the scenes" so for them it "just works", but that means building all of these extra measures.
  - cassepipe 8mo ago
    Would it make sense to advertise to the more technical minded a discount if they set up an IP whitelist with a tutorial you could provide ? A discount in exchange for reduced costs to you ?
    - dolmen 8mo ago
      The right solution is to be registered at Cloudflare, but then getting the customer reach the guy who handles Cloudflare settings (a few clicks) is the hard part.
- dewey 8mo ago
  I'm in a similar boat and getting customers to whitelist IPs is always a big ask. In the best case they call their "tech guy", in the worst case it's a department far away and it has to go through 3 layers of reviews for someone to adapt some Cloudflare / Akamai rules. And then you better make sure your IP is stable and a cloud provider isn't changing any IP assignments in the future, where you'll then have to contact all your clients again with that ask.

dangoodmanUT 8mo ago

> because redis began to hit 120 ops/sec and I’d read that any more would cause issues Suspicious. I don’t think I’ve ever read anything that says redis taps out below tens of thousands of ops…

sunpolice 8mo ago

I was able to get 35k req/sec on a single node with Rust (custom http stack + custom html parser, custom queue, custom kv database) with obsessive optimization. It's possible to scrape Bing size index (say 100B docs) each month with only 10 nodes, under 15k$. Thought about making it public but probably no one would use it.

charlesdenault 8mo ago
please do
- mamsouuu 8mo ago
  Yes! Please do!

gethly 8mo ago

> I also truncated page content to 250KB before passing it to the parser. WTF did I just read?

tengada1 8mo ago
It's just HTML, presumably not requesting JS libraries. So 250K is a large amount.
- gethly 8mo ago
  Exactly - how can a html page need to be trimmed to 250 KB??? That is insane. Something is not right with this article.
  - iggldiggl 8mo ago
    A transcript for a half-hour radio comedy show with some formatting takes up about 60 kB. The English Wikipedia page for Monty Python is about 130 kB in pure UTF-8 text and the actual HTML page takes up around around 660 kB (plus/minus, depending on which Wikipedia theme exactly you use). So large, text-heavy pages don't seem too unlikely to exceed 250 kB, especially if they also include some amount of formatting that's more substantial than just a minimal bunch of <p> tags.

snowhale 8mo ago

[dead]

overfeed 8mo ago
> this scale per-domain politeness queuing also becomes a genuine headache Not really a headache - if you've ever implemented resource-based, server-side rate limiting (per-endpoint, with client-ID and/or IP buckets), that's all the logic that's required, adapted for the client side. One could wrap rate-limiting libraries designed for server-side usage and call it a day. I hate how people who a bad at parallelizing their user-agents across the internet are causing needless pain and giving scrapers a bad name. They are also causing blowback on the more well-behaved scrapers.

corv 8mo ago

Python is obviously too slow for web-scale

T3RMINATED 8mo ago

[dead]

lovelearning 8mo ago

As an experiment, it's interesting. If anyone actually needs such a dataset, look into CommonCrawl first. I feel using something that already exists will be more cooperative and considerate than everyone overloading every website with their spider. https://commoncrawl.org/overview https://commoncrawl.org/overview

umairnadeem123 8mo ago

[dead]

mudkipdev 8mo ago

Does AWS actually allow you to crawl like this? I've been interested in a similar project but the cloud providers I typically use seem to ban it in their terms of service