3 ms·

A couple years ago, I was running a website which allowed the public to view all the US government handouts to small businesses during the COVID-19 pandemic. It

A couple years ago, I was running a website which allowed the public to view all the US government handouts to small businesses during the COVID-19 pandemic. It also tracked all the fraudulent loans being prosecuted by the DOJ, and allowed anyone to run structured queries over the public dataset. There was a "donate" button which took in ~$2k in donations over the lifetime of the site, and you could download the entire underlying dataset (around 10 GB uncompressed) for free directly on the site. Despite the "download all data" link being prominently placed on the front page, the AI scrapers decided it would be more efficient to download terabytes upon terabytes of raw HTML by paginating through every possible facet on the search endpoint. Even with CloudFront caching results and a fairly efficient backend setup, the monthly bill ended up with around $1k just going toward network ingress/egress, so I shut down the site the following month.

tailscaler2026 2mo ago
Building on AWS is a financial time bomb.
- deleted 2mo ago
  [deleted]
- ninalanyon 2mo ago
  Are you not able to put limits on how much the site can spend?
  - shermantanktop 2mo ago
    Sure, but many a hobbyist has discovered the need for that the hard way.
  - mystifyingpoi 2mo ago
    As of 2026, still not, and probably never.
- primitivesuave 2mo ago
  Completely agree, I've worked at several companies where I saw the AWS bill balloon from five to six figures, usually because of pointless over-provisioning. However, it all became worth it when I saw Jeff Bezos go to space. [1] 1. https://www.youtube.com/watch?v=IOmX793-5t4 https://www.youtube.com/watch?v=IOmX793-5t4
deepsun 2mo ago
BigQuery has "public datasets", so users can even run complex SQL on it, but it's them who pays for it, not you. You only pay for data storage.
- dpoloncsak 2mo ago
  The crawlers would have still just hammered their site though, right?
  - pavel_lishin 2mo ago
    I think the idea is that they could store the data in BigQuery, and point users of the site there.
    - PunchyHamster 2mo ago
      The crawlers would have still just hammered their site though, right?
      - 40four 2mo ago
        Why is your comment exactly word for word of another comment just one level above in the comment chain?
        taneq 2mo ago
        Because the crawlers would still have hammered their site, though. (The GP post doesn’t actually meaningfully address the issue being raised. Adding BigQuery or whatever would not change the fact that (a) they already offered a method of getting all of the data in a cost effective way, and (b) the issue was that the crawlers hammered the site hard enough to make it economically unviable.)
        mijowi 2mo ago
        Maybe because they restated what they said instead of addressing to the previous commenter’s point.
BeeOnRope 2mo ago
Do you have any view on why the AI scrapers resulted in a heavier load than existing crawlers from eg search engines? Where they more exhaustive or more frequent?
- ratelimitsteve 2mo ago
  not the OP but I'd say that google crawls you once and AIs scrape your page every time someone asks them a question that they think your page might be relevant to.
- pverheggen 2mo ago
  There's been an explosion of vibe-coded scrapers that behave poorly and ignore robots.txt. Presumably OP disallowed crawling of the search endpoint for the reason stated (that crawling it would result in endless permutations of search filters.)
- primitivesuave 2mo ago
  They were both more exhaustive and more frequent. I don't remember the exact numbers, but it was definitely over 100x the traffic from search engines. By the way, search engines were allowed under robots.txt - I did want all 11.5 million loans to be individually indexed so they would pop up in Google search results, and I actually did receive/forward multiple tips about fraudulent loans because someone searched a business name on Google. All of this traffic was barely a blip, and my AWS bill for my hobby data science account was only ~$50-$100/month. When I looked at the logs after getting the billing alert, 99.99% of the requests were to the "/search" endpoint with virtually every permutation of ~10-12 facets in the query parameters. There was only one scraper, but it triggered an enormous amount of network egress since it ended up missing the cache on the majority of queries.
x3haloed 2mo ago
That’s a traffic design problem. You should be happy that your work is valuable and also protected it against excessive requests. Simple.
- primitivesuave 2mo ago
  I deployed a Lambda function behind CloudFront which rendered a simple HTML page with the query results from executing some SQL over the dataset. I served millions of page views for next to nothing because most pages were already in the cache. I don't know where you get this expectation that people should anticipate that a crawler might try every possible combination of query parameters, thereby missing the cache on each one. Most people consider it a bitter and arrogant perspective, which is why this got downvoted.
sillysaurusx 2mo ago
For what it’s worth, a Hetzner dedicated server has unlimited ingress and egress. It seems like it’s the only provider that does. Egress fees suck. You can get a beefy one for about $40/mo on their server auction site. Just... don’t miss payments. Ever. Or they’ll delete your server within a week or so.
- deleted 2mo ago
  [deleted]
- usernameak 2mo ago
  Only if you get gigabit networking (which can transfer only so much in a month). If you get 10Gbit/s, it's not unlimited anymore.
  - sillysaurusx 2mo ago
    Could you link to anything supporting that claim? I can’t find it.
    - usernameak 2mo ago
      On the 10G Uplink documentation page: https://docs.hetzner.com/robot/dedicated-server/network/10g-uplink/ https://docs.hetzner.com/robot/dedicated-server/network/10g-... > The included traffic volume for outgoing traffic is 20TB. There is no bandwidth limitation. Overusage is billed with € 1.00 ($ 1.20)/TB. Incoming and internal traffic is unlimited. So only egress is paid on 10Gbit, but still.
userbinator 2mo ago
the AI scrapers decided it would be more efficient to download terabytes upon terabytes of raw HTML by paginating through every possible facet on the search endpoint. No, they decided that would be a great way to convince you of the narrative and persuade you to pay for "security" services that further the incumbent browser monopoly. They're not "AI scrapers", they're DDoS'ers manufacturing consent.

tailscaler2026 2mo ago

Building on AWS is a financial time bomb.

deleted 2mo ago
[deleted]
ninalanyon 2mo ago
Are you not able to put limits on how much the site can spend?
- shermantanktop 2mo ago
  Sure, but many a hobbyist has discovered the need for that the hard way.
- mystifyingpoi 2mo ago
  As of 2026, still not, and probably never.
primitivesuave 2mo ago
Completely agree, I've worked at several companies where I saw the AWS bill balloon from five to six figures, usually because of pointless over-provisioning. However, it all became worth it when I saw Jeff Bezos go to space. [1] 1. https://www.youtube.com/watch?v=IOmX793-5t4 https://www.youtube.com/watch?v=IOmX793-5t4

deepsun 2mo ago

BigQuery has "public datasets", so users can even run complex SQL on it, but it's them who pays for it, not you. You only pay for data storage.

dpoloncsak 2mo ago
The crawlers would have still just hammered their site though, right?
- pavel_lishin 2mo ago
  I think the idea is that they could store the data in BigQuery, and point users of the site there.
  - PunchyHamster 2mo ago
    The crawlers would have still just hammered their site though, right?
    - 40four 2mo ago
      Why is your comment exactly word for word of another comment just one level above in the comment chain?
      - taneq 2mo ago
        Because the crawlers would still have hammered their site, though. (The GP post doesn’t actually meaningfully address the issue being raised. Adding BigQuery or whatever would not change the fact that (a) they already offered a method of getting all of the data in a cost effective way, and (b) the issue was that the crawlers hammered the site hard enough to make it economically unviable.)
      - mijowi 2mo ago
        Maybe because they restated what they said instead of addressing to the previous commenter’s point.

BeeOnRope 2mo ago

Do you have any view on why the AI scrapers resulted in a heavier load than existing crawlers from eg search engines? Where they more exhaustive or more frequent?

ratelimitsteve 2mo ago
not the OP but I'd say that google crawls you once and AIs scrape your page every time someone asks them a question that they think your page might be relevant to.
pverheggen 2mo ago
There's been an explosion of vibe-coded scrapers that behave poorly and ignore robots.txt. Presumably OP disallowed crawling of the search endpoint for the reason stated (that crawling it would result in endless permutations of search filters.)
primitivesuave 2mo ago
They were both more exhaustive and more frequent. I don't remember the exact numbers, but it was definitely over 100x the traffic from search engines. By the way, search engines were allowed under robots.txt - I did want all 11.5 million loans to be individually indexed so they would pop up in Google search results, and I actually did receive/forward multiple tips about fraudulent loans because someone searched a business name on Google. All of this traffic was barely a blip, and my AWS bill for my hobby data science account was only ~$50-$100/month. When I looked at the logs after getting the billing alert, 99.99% of the requests were to the "/search" endpoint with virtually every permutation of ~10-12 facets in the query parameters. There was only one scraper, but it triggered an enormous amount of network egress since it ended up missing the cache on the majority of queries.

x3haloed 2mo ago

That’s a traffic design problem. You should be happy that your work is valuable and also protected it against excessive requests. Simple.

primitivesuave 2mo ago
I deployed a Lambda function behind CloudFront which rendered a simple HTML page with the query results from executing some SQL over the dataset. I served millions of page views for next to nothing because most pages were already in the cache. I don't know where you get this expectation that people should anticipate that a crawler might try every possible combination of query parameters, thereby missing the cache on each one. Most people consider it a bitter and arrogant perspective, which is why this got downvoted.

sillysaurusx 2mo ago

For what it’s worth, a Hetzner dedicated server has unlimited ingress and egress. It seems like it’s the only provider that does. Egress fees suck. You can get a beefy one for about $40/mo on their server auction site. Just... don’t miss payments. Ever. Or they’ll delete your server within a week or so.

deleted 2mo ago
[deleted]
usernameak 2mo ago
Only if you get gigabit networking (which can transfer only so much in a month). If you get 10Gbit/s, it's not unlimited anymore.
- sillysaurusx 2mo ago
  Could you link to anything supporting that claim? I can’t find it.
  - usernameak 2mo ago
    On the 10G Uplink documentation page: https://docs.hetzner.com/robot/dedicated-server/network/10g-uplink/ https://docs.hetzner.com/robot/dedicated-server/network/10g-... > The included traffic volume for outgoing traffic is 20TB. There is no bandwidth limitation. Overusage is billed with € 1.00 ($ 1.20)/TB. Incoming and internal traffic is unlimited. So only egress is paid on 10Gbit, but still.

userbinator 2mo ago

the AI scrapers decided it would be more efficient to download terabytes upon terabytes of raw HTML by paginating through every possible facet on the search endpoint. No, they decided that would be a great way to convince you of the narrative and persuade you to pay for "security" services that further the incumbent browser monopoly. They're not "AI scrapers", they're DDoS'ers manufacturing consent.