4 ms·

I created and maintain ProtonDB, a popular Linux gaming resource. I don't do ads, just pay the bills from some Patreon donations. It's a statically generated R

I created and maintain ProtonDB, a popular Linux gaming resource. I don't do ads, just pay the bills from some Patreon donations. It's a statically generated React site I deploy on Netlify. About ten days ago I started incurring 30GB of data per day from user agents indicating they're using Prerender. At this pace almost all of that will push me past the 1TB allotted for my plan, so I'm looking at an extra ~$500USD a month for the extra bandwdith boosters. I'm gonna try the robots.txt options, but I'm doubtful this will be effective in the long run. Many other options aren't available if I want to continue using a SaaS like Netlify. My initial thoughts are to either move to Cloudflare Pages/Workers where bandwidth is unlimited, or make an edge function that parses the user agent and hope it's effective enough. That'd be about $60 in edge function invocations. I've got so many better things to do than play whack-a-mole on user agents and, when failing, pay this scraping ransom. Can I just say fuck all y'all AI harvesters? This is a popular free service that helps get people off of their Microsoft dependency and live their lives on a libre operating system. You wanna leech on that? Fine, download the data dumps I already offer on an ODbL license instead of making me wonder why I fucking bother.

gjsman-1000 1y ago
Your mistake is openly suggesting on HN that you're going to use Cloudflare, increasing the centralization of the internet and contributing to their attestation schemes, while society forces you to be a victim of the tragedy of the commons.
- bdefore 1y ago
  Please believe me that it is not a step I want to take.
  - gjsman-1000 1y ago
    You don't need to apologize - HN needs to get their heads out of the sand that not everything is a tragedy of the commons, there's a reason why centralization exists, and the decentralized internet as it is now comes with serious drawbacks. We're never going to overcome the popularity of big tech if we can't be honest with the problems they solve. Also, sue me, the cathedral has defeated the bazaar. This was predictable, as the bazaar is a bunch of stonecutters competing with each other to sell the best stone for building the cathedral with. We reinvented the farmer's market, and thought that if all the farmers united, they could take down Walmart. It's never happening.
    - hombre_fatal 1y ago
      In this context, the farmers are trying to deal with rampant abuse that is inconceivable to handle on an individual level. It's not clear to me what taking down Cloudflare/Walmart means in this context. Nor how banding together wouldn't just incur the very centralization that is presumably so bad it must be taken down.
  - azdle 1y ago
    Another option that wouldn't contribute to more centralization might be neocities. They give you 3 TB for $5/month. That seems to be _the_ limit though. The dude runs his own CDN just for neocities, so it's not just reselling cloudflare or something. P.S. Thank you for ProtonDB, it has been so incredibly helpful for getting some older games running.
immibis 1y ago
$500 for exceeding 1TB? The problem here isn't the crawlers, it's your price-gouging, extortionate hosting plan. Pick your favourite $5/month VPS platform - I suggest Hetzner with its 20TB limit (if their KYC process lets you in) or Digital Ocean if not (with only 1TB but overage is only a few bucks extra). Even freaking AWS, known for extremely high prices, is cheaper than that (but still too expensive so don't use it).
- deleted 1y ago
  [deleted]
- bdefore 1y ago
  To clarify the math. Netlify bills $50 for each 100GB over the Pro plan limit of 1TB. Which is the barrel I'm looking down just this month before others get the same idea. So yes, I'm squeezed on both side unless I put the work in to rehost.
  - immibis 1y ago
    I went to a Subway shop that charged $50 per lettuce strip past the first 20. As the worker sprinkled lettuce on my sandwich, I counted anxiously, biting my nails. 19, phew, I'm safe. I think I'll come back here tomorrow. Tomorrow, someone in front of me asked for extra lettuce. The worker got confused and put it on my sandwich. I was charged $1000. Drat.
    - KyleTheDev 1y ago
      > The worker got confused and put it on my sandwich. No, this is where you're completely and totally incorrect. There is no 'worker accidentally making a human mistake that costs you money' here. This is a 'multi-billion dollar company routinely runs scripts that they KNOW cost you money, but do it anyways because it generates profit for them'. To fix your example, You RUN a Subway that sells sandwiches. Your lettuce provider charges you $1 per piece of lettuce. Your average customer is given $1 worth of lettuce in their sub. One customer keeps coming in, reaching over the counter, and grabbing handfuls of lettuce. You cannot ban this customer because they routinely put on disguises and ignore your signs saying 'NO EXTRA LETTUCE'. Eventually this bankrupts you, forces you to stop serving lettuce in your subs entirely, or you have to put up bars (eg, Cloudflare) over your lettuce bins.
alright2565 1y ago
I'm not sure what Netlify is doing, but the heaviest assets on your website are your javascript sources. Have you considered hosting those on GitHub pages, which has a generous free tier? The images are from steamcdn-a.akamaihd.net, which I assume is already being hosted by a third-party (Steam)
- bdefore 1y ago
  I'd rather not involve Microsoft but I recognize there are other options. It is additional work/complexity I'll probably have to take on.
beckthompson 1y ago
Proton DB is an amazing website that I use all the time. Thank you for maintaining it!
- bdefore 1y ago
  Thanks. Appreciate your support, and very glad it brings you value.
pbowyer 1y ago
Do you have the ability to block ASNs? I help sysadmin a DIY building forum, and we cut 80% of the load from our server by blocking all Alibaba IPs in ASN 45102. Singapore was sending the most bot traffic.
dlcarrier 1y ago
Please use a default deny on the user agent. It can block a lot of accessability tools and makes privacy difficult.
- danudey 1y ago
  Did you mean to say don't use a default deny?
  - dlcarrier 1y ago
    Yes
mococa 1y ago
Go for Cloudflare pages.
marvinblum 1y ago
Thank you for making ProtonDB! I use it a ton <3
encrypted_bird 1y ago
Hey, I just wanted to say your site is amazing and has helped me SO much and I am incredibly grateful for all your hard work. I would become a patron if I could afford it, but I can barely make ends meet and don't have $447 a month to spare. :(

gjsman-1000 1y ago

Your mistake is openly suggesting on HN that you're going to use Cloudflare, increasing the centralization of the internet and contributing to their attestation schemes, while society forces you to be a victim of the tragedy of the commons.

bdefore 1y ago
Please believe me that it is not a step I want to take.
- gjsman-1000 1y ago
  You don't need to apologize - HN needs to get their heads out of the sand that not everything is a tragedy of the commons, there's a reason why centralization exists, and the decentralized internet as it is now comes with serious drawbacks. We're never going to overcome the popularity of big tech if we can't be honest with the problems they solve. Also, sue me, the cathedral has defeated the bazaar. This was predictable, as the bazaar is a bunch of stonecutters competing with each other to sell the best stone for building the cathedral with. We reinvented the farmer's market, and thought that if all the farmers united, they could take down Walmart. It's never happening.
  - hombre_fatal 1y ago
    In this context, the farmers are trying to deal with rampant abuse that is inconceivable to handle on an individual level. It's not clear to me what taking down Cloudflare/Walmart means in this context. Nor how banding together wouldn't just incur the very centralization that is presumably so bad it must be taken down.
- azdle 1y ago
  Another option that wouldn't contribute to more centralization might be neocities. They give you 3 TB for $5/month. That seems to be _the_ limit though. The dude runs his own CDN just for neocities, so it's not just reselling cloudflare or something. P.S. Thank you for ProtonDB, it has been so incredibly helpful for getting some older games running.

immibis 1y ago

$500 for exceeding 1TB? The problem here isn't the crawlers, it's your price-gouging, extortionate hosting plan. Pick your favourite $5/month VPS platform - I suggest Hetzner with its 20TB limit (if their KYC process lets you in) or Digital Ocean if not (with only 1TB but overage is only a few bucks extra). Even freaking AWS, known for extremely high prices, is cheaper than that (but still too expensive so don't use it).

deleted 1y ago
[deleted]
bdefore 1y ago
To clarify the math. Netlify bills $50 for each 100GB over the Pro plan limit of 1TB. Which is the barrel I'm looking down just this month before others get the same idea. So yes, I'm squeezed on both side unless I put the work in to rehost.
- immibis 1y ago
  I went to a Subway shop that charged $50 per lettuce strip past the first 20. As the worker sprinkled lettuce on my sandwich, I counted anxiously, biting my nails. 19, phew, I'm safe. I think I'll come back here tomorrow. Tomorrow, someone in front of me asked for extra lettuce. The worker got confused and put it on my sandwich. I was charged $1000. Drat.
  - KyleTheDev 1y ago
    > The worker got confused and put it on my sandwich. No, this is where you're completely and totally incorrect. There is no 'worker accidentally making a human mistake that costs you money' here. This is a 'multi-billion dollar company routinely runs scripts that they KNOW cost you money, but do it anyways because it generates profit for them'. To fix your example, You RUN a Subway that sells sandwiches. Your lettuce provider charges you $1 per piece of lettuce. Your average customer is given $1 worth of lettuce in their sub. One customer keeps coming in, reaching over the counter, and grabbing handfuls of lettuce. You cannot ban this customer because they routinely put on disguises and ignore your signs saying 'NO EXTRA LETTUCE'. Eventually this bankrupts you, forces you to stop serving lettuce in your subs entirely, or you have to put up bars (eg, Cloudflare) over your lettuce bins.

alright2565 1y ago

I'm not sure what Netlify is doing, but the heaviest assets on your website are your javascript sources. Have you considered hosting those on GitHub pages, which has a generous free tier? The images are from steamcdn-a.akamaihd.net, which I assume is already being hosted by a third-party (Steam)

bdefore 1y ago
I'd rather not involve Microsoft but I recognize there are other options. It is additional work/complexity I'll probably have to take on.

beckthompson 1y ago

Proton DB is an amazing website that I use all the time. Thank you for maintaining it!

bdefore 1y ago
Thanks. Appreciate your support, and very glad it brings you value.

pbowyer 1y ago

Do you have the ability to block ASNs? I help sysadmin a DIY building forum, and we cut 80% of the load from our server by blocking all Alibaba IPs in ASN 45102. Singapore was sending the most bot traffic.

dlcarrier 1y ago

Please use a default deny on the user agent. It can block a lot of accessability tools and makes privacy difficult.

danudey 1y ago
Did you mean to say don't use a default deny?
- dlcarrier 1y ago
  Yes

mococa 1y ago

Go for Cloudflare pages.

marvinblum 1y ago

Thank you for making ProtonDB! I use it a ton <3

encrypted_bird 1y ago

Hey, I just wanted to say your site is amazing and has helped me SO much and I am incredibly grateful for all your hard work. I would become a patron if I could afford it, but I can barely make ends meet and don't have $447 a month to spare. :(