2 ms·

No. It's not a lot. 20-30k req/s is easy for serving simple, small files. If you have beefy machine (say, a $50 hetzner rootserver), you get a few TB of storage

No. It's not a lot. 20-30k req/s is easy for serving simple, small files. If you have beefy machine (say, a $50 hetzner rootserver), you get a few TB of storage and unlimited or cheap bandwidth. 8-16 cores can easily(!!!) push this kind of data without even heating up, not sure wtf OP is doing. Well, I know what OP is doing - they fell for the idea that the cloud is more scalable. The issue with this is that the cloud™ starts "scaling" at the first user, whereas a baremetal server needs to scale when you have saturated a 8-16 core modern CPU, a 1-10 Gb/s NiC, 30-60 GB of RAM. In other words, baremetal needs to scale when you actually run out of hardware resources, which is tens or hundreds of thousands of users later. Edit: for example, at BeamMP, we run on a few bare metal servers, and serve 22k unique users per day in a multiplayer videogame service. Funded by around 800 people donating.

nirui 2y ago
I'm not a fan of cloud too, but I have to admit that the networks of these big cloud providers built is just better than self-hosted ones. When they say they'll distribute your file globally, they mean it, as long as you pay of course. But I would rather say, cloud is not for everyone. Especially in the case mentioned in the article. Think this: do you really REALLY need to distribute enlarged images globally at top speed? I bet most people just don't. Same thing goes for "scaling", it's true the cloud can do that very well, but do you really need it that bad?
jeroenhd 2y ago
Based on the names of the endpoints, I get the idea that they're altering the image files on the fly (and probably caching processed files) based on the URL. I've seen this quite often on blogs and such. Serving files shouldn't take much CPU power, but resizing images can get quite expensive, especially if you want to achieve lower egress fees by using better compression methods. Still, you need to deal with bad scrapers. Plus, this scraper downloaded at a consistent 650mbps, taking up half the unlimited Hetzner pipe by itself; if you'd go for a 10gbps Hetzner machine, you suddenly start paying egress fees once you hit 20TB of traffic. Even then, if you go the cheapo Hetzner server route, you probably still want at least some kind of CDN to keep latency down. Add to that costs of backups and synchronising failovers, and you may end up with more traffic than you'd expect. I think going bare metal would save more than the signed URLs would, at least until the ten thousanth customer, but not everyone is proficient in maintaining servers. A lot of cloud projects I see are coming from programmers who don't want to/don't know how to maintain a Linux server and just want to run their code. If you're in that category, taking time off to learn server maintenance or hiring a sysadmin can easily be a lot more expensive than paying the extortionate rates cloud providers demand.

nirui 2y ago

I'm not a fan of cloud too, but I have to admit that the networks of these big cloud providers built is just better than self-hosted ones. When they say they'll distribute your file globally, they mean it, as long as you pay of course. But I would rather say, cloud is not for everyone. Especially in the case mentioned in the article. Think this: do you really REALLY need to distribute enlarged images globally at top speed? I bet most people just don't. Same thing goes for "scaling", it's true the cloud can do that very well, but do you really need it that bad?

jeroenhd 2y ago

Based on the names of the endpoints, I get the idea that they're altering the image files on the fly (and probably caching processed files) based on the URL. I've seen this quite often on blogs and such. Serving files shouldn't take much CPU power, but resizing images can get quite expensive, especially if you want to achieve lower egress fees by using better compression methods. Still, you need to deal with bad scrapers. Plus, this scraper downloaded at a consistent 650mbps, taking up half the unlimited Hetzner pipe by itself; if you'd go for a 10gbps Hetzner machine, you suddenly start paying egress fees once you hit 20TB of traffic. Even then, if you go the cheapo Hetzner server route, you probably still want at least some kind of CDN to keep latency down. Add to that costs of backups and synchronising failovers, and you may end up with more traffic than you'd expect. I think going bare metal would save more than the signed URLs would, at least until the ten thousanth customer, but not everyone is proficient in maintaining servers. A lot of cloud projects I see are coming from programmers who don't want to/don't know how to maintain a Linux server and just want to run their code. If you're in that category, taking time off to learn server maintenance or hiring a sysadmin can easily be a lot more expensive than paying the extortionate rates cloud providers demand.