4 ms·

Sure you are for something like serving large image files (like, say 3D asset files?)... I serve >30TB/month off a Hetzner server with rotating rust drives for

by gwern 5y ago

Sure you are for something like serving large image files (like, say 3D asset files?)... I serve >30TB/month off a Hetzner server with rotating rust drives for like $50/month doing pretty much that, multi-terabyte image datasets and gigabyte-sized DL models. Plus a few websites, which you've probably seen. Zero optimization, just rsync and nginx with defaults, works fine. (It'd be way cheaper except I wanted another 20TB or so of rust to get room to add more datasets to serve - after all, the server is barely breaking a sweat.)

PragmaticPulp 5y ago
> I serve >30TB/month off a Hetzner server with rotating rust drives for like $50/month You could probably do it with Cloudflare for a similar price, but that's not what this article is about. The article says that their Cloudflare (not Argo) bill is only $40/month, and that's only because they have two domains. It's $20/month per domain. The $400/month figure isn't just for static file hosting. It includes their API, database, and global CDN. They make a point to say that low-latency, global CDN is a priority for them, which is why they're paying extra for Cloudflare Argo. I'm not sure why everyone is comparing what they're doing to a single fileserver hosted somewhere, serving up static files without regard to global latency. It's apples and oranges.
- mike_d 5y ago
  > They make a point to say that low-latency, global CDN is a priority for them Which, franky, is dumb. File downloads depend on throughput, not time to first byte like a small javascript file. Once bits start flowing it doesn't matter if they travel half way around the globe. It is like saying you are trying to optimize an ocean cargo ship for acceleration time.
  - PragmaticPulp 5y ago
    > Which, franky, is dumb. File downloads depend on throughput, not time to first byte like a small javascript file. Once bits start flowing it doesn't matter if they travel half way around the globe. Round-trip latency has a significant impact on TCP throughput. It's not just about time to first byte. They're also not a static file serving website. If they were just serving static files and they didn't care about latency, they could pay $20/month for Cloudflare Pro and be done with it. I don't understand why you continue to ignore the fact that this for running a high-traffic, global, dynamic website, not just a static file server.
    - mike_d 5y ago
      > Round-trip latency has a significant impact on TCP throughput. It's not just about time to first byte. I've built three CDNs in my career, one at the Tbps scale. Latency only has two factors in throughput: time to scale window size, and retransmits. Modern TCP stacks handle the prior just fine, and the latter is only an issue with packet loss. You can also turn on HTTP/3 and remove your reliance on TCP entirely. > I don't understand why you continue to ignore the fact that this for running a high-traffic, global, dynamic website, not just a static file server. Because I actually looked at the site and read the article. The vast majority of the site is large static files. The frontend is a static JavaScript app that calls an API server. That API server is running on a $5/mo VM. My offer still stands if you'd like a tour of a datacenter and see the sausage being made at scale. I'll throw in lunch and answer all your scalability questions if you like. But this thread is growing quite long and veering off topic.
      - csdvrx 5y ago
        > My offer still stands if you'd like a tour of a datacenter and see the sausage being made at scale. I might take you up on that offer, as I might need that for some new project I'm preparing. Where are you located?
        mike_d 5y ago
        Email is in my profile.
    - zekica 5y ago
      You can use different TCP congestion control algorithms, not just cubic. BBR2 has no issues with high RTT even with packet loss and is fair to other TCP streams. It serves 25Mbps stream in real time to users in Japan and the US from a data-centre in Europe with no issues.

by gwern 5y ago

PragmaticPulp 5y ago

> I serve >30TB/month off a Hetzner server with rotating rust drives for like $50/month You could probably do it with Cloudflare for a similar price, but that's not what this article is about. The article says that their Cloudflare (not Argo) bill is only $40/month, and that's only because they have two domains. It's $20/month per domain. The $400/month figure isn't just for static file hosting. It includes their API, database, and global CDN. They make a point to say that low-latency, global CDN is a priority for them, which is why they're paying extra for Cloudflare Argo. I'm not sure why everyone is comparing what they're doing to a single fileserver hosted somewhere, serving up static files without regard to global latency. It's apples and oranges.

mike_d 5y ago
> They make a point to say that low-latency, global CDN is a priority for them Which, franky, is dumb. File downloads depend on throughput, not time to first byte like a small javascript file. Once bits start flowing it doesn't matter if they travel half way around the globe. It is like saying you are trying to optimize an ocean cargo ship for acceleration time.
- PragmaticPulp 5y ago
  > Which, franky, is dumb. File downloads depend on throughput, not time to first byte like a small javascript file. Once bits start flowing it doesn't matter if they travel half way around the globe. Round-trip latency has a significant impact on TCP throughput. It's not just about time to first byte. They're also not a static file serving website. If they were just serving static files and they didn't care about latency, they could pay $20/month for Cloudflare Pro and be done with it. I don't understand why you continue to ignore the fact that this for running a high-traffic, global, dynamic website, not just a static file server.
  - mike_d 5y ago
    > Round-trip latency has a significant impact on TCP throughput. It's not just about time to first byte. I've built three CDNs in my career, one at the Tbps scale. Latency only has two factors in throughput: time to scale window size, and retransmits. Modern TCP stacks handle the prior just fine, and the latter is only an issue with packet loss. You can also turn on HTTP/3 and remove your reliance on TCP entirely. > I don't understand why you continue to ignore the fact that this for running a high-traffic, global, dynamic website, not just a static file server. Because I actually looked at the site and read the article. The vast majority of the site is large static files. The frontend is a static JavaScript app that calls an API server. That API server is running on a $5/mo VM. My offer still stands if you'd like a tour of a datacenter and see the sausage being made at scale. I'll throw in lunch and answer all your scalability questions if you like. But this thread is growing quite long and veering off topic.
    - csdvrx 5y ago
      > My offer still stands if you'd like a tour of a datacenter and see the sausage being made at scale. I might take you up on that offer, as I might need that for some new project I'm preparing. Where are you located?
      - mike_d 5y ago
        Email is in my profile.
  - zekica 5y ago
    You can use different TCP congestion control algorithms, not just cubic. BBR2 has no issues with high RTT even with packet loss and is fair to other TCP streams. It serves 25Mbps stream in real time to users in Japan and the US from a data-centre in Europe with no issues.