4 ms·

The bulk of the traffic is serving static files. We figured out how to do this with "old and weak" hardware in the 90s. Here is a 3.3 GHz Xeon, 24 GB of RAM, w

by mike_d 5y ago

The bulk of the traffic is serving static files. We figured out how to do this with "old and weak" hardware in the 90s. Here is a 3.3 GHz Xeon, 24 GB of RAM, with the required disk and bandwidth for $33: https://oneprovider.com/order/item/dediconf/59 https://oneprovider.com/order/item/dediconf/59

PragmaticPulp 5y ago
> The bulk of the traffic is serving static files. The bulk of the data transferring is static files because they're hosting large assets, but the rest of the article is about their API, database, etc. A single server in a single datacenter isn't comparable to a global CDN. They made it clear in the article that they value global latency, and they're willing to pay more for it. > Here is a 3.3 GHz Xeon, 24 GB of RAM, with the required disk and bandwidth for $33: https://oneprovider.com/order/item/dediconf/59 https://oneprovider.com/order/item/dediconf/59 That has mechanical spinning hard disks and a CPU that was released literally a decade ago. You're not going to replace Cloudflare, Firebase, their API server, and a global CDN with a single 10-year old machine serving files from a mechanical spinning hard disk, hosted by a company that almost nobody has heard of.
- mike_d 5y ago
  Yahoo, Google, and eBay were built with gasp spinning hard disks. But fine, if you don't believe it I will happily extend an invitation to come see in person my racks of spinning rust and 10 year old servers in Los Angeles or Fremont that are running services that are used by millions of end users. My email is in my profile.
  - PragmaticPulp 5y ago
    > Yahoo, Google, and eBay were built with gasp spinning hard disks. Which they ditched as soon as physically possible. The internet has changed a lot since then. That much should be obvious. You can't reduce this article to "millions of end users" and then equate it with every other service. There's more to this article/service than you're suggesting, and there's more to a website than just users/month.
- sroussey 5y ago
  Why not? We did that for years if not decades.
- herpderperator 5y ago
  What's wrong with a spinning hard disk? The files are super easy to read, we're talking milliseconds, and once done it stays in the page cache in RAM... that completely negates any SSD advantage. SSDs have little to no benefit in serving static files. If you're really strict on this it's trivial to preload these files in RAM after compiling your assets so you don't have to wait for a request.
  - PragmaticPulp 5y ago
    > What's wrong with a spinning hard disk? The files are super easy to read, we're talking milliseconds, and once done it stays in the page cache in RAM They're running a dynamic website with an API and other dynamic functions. It's not a static fileserver. It might make more sense if you visit the website in question: https://polyhaven.com/ https://polyhaven.com/
  - prirun 5y ago
    Higher up someone calculated that they have 2TB of data, considering their $11/mo Backblaze bill. I calculate $11/$0.004/GB and get 2750 GB. If you have 2.75TB of files to serve and have a bunch of simultaneous requests for different files: - everything doesn't fit into the RAM page cache, unless you have 3TB of RAM in your server - for files that aren't cached, you have to read from the drive - when you have a bunch of different requests for different files, you're going to have a lot of disk seeks. Seeks are expensive on rotating media. Serving a large collection of static files at scale can be quite challenging. At my previous e-commerce company, back in the early 2000's, we did our own image serving with RAID1 multiply-mirrored drives (before SSDs!) so we could multiply the seek capacity by the number of drives.
- gwern 5y ago
  Sure you are for something like serving large image files (like, say 3D asset files?)... I serve >30TB/month off a Hetzner server with rotating rust drives for like $50/month doing pretty much that, multi-terabyte image datasets and gigabyte-sized DL models. Plus a few websites, which you've probably seen. Zero optimization, just rsync and nginx with defaults, works fine. (It'd be way cheaper except I wanted another 20TB or so of rust to get room to add more datasets to serve - after all, the server is barely breaking a sweat.)
  - PragmaticPulp 5y ago
    > I serve >30TB/month off a Hetzner server with rotating rust drives for like $50/month You could probably do it with Cloudflare for a similar price, but that's not what this article is about. The article says that their Cloudflare (not Argo) bill is only $40/month, and that's only because they have two domains. It's $20/month per domain. The $400/month figure isn't just for static file hosting. It includes their API, database, and global CDN. They make a point to say that low-latency, global CDN is a priority for them, which is why they're paying extra for Cloudflare Argo. I'm not sure why everyone is comparing what they're doing to a single fileserver hosted somewhere, serving up static files without regard to global latency. It's apples and oranges.
    - mike_d 5y ago
      > They make a point to say that low-latency, global CDN is a priority for them Which, franky, is dumb. File downloads depend on throughput, not time to first byte like a small javascript file. Once bits start flowing it doesn't matter if they travel half way around the globe. It is like saying you are trying to optimize an ocean cargo ship for acceleration time.
      - PragmaticPulp 5y ago
        > Which, franky, is dumb. File downloads depend on throughput, not time to first byte like a small javascript file. Once bits start flowing it doesn't matter if they travel half way around the globe. Round-trip latency has a significant impact on TCP throughput. It's not just about time to first byte. They're also not a static file serving website. If they were just serving static files and they didn't care about latency, they could pay $20/month for Cloudflare Pro and be done with it. I don't understand why you continue to ignore the fact that this for running a high-traffic, global, dynamic website, not just a static file server.
- est31 5y ago
  24 GB of RAM means a lot of ability to cache data in RAM, which the OS does already by default. It of course depends on the type of content you have but generally traffic follows a power law, so most content can be served from RAM directly. The write side I'm a bit more worried about, as e.g. SQLite issues a sync request on each transaction (not sure about the other SQL implementations, should be similar there tho), and those might require seeking.
  - PragmaticPulp 5y ago
    The article explains that they have database and API usage. You can't run a high-throughput database on spinning disks unless you disregard data integrity. It's not just a static fileserver, despite some of the comparisons in this thread.
    - lstodd 5y ago
      Like SSDs bring any new angle to this. > You can't run a high-throughput database on spinning disks unless you disregard data integrity Of course you can. That's the way it was and still is done. I'd say one can't run a database with any data integrity unless there are disks spinning somewhere very near. I'd take another box for like $40 there just for the replica, which would sit mostly idle anyways. But piling on CDNs, clouds and variuos other opex and indeterminate risks instead of some thinking out of this little cloud shoebox is just baffling.
    - xfer 5y ago
      Spinning disks lose data integrity? You will be surprised where the cloud servers store your backups then. They are doing 80TB/month, you don't need 1GB/s nvme for that(assuming everything is uncached).

by mike_d 5y ago

PragmaticPulp 5y ago

> The bulk of the traffic is serving static files. The bulk of the data transferring is static files because they're hosting large assets, but the rest of the article is about their API, database, etc. A single server in a single datacenter isn't comparable to a global CDN. They made it clear in the article that they value global latency, and they're willing to pay more for it. > Here is a 3.3 GHz Xeon, 24 GB of RAM, with the required disk and bandwidth for $33: https://oneprovider.com/order/item/dediconf/59 https://oneprovider.com/order/item/dediconf/59 That has mechanical spinning hard disks and a CPU that was released literally a decade ago. You're not going to replace Cloudflare, Firebase, their API server, and a global CDN with a single 10-year old machine serving files from a mechanical spinning hard disk, hosted by a company that almost nobody has heard of.

mike_d 5y ago
Yahoo, Google, and eBay were built with gasp spinning hard disks. But fine, if you don't believe it I will happily extend an invitation to come see in person my racks of spinning rust and 10 year old servers in Los Angeles or Fremont that are running services that are used by millions of end users. My email is in my profile.
- PragmaticPulp 5y ago
  > Yahoo, Google, and eBay were built with gasp spinning hard disks. Which they ditched as soon as physically possible. The internet has changed a lot since then. That much should be obvious. You can't reduce this article to "millions of end users" and then equate it with every other service. There's more to this article/service than you're suggesting, and there's more to a website than just users/month.
sroussey 5y ago
Why not? We did that for years if not decades.
herpderperator 5y ago
What's wrong with a spinning hard disk? The files are super easy to read, we're talking milliseconds, and once done it stays in the page cache in RAM... that completely negates any SSD advantage. SSDs have little to no benefit in serving static files. If you're really strict on this it's trivial to preload these files in RAM after compiling your assets so you don't have to wait for a request.
- PragmaticPulp 5y ago
  > What's wrong with a spinning hard disk? The files are super easy to read, we're talking milliseconds, and once done it stays in the page cache in RAM They're running a dynamic website with an API and other dynamic functions. It's not a static fileserver. It might make more sense if you visit the website in question: https://polyhaven.com/ https://polyhaven.com/
- prirun 5y ago
  Higher up someone calculated that they have 2TB of data, considering their $11/mo Backblaze bill. I calculate $11/$0.004/GB and get 2750 GB. If you have 2.75TB of files to serve and have a bunch of simultaneous requests for different files: - everything doesn't fit into the RAM page cache, unless you have 3TB of RAM in your server - for files that aren't cached, you have to read from the drive - when you have a bunch of different requests for different files, you're going to have a lot of disk seeks. Seeks are expensive on rotating media. Serving a large collection of static files at scale can be quite challenging. At my previous e-commerce company, back in the early 2000's, we did our own image serving with RAID1 multiply-mirrored drives (before SSDs!) so we could multiply the seek capacity by the number of drives.
gwern 5y ago
Sure you are for something like serving large image files (like, say 3D asset files?)... I serve >30TB/month off a Hetzner server with rotating rust drives for like $50/month doing pretty much that, multi-terabyte image datasets and gigabyte-sized DL models. Plus a few websites, which you've probably seen. Zero optimization, just rsync and nginx with defaults, works fine. (It'd be way cheaper except I wanted another 20TB or so of rust to get room to add more datasets to serve - after all, the server is barely breaking a sweat.)
- PragmaticPulp 5y ago
  > I serve >30TB/month off a Hetzner server with rotating rust drives for like $50/month You could probably do it with Cloudflare for a similar price, but that's not what this article is about. The article says that their Cloudflare (not Argo) bill is only $40/month, and that's only because they have two domains. It's $20/month per domain. The $400/month figure isn't just for static file hosting. It includes their API, database, and global CDN. They make a point to say that low-latency, global CDN is a priority for them, which is why they're paying extra for Cloudflare Argo. I'm not sure why everyone is comparing what they're doing to a single fileserver hosted somewhere, serving up static files without regard to global latency. It's apples and oranges.
  - mike_d 5y ago
    > They make a point to say that low-latency, global CDN is a priority for them Which, franky, is dumb. File downloads depend on throughput, not time to first byte like a small javascript file. Once bits start flowing it doesn't matter if they travel half way around the globe. It is like saying you are trying to optimize an ocean cargo ship for acceleration time.
    - PragmaticPulp 5y ago
      > Which, franky, is dumb. File downloads depend on throughput, not time to first byte like a small javascript file. Once bits start flowing it doesn't matter if they travel half way around the globe. Round-trip latency has a significant impact on TCP throughput. It's not just about time to first byte. They're also not a static file serving website. If they were just serving static files and they didn't care about latency, they could pay $20/month for Cloudflare Pro and be done with it. I don't understand why you continue to ignore the fact that this for running a high-traffic, global, dynamic website, not just a static file server.
est31 5y ago
24 GB of RAM means a lot of ability to cache data in RAM, which the OS does already by default. It of course depends on the type of content you have but generally traffic follows a power law, so most content can be served from RAM directly. The write side I'm a bit more worried about, as e.g. SQLite issues a sync request on each transaction (not sure about the other SQL implementations, should be similar there tho), and those might require seeking.
- PragmaticPulp 5y ago
  The article explains that they have database and API usage. You can't run a high-throughput database on spinning disks unless you disregard data integrity. It's not just a static fileserver, despite some of the comparisons in this thread.
  - lstodd 5y ago
    Like SSDs bring any new angle to this. > You can't run a high-throughput database on spinning disks unless you disregard data integrity Of course you can. That's the way it was and still is done. I'd say one can't run a database with any data integrity unless there are disks spinning somewhere very near. I'd take another box for like $40 there just for the replica, which would sit mostly idle anyways. But piling on CDNs, clouds and variuos other opex and indeterminate risks instead of some thinking out of this little cloud shoebox is just baffling.
  - xfer 5y ago
    Spinning disks lose data integrity? You will be surprised where the cloud servers store your backups then. They are doing 80TB/month, you don't need 1GB/s nvme for that(assuming everything is uncached).