8 ms·

Half a petabyte in storage and 3 TB transfer a day? Shit. That's nothing, unless you're "saving money by using the cloud". While I feel for them, they put them

Half a petabyte in storage and 3 TB transfer a day? Shit. That's nothing, unless you're "saving money by using the cloud". While I feel for them, they put themselves in this position. Even when someone gives you something apparently for free, you have to consider what might happen when it disappears, and it seems that nobody thought about it, or if they did, they didn't care. People often learn how expensive "the cloud" is the hard way.

paulddraper 3y ago
> they put themselves in this position You say this like they're hopeless.
- simonjgreen 3y ago
  I think the implication is either naive or wilfully ignorant, rather than hopeless
n2d4 3y ago
I'm curious. What would you do, host on-prem? Even if I had the hardware, as a pure software person, I have no clue how I'd even start setting up a physical location to serve data on that scale. And even if the people on the NixOS team know, it sounds like their time would be better spent working on NixOS than building & maintaining a data center (even if small). Bare metal storage services seem to be on a similar price level as lower-cost cloud services like Cloudflare or Backblaze, but I'm very interested if you happen to know a cheaper alternative! Plus, even if you deal with the storage itself, how do you deal with serving the traffic globally? Optimally you'd want a distributed CDN, but that doesn't really sound feasible without the cloud. Would you still use a cloud CDN, or just serve directly from your own server?
- jtolds 3y ago
  (piping up from another thread, but) > I'm very interested if you happen to know a cheaper alternative! and > Optimally you'd want a distributed CDN https://www.storj.io/ https://www.storj.io/ is this thing!
  - n2d4 3y ago
    Using the pricing from your other comment and the numbers from the parent, if I estimate correctly, Storj comes in at about $8k/month? That's cheaper than AWS (considering there are no CDN costs), roughly on the same level as Cloudflare and Backblaze, but still more than the entirety of NixOS Foundation's (monthly) incoming funds in 2022.
    - jtolds 3y ago
      According to https://discourse.nixos.org/t/nixos-foundations-financial-summary-a-transparent-look-into-2022/28107/16?u=jtolio https://discourse.nixos.org/t/nixos-foundations-financial-su..., they have 450TB and 29.5TB of S3 egress/month. By my math, assuming the Nix Foundation goes in through our front door, talks to no sales people, and pays list price, that's $2k/month.
      - n2d4 3y ago
        You would replace the Fastly CDN with Storj, correct? Your link mentions 1500TB of Fastly traffic, which would cost $12k with Storj. Not sure if this is related to parent's 3TB/day number, but I went with that when I estimated the $8k. I think the best cloud solution might be something like Storj + Fastly or Cloudflare end-to-end. But I'm really curious how much could be chopped off if you tried to solve this on-prem.
        jtolds 3y ago
        Storj + Fastly is genuinely a pretty good combo. There are certainly some things Fastly does that we don't do (besides being more specifically a CDN, they also have compute@edge, etc). We have a good relationship with Fastly (e.g. https://docs.fastly.com/en/guides/log-streaming-storj-dcs https://docs.fastly.com/en/guides/log-streaming-storj-dcs) But yeah, I don't know how the Nix Foundation is getting their Fastly traffic paid for, maybe that's in/out of scope. Certainly they should keep Fastly if it makes the overall thing cheaper. Having a distributed origin (us) to dedicated edge cache (Fastly) makes for a strong distribution story.
- lordgilman 3y ago
  You colocate, like every other distro has done since the 90s.
  - herewulf 3y ago
    And throw in some rsync mirrors for good measure.
- toyg 3y ago
  Actually, for someone building an OS, having to face very practical challenges in the server space would likely be extremely beneficial. Dogfood, people, dogfood.
- 411111111111111 3y ago
  > Plus, even if you deal with the storage itself, how do you deal with serving the traffic globally? As a simple example 1 of the cheapest hetzner baremetal nodes gives you 500gb nvme or 2tb HDD storage with an unlimited gigabit Connection in Europe. This sums up to i believe 10TB of data per day. Given the provided numbers, a single 50€ instance could handle the load, even if it's located in Europe. CDNs make sense for systems which are sensitive to high latency, or you're pushing multiple TB per hour, which this just isn't.
  - usr1106 3y ago
    I wonder how unlimited such cheap unlimited connections really are. In the simplest case insufficient capacity in the provider network might become a bottleneck. They might notice that you are the noisiest one in the whole neighboorhood and either ask more money or throttle you. Offers that are too good to be true typically don't last very long.
    - CodesInChaos 3y ago
      At 1 EUR/TB, even the instance types that pay for traffic (e.g. the 10GBit dedicated servers) hardly break the bank. However Hetzner does not have the best peerings, so bandwidth to some locations might be worse than you'd like.
      - 411111111111111 3y ago
        Thats true, and I didn't actually want to suggest using the cheapest server available as a singular node either. I just wanted to put the given numbers into the scale of the cheapest baremetal servers available on the market, to illustrate that this is not an amount of data you'd have to consider for scaling. Realistically speaking, you'd want to have more then 2 nodes so you don't have to worry about rebooting/updates etc
    - _a9 3y ago
      I have quite a few servers with hetzner. I don't know what the exact criteria are but once you hit around 250TB/month per server with multiple servers for (likely) multiple months in a row then they will tell you to chill out or they will terminate. I've gotten it and I've seen a few others on lowendtalk mention it too. I've been using hetzner for a couple years and have always used them for my high bandwidth services but I have only gotten that notice once despite being a very heavy user, and around the same time everyone else did around a year ago. Maybe they're gonna start to care? With all the price hikes and all. I ended up just getting more servers to level it all out.
- supriyo-biswas 3y ago
  It is astonishing as to how marketers from the three big clouds have successfully convinced people that there is no middle ground between operating your data center and using one of their cloud services. Here's what you do: 1. Buy two servers equipped with a large amount of storage. There are many providers[1][2] that offer this. (You could technically do it with one, but two allows you to have high availability and prevent data loss in case of accidents, imagine `rm -rf *`.) 2. Use Minio to set up erasure-coded storage on one of the servers, and replicate objects[3] to the other server. 4. Use Cloudflare (because of their open source sponsorship program[4]) to serve the objects from the primary server, and when performing maintenance tasks, fall back to the secondary. You could also serve objects directly out of the primary/secondary, but using Cloudflare takes care of low-effort DDoS attacks by hiding the origin, and provides some caching which helps prevent hitting the egress limits of the server provider. [1] https://www.netcup.eu/vserver/vstorage.php https://www.netcup.eu/vserver/vstorage.php [2] https://www.hetzner.com/dedicated-rootserver https://www.hetzner.com/dedicated-rootserver [3] https://blog.min.io/active-active-replication/ https://blog.min.io/active-active-replication/ [4] https://blog.cloudflare.com/cloudflare-new-oss-sponsorships-program/ https://blog.cloudflare.com/cloudflare-new-oss-sponsorships-...
  - usr1106 3y ago
    Yes, perfectly doable, but the expertise and time needed to do it (without fatal mistakes) and keep it running 365 days a year is non-negligible. Doing this by unpaid volunteers is more than challenging.
    - jacquesm 3y ago
      Not for the people behind NixOS. If it is challenging then that would cause me to re-calibrate my opinion of the project as a whole.
    - simonjgreen 3y ago
      This is, in my opinion, a falacy perpetuated by the marketing departments of "big cloud". Don't forget that pre-2006 there was no S3, and we managed just fine. They are not challenging skills to learn, the zeitgeist has just convinced itself they are not worth learning
- AnthonyMouse 3y ago
  > Even if I had the hardware, as a pure software person, I have no clue how I'd even start setting up a physical location to serve data on that scale. 3TB/day is about a third of what you can serve out of any run of the mill PC with a gigabit ethernet connection. There are 22TB drives and individual servers with 24 drive bays, which exceeds 500TB. So you could do this with one machine connected to a gigabit fiber connection, and the machine would cost approximately what they're paying monthly (almost all of that for the drives). In practice a 24-bay chassis is silly and you might use a separate storage array etc., but that's just details. > Plus, even if you deal with the storage itself, how do you deal with serving the traffic globally? At 3TB/day you're not Netflix. Just serve it directly from your own server.
  - ptman 3y ago
    https://www.supermicro.com/en/products/system/4U/6048/SSG-6048R-E1CR72L.cfm https://www.supermicro.com/en/products/system/4U/6048/SSG-60... - 4U 72 drives
    - AnthonyMouse 3y ago
      It's not silly because it doesn't exist. It's silly because it weighs a hundred pounds and makes design trade offs to achieve a level of density you probably don't even care about if you only need one of them. The people who want that thing are the people who are going to fill a rack with them or more instead of several times that many racks with something else. In practice if you need ~24 drives you get a couple of 2U array containers that each hold 12 drives and then buy a 2U server or two to plug them into and don't care at all that you used 8U for 24 drives instead of 4U for 72 because your 42U rack still has 34U to spare.
- Roark66 3y ago
  Well, having a $9k per month budget you could hire an entire rack in a premium data center location (let's say Telehouse London-Docklands that hosts London internet exchange - You can connect to it in a very cost effective way). This would still leave 80%~90% of the budget (depending on power use) for hardware/software etc. But a full rack is a huge overkill. How about a quarter rack for few hundred $ per month? What about connectivity? Well, as mentioned Telehouse Docklands hosts LINX so you can get an unlimited transfer gigabit right into the Internet backbone for a couple hundred $ (mind I dealt with them few years ago, but I doubt prices changed that much - probably they're lower if anything). Also there is a cluster of data centers in docklands so they compete on price heavily as well as resellers that often have good deals. This is all just location and connectivity. How about hardware? Let's say they wanted proper support with their hardware not diy so they were looking for a "solution provider". Send a request for quote to 10 of them including fujitsu, ibm, hp etc. Then tell them you're looking for a "charitable organisation discount". Unfortunately we're missing the most important metric to price this "solution". IOPS and if its mostly small or large files (random or sequential Access), but they host most of it in "infrequent access" s3 so let's assume it's not a lot, just to find the lower pricing boundary. So let's say a pair of 2u servers with two hba cards each and 4 12 disc jbods with 22tb Sas drives a total of a petabyte in raw storage or half a PB mirrored (not that they would run it mirrored). A system like this from a big name (it's rather low performance, but we're after cheap right, their s3 is likely very similar) might cost $80k. Over 5 years of lease that might be ~$3k per month plus their hosting and connectivity it's about half of their $9k.... Then there is software, but that's a separate thing. Also one could DIY such system for half the price, one could add SSDs etc. This is just an example. To design this properly one would need usage data from their s3 and few days of work to put it nicely together.
  - traceroute66 3y ago
    > mentioned Telehouse Docklands hosts LINX so you can get an unlimited transfer gigabit right into the Internet backbone First, look at the LINX network map. They are not just in Telehouse Docklands. Second, with all due respect, do you know anything about how the internet works ? "internet backbone" huh ? LINX is a peering exchange. A connection to LINX gets you nothing. Sure you can peer with the route servers. But you won't find any Tier 1 routes on the route servers, and the Tier 1s will also refuse to peer with you because you're too small or you don't meet their ultra-restrictive peering policies (for example, good luck getting BT to peer with you !). Most people are better off getting a good connection to either a Tier 1 or a high-quality Tier 2 network and leaving it at that. > Also there is a cluster of data centers in docklands so they compete on price heavily London Docklands is the most expensive place in the UK to colocate, and probably in the Top 5 most expensive globally. Its prime real estate, all of the big-name colo facilities know that and will charge you accordingly, ESPECIALLY Telehouse ! And trust me, you don't want to go anywhere near a "cheap" Docklands facility, they are cheap for a reason. The days of £300 a month for a full rack in Docklands are long gone my friend. There are many other perfectly good places both in the Greater London area and elsewhere in the UK and Europe where you will get more bang for your buck.
- citrin_ru 3y ago
  > Optimally you'd want a distributed CDN, but that doesn't really sound feasible without the cloud CDN is a good choice when you ether need to sever small volume (i. e. not 500Tb) with low latency, e. g. to reduce web page load time by hosting static assets (js/css/img) close to the customer or when you have really huge traffic (e. g. Netflix, but they decided public cloud is not cost effective for CDN and built their own). For NixOS mirrors latency is not important, traffic is not huge (withing capacity of a single commodity server) and just two mirrors - in US and EU would be close enough to majority of users.
- tecleandor 3y ago
  I would split it in pieces and create a DIY CDN (like lots of Linux distros have). From the numbers I've seen in the article, the big problem is the storage, not the transfer, although they say that the space can be reduce by garbage collecting the less used caches (although not ideal). Note that they are using Fastly as CDN in the front, but that's not important for them (monetary wise at least). The big part of the bill is just the storage part. With little detail about their data I would chose one of these architectures... Debian (or other distro) mirror style: - Split the cache storage in pieces (by architecture, release or other categories)... - Deploy a bunch (10-30) of cheap servers in affordable hosting around the world. The most popular data is distributed in more servers (for example, the x86_64 build caches for the latest version of the packages) - Get some people to volunteer with small mirrors CDN style: - Get a couple or three medium/big sized (in disk, not CPU) servers for storing all the cache, for example, with Minio, so they don't even have to stop using S3 storage libraries. And then either: - Keep using Fastly or - Distribute tiny servers with Varnish or the like - Get some people to volunteer tiny edge cache servers Some weeks ago this was on the front page: - Serving 90TB/Day of Linux Updates from Thin Clients: https://news.ycombinator.com/item?id=35883053 https://news.ycombinator.com/item?id=35883053 - The scripts they use: https://github.com/PhirePhly/micromirrors https://github.com/PhirePhly/micromirrors
- johnklos 3y ago
  Surely you know what colocation is, don't you? Suggesting someone has to build a datacenter for it is like saying that programming is time consuming because people have to write their own compilers. If someone asked me to do this, I'd colocate in two locations (likely Europe and west coast USA) two full copies of everything in 1/4 rack. After physically building, testing, installing, then testing some more, which likely would take a month, it'd be a part time job to monitor and babysit for the first few months of use, then it'd be an hour a month maintenance job. CDNs cost money and come with tons of problems. For non-interactive static files, they're not necessary. If it requires constant attention, you're doing it wrong :)
tstrimple 3y ago
> That's nothing, unless you're "saving money by using the cloud". I've helped a number of clients save money by using the cloud. It's not that hard when you use the tools for the purposes they are suited for. It's not the hammer's fault that it does a poor job of pounding in that screw. At some point the "craftsman" needs to take the time to learn what their tools do and what they are best used for.
esjeon 3y ago
3 TB a day is roughly 36 MB/s, so yeah it's pretty much doable. 10 Gbps connection is already common, and it's even possible on 1 Gbps connection if people don't mind some saturation during peak time. > People often learn how expensive "the cloud" is the hard way. Yeah, people really go a long way around just to get back to on-premise and colocation...
- vasco 3y ago
  > Yeah, people really go a long way around just to get back to on-premise and colocation. Outside of pet projects I'm not sure many people are doing that, but happy to read any research. From all I've seen the numbers show an overwhelming move to the cloud, and a few grumpy cats annoyed that the pricing structure is different.
  - BiteCode_dev 3y ago
    37signals enters the chat
    - WJW 3y ago
      I've read the article from DHH he makes some decent arguments, but as I see it that article is mostly notable because most people don't self-host. At least in my corner of the tech scene (ie EU startups) there are probably hundreds of companies hosting in the cloud for every company choosing to self-host and I haven't seen a shift in that trend yet. If anything it seems to be shifting more towards the cloud.
      - BiteCode_dev 3y ago
        True, the wake up call is mostly in articles and social medias, not in the trend, in the same way most people use Windows despite most technical writers using Mac or Linux. Personally I've been self-hosting forever, and that's the only reason I could stay afloat. There is a whole category of business that is simply not profitable if you use the cloud.
      - 3y ago
mrweasel 3y ago
There's still open source hosting providers that could provide the bandwidth and maybe the storage for such a project, but I doubt many of them could provide something like S3, it would either be git, https, ftp or maybe rsync. Building something like S3 yourself on Swift or Minio isn't easily done, the management of these solution can be tricky, especially for multi-petabyte systems. S3 most likely got them going and scaling really easily, but then at some point the cost comes back to bit you. Still I don't see any good alternatives right now. I don't completely disagree with your, but the overhead of building an S3 like storage solution upfront would have actively prevented the NixOS project getting off the ground.
usr1106 3y ago
They did not make the decision that the cloud is cost-efficient. They had someone else paying their cloud bill and the cloud is easy for them. Now that the someone else is pulling out they need to think for the first time what would be cost-efficient.
- traceroute66 3y ago
  > Now that the someone else is pulling out they need to think for the first time They should have thought about that on day zero. If someone else is picking up the tab then there is ALWAYS the possibility that someone will pull out for whatever reason.
  - usr1106 3y ago
    When you start an open-source project you don't start with thinking what will happen when you have Terabytes of traffic. Even for a start-up it's generally not advised to spend considerable effort on scalability before you have any customers.
    - traceroute66 3y ago
      >When you start an open-source project ... Rubbish. Its nothing to do with "start-up" or "open-source project". As I clearly said. Its Dummies 101 ... if "somebody" is giving me money, then only an idiot doesn't consider what happens if that "somebody" stops giving me money. Its not rocket science, its basic planning.
      - regularjack 3y ago
        What you're describing is premature optimisation. They made the right call going with the solution that required the least effort. Now that that solution is no longer viable, they can spend resources addressing it. Not fixing problems that don't exist is the right call.
        traceroute66 3y ago
traceroute66 3y ago
> While I feel for them, they put themselves in this position. I agree entirely. I feel the same way about OpenBSD when they come round begging for money. I'm like "sure, great cause, but can we first have chat about your totally energy inefficient hosting of multiple racks full of ancient power-hungry kit in Theo's basement". Money doesn't grow on trees, and especially in this era of inflation and high energy prices, I'm afraid I don't think it is unreasonable for me to expect the organisation asking me for money to get their affairs in order first.
- anotherhue 3y ago
  Well they're hardly going to find a colo that'll take those machines, so you must mean that they should stop building for them? They've said before they hardware heterogeneity has been useful in bug hunting, and at worst it can be viewed as a heating expense for the fairly prolific guy. Does qemu support all those architectures ?
  - traceroute66 3y ago
    > Does qemu support all those architectures ? Probably. But my understanding is that the OpenBSD core developers consider qemu as the antithesis to their beliefs. At least that's what I recall reading when it was discussed at the time, i.e. it was met with one of Theo's famous hard "no".
tecleandor 3y ago
Yep, the thing is they're using Fastly as CDN, moving 1.5PB a month, and its cost doesn't seem to be a problem for them (I don't know if its covered by Fastly or they have a good price). The gross of their problem is the storage, and S3 is probably the most expensive place to put raw storage. If they start paying the bill, their best option is moving out. Either to a cheaper provider (be it R2, B2 or whatever) or self managing it (Minio...). And seeing the $32K price for the migration to R2, and guessing that's transfer costs, I would garbage collect and move only hot data and let the rest be rebuilt during some months.
- adisbladis 3y ago
  Fastly sponsors NixOS.
  - tecleandor 3y ago
    That's good, because that bill would be even bigger than the S3 one :P
ThinkBeat 3y ago
I think that is "something" for a huge amount of business operating around the world. For everyone here saying its trivial and they can do it cheap. Get together and form a non-profit organization that offers super cheap infrastructure to the NixOS foundation. With a 99.99% SLA and around the clock staff ready if something goes wrong. Which I think if you just co locate at a datacenter means going to the physical racks when shit breaks. (Or pay for "remote hands" and "remote spare parts". Or just pay for managed hosting. I think that sounds like a lot of work for volunteers who have other jobs as wel.
- nine_k 3y ago
  A perfect case for something like Backblaze to sponsor it and get some more publicity.

paulddraper 3y ago

> they put themselves in this position You say this like they're hopeless.

simonjgreen 3y ago
I think the implication is either naive or wilfully ignorant, rather than hopeless

n2d4 3y ago

I'm curious. What would you do, host on-prem? Even if I had the hardware, as a pure software person, I have no clue how I'd even start setting up a physical location to serve data on that scale. And even if the people on the NixOS team know, it sounds like their time would be better spent working on NixOS than building & maintaining a data center (even if small). Bare metal storage services seem to be on a similar price level as lower-cost cloud services like Cloudflare or Backblaze, but I'm very interested if you happen to know a cheaper alternative! Plus, even if you deal with the storage itself, how do you deal with serving the traffic globally? Optimally you'd want a distributed CDN, but that doesn't really sound feasible without the cloud. Would you still use a cloud CDN, or just serve directly from your own server?

jtolds 3y ago
(piping up from another thread, but) > I'm very interested if you happen to know a cheaper alternative! and > Optimally you'd want a distributed CDN https://www.storj.io/ https://www.storj.io/ is this thing!
- n2d4 3y ago
  Using the pricing from your other comment and the numbers from the parent, if I estimate correctly, Storj comes in at about $8k/month? That's cheaper than AWS (considering there are no CDN costs), roughly on the same level as Cloudflare and Backblaze, but still more than the entirety of NixOS Foundation's (monthly) incoming funds in 2022.
  - jtolds 3y ago
    According to https://discourse.nixos.org/t/nixos-foundations-financial-summary-a-transparent-look-into-2022/28107/16?u=jtolio https://discourse.nixos.org/t/nixos-foundations-financial-su..., they have 450TB and 29.5TB of S3 egress/month. By my math, assuming the Nix Foundation goes in through our front door, talks to no sales people, and pays list price, that's $2k/month.
    - n2d4 3y ago
      You would replace the Fastly CDN with Storj, correct? Your link mentions 1500TB of Fastly traffic, which would cost $12k with Storj. Not sure if this is related to parent's 3TB/day number, but I went with that when I estimated the $8k. I think the best cloud solution might be something like Storj + Fastly or Cloudflare end-to-end. But I'm really curious how much could be chopped off if you tried to solve this on-prem.
      - jtolds 3y ago
        Storj + Fastly is genuinely a pretty good combo. There are certainly some things Fastly does that we don't do (besides being more specifically a CDN, they also have compute@edge, etc). We have a good relationship with Fastly (e.g. https://docs.fastly.com/en/guides/log-streaming-storj-dcs https://docs.fastly.com/en/guides/log-streaming-storj-dcs) But yeah, I don't know how the Nix Foundation is getting their Fastly traffic paid for, maybe that's in/out of scope. Certainly they should keep Fastly if it makes the overall thing cheaper. Having a distributed origin (us) to dedicated edge cache (Fastly) makes for a strong distribution story.
lordgilman 3y ago
You colocate, like every other distro has done since the 90s.
- herewulf 3y ago
  And throw in some rsync mirrors for good measure.
toyg 3y ago
Actually, for someone building an OS, having to face very practical challenges in the server space would likely be extremely beneficial. Dogfood, people, dogfood.
411111111111111 3y ago
> Plus, even if you deal with the storage itself, how do you deal with serving the traffic globally? As a simple example 1 of the cheapest hetzner baremetal nodes gives you 500gb nvme or 2tb HDD storage with an unlimited gigabit Connection in Europe. This sums up to i believe 10TB of data per day. Given the provided numbers, a single 50€ instance could handle the load, even if it's located in Europe. CDNs make sense for systems which are sensitive to high latency, or you're pushing multiple TB per hour, which this just isn't.
- usr1106 3y ago
  I wonder how unlimited such cheap unlimited connections really are. In the simplest case insufficient capacity in the provider network might become a bottleneck. They might notice that you are the noisiest one in the whole neighboorhood and either ask more money or throttle you. Offers that are too good to be true typically don't last very long.
  - CodesInChaos 3y ago
    At 1 EUR/TB, even the instance types that pay for traffic (e.g. the 10GBit dedicated servers) hardly break the bank. However Hetzner does not have the best peerings, so bandwidth to some locations might be worse than you'd like.
    - 411111111111111 3y ago
      Thats true, and I didn't actually want to suggest using the cheapest server available as a singular node either. I just wanted to put the given numbers into the scale of the cheapest baremetal servers available on the market, to illustrate that this is not an amount of data you'd have to consider for scaling. Realistically speaking, you'd want to have more then 2 nodes so you don't have to worry about rebooting/updates etc
  - _a9 3y ago
    I have quite a few servers with hetzner. I don't know what the exact criteria are but once you hit around 250TB/month per server with multiple servers for (likely) multiple months in a row then they will tell you to chill out or they will terminate. I've gotten it and I've seen a few others on lowendtalk mention it too. I've been using hetzner for a couple years and have always used them for my high bandwidth services but I have only gotten that notice once despite being a very heavy user, and around the same time everyone else did around a year ago. Maybe they're gonna start to care? With all the price hikes and all. I ended up just getting more servers to level it all out.
supriyo-biswas 3y ago
It is astonishing as to how marketers from the three big clouds have successfully convinced people that there is no middle ground between operating your data center and using one of their cloud services. Here's what you do: 1. Buy two servers equipped with a large amount of storage. There are many providers[1][2] that offer this. (You could technically do it with one, but two allows you to have high availability and prevent data loss in case of accidents, imagine `rm -rf *`.) 2. Use Minio to set up erasure-coded storage on one of the servers, and replicate objects[3] to the other server. 4. Use Cloudflare (because of their open source sponsorship program[4]) to serve the objects from the primary server, and when performing maintenance tasks, fall back to the secondary. You could also serve objects directly out of the primary/secondary, but using Cloudflare takes care of low-effort DDoS attacks by hiding the origin, and provides some caching which helps prevent hitting the egress limits of the server provider. [1] https://www.netcup.eu/vserver/vstorage.php https://www.netcup.eu/vserver/vstorage.php [2] https://www.hetzner.com/dedicated-rootserver https://www.hetzner.com/dedicated-rootserver [3] https://blog.min.io/active-active-replication/ https://blog.min.io/active-active-replication/ [4] https://blog.cloudflare.com/cloudflare-new-oss-sponsorships-program/ https://blog.cloudflare.com/cloudflare-new-oss-sponsorships-...
- usr1106 3y ago
  Yes, perfectly doable, but the expertise and time needed to do it (without fatal mistakes) and keep it running 365 days a year is non-negligible. Doing this by unpaid volunteers is more than challenging.
  - jacquesm 3y ago
    Not for the people behind NixOS. If it is challenging then that would cause me to re-calibrate my opinion of the project as a whole.
  - simonjgreen 3y ago
    This is, in my opinion, a falacy perpetuated by the marketing departments of "big cloud". Don't forget that pre-2006 there was no S3, and we managed just fine. They are not challenging skills to learn, the zeitgeist has just convinced itself they are not worth learning
AnthonyMouse 3y ago
> Even if I had the hardware, as a pure software person, I have no clue how I'd even start setting up a physical location to serve data on that scale. 3TB/day is about a third of what you can serve out of any run of the mill PC with a gigabit ethernet connection. There are 22TB drives and individual servers with 24 drive bays, which exceeds 500TB. So you could do this with one machine connected to a gigabit fiber connection, and the machine would cost approximately what they're paying monthly (almost all of that for the drives). In practice a 24-bay chassis is silly and you might use a separate storage array etc., but that's just details. > Plus, even if you deal with the storage itself, how do you deal with serving the traffic globally? At 3TB/day you're not Netflix. Just serve it directly from your own server.
- ptman 3y ago
  https://www.supermicro.com/en/products/system/4U/6048/SSG-6048R-E1CR72L.cfm https://www.supermicro.com/en/products/system/4U/6048/SSG-60... - 4U 72 drives
  - AnthonyMouse 3y ago
    It's not silly because it doesn't exist. It's silly because it weighs a hundred pounds and makes design trade offs to achieve a level of density you probably don't even care about if you only need one of them. The people who want that thing are the people who are going to fill a rack with them or more instead of several times that many racks with something else. In practice if you need ~24 drives you get a couple of 2U array containers that each hold 12 drives and then buy a 2U server or two to plug them into and don't care at all that you used 8U for 24 drives instead of 4U for 72 because your 42U rack still has 34U to spare.
Roark66 3y ago
Well, having a $9k per month budget you could hire an entire rack in a premium data center location (let's say Telehouse London-Docklands that hosts London internet exchange - You can connect to it in a very cost effective way). This would still leave 80%~90% of the budget (depending on power use) for hardware/software etc. But a full rack is a huge overkill. How about a quarter rack for few hundred $ per month? What about connectivity? Well, as mentioned Telehouse Docklands hosts LINX so you can get an unlimited transfer gigabit right into the Internet backbone for a couple hundred $ (mind I dealt with them few years ago, but I doubt prices changed that much - probably they're lower if anything). Also there is a cluster of data centers in docklands so they compete on price heavily as well as resellers that often have good deals. This is all just location and connectivity. How about hardware? Let's say they wanted proper support with their hardware not diy so they were looking for a "solution provider". Send a request for quote to 10 of them including fujitsu, ibm, hp etc. Then tell them you're looking for a "charitable organisation discount". Unfortunately we're missing the most important metric to price this "solution". IOPS and if its mostly small or large files (random or sequential Access), but they host most of it in "infrequent access" s3 so let's assume it's not a lot, just to find the lower pricing boundary. So let's say a pair of 2u servers with two hba cards each and 4 12 disc jbods with 22tb Sas drives a total of a petabyte in raw storage or half a PB mirrored (not that they would run it mirrored). A system like this from a big name (it's rather low performance, but we're after cheap right, their s3 is likely very similar) might cost $80k. Over 5 years of lease that might be ~$3k per month plus their hosting and connectivity it's about half of their $9k.... Then there is software, but that's a separate thing. Also one could DIY such system for half the price, one could add SSDs etc. This is just an example. To design this properly one would need usage data from their s3 and few days of work to put it nicely together.
- traceroute66 3y ago
  > mentioned Telehouse Docklands hosts LINX so you can get an unlimited transfer gigabit right into the Internet backbone First, look at the LINX network map. They are not just in Telehouse Docklands. Second, with all due respect, do you know anything about how the internet works ? "internet backbone" huh ? LINX is a peering exchange. A connection to LINX gets you nothing. Sure you can peer with the route servers. But you won't find any Tier 1 routes on the route servers, and the Tier 1s will also refuse to peer with you because you're too small or you don't meet their ultra-restrictive peering policies (for example, good luck getting BT to peer with you !). Most people are better off getting a good connection to either a Tier 1 or a high-quality Tier 2 network and leaving it at that. > Also there is a cluster of data centers in docklands so they compete on price heavily London Docklands is the most expensive place in the UK to colocate, and probably in the Top 5 most expensive globally. Its prime real estate, all of the big-name colo facilities know that and will charge you accordingly, ESPECIALLY Telehouse ! And trust me, you don't want to go anywhere near a "cheap" Docklands facility, they are cheap for a reason. The days of £300 a month for a full rack in Docklands are long gone my friend. There are many other perfectly good places both in the Greater London area and elsewhere in the UK and Europe where you will get more bang for your buck.
citrin_ru 3y ago
> Optimally you'd want a distributed CDN, but that doesn't really sound feasible without the cloud CDN is a good choice when you ether need to sever small volume (i. e. not 500Tb) with low latency, e. g. to reduce web page load time by hosting static assets (js/css/img) close to the customer or when you have really huge traffic (e. g. Netflix, but they decided public cloud is not cost effective for CDN and built their own). For NixOS mirrors latency is not important, traffic is not huge (withing capacity of a single commodity server) and just two mirrors - in US and EU would be close enough to majority of users.
tecleandor 3y ago
I would split it in pieces and create a DIY CDN (like lots of Linux distros have). From the numbers I've seen in the article, the big problem is the storage, not the transfer, although they say that the space can be reduce by garbage collecting the less used caches (although not ideal). Note that they are using Fastly as CDN in the front, but that's not important for them (monetary wise at least). The big part of the bill is just the storage part. With little detail about their data I would chose one of these architectures... Debian (or other distro) mirror style: - Split the cache storage in pieces (by architecture, release or other categories)... - Deploy a bunch (10-30) of cheap servers in affordable hosting around the world. The most popular data is distributed in more servers (for example, the x86_64 build caches for the latest version of the packages) - Get some people to volunteer with small mirrors CDN style: - Get a couple or three medium/big sized (in disk, not CPU) servers for storing all the cache, for example, with Minio, so they don't even have to stop using S3 storage libraries. And then either: - Keep using Fastly or - Distribute tiny servers with Varnish or the like - Get some people to volunteer tiny edge cache servers Some weeks ago this was on the front page: - Serving 90TB/Day of Linux Updates from Thin Clients: https://news.ycombinator.com/item?id=35883053 https://news.ycombinator.com/item?id=35883053 - The scripts they use: https://github.com/PhirePhly/micromirrors https://github.com/PhirePhly/micromirrors
johnklos 3y ago
Surely you know what colocation is, don't you? Suggesting someone has to build a datacenter for it is like saying that programming is time consuming because people have to write their own compilers. If someone asked me to do this, I'd colocate in two locations (likely Europe and west coast USA) two full copies of everything in 1/4 rack. After physically building, testing, installing, then testing some more, which likely would take a month, it'd be a part time job to monitor and babysit for the first few months of use, then it'd be an hour a month maintenance job. CDNs cost money and come with tons of problems. For non-interactive static files, they're not necessary. If it requires constant attention, you're doing it wrong :)

tstrimple 3y ago

> That's nothing, unless you're "saving money by using the cloud". I've helped a number of clients save money by using the cloud. It's not that hard when you use the tools for the purposes they are suited for. It's not the hammer's fault that it does a poor job of pounding in that screw. At some point the "craftsman" needs to take the time to learn what their tools do and what they are best used for.

esjeon 3y ago

3 TB a day is roughly 36 MB/s, so yeah it's pretty much doable. 10 Gbps connection is already common, and it's even possible on 1 Gbps connection if people don't mind some saturation during peak time. > People often learn how expensive "the cloud" is the hard way. Yeah, people really go a long way around just to get back to on-premise and colocation...

vasco 3y ago
> Yeah, people really go a long way around just to get back to on-premise and colocation. Outside of pet projects I'm not sure many people are doing that, but happy to read any research. From all I've seen the numbers show an overwhelming move to the cloud, and a few grumpy cats annoyed that the pricing structure is different.
- BiteCode_dev 3y ago
  37signals enters the chat
  - WJW 3y ago
    I've read the article from DHH he makes some decent arguments, but as I see it that article is mostly notable because most people don't self-host. At least in my corner of the tech scene (ie EU startups) there are probably hundreds of companies hosting in the cloud for every company choosing to self-host and I haven't seen a shift in that trend yet. If anything it seems to be shifting more towards the cloud.
    - BiteCode_dev 3y ago
      True, the wake up call is mostly in articles and social medias, not in the trend, in the same way most people use Windows despite most technical writers using Mac or Linux. Personally I've been self-hosting forever, and that's the only reason I could stay afloat. There is a whole category of business that is simply not profitable if you use the cloud.
    - 3y ago

mrweasel 3y ago

There's still open source hosting providers that could provide the bandwidth and maybe the storage for such a project, but I doubt many of them could provide something like S3, it would either be git, https, ftp or maybe rsync. Building something like S3 yourself on Swift or Minio isn't easily done, the management of these solution can be tricky, especially for multi-petabyte systems. S3 most likely got them going and scaling really easily, but then at some point the cost comes back to bit you. Still I don't see any good alternatives right now. I don't completely disagree with your, but the overhead of building an S3 like storage solution upfront would have actively prevented the NixOS project getting off the ground.

usr1106 3y ago

They did not make the decision that the cloud is cost-efficient. They had someone else paying their cloud bill and the cloud is easy for them. Now that the someone else is pulling out they need to think for the first time what would be cost-efficient.

traceroute66 3y ago
> Now that the someone else is pulling out they need to think for the first time They should have thought about that on day zero. If someone else is picking up the tab then there is ALWAYS the possibility that someone will pull out for whatever reason.
- usr1106 3y ago
  When you start an open-source project you don't start with thinking what will happen when you have Terabytes of traffic. Even for a start-up it's generally not advised to spend considerable effort on scalability before you have any customers.
  - traceroute66 3y ago
    >When you start an open-source project ... Rubbish. Its nothing to do with "start-up" or "open-source project". As I clearly said. Its Dummies 101 ... if "somebody" is giving me money, then only an idiot doesn't consider what happens if that "somebody" stops giving me money. Its not rocket science, its basic planning.
    - regularjack 3y ago
      What you're describing is premature optimisation. They made the right call going with the solution that required the least effort. Now that that solution is no longer viable, they can spend resources addressing it. Not fixing problems that don't exist is the right call.
      - traceroute66 3y ago

traceroute66 3y ago

> While I feel for them, they put themselves in this position. I agree entirely. I feel the same way about OpenBSD when they come round begging for money. I'm like "sure, great cause, but can we first have chat about your totally energy inefficient hosting of multiple racks full of ancient power-hungry kit in Theo's basement". Money doesn't grow on trees, and especially in this era of inflation and high energy prices, I'm afraid I don't think it is unreasonable for me to expect the organisation asking me for money to get their affairs in order first.

anotherhue 3y ago
Well they're hardly going to find a colo that'll take those machines, so you must mean that they should stop building for them? They've said before they hardware heterogeneity has been useful in bug hunting, and at worst it can be viewed as a heating expense for the fairly prolific guy. Does qemu support all those architectures ?
- traceroute66 3y ago
  > Does qemu support all those architectures ? Probably. But my understanding is that the OpenBSD core developers consider qemu as the antithesis to their beliefs. At least that's what I recall reading when it was discussed at the time, i.e. it was met with one of Theo's famous hard "no".

tecleandor 3y ago

Yep, the thing is they're using Fastly as CDN, moving 1.5PB a month, and its cost doesn't seem to be a problem for them (I don't know if its covered by Fastly or they have a good price). The gross of their problem is the storage, and S3 is probably the most expensive place to put raw storage. If they start paying the bill, their best option is moving out. Either to a cheaper provider (be it R2, B2 or whatever) or self managing it (Minio...). And seeing the $32K price for the migration to R2, and guessing that's transfer costs, I would garbage collect and move only hot data and let the rest be rebuilt during some months.

adisbladis 3y ago
Fastly sponsors NixOS.
- tecleandor 3y ago
  That's good, because that bill would be even bigger than the S3 one :P

ThinkBeat 3y ago

I think that is "something" for a huge amount of business operating around the world. For everyone here saying its trivial and they can do it cheap. Get together and form a non-profit organization that offers super cheap infrastructure to the NixOS foundation. With a 99.99% SLA and around the clock staff ready if something goes wrong. Which I think if you just co locate at a datacenter means going to the physical racks when shit breaks. (Or pay for "remote hands" and "remote spare parts". Or just pay for managed hosting. I think that sounds like a lot of work for volunteers who have other jobs as wel.

nine_k 3y ago
A perfect case for something like Backblaze to sponsor it and get some more publicity.