4 ms·

It is astonishing as to how marketers from the three big clouds have successfully convinced people that there is no middle ground between operating your data ce

It is astonishing as to how marketers from the three big clouds have successfully convinced people that there is no middle ground between operating your data center and using one of their cloud services. Here's what you do: 1. Buy two servers equipped with a large amount of storage. There are many providers[1][2] that offer this. (You could technically do it with one, but two allows you to have high availability and prevent data loss in case of accidents, imagine `rm -rf *`.) 2. Use Minio to set up erasure-coded storage on one of the servers, and replicate objects[3] to the other server. 4. Use Cloudflare (because of their open source sponsorship program[4]) to serve the objects from the primary server, and when performing maintenance tasks, fall back to the secondary. You could also serve objects directly out of the primary/secondary, but using Cloudflare takes care of low-effort DDoS attacks by hiding the origin, and provides some caching which helps prevent hitting the egress limits of the server provider. [1] https://www.netcup.eu/vserver/vstorage.php https://www.netcup.eu/vserver/vstorage.php [2] https://www.hetzner.com/dedicated-rootserver https://www.hetzner.com/dedicated-rootserver [3] https://blog.min.io/active-active-replication/ https://blog.min.io/active-active-replication/ [4] https://blog.cloudflare.com/cloudflare-new-oss-sponsorships-program/ https://blog.cloudflare.com/cloudflare-new-oss-sponsorships-...

usr1106 3y ago
Yes, perfectly doable, but the expertise and time needed to do it (without fatal mistakes) and keep it running 365 days a year is non-negligible. Doing this by unpaid volunteers is more than challenging.
- jacquesm 3y ago
  Not for the people behind NixOS. If it is challenging then that would cause me to re-calibrate my opinion of the project as a whole.
- simonjgreen 3y ago
  This is, in my opinion, a falacy perpetuated by the marketing departments of "big cloud". Don't forget that pre-2006 there was no S3, and we managed just fine. They are not challenging skills to learn, the zeitgeist has just convinced itself they are not worth learning
  - vasco 3y ago
    It's condescending to think that someone can't reasonably choose to pay for a service they are capable of doing. Cooking isn't hard either but I still go to restaurants, and don't go around laughing at patrons telling them that they never bothered to learn how to cook if they are there.
    - heisenbit 3y ago
      Eating in the restaurant once in a while is fine. Lage range of choices delivered quickly. But is it sustainable? Price higher, usually less healthy and not tailored to special needs. In any case one does it professional IT done right is costly but cheaper than e.g. IT without offsite backup in case of a disaster.
    - simonjgreen 3y ago
      Do you eat every meal at a restaurant though? That would not be sustainable. To roll with your analogy it would be like the hospitality sector persuading the majority to ditch home meals entirely
      - vasco 3y ago
        The problem was your assumption which is condescending. I gave you an analogy to make it clearer. Taking the analogy in the second paragraph apart doesn't change the point in the first - it was simply to help illustrate it, but it stands on its own.
        simonjgreen 3y ago
        It's not condescending at all. You are saying they could reasonably pay for it, but they quite obviously cannot (hence the existence of this whole post). What I'm principally challenging is the assumption that this required effort is substantial. I'm making the point that it is not, certainly (in my view) compared to the effort to run an entire operating system project and all of the ancillaries that come with that. This is an area they have chosen to pay for that (again, in my view) they don't need to. They have gone to great lengths to architect a system that allows deep pinning of very specific versions, it's not a stretch to think about the architecture required to deliver that. Instead they have stopped that line of thinking at [insert S3 here] rather than carry on solving. A direct consequence of that now is a $9k/m hole they need to fill in their budget. I'm being critical, sure, but not condescending. Overall, it frustrates me seeing opinions that operating infrastructure is difficult and therefore must automatically be avoided. In this particular example avoiding operating said infrastructure potentially is putting the project in the red and at risk, but nowhere on the NixOS call for help does it suggest they have considered operating this themselves. And there in is the crux of my point. The zeitgeist has turned against operating infrastructure, and yet is frequently caught out by the cost of this decision.
- makkesk8 3y ago
  We do this at work, albeit much larger scale than this and I'm the only one who spends any time on upgrades, replacing failed disks and so on. Honestly, apart from upgrades every 3-ish months I don't spend any time at all on this, it's all chugging along without intervention. Things don't fail as much as they used to, at least not from using something mature as minio and of course hardware can fail as they do specifically spinning rust, which is non trivial to replace if you have hot-swap capable servers in the first place. I don't possess any decade worth expertise in either minio or running this kind of setup so personally I believe you are exaggerating that this is even worth calling challenge at all. But that's just my take :)
  - mynameisvlad 3y ago
    So I’m sure you’d be more than happy to volunteer your services to NixOS and be responsible for the setup, including when there are issues (which will inevitably happen regardless of tools and technologies used), right? It’s so easy, after all.
    - johnklos 3y ago
      I'd volunteer to do it, if they paid for hardware, colo and transportation for initial install. After things settle down, it takes no more than half an hour or so every month. Pretending to know how much work something is when it's not your area of expertise is a bit silly. Actually, yes, it is easy, after all :)
- nine_k 3y ago
  The money that are paid as the exorbitant AWS egress fees could be used to pay a competent SRE to handle that. I don't think that running the package mirror would require a full-time dedicated paid position. Initial setup, maybe some heavy restructuring operations as storage requirements grow, maybe some firefighting, but not the daily operation.
lacasito25 3y ago
I mean, you make S3 look easy compared to that
- danielrmay 3y ago
  That's the business model.
- supriyo-biswas 3y ago
  If `minio server /mnt/sd[abcd]1` and the occasional `apt-get upgrade` and changeover of DNS records is so much work, maybe you should just spend 100x for S3.
  - bsagdiyev 3y ago
    Seriously, there is a large amount of comments of people who seem terrified of this or throw out "but but updates and failing hardware". We run multi PB storage on prem at work and it's just fine. Not some massive team running it.
vasco 3y ago
You do all of this and you still don't have nearly the availability or durability of data you get from S3, nor people on call to fix it once anything happens to it. It's like comparing leasing a Mercedes-Benz with full service and all inclusive insurance to building your own bicycle. Some people don't need the car, sure, but it's not the same thing.
- debugnik 3y ago
  But they're complaining about the cost of leasing the Mercedes, so they need to lower their expectations if they want a cheaper solution. Although having to spend some time to attain good enough availability/durability is not a big deal when the rush to launch was over long ago. Unlike in your example, you can turn the bicycle into a car.
- megous 3y ago
  The usecase is a cache. http://cache.nixos.org/ http://cache.nixos.org/ All it needs to do is to get you the data faster than the compilation itself - most of the time - to be useful. I also wonder at the original thinking, given the use case. Cache usually has high ratio of download to upload and doesn't need high reliability. S3 doesn't excel at the costs of fetching the data from it, at all.
  - Filligree 3y ago
    It also “caches” the source code downloads, many of which are no longer available from the URL listed in the repository, if anywhere at all.
- johnklos 3y ago
  It only seems like that to people who don't know any better. Do you ACTUALLY think you can communicate with people at Amazon who truly know what's going on? If so, why do we see countless stories about people who have problems with Amazon, Google, et cetera, who can't communicate with them? Do you know of some special, magic way to get the skilled people to talk with you that the rest of us don't? You act like being able to call Amazon, even if you could get someone on the line who knows something, is somehow better than having even just one good systems administrator. All of your ridiculous assumptions go right out the window when you have a sysadmin build, test and deploy your own hardware that you (via your sysadmin) have complete control over. Security? You can't secure Amazon. You literally can't. If you wanted to, you'd have to buy the company, then you'd have to evaluate your whole S3 staff. If you hire a systems administrator, you have to vet your systems administrator and anyone else who has admin access, not a collection of tens of thousands of employees across the planet, any of whom could be stupid enough to visit the wrong web site using the wrong browser on the same computer as their keys. Availability? My side projects have better uptime than Amazon. Durability? Two geographically disparate locations and LTO, plus many mirrors, make things plenty durable. When you're not a systems administrator, you can be forgiven for not understanding the big picture, but not for acting like you do.
baq 3y ago
In the olden days you’d find someone to donate you a server and have a few volunteers around the world mirror it.
- not_your_vase 3y ago
  It still happens. You need storage a TB or two with unlimited (or at least very high) traffic? That's not an issue. Even I could help being a mirror from home. But I wouldn't be surprised if there would be some free tiers already from cloud provider. Half PB is a bit more than that, and I guess it is expected to grow... in 2023 it is not realistic to throw such storage around for free (+mirrors). It will be realistic in a few years considering that the storage prices have been in free fall for the past 30 years, but that time hasn't came yet. (As a maybe interesting comparison, Debian's mirror is around half TB: https://people.debian.org/~koster/www/mirror/size https://people.debian.org/~koster/www/mirror/size )
  - mcbits 3y ago
    The falling cost of storage is at least partly the reason hosting a few thousand packages takes up hundreds of TB. "Storage is practically free" from the perspective of each of those projects. By the time .5 PB is a trivial cost, they'll need 50 PB.
    - not_your_vase 3y ago
      Yeah, that too.
    - est31 3y ago
      Yeah NixOS has done some technical choices to require tons of data downloads, even if there is little modification of the content of the packages. Regularly my computer has 10 Gb worth of downloads because some high level package got upgraded and even if it's dynamically linked, the lower level packages still get re-downloaded from cache.nixos.org. It doesn't matter because storage is free right?
goodpoint 3y ago
You really don't need al that mess, just a bunch of mirrors around the world.
chpatrick 3y ago
Or a couple SX servers from Hetzner https://www.hetzner.com/dedicated-rootserver/matrix-sx https://www.hetzner.com/dedicated-rootserver/matrix-sx (You would need 4 SX294s for half a petabyte with 2x replication) And put ceph on them which gives you an S3-compatible API: https://docs.ceph.com/en/quincy/radosgw/index.html https://docs.ceph.com/en/quincy/radosgw/index.html It's not trivial to set up but once you do it's much cheaper than AWS. It's even cheaper if you can find the machines in the server auction. There are some 150 TB machines on there right now for 174 euro/month.
- antongribok 3y ago
  While I totally agree you can easily handle this with Ceph, I would never do it with 2x replication on spinning rust. You can maybe do 2x replication with SSDs if you really know what you're doing and you have data replicated on multiple Ceph clusters. Also, your math does not quite add up if you want to store 425TiB. You'd be filling up the Ceph cluster to 95% which would be impossible (you'll start losing OSDs way before that). If this was me, I would definitely do this on Ceph, but I would opt for a more traditional erasure coding setup of 8+3. For that, you'd need 12x SX134s as a minimum starting point. I suppose you could start with an EC profile of 2+2 or 4+2... I do have a serious question on Hetzner: Do they offer any private networking options? That page only lists a single 1 Gbit connection. I don't think that would work well with Ceph. It's doable, sure, but I wouldn't want to support such a cluster, not with 160TB raw per box.
  - justinclift 3y ago
    > Do they offer any private networking options? Doesn't seem like it for their dedicated servers, not 100% sure though. Their docs indicate that upgrading the 1Gbit ethernet link to 10GBe is an available option: https://docs.hetzner.com/robot/dedicated-server/network/10g-uplink https://docs.hetzner.com/robot/dedicated-server/network/10g-...
  - chpatrick 3y ago
    You can upgrade them to 10gb NICs.
  - 3y ago
rakoo 3y ago
On the technical side, garage (https://garagehq.deuxfleurs.fr/ https://garagehq.deuxfleurs.fr/) does multi master replication by default, so is probably better for this use case. Still with S3 API.
- slondr 3y ago
  I have a few projects running off of a self-hosted garage instance, it’s very nice software.

usr1106 3y ago

Yes, perfectly doable, but the expertise and time needed to do it (without fatal mistakes) and keep it running 365 days a year is non-negligible. Doing this by unpaid volunteers is more than challenging.

jacquesm 3y ago
Not for the people behind NixOS. If it is challenging then that would cause me to re-calibrate my opinion of the project as a whole.
simonjgreen 3y ago
This is, in my opinion, a falacy perpetuated by the marketing departments of "big cloud". Don't forget that pre-2006 there was no S3, and we managed just fine. They are not challenging skills to learn, the zeitgeist has just convinced itself they are not worth learning
- vasco 3y ago
  It's condescending to think that someone can't reasonably choose to pay for a service they are capable of doing. Cooking isn't hard either but I still go to restaurants, and don't go around laughing at patrons telling them that they never bothered to learn how to cook if they are there.
  - heisenbit 3y ago
    Eating in the restaurant once in a while is fine. Lage range of choices delivered quickly. But is it sustainable? Price higher, usually less healthy and not tailored to special needs. In any case one does it professional IT done right is costly but cheaper than e.g. IT without offsite backup in case of a disaster.
  - simonjgreen 3y ago
    Do you eat every meal at a restaurant though? That would not be sustainable. To roll with your analogy it would be like the hospitality sector persuading the majority to ditch home meals entirely
    - vasco 3y ago
      The problem was your assumption which is condescending. I gave you an analogy to make it clearer. Taking the analogy in the second paragraph apart doesn't change the point in the first - it was simply to help illustrate it, but it stands on its own.
      - simonjgreen 3y ago
        It's not condescending at all. You are saying they could reasonably pay for it, but they quite obviously cannot (hence the existence of this whole post). What I'm principally challenging is the assumption that this required effort is substantial. I'm making the point that it is not, certainly (in my view) compared to the effort to run an entire operating system project and all of the ancillaries that come with that. This is an area they have chosen to pay for that (again, in my view) they don't need to. They have gone to great lengths to architect a system that allows deep pinning of very specific versions, it's not a stretch to think about the architecture required to deliver that. Instead they have stopped that line of thinking at [insert S3 here] rather than carry on solving. A direct consequence of that now is a $9k/m hole they need to fill in their budget. I'm being critical, sure, but not condescending. Overall, it frustrates me seeing opinions that operating infrastructure is difficult and therefore must automatically be avoided. In this particular example avoiding operating said infrastructure potentially is putting the project in the red and at risk, but nowhere on the NixOS call for help does it suggest they have considered operating this themselves. And there in is the crux of my point. The zeitgeist has turned against operating infrastructure, and yet is frequently caught out by the cost of this decision.
makkesk8 3y ago
We do this at work, albeit much larger scale than this and I'm the only one who spends any time on upgrades, replacing failed disks and so on. Honestly, apart from upgrades every 3-ish months I don't spend any time at all on this, it's all chugging along without intervention. Things don't fail as much as they used to, at least not from using something mature as minio and of course hardware can fail as they do specifically spinning rust, which is non trivial to replace if you have hot-swap capable servers in the first place. I don't possess any decade worth expertise in either minio or running this kind of setup so personally I believe you are exaggerating that this is even worth calling challenge at all. But that's just my take :)
- mynameisvlad 3y ago
  So I’m sure you’d be more than happy to volunteer your services to NixOS and be responsible for the setup, including when there are issues (which will inevitably happen regardless of tools and technologies used), right? It’s so easy, after all.
  - johnklos 3y ago
    I'd volunteer to do it, if they paid for hardware, colo and transportation for initial install. After things settle down, it takes no more than half an hour or so every month. Pretending to know how much work something is when it's not your area of expertise is a bit silly. Actually, yes, it is easy, after all :)
nine_k 3y ago
The money that are paid as the exorbitant AWS egress fees could be used to pay a competent SRE to handle that. I don't think that running the package mirror would require a full-time dedicated paid position. Initial setup, maybe some heavy restructuring operations as storage requirements grow, maybe some firefighting, but not the daily operation.

lacasito25 3y ago

I mean, you make S3 look easy compared to that

danielrmay 3y ago
That's the business model.
supriyo-biswas 3y ago
If `minio server /mnt/sd[abcd]1` and the occasional `apt-get upgrade` and changeover of DNS records is so much work, maybe you should just spend 100x for S3.
- bsagdiyev 3y ago
  Seriously, there is a large amount of comments of people who seem terrified of this or throw out "but but updates and failing hardware". We run multi PB storage on prem at work and it's just fine. Not some massive team running it.

vasco 3y ago

You do all of this and you still don't have nearly the availability or durability of data you get from S3, nor people on call to fix it once anything happens to it. It's like comparing leasing a Mercedes-Benz with full service and all inclusive insurance to building your own bicycle. Some people don't need the car, sure, but it's not the same thing.

debugnik 3y ago
But they're complaining about the cost of leasing the Mercedes, so they need to lower their expectations if they want a cheaper solution. Although having to spend some time to attain good enough availability/durability is not a big deal when the rush to launch was over long ago. Unlike in your example, you can turn the bicycle into a car.
megous 3y ago
The usecase is a cache. http://cache.nixos.org/ http://cache.nixos.org/ All it needs to do is to get you the data faster than the compilation itself - most of the time - to be useful. I also wonder at the original thinking, given the use case. Cache usually has high ratio of download to upload and doesn't need high reliability. S3 doesn't excel at the costs of fetching the data from it, at all.
- Filligree 3y ago
  It also “caches” the source code downloads, many of which are no longer available from the URL listed in the repository, if anywhere at all.
johnklos 3y ago
It only seems like that to people who don't know any better. Do you ACTUALLY think you can communicate with people at Amazon who truly know what's going on? If so, why do we see countless stories about people who have problems with Amazon, Google, et cetera, who can't communicate with them? Do you know of some special, magic way to get the skilled people to talk with you that the rest of us don't? You act like being able to call Amazon, even if you could get someone on the line who knows something, is somehow better than having even just one good systems administrator. All of your ridiculous assumptions go right out the window when you have a sysadmin build, test and deploy your own hardware that you (via your sysadmin) have complete control over. Security? You can't secure Amazon. You literally can't. If you wanted to, you'd have to buy the company, then you'd have to evaluate your whole S3 staff. If you hire a systems administrator, you have to vet your systems administrator and anyone else who has admin access, not a collection of tens of thousands of employees across the planet, any of whom could be stupid enough to visit the wrong web site using the wrong browser on the same computer as their keys. Availability? My side projects have better uptime than Amazon. Durability? Two geographically disparate locations and LTO, plus many mirrors, make things plenty durable. When you're not a systems administrator, you can be forgiven for not understanding the big picture, but not for acting like you do.

baq 3y ago

In the olden days you’d find someone to donate you a server and have a few volunteers around the world mirror it.

not_your_vase 3y ago
It still happens. You need storage a TB or two with unlimited (or at least very high) traffic? That's not an issue. Even I could help being a mirror from home. But I wouldn't be surprised if there would be some free tiers already from cloud provider. Half PB is a bit more than that, and I guess it is expected to grow... in 2023 it is not realistic to throw such storage around for free (+mirrors). It will be realistic in a few years considering that the storage prices have been in free fall for the past 30 years, but that time hasn't came yet. (As a maybe interesting comparison, Debian's mirror is around half TB: https://people.debian.org/~koster/www/mirror/size https://people.debian.org/~koster/www/mirror/size )
- mcbits 3y ago
  The falling cost of storage is at least partly the reason hosting a few thousand packages takes up hundreds of TB. "Storage is practically free" from the perspective of each of those projects. By the time .5 PB is a trivial cost, they'll need 50 PB.
  - not_your_vase 3y ago
    Yeah, that too.
  - est31 3y ago
    Yeah NixOS has done some technical choices to require tons of data downloads, even if there is little modification of the content of the packages. Regularly my computer has 10 Gb worth of downloads because some high level package got upgraded and even if it's dynamically linked, the lower level packages still get re-downloaded from cache.nixos.org. It doesn't matter because storage is free right?

goodpoint 3y ago

You really don't need al that mess, just a bunch of mirrors around the world.

chpatrick 3y ago

Or a couple SX servers from Hetzner https://www.hetzner.com/dedicated-rootserver/matrix-sx https://www.hetzner.com/dedicated-rootserver/matrix-sx (You would need 4 SX294s for half a petabyte with 2x replication) And put ceph on them which gives you an S3-compatible API: https://docs.ceph.com/en/quincy/radosgw/index.html https://docs.ceph.com/en/quincy/radosgw/index.html It's not trivial to set up but once you do it's much cheaper than AWS. It's even cheaper if you can find the machines in the server auction. There are some 150 TB machines on there right now for 174 euro/month.

antongribok 3y ago
While I totally agree you can easily handle this with Ceph, I would never do it with 2x replication on spinning rust. You can maybe do 2x replication with SSDs if you really know what you're doing and you have data replicated on multiple Ceph clusters. Also, your math does not quite add up if you want to store 425TiB. You'd be filling up the Ceph cluster to 95% which would be impossible (you'll start losing OSDs way before that). If this was me, I would definitely do this on Ceph, but I would opt for a more traditional erasure coding setup of 8+3. For that, you'd need 12x SX134s as a minimum starting point. I suppose you could start with an EC profile of 2+2 or 4+2... I do have a serious question on Hetzner: Do they offer any private networking options? That page only lists a single 1 Gbit connection. I don't think that would work well with Ceph. It's doable, sure, but I wouldn't want to support such a cluster, not with 160TB raw per box.
- justinclift 3y ago
  > Do they offer any private networking options? Doesn't seem like it for their dedicated servers, not 100% sure though. Their docs indicate that upgrading the 1Gbit ethernet link to 10GBe is an available option: https://docs.hetzner.com/robot/dedicated-server/network/10g-uplink https://docs.hetzner.com/robot/dedicated-server/network/10g-...
- chpatrick 3y ago
  You can upgrade them to 10gb NICs.
- 3y ago

rakoo 3y ago

On the technical side, garage (https://garagehq.deuxfleurs.fr/ https://garagehq.deuxfleurs.fr/) does multi master replication by default, so is probably better for this use case. Still with S3 API.

slondr 3y ago
I have a few projects running off of a self-hosted garage instance, it’s very nice software.