3 ms·

I am no fan of the cloud, but with that many disks, failure rate goes up. So you also need someone to go to the DC. All the disks will bought around the same ti

I am no fan of the cloud, but with that many disks, failure rate goes up. So you also need someone to go to the DC. All the disks will bought around the same time, which means failures start to cluster up as well. The machine also needs to be replaced every $interval. After 4 or 5 years, keep it up is more hassle that replacing it. A single machine patches and will need to reboot. What should happen to people trying to access the store? Keep replicas help, ensuring a petabyte or even half a petabyte of small files is in sync across multiple sites with various link speeds is not as trivial as rsync.

bsagdiyev 3y ago
If the NixOS team cannot handle this then I question their capabilities. This is cake work.
- quickthrower2 3y ago
  Handle yes, but is this “cheaper”
  - deleted 3y ago
    [deleted]
  - toomuchtodo 3y ago
    As is, Nix would have to spend $108k/year to continue to host from S3. For comparison, OpenStreetMap's entire budget for infra/ops is 169,197/year (FY2023).
    - georgyo 3y ago
      Of the cloud storages, AWS S3 is the most expensive for both storage and bandwidth. A half petabyte at other places such as wasabi would cost 35k a year, including bandwidth. Open street maps by comparison in 196GB compressed including full history. 20 years of full snapshots would amount to only .2PB, or 20k a month to host on cheaper storage providers. I don't think using OSM's budget was the best example.
welterde 3y ago
Most datacenters offer remote hands, so there is no need to travel to the DC for those kinds of things (just ship them the drives and have them replace it). One can also easily add extra spare hard drives to the pool that ZFS or mdraid can automatically put into production in case of drive failure, so there is no need for active intervention after each drive failure. And if any downtime for upgrades is unacceptable one can go for distributed storage solutions like ceph, seaweedfs or minio. Or simply 1:1 mirror everything to a second machine (which you want for disaster recovery anyway) and switch over during the maintenance period. And since for this specific usecase the server(s) would only serve as CDN origin, a brief downtime might even have very little actual impact.

bsagdiyev 3y ago

If the NixOS team cannot handle this then I question their capabilities. This is cake work.

quickthrower2 3y ago
Handle yes, but is this “cheaper”
- deleted 3y ago
  [deleted]
- toomuchtodo 3y ago
  As is, Nix would have to spend $108k/year to continue to host from S3. For comparison, OpenStreetMap's entire budget for infra/ops is 169,197/year (FY2023).
  - georgyo 3y ago
    Of the cloud storages, AWS S3 is the most expensive for both storage and bandwidth. A half petabyte at other places such as wasabi would cost 35k a year, including bandwidth. Open street maps by comparison in 196GB compressed including full history. 20 years of full snapshots would amount to only .2PB, or 20k a month to host on cheaper storage providers. I don't think using OSM's budget was the best example.

welterde 3y ago

Most datacenters offer remote hands, so there is no need to travel to the DC for those kinds of things (just ship them the drives and have them replace it). One can also easily add extra spare hard drives to the pool that ZFS or mdraid can automatically put into production in case of drive failure, so there is no need for active intervention after each drive failure. And if any downtime for upgrades is unacceptable one can go for distributed storage solutions like ceph, seaweedfs or minio. Or simply 1:1 mirror everything to a second machine (which you want for disaster recovery anyway) and switch over during the maintenance period. And since for this specific usecase the server(s) would only serve as CDN origin, a brief downtime might even have very little actual impact.