5 ms·

A vague description or white lie might have been better here - a storage company with a fleet of servers shouldn’t ever be in this situation for the reason give

A vague description or white lie might have been better here - a storage company with a fleet of servers shouldn’t ever be in this situation for the reason given

smcleod 3y ago
Personally I appreciate the honesty and look for it in vendors, there are too many that cover up the truth (MS/Github etc...). However I agree that it doesn't look good for a storage company.
- TheNewsIsHere 3y ago
  I agree with you. I’d like further transparency on this: why didn’t they anticipate that the array would be offline for this expansion? Expanding a storage environment can certainly be a mine-filled exercise for even the most experienced. But why was this expansion unexpectedly an offline operation? I’m not throwing shade here. I’m genuinely curious because I’ve been considering Borg and Borgbase for an upcoming project.
  - SteveNuts 3y ago
    I've been in this exact situation before in a storage expansion. With enough 22TB spinning drives nowadays you can get in to a scenario where, with the new data being written into the array and the expansion process going on at the same time the rebuild essentially will never complete. This is especially true with lower end CPU servers and without dedicated RAID cards. It is dangerous because the rebuild stresses the drive which makes a failure more likely, and a failure during the rebuild process is not great, on top of that you've got a months long ETA... No fun
    - noAnswer 3y ago
      > I've been in this exact situation before in a storage expansion. Can you tell us what solution that was? Something with ZFS or BTRFS? My experience with classic RAID system is, you can't expand them without reinitializing them. (But that comes with obvious warnings about data loss.)
      - wtallis 3y ago
        Traditional hardware RAID requires you to start over from scratch to expand an array. ZFS has some limited ability to expand an array, but there are still a lot of scenarios where the recommended solution is to start over from scratch. BTRFS has full flexibility for online resizing of arrays and changing between RAID modes and rebalancing the array, but its parity RAID modes aren't really trustworthy enough yet for a backup provider to be relying on.
        SteveNuts 3y ago
        Not always true, Dell PERC allows online expansion (you can even change raid levels on the fly in certain circumstances) but the controller switches from buffering writes to write-through cache policy during the operation which further slows things down.
      - jnsaff2 3y ago
        This may not be the answer you are looking for but one option is Ceph. Several cheap, low-power NAS boxes running Linux/Ceph and throw disk at them. Depending on the data you store you can have 3-way replication of Erasure Coding at your preferred risk/performance level. Disk failures are painless, box failures don't lose data, expansion and re-balancing scales with the number of disks and pretty quickly the limiting factor is your network (4-5 modern spinning disks can easily saturate a 10G network link).
  - kiwijamo 3y ago
    I've used borg with rsync.net for several years for my primary off-site backup and have had fantastic reliability and support from them. They cover the usage of borg on their website[1] and support a range of other backup tools as well (including even just simply copying data over SFTP and relying on automatic ZFS snapshots on their end). I pay something in the region of $100 USD a year for this service. My secondary backup is borg as well, this one hosted on a Hetzner Storage Box[2]. Having seen some bad practice in this thread, I want to make it clear that I am not duplicating the rsync instance to Hetzner but have created a separate instance entirely to avoid issues with the primary borg instance being replicated. From what I understand this is best practice (although it does mean I have to run the backup process twice--once to rsync.net and then again to Hetzner). Hetzner also provides info[3] on using their service with borg. I pay around 3.50 EUR a month for this service. I chose rsync.net due to their CEO being active on HN and also because they're based in North America (away from me in the Pacific). Hetzner , I chose because they are based in Europe giving me both operational diversity (operated by two different unrelated companies) and locational diversity (Western US for rsync.net and Northern Europe for Hetzner. Hetzner sharp pricing also helped as well. I'm sure I looked at BorgBase at one point for my EU-based backup location but something must have put me in Hetzner's direction instead. I also store photos (not my general backup, for now) in AWS S3 (largely because that makes it easier to deploy my static photo album site which it itself deployed via S3+CloudFront). I use S3 standard storage (for the time being) for a bucket in Sydney (nearest to my location) and then a glacier storage in Ireland for a bucket which is automatically replicated from my Sydney bucket. Hopefully all this is enough redundancy! But all up I only spend ~ USD20 or so a month for this peace of mind. [1]: https://www.rsync.net/products/borg.html https://www.rsync.net/products/borg.html [2]: https://www.hetzner.com/storage/storage-box https://www.hetzner.com/storage/storage-box [3]: https://community.hetzner.com/tutorials/install-and-configure-borgbackup https://community.hetzner.com/tutorials/install-and-configur...
    - nieve 3y ago
      For me everything about rsync.net was great except for the throughput. It didn't matter which continent, isp, or operating system I tested, I couldn't get past single digit Mbps and sometimes had trouble reaching that. Support tried moving me to another server, but the problem persisted. Other than that I was pretty happy, but it was completely infeasible to store PostgreSQL backups there, much less server & laptop backups. Every now and then I consider going back in case they've fixed the root problem.
  - seba_dos1 3y ago
    > I’d like further transparency on this: why didn’t they anticipate that the array would be offline for this expansion? People who can give solid explanations are probably busy right now, and those who aren't can only write what you already seen.
- sparrish 3y ago
  And it's a practical guarantee that something like this will never happen again. The wise man learns from his mistakes.
  - mkoryak 3y ago
    That man was not very wise to begin with. I wonder what else he needs to learn about _in prod_.
bartvk 3y ago
A white lie? Like what? How would you word it? You mean to downplay what actually happened?

A vague description or white lie might have been better here - a storage company with a fleet of servers shouldn’t ever be in this situation for the reason given

smcleod 3y ago

Personally I appreciate the honesty and look for it in vendors, there are too many that cover up the truth (MS/Github etc...). However I agree that it doesn't look good for a storage company.

TheNewsIsHere 3y ago
I agree with you. I’d like further transparency on this: why didn’t they anticipate that the array would be offline for this expansion? Expanding a storage environment can certainly be a mine-filled exercise for even the most experienced. But why was this expansion unexpectedly an offline operation? I’m not throwing shade here. I’m genuinely curious because I’ve been considering Borg and Borgbase for an upcoming project.
- SteveNuts 3y ago
  I've been in this exact situation before in a storage expansion. With enough 22TB spinning drives nowadays you can get in to a scenario where, with the new data being written into the array and the expansion process going on at the same time the rebuild essentially will never complete. This is especially true with lower end CPU servers and without dedicated RAID cards. It is dangerous because the rebuild stresses the drive which makes a failure more likely, and a failure during the rebuild process is not great, on top of that you've got a months long ETA... No fun
  - noAnswer 3y ago
    > I've been in this exact situation before in a storage expansion. Can you tell us what solution that was? Something with ZFS or BTRFS? My experience with classic RAID system is, you can't expand them without reinitializing them. (But that comes with obvious warnings about data loss.)
    - wtallis 3y ago
      Traditional hardware RAID requires you to start over from scratch to expand an array. ZFS has some limited ability to expand an array, but there are still a lot of scenarios where the recommended solution is to start over from scratch. BTRFS has full flexibility for online resizing of arrays and changing between RAID modes and rebalancing the array, but its parity RAID modes aren't really trustworthy enough yet for a backup provider to be relying on.
      - SteveNuts 3y ago
        Not always true, Dell PERC allows online expansion (you can even change raid levels on the fly in certain circumstances) but the controller switches from buffering writes to write-through cache policy during the operation which further slows things down.
    - jnsaff2 3y ago
      This may not be the answer you are looking for but one option is Ceph. Several cheap, low-power NAS boxes running Linux/Ceph and throw disk at them. Depending on the data you store you can have 3-way replication of Erasure Coding at your preferred risk/performance level. Disk failures are painless, box failures don't lose data, expansion and re-balancing scales with the number of disks and pretty quickly the limiting factor is your network (4-5 modern spinning disks can easily saturate a 10G network link).
- kiwijamo 3y ago
  I've used borg with rsync.net for several years for my primary off-site backup and have had fantastic reliability and support from them. They cover the usage of borg on their website[1] and support a range of other backup tools as well (including even just simply copying data over SFTP and relying on automatic ZFS snapshots on their end). I pay something in the region of $100 USD a year for this service. My secondary backup is borg as well, this one hosted on a Hetzner Storage Box[2]. Having seen some bad practice in this thread, I want to make it clear that I am not duplicating the rsync instance to Hetzner but have created a separate instance entirely to avoid issues with the primary borg instance being replicated. From what I understand this is best practice (although it does mean I have to run the backup process twice--once to rsync.net and then again to Hetzner). Hetzner also provides info[3] on using their service with borg. I pay around 3.50 EUR a month for this service. I chose rsync.net due to their CEO being active on HN and also because they're based in North America (away from me in the Pacific). Hetzner , I chose because they are based in Europe giving me both operational diversity (operated by two different unrelated companies) and locational diversity (Western US for rsync.net and Northern Europe for Hetzner. Hetzner sharp pricing also helped as well. I'm sure I looked at BorgBase at one point for my EU-based backup location but something must have put me in Hetzner's direction instead. I also store photos (not my general backup, for now) in AWS S3 (largely because that makes it easier to deploy my static photo album site which it itself deployed via S3+CloudFront). I use S3 standard storage (for the time being) for a bucket in Sydney (nearest to my location) and then a glacier storage in Ireland for a bucket which is automatically replicated from my Sydney bucket. Hopefully all this is enough redundancy! But all up I only spend ~ USD20 or so a month for this peace of mind. [1]: https://www.rsync.net/products/borg.html https://www.rsync.net/products/borg.html [2]: https://www.hetzner.com/storage/storage-box https://www.hetzner.com/storage/storage-box [3]: https://community.hetzner.com/tutorials/install-and-configure-borgbackup https://community.hetzner.com/tutorials/install-and-configur...
  - nieve 3y ago
    For me everything about rsync.net was great except for the throughput. It didn't matter which continent, isp, or operating system I tested, I couldn't get past single digit Mbps and sometimes had trouble reaching that. Support tried moving me to another server, but the problem persisted. Other than that I was pretty happy, but it was completely infeasible to store PostgreSQL backups there, much less server & laptop backups. Every now and then I consider going back in case they've fixed the root problem.
- seba_dos1 3y ago
  > I’d like further transparency on this: why didn’t they anticipate that the array would be offline for this expansion? People who can give solid explanations are probably busy right now, and those who aren't can only write what you already seen.
sparrish 3y ago
And it's a practical guarantee that something like this will never happen again. The wise man learns from his mistakes.
- mkoryak 3y ago
  That man was not very wise to begin with. I wonder what else he needs to learn about _in prod_.

bartvk 3y ago

A white lie? Like what? How would you word it? You mean to downplay what actually happened?