8 ms·

Can confirm. We run 26 services in production on DigitalOcean. Every single VPS in our setup uses block storage feature as a persistence layer (system logs, ap

Can confirm. We run 26 services in production on DigitalOcean. Every single VPS in our setup uses block storage feature as a persistence layer (system logs, app logs, databases, etc). Thanks to this architecture we can rebuild machines at will. The _function_ and _state_ are nicely separated. The downside is that we are now fucked.

mattbillenstein 9y ago
You should re-architect without network block storage as a requirement imho. Ever since that big AWS EBS outage in 2012 or whatever, I avoid it like the plague. Databases like low-latency local storage (the newer nvme instances on aws are very good) and logs and whatnot are aggregated with other systems (fluent, logstash, etc). I actually do not miss EBS much at all really -- if I have a problem with a VM, they're disposable and redundant and I've designed out most of the SPOFs.
- qertoip 9y ago
  We are actually very happy with database performance over block storage. As for the architecture, in DigitalOcean the external block storage is the only way to separate function from state. We need this to be able to rebuild VPS-es regularly. This is to avoid configuration drift and to prove/enforce reproducibility.
  - mattbillenstein 9y ago
    I guess I've solved that in other ways -- configuration drift is really a config management and orchestration problem. And recycling or OS upgrades can happen if your story around replication is sound -- I only typically do that every two years anyway following the Ubuntu LTS releases.
- joelhaasnoot 9y ago
  Coming from an enterprise background for a couple of years: network block storage screws you one way or another either way. It's a single point of failure that no matter how reliable it is, will always go down at some point or have failures that affect your application layer.
  - llama052 9y ago
    Sounds like you've been in environments that have done network block storage incorrectly. If you do multi-path, redundant/replicated network storage with proper configurations in place, you shouldn't have a single point of failure. Ceph oddities aside. Everything has issues given a long enough time span though. Each layer has an accounted level of risk, and it definitely is a balancing act.
  - antongribok 9y ago
    That blanket statement is absolutely not true. With Ceph you can distribute data across hosts, switches, racks, rows, fabrics, rooms, datacenters. If you design correctly, you can have a very resilient storage system. Source: I come from enterprise storage and have been running Ceph in production (successfully) for ~5 years.
    - halbritt 9y ago
      Ceph has some weird failure modes, though. Your radius of failure absolutely includes the entire cluster, regardless of how resilient its been made. Sure, you're insulated from hardware failures, but not from bugs in the underlying system. Data loss is rare, but having a cluster go unavailable does happen.
      - antongribok 9y ago
        > Ceph has some weird failure modes No argument. > radius of failure absolutely includes the entire cluster Completely agree, same goes for any other storage system, network-based or otherwise. I'm simply disagreeing with a blanket statement that all network block storage is unreliable, and that is simply not true. Don't do silly things with Ceph, don't just have a single network fabric, don't buy cheap switches that drop packets on the floor at medium load, don't test things in production.
        halbritt 9y ago
        Oh, do test things in production. Can’t understate the need for good network, especially with the traffic amplification of replication. I was running 2x10GE and had plans to scale up the cluster network to 2x or 3x10GE. My biggest issue was the MTTR of a failed cluster. If one is RBD mirroring to a different cluster, recovery time may be hours or days. That said, I had one cluster run continuously for a couple years with basically zero administration. It had great performance and good reliability other than that one week-long service impacting incident as a result of a bug. I’d definitely run it again.
- boulos 9y ago
  Disclosure: I work on Google Cloud. I wouldn't throw the baby out with the bathwater. Across Google, we rely almost exclusively on networked storage (Colossus) unless we need extremely high-performance local flash. Being able to separate compute from storage is a huge part of our ability to both scale out, as well as to do live migration for GCE (where Persistent Disk, our equivalent of EBS is built upon Colossus). Persistent Disk has never had an outage like the EBS one, but I attribute that to the Colossus and underlying teams having run this at Google for a really long time. Fwiw, the AWS folks have also massively improved EBS over the years. You can still worry, but I prefer to think about overall MTBF rather than consider networked storage in particular as the plague :).
  - mattbillenstein 9y ago
    To each their own. I don't like getting paged at 3am, so I choose to remove complexity and external dependencies where I can. And not operating at Google-scale, this is one of those things I can live without.
  - butwhythough 9y ago
    Disclosure: I use both gce and ec2 Unfortunately unlike EBS, persistent disks (Colossus) in Google cloud share a network plane with the VM. To quote the docs. "Each persistent disk write operation contributes to your virtual machine instance's cumulative network egress cap." https://cloud.google.com/compute/docs/disks/performance https://cloud.google.com/compute/docs/disks/performance If you haven't noticed the mtu in Google cloud is < 1500 vs AWS where you can get jumbo frames (9k). I have no reason to believe persistent disks are different. Enabling live migrations? You mean the choice of having an instance terminated with zero notice or migrate with a 60 second warning if you subscribe to the right API? Oh yeah and this is happening constantly. Live migrations (and by connection VM attrition), persistent disks, and network performance are my least favorite aspects of google cloud today. That being said google cloud does have alot of advantages over ec2. These are just not them.
    - cube2222 9y ago
      No, that's not that actually. On google cloud platform your virtual machine usually gets migrated between machines without any downtime. Aren't you talking about preemptible instances?
- pmlnr 9y ago
  Add a cache layer for read-only content which doesn't invalidate in case of a failure like this, so it allows your system to go into read-only mode. Local copies of puppet/ansible can be very useful as well. For logs and metrics, I'm not aware of an out of the box solution that could replay from the last, successfully transmitted line; this is something rsyslog, graphite, etc. could certainly benefit from. (Please let me know if you are aware of these kinds of buffers.) However, distributed network block storage is usually something unavoidable after a certain size; local filers are really expensive.
  - jlgaddis 9y ago
    FWIW, rsyslog can queue up log messages locally if the central server it is sending to is down (and will send them when it comes back up, of course).
  - lima 9y ago
    > For logs and metrics, I'm not aware of an out of the box solution that could replay from the last, successfully transmitted line; this is something rsyslog, graphite, etc. could certainly benefit from. (Please let me know if you are aware of these kinds of buffers.) filebeat + kafka does exactly this.
- notatoad 9y ago
  Avoiding an entire class of storage technology because one vendor's implementation went down in 2012 seems a little questionable.

mattbillenstein 9y ago

You should re-architect without network block storage as a requirement imho. Ever since that big AWS EBS outage in 2012 or whatever, I avoid it like the plague. Databases like low-latency local storage (the newer nvme instances on aws are very good) and logs and whatnot are aggregated with other systems (fluent, logstash, etc). I actually do not miss EBS much at all really -- if I have a problem with a VM, they're disposable and redundant and I've designed out most of the SPOFs.

qertoip 9y ago
We are actually very happy with database performance over block storage. As for the architecture, in DigitalOcean the external block storage is the only way to separate function from state. We need this to be able to rebuild VPS-es regularly. This is to avoid configuration drift and to prove/enforce reproducibility.
- mattbillenstein 9y ago
  I guess I've solved that in other ways -- configuration drift is really a config management and orchestration problem. And recycling or OS upgrades can happen if your story around replication is sound -- I only typically do that every two years anyway following the Ubuntu LTS releases.
joelhaasnoot 9y ago
Coming from an enterprise background for a couple of years: network block storage screws you one way or another either way. It's a single point of failure that no matter how reliable it is, will always go down at some point or have failures that affect your application layer.
- llama052 9y ago
  Sounds like you've been in environments that have done network block storage incorrectly. If you do multi-path, redundant/replicated network storage with proper configurations in place, you shouldn't have a single point of failure. Ceph oddities aside. Everything has issues given a long enough time span though. Each layer has an accounted level of risk, and it definitely is a balancing act.
- antongribok 9y ago
  That blanket statement is absolutely not true. With Ceph you can distribute data across hosts, switches, racks, rows, fabrics, rooms, datacenters. If you design correctly, you can have a very resilient storage system. Source: I come from enterprise storage and have been running Ceph in production (successfully) for ~5 years.
  - halbritt 9y ago
    Ceph has some weird failure modes, though. Your radius of failure absolutely includes the entire cluster, regardless of how resilient its been made. Sure, you're insulated from hardware failures, but not from bugs in the underlying system. Data loss is rare, but having a cluster go unavailable does happen.
    - antongribok 9y ago
      > Ceph has some weird failure modes No argument. > radius of failure absolutely includes the entire cluster Completely agree, same goes for any other storage system, network-based or otherwise. I'm simply disagreeing with a blanket statement that all network block storage is unreliable, and that is simply not true. Don't do silly things with Ceph, don't just have a single network fabric, don't buy cheap switches that drop packets on the floor at medium load, don't test things in production.
      - halbritt 9y ago
        Oh, do test things in production. Can’t understate the need for good network, especially with the traffic amplification of replication. I was running 2x10GE and had plans to scale up the cluster network to 2x or 3x10GE. My biggest issue was the MTTR of a failed cluster. If one is RBD mirroring to a different cluster, recovery time may be hours or days. That said, I had one cluster run continuously for a couple years with basically zero administration. It had great performance and good reliability other than that one week-long service impacting incident as a result of a bug. I’d definitely run it again.
boulos 9y ago
Disclosure: I work on Google Cloud. I wouldn't throw the baby out with the bathwater. Across Google, we rely almost exclusively on networked storage (Colossus) unless we need extremely high-performance local flash. Being able to separate compute from storage is a huge part of our ability to both scale out, as well as to do live migration for GCE (where Persistent Disk, our equivalent of EBS is built upon Colossus). Persistent Disk has never had an outage like the EBS one, but I attribute that to the Colossus and underlying teams having run this at Google for a really long time. Fwiw, the AWS folks have also massively improved EBS over the years. You can still worry, but I prefer to think about overall MTBF rather than consider networked storage in particular as the plague :).
- mattbillenstein 9y ago
  To each their own. I don't like getting paged at 3am, so I choose to remove complexity and external dependencies where I can. And not operating at Google-scale, this is one of those things I can live without.
- butwhythough 9y ago
  Disclosure: I use both gce and ec2 Unfortunately unlike EBS, persistent disks (Colossus) in Google cloud share a network plane with the VM. To quote the docs. "Each persistent disk write operation contributes to your virtual machine instance's cumulative network egress cap." https://cloud.google.com/compute/docs/disks/performance https://cloud.google.com/compute/docs/disks/performance If you haven't noticed the mtu in Google cloud is < 1500 vs AWS where you can get jumbo frames (9k). I have no reason to believe persistent disks are different. Enabling live migrations? You mean the choice of having an instance terminated with zero notice or migrate with a 60 second warning if you subscribe to the right API? Oh yeah and this is happening constantly. Live migrations (and by connection VM attrition), persistent disks, and network performance are my least favorite aspects of google cloud today. That being said google cloud does have alot of advantages over ec2. These are just not them.
  - cube2222 9y ago
    No, that's not that actually. On google cloud platform your virtual machine usually gets migrated between machines without any downtime. Aren't you talking about preemptible instances?
pmlnr 9y ago
Add a cache layer for read-only content which doesn't invalidate in case of a failure like this, so it allows your system to go into read-only mode. Local copies of puppet/ansible can be very useful as well. For logs and metrics, I'm not aware of an out of the box solution that could replay from the last, successfully transmitted line; this is something rsyslog, graphite, etc. could certainly benefit from. (Please let me know if you are aware of these kinds of buffers.) However, distributed network block storage is usually something unavoidable after a certain size; local filers are really expensive.
- jlgaddis 9y ago
  FWIW, rsyslog can queue up log messages locally if the central server it is sending to is down (and will send them when it comes back up, of course).
- lima 9y ago
  > For logs and metrics, I'm not aware of an out of the box solution that could replay from the last, successfully transmitted line; this is something rsyslog, graphite, etc. could certainly benefit from. (Please let me know if you are aware of these kinds of buffers.) filebeat + kafka does exactly this.
notatoad 9y ago
Avoiding an entire class of storage technology because one vendor's implementation went down in 2012 seems a little questionable.