6 ms·

It's very nice to see someone putting in the work to document this and show others how they were able to get it to work. But I unfortunately believe, quite sin

It's very nice to see someone putting in the work to document this and show others how they were able to get it to work. But I unfortunately believe, quite sincerely, that redundancy at this level is misguided at best and dangerous at worst. Instead of building storage that can fail between nodes, consider building applications that can survive node loss and are not relying on persisting their state in a black box that is assumed to be 100% available. These assumptions about reliablity of systems and their consistent behavior under network partitions has bitten countless people before and will continue to claim victims unless application paradigms are fundamentally rethought. Many people already know this. Even people who build these systems know this, but too many applications already exist that can't be change. Nonetheless, it's important for people building such systems for the first time to be aware that they can make a choice early in the design process and that the status quo is not a good choice. In other words, "Every day, somebody's born who's never seen The Flintstones." (https://mobile.twitter.com/hotdogsladies/status/760580954532433920 https://mobile.twitter.com/hotdogsladies/status/760580954532...)

windowsworkstoo 10y ago
Given the target use case of this kind of system, you control what you can control, which is the infra, because the software will overwhelmingly be industry specific COTS stuff that realistically wont be changed. Further its likely that the types of internal apps running could handle a recovery period, makes total loss less of an issue...this is "old school" operations - having DR plans, backups and recovery time targets
qaq 10y ago
There are legitimate reasons to have a CP system and not an AP system, it's obviously application specific. Also over-engineered systems come with their own risks and can experience as much down time as a well setup CP system due to human error stemming from the system's complexity (AWS is a good example).
- zzzcpan 10y ago
  NASes are not CP systems, they are noCAP systems. They cannot guarantee neither consistency nor availability in the event of network partition.
  - deleted 10y ago
    [deleted]
  - deleted 10y ago
    [deleted]
  - rdtsc 10y ago
    Very good point. Quite often when talking about distributed system, everyone goes to the CAP theorem and automatically assumes if it is not AP, it must CP then. While in reality (and maybe for good reasons) it might be neither.
    - qaq 10y ago
      You sure can but it seems there are not that many practical applications for distributed system that are not tolerant to partitions.
  - qaq 10y ago
    Setup DRBD on your NAS boxes and you have AP :).
guitarbill 10y ago
> Instead of building storage that can fail between nodes, consider building applications that can survive node loss and are not relying on persisting their state in a black box that is assumed to be 100% available. These assumptions about reliablity of systems and their consistent behavior under network partitions has bitten countless people before and will continue to claim victims unless application paradigms are fundamentally rethought. I'd recommend this... except that it's really, really hard [0]. And if you think about it, you have to do this for every new application, which is a bit wasteful. So IMO, doing it at filesystem level is a good trade off. It's abstract enough to be useful over a wide range of use-cases (unlike block storage), because if you think about it file systems are just frameworks. To be fair, block can work really well. I think block storage vendors have themselves to blame. Half the "standards" aren't, interop can be terrible. I know I'd rather have simple, reliable storage - stop stuffing "enterprise" features into a rotting, increasingly technical-debt laden codebase. All back-up solutions for block suck - you'd think this problem would be fixed, especially with cloud storage. I could go on. [0] https://www.somethingsimilar.com/2013/01/14/notes-on-distributed-systems-for-young-bloods/ https://www.somethingsimilar.com/2013/01/14/notes-on-distrib...
Sanddancer 10y ago
This is about a completely different type of VM application, and a completely different kind of storage architecture. This is about everything being on the storage box, not just any application state. The two nodes export NFS shares, which host the root filesystems of the nodes being run on the VM servers. Most likely, the two storage heads will be existing on the same subnet as the VMWare boxes, thus making a network partition that affects both nodes much, much less likely. Additionally, by having the sort of architecture being presented, you can do things like live migrations of VMs, which is useful when the underlying OS needs to be brought down for maintenance. There are trade-offs with any architectural decisions, and your idea provides a much more byzantine, complex, and expensive system for little to no trade-off at the scale presented. There are quite a few places where they just need a few servers, and going to your idea just makes things more expensive and more complicated.
- insaneirish 10y ago
  > This is about a completely different type of VM application, and a completely different kind of storage architecture. I understand it completely, and I vehemently disagree with everything about it. Providing networked block storage for virtual disks on top of NFS is a recipe for disaster and performance degradation. Just because "everyone" does it doesn't make me like it any more. Using redundant local disk (preferably with ZFS) gets around this problem. Blocks are actually blocks and failures and performance issues are confined to one node. > Additionally, by having the sort of architecture being presented, you can do things like live migrations of VMs, which is useful when the underlying OS needs to be brought down for maintenance. I find live migration to be a hack. Once again, solving the wrong problem at the wrong level. > There are quite a few places where they just need a few servers, and going to your idea just makes things more expensive and more complicated. Local disk with appropriate backups and application level failover makes a lot more sense and is fundamentally less complicated.
  - user5994461 10y ago
    I really don't understand that fight. SAN storage and VmWare (ESXi) have been around forever. VmWare will gladly provision VMs over any SAN storage. Live migration has been available for more than 10 years (that's called VMotion and it requires any of paid edition). Prefer local storage? Put any sort of local storage on the server, VmWare will gladly provision VM on it. What's the most important thing in a storage... THE BACKUP! For backups, there are snapshots capabilities well integrated into VmWare (cold snapshots, live snapshots, incremental snapsnots, whatever you name it). All of this has existed for more than ten years. Standard protocols (iSCSI, FiberChannel, other) + robust enclosures (all vendors have SAN) + battle tested software and drivers (VmWare). A couple servers and some TB of storage is a nothing fancy. The cost of the setup is shared between the servers + the labor + many SAS hard drives + the enclosure (a very little part). It's really hard to take seriously an overly complex setup that is trying to imitate that over cheap untested hardware and software (NFS? seriously?). Not only it is a recipe for disaster in production but there isn't even any cost savings to be done. The cost of complexity and labors will be going wild with all the configuration. The hardware costs won't go down (still need many drives and servers). Good luck debugging that and retrieving data when things will go wrong. Well. I suppose it's fun for home labs and that's about it :D

windowsworkstoo 10y ago

Given the target use case of this kind of system, you control what you can control, which is the infra, because the software will overwhelmingly be industry specific COTS stuff that realistically wont be changed. Further its likely that the types of internal apps running could handle a recovery period, makes total loss less of an issue...this is "old school" operations - having DR plans, backups and recovery time targets

qaq 10y ago

There are legitimate reasons to have a CP system and not an AP system, it's obviously application specific. Also over-engineered systems come with their own risks and can experience as much down time as a well setup CP system due to human error stemming from the system's complexity (AWS is a good example).

zzzcpan 10y ago
NASes are not CP systems, they are noCAP systems. They cannot guarantee neither consistency nor availability in the event of network partition.
- deleted 10y ago
  [deleted]
- deleted 10y ago
  [deleted]
- rdtsc 10y ago
  Very good point. Quite often when talking about distributed system, everyone goes to the CAP theorem and automatically assumes if it is not AP, it must CP then. While in reality (and maybe for good reasons) it might be neither.
  - qaq 10y ago
    You sure can but it seems there are not that many practical applications for distributed system that are not tolerant to partitions.
- qaq 10y ago
  Setup DRBD on your NAS boxes and you have AP :).

guitarbill 10y ago

> Instead of building storage that can fail between nodes, consider building applications that can survive node loss and are not relying on persisting their state in a black box that is assumed to be 100% available. These assumptions about reliablity of systems and their consistent behavior under network partitions has bitten countless people before and will continue to claim victims unless application paradigms are fundamentally rethought. I'd recommend this... except that it's really, really hard [0]. And if you think about it, you have to do this for every new application, which is a bit wasteful. So IMO, doing it at filesystem level is a good trade off. It's abstract enough to be useful over a wide range of use-cases (unlike block storage), because if you think about it file systems are just frameworks. To be fair, block can work really well. I think block storage vendors have themselves to blame. Half the "standards" aren't, interop can be terrible. I know I'd rather have simple, reliable storage - stop stuffing "enterprise" features into a rotting, increasingly technical-debt laden codebase. All back-up solutions for block suck - you'd think this problem would be fixed, especially with cloud storage. I could go on. [0] https://www.somethingsimilar.com/2013/01/14/notes-on-distributed-systems-for-young-bloods/ https://www.somethingsimilar.com/2013/01/14/notes-on-distrib...

Sanddancer 10y ago

This is about a completely different type of VM application, and a completely different kind of storage architecture. This is about everything being on the storage box, not just any application state. The two nodes export NFS shares, which host the root filesystems of the nodes being run on the VM servers. Most likely, the two storage heads will be existing on the same subnet as the VMWare boxes, thus making a network partition that affects both nodes much, much less likely. Additionally, by having the sort of architecture being presented, you can do things like live migrations of VMs, which is useful when the underlying OS needs to be brought down for maintenance. There are trade-offs with any architectural decisions, and your idea provides a much more byzantine, complex, and expensive system for little to no trade-off at the scale presented. There are quite a few places where they just need a few servers, and going to your idea just makes things more expensive and more complicated.

insaneirish 10y ago
> This is about a completely different type of VM application, and a completely different kind of storage architecture. I understand it completely, and I vehemently disagree with everything about it. Providing networked block storage for virtual disks on top of NFS is a recipe for disaster and performance degradation. Just because "everyone" does it doesn't make me like it any more. Using redundant local disk (preferably with ZFS) gets around this problem. Blocks are actually blocks and failures and performance issues are confined to one node. > Additionally, by having the sort of architecture being presented, you can do things like live migrations of VMs, which is useful when the underlying OS needs to be brought down for maintenance. I find live migration to be a hack. Once again, solving the wrong problem at the wrong level. > There are quite a few places where they just need a few servers, and going to your idea just makes things more expensive and more complicated. Local disk with appropriate backups and application level failover makes a lot more sense and is fundamentally less complicated.
- user5994461 10y ago
  I really don't understand that fight. SAN storage and VmWare (ESXi) have been around forever. VmWare will gladly provision VMs over any SAN storage. Live migration has been available for more than 10 years (that's called VMotion and it requires any of paid edition). Prefer local storage? Put any sort of local storage on the server, VmWare will gladly provision VM on it. What's the most important thing in a storage... THE BACKUP! For backups, there are snapshots capabilities well integrated into VmWare (cold snapshots, live snapshots, incremental snapsnots, whatever you name it). All of this has existed for more than ten years. Standard protocols (iSCSI, FiberChannel, other) + robust enclosures (all vendors have SAN) + battle tested software and drivers (VmWare). A couple servers and some TB of storage is a nothing fancy. The cost of the setup is shared between the servers + the labor + many SAS hard drives + the enclosure (a very little part). It's really hard to take seriously an overly complex setup that is trying to imitate that over cheap untested hardware and software (NFS? seriously?). Not only it is a recipe for disaster in production but there isn't even any cost savings to be done. The cost of complexity and labors will be going wild with all the configuration. The hardware costs won't go down (still need many drives and servers). Good luck debugging that and retrieving data when things will go wrong. Well. I suppose it's fun for home labs and that's about it :D