Just want to say for anyone running a clusters of machinese (with kubernetes for example) -- OpenEBS provides one of the most approachable and easily to use sol

Just want to say for anyone running a clusters of machinese (with kubernetes for example) -- OpenEBS provides one of the most approachable and easily to use solutions for storage out there. It scales from hobby grade to enterprise grade in my opinion, and although I can't point you to a 1000+ node cluster I run or anything (maybe OpenEBS can), they offer enterprise editions and support which is usually enough to qualify as the minimum viable definition of "enterprise grade". OpenEBS has many different drivers[0] (which is a good sign IMO) I only use one of them but am planning a post (that may never actually get written) to expand on all of them if/when I can: - Jiva[1] (based on Longhorn[1], iSCSI-based, bring your own disk driver) - cStor[2] (baked by OpenEBS, zfs-based used to be the golden child before MayaStor) - MayaStor (written in rust, iSCSI/NVMe-oF based, bring your own lower layer) - zfs-localpv[3] Funnily enough, I've been really interested in zfs and btrfs lately so have been learning a lot about the field, but cStor didn't work when I first ran it. I know "written in rust" has become somewhat of a buzzword, but don't forget that it has actual meaning -- it means that they will be able to get the raw performance possible without a language runtime or garbage collection to hold them back. This is critical for "systems" programming, and is the domain rust was built for. When I see rust, I basically think -- this could be as fast as average C (and safer) if you get even just pretty good developers. Also a note on this -- one of the things that's nice about OpenEBS's offerings is that writes are synchronous. This is obviously a problem[4] for performance of course, but it's a massive relief for building resiliency into systems. I'm not a classical sysadmin, but it puts me at ease that I can give up some throughput to know that if one of my servers goes completely kaput, the other can pick up at the last write on the other node, with zero guesswork on my end. The switchover is of course a little hairy but this is what OpenEBS smoothes out for you. For this not to be completely an ad for OpenEBS, I do want to note that the other prominent option that scales from hobbyist to enterprise (and arguably more so) is Rook[5] which manages Rook clusters. Rook the organization seems to be branching out to position themselves as Cassandra storage and a bunch of other stuff, but the key thing they brought to the table was a mostly hands-off operator for operating Ceph[6] which is certified enterprise grade(tm) FREE software. Ceph is basically the linux of the storage appliance world. Ceph (via Rook) is another really good choice for getting storage up and running on a kubernetes (or whatever else) because it offers similar benefits to OpenEBS's solutions but does it slightly different. Instead of container-attached-storage, it solves the features come at the filesystem layer-- you get a tunable "ceph" installation (you should definitely choose BlueStore btw) which gives you: - replication (like the iSCSI approaches of OpenEBS) - checksumming (like ZFS and zfs-backed OpenEBS)[7] - striping (like ZFS/btrfs and zfs-backed OpenEBS depending on setup) - compression (like ZFS)[7] Anyway, I don't run any servers that use NVMe but nice to see this article, hope the people at OpenEBS are still doing well, love their entry into the space. [EDIT] - I totally forgot and should mention, you can obviously always bring your own lower layer, ex creating a xfs-formatted ZVOL on top of ZFS on the host and then exposing that drive to whatever. Excellent example is a recent talk from Japan which I've mentioned before[8][9] which uses this to run Ceph on ZFS. [0]: https://docs.openebs.io/docs/next/casengines.html https://docs.openebs.io/docs/next/casengines.html [1]: https://docs.openebs.io/docs/next/jiva.html https://docs.openebs.io/docs/next/jiva.html [2]: https://docs.openebs.io/docs/next/alphafeatures.html#mayastor https://docs.openebs.io/docs/next/alphafeatures.html#mayasto... [3]: https://github.com/openebs/zfs-localpv/ https://github.com/openebs/zfs-localpv/ [4]: https://github.com/longhorn/longhorn/issues/1242#issuecomment-682066185 https://github.com/longhorn/longhorn/issues/1242#issuecommen... [5]: https://rook.io/docs/rook/ https://rook.io/docs/rook/ [6]: https://docs.ceph.com/en/latest/architecture/ https://docs.ceph.com/en/latest/architecture/ [7]: https://ceph.io/community/new-luminous-bluestore/ https://ceph.io/community/new-luminous-bluestore/ [8]: https://speakerdeck.com/takutakahashi/ceph-on-zfs https://speakerdeck.com/takutakahashi/ceph-on-zfs [9]: https://news.ycombinator.com/item?id=24784596 https://news.ycombinator.com/item?id=24784596