3 ms·

> Disks are cheap, storage is not. I really like how you worded this, because I'm running into the same issue with my current fleet of servers. They all run ku

> Disks are cheap, storage is not. I really like how you worded this, because I'm running into the same issue with my current fleet of servers. They all run kubernetes, have 100gb interconnects, are stuffed with NVMe drives and spinning rust (28tb per node, 2x2Tb nvme, and 4x6tb spinning drives) and, I've had the hardest time actually turning all those disks into a reliable, performant and easy to manage storage system.

PaywallBuster 5y ago
Was doing that just recently, dedicated server stuffed with spinning rust. ZFS to the rescue > zpool create tank raidz2 sda sdc sdd sde sdf sdg sdh sdi sdj sdk sdl sdm sdn sdo > zpool list NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT tank 127T 1.30M 127T - - 0% 0% 1.00x ONLINE - > df -h | grep tank tank 102T 256K 102T 1% /tank PS: not an expert, just my 2 cents. do your homework
- edude03 5y ago
  Oh yes, I’ma big zfs fan as well. To be clear though I need to be able to survive a node failure in addition to a disk failure
washadjeffmad 5y ago
Any specific challenges? I'm running something similar with a little more storage density.
linsomniac 5y ago
Have you tried Ceph? Seems purpose built for just that sort of setup (metadata on NVMe, data on spinning). I've been watching it for years, but only recently set up a test, and it worked well even through a power failure at home. My previous company had been using it after I left, and they seemed happy with it, but I haven't asked too much about it. I'd like to start doing some testing with it at work.
- edude03 5y ago
  I am currently running ceph actually! Out of the box, only having NVME in my storage pool, with 100GB interconnects, I'm getting about 50Mbit/s in Fio write tests. I'm not a storage expert, or a ceph expert clearly, but so far of the options I've tried (Portworx, Rancher) it's the worst performing, and I don't know why.

PaywallBuster 5y ago

Was doing that just recently, dedicated server stuffed with spinning rust. ZFS to the rescue > zpool create tank raidz2 sda sdc sdd sde sdf sdg sdh sdi sdj sdk sdl sdm sdn sdo > zpool list NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT tank 127T 1.30M 127T - - 0% 0% 1.00x ONLINE - > df -h | grep tank tank 102T 256K 102T 1% /tank PS: not an expert, just my 2 cents. do your homework

edude03 5y ago
Oh yes, I’ma big zfs fan as well. To be clear though I need to be able to survive a node failure in addition to a disk failure

washadjeffmad 5y ago

Any specific challenges? I'm running something similar with a little more storage density.

linsomniac 5y ago

Have you tried Ceph? Seems purpose built for just that sort of setup (metadata on NVMe, data on spinning). I've been watching it for years, but only recently set up a test, and it worked well even through a power failure at home. My previous company had been using it after I left, and they seemed happy with it, but I haven't asked too much about it. I'd like to start doing some testing with it at work.

edude03 5y ago
I am currently running ceph actually! Out of the box, only having NVME in my storage pool, with 100GB interconnects, I'm getting about 50Mbit/s in Fio write tests. I'm not a storage expert, or a ceph expert clearly, but so far of the options I've tried (Portworx, Rancher) it's the worst performing, and I don't know why.