6 ms·

I can answer this! It has nothing to do with raid controller quality so much as NVMe doesn’t go to a disk controller. It lives on the PCIe bus itself. In IDE,

I can answer this! It has nothing to do with raid controller quality so much as NVMe doesn’t go to a disk controller. It lives on the PCIe bus itself. In IDE, SATA, SAS, you wire up to RAID. In NVMe, you skip the raid controller. So you can still RAID, but it’s either software or some device with a “private” bus like those PCIe cards that hold 4 m2 sticks. Or IDK how others work. I just know for Enterprise right now, you get NVMe -OR- RAID. Sucks because I had some screaming drives picked out super cheap but had to ditch them in favor of slower SAS drives. But... those slower and smaller SAS drives in RAID10 will be faster than the RAID1 NMVe plan I had.

kbenson 7y ago
Seems like there's enough cores available in most systems you'd want an NVMe system that you probably don't lose much doing software RAID (or pooling, in some more complex FS like ZFS). I just found some simplistic software RAID benchmarks on Phoronix[1][2], there may be better ones (I would be interested to see how much CPU the software raid processing takes during some of these tests, even if just to confirm to myself it doesn't take much). 1: https://www.phoronix.com/scan.php?page=article&item=samsung-960-raid&num=1 https://www.phoronix.com/scan.php?page=article&item=samsung-... 2: https://www.phoronix.com/scan.php?page=article&item=linux418-nvme-raid&num=1 https://www.phoronix.com/scan.php?page=article&item=linux418...
- concerned_user 7y ago
  Software raid (implemented in os, not the fakeraid/softraid chip on motherboards) in this case is better because it can use as much memory as it needs and also since it is part of the OS it can receive additional information on what kind of workload OS wants from it, i.e. readahead or not and how much. Hardware raid generally doesn't know about those things and tries to infer them so it can't optimize fully.
  - SlowRobotAhead 7y ago
    Sort of, I’d have no issue with software RAID. VMWare however has exactly ZERO software raid options so on servers you are limited to HyperV it seems.
    - grosswait 7y ago
      Or the myriad of Linux based options
    - ody4242 7y ago
      > so on servers you are limited to HyperV you are limited to HyperV, Xen, KVM or insert your favorite hypervisor except for ESXi
      - SlowRobotAhead 7y ago
        Let me know when they approach VMWare’s market share, texture, third party acceptance, and support. I’ll switch all our servers over right away.
        kbenson 7y ago
        Xen/KVM market share probably dwarfs VMware if you look at total systems. It's what a lot of the large cloud providers use (AWS/GCE at least). I assume Azure uses HyperV, and also that a lot of shops are using it because it's the supported MS product. Our company has VMware and HyperV deployments for just that reason (Office IT has their own MS cluster, systems uses a VMware cluster we admin. Yes, it's silly, but does illustrate a point).
- cm2187 7y ago
  On windows, Windows Storage Space sucks. The RAID 5-like write performance is unusable. And Microsoft disabled the old mirroring alternative on most windows SKUs.
  - concerned_user 7y ago
    Storage spaces are not a RAID though, their purpose is to allow you to have a pool and manage it easier, the same thing in linux is LVM. Storage spaces do offer redundancy but how it works is a mystery I never had the courage to trust production server to it.
    - cm2187 7y ago
      My understanding is that it is an implementation of software RAID that enables pools of drives of multiple sizes, plus some tiering for caching. Which is why they disabled the traditional windows software RAID.
- etherael 7y ago
  Using https://old.reddit.com/r/VFIO/comments/eryb71/best_storage_for_vfiokvmqemu_experiment_results/ https://old.reddit.com/r/VFIO/comments/eryb71/best_storage_f... setup and truecrypt + zfs raid with lz4 compression I never see cores even visibly affected by either compression or encryption, the speed seems to hit the actual underlying hardware speeds pretty reliably, and this machine CPU is not anything to be amazed about in current year (Ryzen 7 2700 stock clocks with the performance governor) Actually I was a bit curious about this so just tried reading some raw images off the zfs pool with no directives against caching or anything to hit the absolute maximum possible read speeds, even with the read speeds getting well above 1GB/sec it still wasn't showing any visible impact on any cores in terms of either truecrypt or zfs. Almost wondering if they're hidden somehow at this stage. If they're not though it seems the performance impact is so low it gets nowhere near restricting throughput compared to uncompressed / unencrypted speeds.
  - kbenson 7y ago
    If it's a Ryzen Pro, it has dedicated encryption/decryption hardware which might be being used. The old Via mini-itx systems had this, which allowed them to speed up AES operation massively, to the point that it was basically free to run an encrypted FS (and in their cases, it definitely had a CPU cost, around 10-20% of their single core IIRC, but that was a decade ago, and I'm not sure I could find the blog entry that person did now).
    - etherael 7y ago
      Brief search on Ryzen pro seems to imply it's some kind of mobile integrated graphics / cpu solution? I don't think the Ryzen 7 2700 is one of those as it's just a plain old am4 desktop board CPU of the ryzen2+ generation, that said, it does have AES hardware encryption support in the instruction set according to https://en.wikichip.org/wiki/amd/ryzen_7/2700#Features https://en.wikichip.org/wiki/amd/ryzen_7/2700#Features so that may well be what's going on with the seeming complete lack of overhead there.

kbenson 7y ago

Seems like there's enough cores available in most systems you'd want an NVMe system that you probably don't lose much doing software RAID (or pooling, in some more complex FS like ZFS). I just found some simplistic software RAID benchmarks on Phoronix[1][2], there may be better ones (I would be interested to see how much CPU the software raid processing takes during some of these tests, even if just to confirm to myself it doesn't take much). 1: https://www.phoronix.com/scan.php?page=article&item=samsung-960-raid&num=1 https://www.phoronix.com/scan.php?page=article&item=samsung-... 2: https://www.phoronix.com/scan.php?page=article&item=linux418-nvme-raid&num=1 https://www.phoronix.com/scan.php?page=article&item=linux418...

concerned_user 7y ago
Software raid (implemented in os, not the fakeraid/softraid chip on motherboards) in this case is better because it can use as much memory as it needs and also since it is part of the OS it can receive additional information on what kind of workload OS wants from it, i.e. readahead or not and how much. Hardware raid generally doesn't know about those things and tries to infer them so it can't optimize fully.
- SlowRobotAhead 7y ago
  Sort of, I’d have no issue with software RAID. VMWare however has exactly ZERO software raid options so on servers you are limited to HyperV it seems.
  - grosswait 7y ago
    Or the myriad of Linux based options
  - ody4242 7y ago
    > so on servers you are limited to HyperV you are limited to HyperV, Xen, KVM or insert your favorite hypervisor except for ESXi
    - SlowRobotAhead 7y ago
      Let me know when they approach VMWare’s market share, texture, third party acceptance, and support. I’ll switch all our servers over right away.
      - kbenson 7y ago
        Xen/KVM market share probably dwarfs VMware if you look at total systems. It's what a lot of the large cloud providers use (AWS/GCE at least). I assume Azure uses HyperV, and also that a lot of shops are using it because it's the supported MS product. Our company has VMware and HyperV deployments for just that reason (Office IT has their own MS cluster, systems uses a VMware cluster we admin. Yes, it's silly, but does illustrate a point).
cm2187 7y ago
On windows, Windows Storage Space sucks. The RAID 5-like write performance is unusable. And Microsoft disabled the old mirroring alternative on most windows SKUs.
- concerned_user 7y ago
  Storage spaces are not a RAID though, their purpose is to allow you to have a pool and manage it easier, the same thing in linux is LVM. Storage spaces do offer redundancy but how it works is a mystery I never had the courage to trust production server to it.
  - cm2187 7y ago
    My understanding is that it is an implementation of software RAID that enables pools of drives of multiple sizes, plus some tiering for caching. Which is why they disabled the traditional windows software RAID.
etherael 7y ago
Using https://old.reddit.com/r/VFIO/comments/eryb71/best_storage_for_vfiokvmqemu_experiment_results/ https://old.reddit.com/r/VFIO/comments/eryb71/best_storage_f... setup and truecrypt + zfs raid with lz4 compression I never see cores even visibly affected by either compression or encryption, the speed seems to hit the actual underlying hardware speeds pretty reliably, and this machine CPU is not anything to be amazed about in current year (Ryzen 7 2700 stock clocks with the performance governor) Actually I was a bit curious about this so just tried reading some raw images off the zfs pool with no directives against caching or anything to hit the absolute maximum possible read speeds, even with the read speeds getting well above 1GB/sec it still wasn't showing any visible impact on any cores in terms of either truecrypt or zfs. Almost wondering if they're hidden somehow at this stage. If they're not though it seems the performance impact is so low it gets nowhere near restricting throughput compared to uncompressed / unencrypted speeds.
- kbenson 7y ago
  If it's a Ryzen Pro, it has dedicated encryption/decryption hardware which might be being used. The old Via mini-itx systems had this, which allowed them to speed up AES operation massively, to the point that it was basically free to run an encrypted FS (and in their cases, it definitely had a CPU cost, around 10-20% of their single core IIRC, but that was a decade ago, and I'm not sure I could find the blog entry that person did now).
  - etherael 7y ago
    Brief search on Ryzen pro seems to imply it's some kind of mobile integrated graphics / cpu solution? I don't think the Ryzen 7 2700 is one of those as it's just a plain old am4 desktop board CPU of the ryzen2+ generation, that said, it does have AES hardware encryption support in the instruction set according to https://en.wikichip.org/wiki/amd/ryzen_7/2700#Features https://en.wikichip.org/wiki/amd/ryzen_7/2700#Features so that may well be what's going on with the seeming complete lack of overhead there.