6 ms·

Anecdote. I swore off spinning disks for my PC years ago, but I've been a bit cheap and recently got fed up with my 500GB "storage" drive always being near full

by Rapzid 7y ago

Anecdote. I swore off spinning disks for my PC years ago, but I've been a bit cheap and recently got fed up with my 500GB "storage" drive always being near full and my work VM as well :/ So I finally went onto Google to look into storage and was absolutely, postively, shocked that I somehow missed the whole m.2 nvme surge. So much so that I got a 1TB 970 EVO for just $170. Nearly 500k random IOPS with qd32, and 3.5GB/s and 2.5GB/s sequential read/write?! Yes please. I was even able to jump through hoops to get bitlocker working with hardware encryption, so I'm getting full disk encryption at nearly spec speeds. Awesomeness. EDIT: It's a shame the situation with RAID controller software, drivers, and physical controllers is such a crap show ATM. Doing anything "enterprise" with them seems a bit of a cluster F ATM. Anyone know what's up with that?

SlowRobotAhead 7y ago
I can answer this! It has nothing to do with raid controller quality so much as NVMe doesn’t go to a disk controller. It lives on the PCIe bus itself. In IDE, SATA, SAS, you wire up to RAID. In NVMe, you skip the raid controller. So you can still RAID, but it’s either software or some device with a “private” bus like those PCIe cards that hold 4 m2 sticks. Or IDK how others work. I just know for Enterprise right now, you get NVMe -OR- RAID. Sucks because I had some screaming drives picked out super cheap but had to ditch them in favor of slower SAS drives. But... those slower and smaller SAS drives in RAID10 will be faster than the RAID1 NMVe plan I had.
- kbenson 7y ago
  Seems like there's enough cores available in most systems you'd want an NVMe system that you probably don't lose much doing software RAID (or pooling, in some more complex FS like ZFS). I just found some simplistic software RAID benchmarks on Phoronix[1][2], there may be better ones (I would be interested to see how much CPU the software raid processing takes during some of these tests, even if just to confirm to myself it doesn't take much). 1: https://www.phoronix.com/scan.php?page=article&item=samsung-960-raid&num=1 https://www.phoronix.com/scan.php?page=article&item=samsung-... 2: https://www.phoronix.com/scan.php?page=article&item=linux418-nvme-raid&num=1 https://www.phoronix.com/scan.php?page=article&item=linux418...
  - concerned_user 7y ago
    Software raid (implemented in os, not the fakeraid/softraid chip on motherboards) in this case is better because it can use as much memory as it needs and also since it is part of the OS it can receive additional information on what kind of workload OS wants from it, i.e. readahead or not and how much. Hardware raid generally doesn't know about those things and tries to infer them so it can't optimize fully.
    - SlowRobotAhead 7y ago
      Sort of, I’d have no issue with software RAID. VMWare however has exactly ZERO software raid options so on servers you are limited to HyperV it seems.
      - grosswait 7y ago
        Or the myriad of Linux based options
      - ody4242 7y ago
        > so on servers you are limited to HyperV you are limited to HyperV, Xen, KVM or insert your favorite hypervisor except for ESXi
        SlowRobotAhead 7y ago
        Let me know when they approach VMWare’s market share, texture, third party acceptance, and support. I’ll switch all our servers over right away.
        kbenson 7y ago
        Xen/KVM market share probably dwarfs VMware if you look at total systems. It's what a lot of the large cloud providers use (AWS/GCE at least). I assume Azure uses HyperV, and also that a lot of shops are using it because it's the supported MS product. Our company has VMware and HyperV deployments for just that reason (Office IT has their own MS cluster, systems uses a VMware cluster we admin. Yes, it's silly, but does illustrate a point).
wtallis 7y ago
> It's a shame the situation with RAID controller software, drivers, and physical controllers is such a crap show ATM. Doing anything "enterprise" with them seems a bit of a cluster F ATM. Anyone know what's up with that? The guys making storage controller ASICs weren't really prepared to start pushing 3M+ IOPS through their entry-level products. If you want to get your money's worth of performance out of NVMe SSDs, you can't have much of an abstraction layer sitting in front of them, especially not one that translates them into SCSI devices. It also doesn't take many drives before the RAID controller's host interface is the bottleneck, because the RAID card can't go into a slot wider than PCIe x16, and most are just x8. On the other hand, actually using that much storage performance requires a ton of software engineering effort on the application side, and quickly pushes you toward a clustered approach where a traditional RAID controller isn't all that useful. "Software Defined Storage", NVMe over Fabrics, etc. are popular because it's hard to actually keep 24 NVMe SSDs busy with just a 2P 2U server.
gregmac 7y ago
FYI, at this point, hardware-based drive encryption is considered utterly broken and you shouldn't use it: https://news.ycombinator.com/item?id=18382975 https://news.ycombinator.com/item?id=18382975
- Rapzid 7y ago
  I will look into it but the 970 Evo I have supposedly has fixes in for a lot of the stuff from a couple years back.
- londons_explore 7y ago
  Which is a real shame, because 'in the hardware, with no software complexity or overhead' is exactly where encryption should live. I wish some SSD manufacturers would get third party audits of their firmware which can say "We believe data stored on this SSD is not decryptable without an externally provided key". It ought to be pretty trivial to inspect the source to verify all keys which can derive the master key are never written to persistent storage.
  - dmos62 7y ago
    Audit-based security is ok, but the ideal solution is open hardware and firmware.
    - londons_explore 7y ago
      My ranking of security: "It's secure because we say so" "It's secure, and we have a published design and audit" "It's open source" "We're offering a reward to anyone who can break it" "Our company would go under if anyone found a flaw in it" SSD's are currently at the first stage.
      - dmos62 7y ago
        I'd put "it's open-source and simple" somewhere below "it's open source". There are quite a few open source codebases that are very hard to trust due to the sheer complexity.
StavrosK 7y ago
Does NVMe make a difference over SATA2? Isn't the latter plenty fast anyway? I've been holding off on upgrading because it doesn't seem like I'd see a benefit.
- TsiCClawOfLight 7y ago
  We are already past the SATA II bottleneck, so yes.
- mtreis86 7y ago
  Yes, NVMe is on a 4x pcie pipeline right to the CPU, where SATA runs through another chip.
- lizknope 7y ago
  SATA2 is really old. The SATA revisions are: SATA1 - 1.5Gb/s SATA2 - 3.0Gb/s SATA3 - 6.0Gb/s SATA3 is often referred to as SATA6G although because of 8b/10b encoding overhead you a SATA6G link maxes out at 4.8Gb/s There are many NVMe SSDs that use 4 lanes of PCI-E Gen 3 which is just under 32Gb/s If you read this review you can see read speeds of 3500MB/s which is 28Gb/s or just under the 32Gb/s limit. https://www.anandtech.com/show/13761/the-samsung-970-evo-plus-ssd-review https://www.anandtech.com/show/13761/the-samsung-970-evo-plu... I am using lowercase "b" for bit and capital "B" for byte
  - StavrosK 7y ago
    Hmm, maybe my disk is using SATA3 then, I have an Evo and a not-ancient motherboard. However, I'm wondering whether I'll see a practical difference in my day to day workloads, which is mostly web development and no gaming.
    - barrkel 7y ago
      IMO - not really. You might if you're memory constrained and you're doing analysis of logs or something like that.
      - 7y ago
ryanmercer 7y ago
> was absolutely, postively, shocked that I somehow missed the whole m.2 nvme surge. I only stay subscribed to the NewEgg emails to watch retail HDD/SSD prices. The prices always amaze me, I member paying over 100$ for a 32mb usb drive and our first HDD in a home PC was a 200 something megabyte caviar. It makes me wonder what things will be like in another 25 years.
RobRivera 7y ago
Do you have a recommended vendor? Last time i bought anything was prior to Newegg doing its marketplace change permitting 3rd party vendors.
AtlasBarfed 7y ago
Software RAID doesn't work in linux with SSDs? Remember that motherboard RAID isn't RAID if your motherboard dies. You might as well do software RAID. Edit: Oh right, probably TRIM problems.

by Rapzid 7y ago

SlowRobotAhead 7y ago

I can answer this! It has nothing to do with raid controller quality so much as NVMe doesn’t go to a disk controller. It lives on the PCIe bus itself. In IDE, SATA, SAS, you wire up to RAID. In NVMe, you skip the raid controller. So you can still RAID, but it’s either software or some device with a “private” bus like those PCIe cards that hold 4 m2 sticks. Or IDK how others work. I just know for Enterprise right now, you get NVMe -OR- RAID. Sucks because I had some screaming drives picked out super cheap but had to ditch them in favor of slower SAS drives. But... those slower and smaller SAS drives in RAID10 will be faster than the RAID1 NMVe plan I had.

kbenson 7y ago
Seems like there's enough cores available in most systems you'd want an NVMe system that you probably don't lose much doing software RAID (or pooling, in some more complex FS like ZFS). I just found some simplistic software RAID benchmarks on Phoronix[1][2], there may be better ones (I would be interested to see how much CPU the software raid processing takes during some of these tests, even if just to confirm to myself it doesn't take much). 1: https://www.phoronix.com/scan.php?page=article&item=samsung-960-raid&num=1 https://www.phoronix.com/scan.php?page=article&item=samsung-... 2: https://www.phoronix.com/scan.php?page=article&item=linux418-nvme-raid&num=1 https://www.phoronix.com/scan.php?page=article&item=linux418...
- concerned_user 7y ago
  Software raid (implemented in os, not the fakeraid/softraid chip on motherboards) in this case is better because it can use as much memory as it needs and also since it is part of the OS it can receive additional information on what kind of workload OS wants from it, i.e. readahead or not and how much. Hardware raid generally doesn't know about those things and tries to infer them so it can't optimize fully.
  - SlowRobotAhead 7y ago
    Sort of, I’d have no issue with software RAID. VMWare however has exactly ZERO software raid options so on servers you are limited to HyperV it seems.
    - grosswait 7y ago
      Or the myriad of Linux based options
    - ody4242 7y ago
      > so on servers you are limited to HyperV you are limited to HyperV, Xen, KVM or insert your favorite hypervisor except for ESXi
      - SlowRobotAhead 7y ago
        Let me know when they approach VMWare’s market share, texture, third party acceptance, and support. I’ll switch all our servers over right away.
        kbenson 7y ago
        Xen/KVM market share probably dwarfs VMware if you look at total systems. It's what a lot of the large cloud providers use (AWS/GCE at least). I assume Azure uses HyperV, and also that a lot of shops are using it because it's the supported MS product. Our company has VMware and HyperV deployments for just that reason (Office IT has their own MS cluster, systems uses a VMware cluster we admin. Yes, it's silly, but does illustrate a point).

wtallis 7y ago

> It's a shame the situation with RAID controller software, drivers, and physical controllers is such a crap show ATM. Doing anything "enterprise" with them seems a bit of a cluster F ATM. Anyone know what's up with that? The guys making storage controller ASICs weren't really prepared to start pushing 3M+ IOPS through their entry-level products. If you want to get your money's worth of performance out of NVMe SSDs, you can't have much of an abstraction layer sitting in front of them, especially not one that translates them into SCSI devices. It also doesn't take many drives before the RAID controller's host interface is the bottleneck, because the RAID card can't go into a slot wider than PCIe x16, and most are just x8. On the other hand, actually using that much storage performance requires a ton of software engineering effort on the application side, and quickly pushes you toward a clustered approach where a traditional RAID controller isn't all that useful. "Software Defined Storage", NVMe over Fabrics, etc. are popular because it's hard to actually keep 24 NVMe SSDs busy with just a 2P 2U server.

gregmac 7y ago

FYI, at this point, hardware-based drive encryption is considered utterly broken and you shouldn't use it: https://news.ycombinator.com/item?id=18382975 https://news.ycombinator.com/item?id=18382975

Rapzid 7y ago
I will look into it but the 970 Evo I have supposedly has fixes in for a lot of the stuff from a couple years back.
londons_explore 7y ago
Which is a real shame, because 'in the hardware, with no software complexity or overhead' is exactly where encryption should live. I wish some SSD manufacturers would get third party audits of their firmware which can say "We believe data stored on this SSD is not decryptable without an externally provided key". It ought to be pretty trivial to inspect the source to verify all keys which can derive the master key are never written to persistent storage.
- dmos62 7y ago
  Audit-based security is ok, but the ideal solution is open hardware and firmware.
  - londons_explore 7y ago
    My ranking of security: "It's secure because we say so" "It's secure, and we have a published design and audit" "It's open source" "We're offering a reward to anyone who can break it" "Our company would go under if anyone found a flaw in it" SSD's are currently at the first stage.
    - dmos62 7y ago
      I'd put "it's open-source and simple" somewhere below "it's open source". There are quite a few open source codebases that are very hard to trust due to the sheer complexity.

StavrosK 7y ago

Does NVMe make a difference over SATA2? Isn't the latter plenty fast anyway? I've been holding off on upgrading because it doesn't seem like I'd see a benefit.

TsiCClawOfLight 7y ago
We are already past the SATA II bottleneck, so yes.
mtreis86 7y ago
Yes, NVMe is on a 4x pcie pipeline right to the CPU, where SATA runs through another chip.
lizknope 7y ago
SATA2 is really old. The SATA revisions are: SATA1 - 1.5Gb/s SATA2 - 3.0Gb/s SATA3 - 6.0Gb/s SATA3 is often referred to as SATA6G although because of 8b/10b encoding overhead you a SATA6G link maxes out at 4.8Gb/s There are many NVMe SSDs that use 4 lanes of PCI-E Gen 3 which is just under 32Gb/s If you read this review you can see read speeds of 3500MB/s which is 28Gb/s or just under the 32Gb/s limit. https://www.anandtech.com/show/13761/the-samsung-970-evo-plus-ssd-review https://www.anandtech.com/show/13761/the-samsung-970-evo-plu... I am using lowercase "b" for bit and capital "B" for byte
- StavrosK 7y ago
  Hmm, maybe my disk is using SATA3 then, I have an Evo and a not-ancient motherboard. However, I'm wondering whether I'll see a practical difference in my day to day workloads, which is mostly web development and no gaming.
  - barrkel 7y ago
    IMO - not really. You might if you're memory constrained and you're doing analysis of logs or something like that.
    - 7y ago

ryanmercer 7y ago

> was absolutely, postively, shocked that I somehow missed the whole m.2 nvme surge. I only stay subscribed to the NewEgg emails to watch retail HDD/SSD prices. The prices always amaze me, I member paying over 100$ for a 32mb usb drive and our first HDD in a home PC was a 200 something megabyte caviar. It makes me wonder what things will be like in another 25 years.

RobRivera 7y ago

Do you have a recommended vendor? Last time i bought anything was prior to Newegg doing its marketplace change permitting 3rd party vendors.

AtlasBarfed 7y ago

Software RAID doesn't work in linux with SSDs? Remember that motherboard RAID isn't RAID if your motherboard dies. You might as well do software RAID. Edit: Oh right, probably TRIM problems.