6 ms·

The most exciting recent development in SSDs (until 3DXpoint is released), is bypassing the SATA interface, connecting drives straight into the PCIe bus (no mor

The most exciting recent development in SSDs (until 3DXpoint is released), is bypassing the SATA interface, connecting drives straight into the PCIe bus (no more expensive raid controllers). Just a shame hardly any server motherboards come with M.2 slots right now. The 4x speed increase and lower CPU overhead means it is now possible to move RAM only applications (for instance in-memory databases) to SSDs, keeping only the indexes in memory. Yeah, we've been going that way for a while, just seems we've come a long way from expensive Sun e6500's I was working with in just over a decade ago.

wtallis 11y ago
M.2 slots don't make much sense for servers, at least if you're trying to take advantage of the performance benefits possible with a PCIe interface. Current M.2 drives aren't even close to saturating the PCIe 3.0 x4 link but they're severely thermally limited for sustained use and they're restricted in capacity due to lack of PCB area. Server SSDs should stick with the traditional half-height add-in card form factor with nice big heatsinks.
- scurvy 11y ago
  2.5" NVMe is fine and more scalable than add-in cards (more slots) with increased serviceability. The equivalent add-in card is usually much more expensive than the 2.5" version.
- lsc 11y ago
  most of the NVMe backplanes I've seen give full 'enterprise 2.5" drive clearance to the thing, so if they are actually as thick as consumer SSDs, as most current SATA 'enterprise SSD' are, there's plenty of room for a heatsink without expanding the slot. The supermicro chassis (and I've only explored the NVMe backplanes from supermicro) usually put a lot of effort into drawing air through the drives, so assuming you put in blanks and stuff, the airflow should be there, if the SSD are setup to take advantage of it.
  - wtallis 11y ago
    You need to be more precise with your terminology. There is no such thing as a NVMe backplane. NVMe is a software protocol. The backplanes for 2.5" SSDs to which you refer would be PCIe backplanes using the SFF-8639 aka U.2 connector. None of the above is synonymous with the M.2 connector/form factor standard, which is what I was talking about.
    - lsc 11y ago
      edit: okay, I re-read what you said and yes, these won't support M.2 drives, if I understand what's going on here, and it's possible I still don't. (I have yet to buy any non-sata SSD, though I will soon be making experimental purchases.) I was talking about these: http://www.supermicro.com/products/nfo/NVMe.cfm http://www.supermicro.com/products/nfo/NVMe.cfm Note, though, it looks like if you are willing to pay for a U.2 connected drive, you can get 'em with the giant heatsinks you want: http://www.pcper.com/news/Storage/Connector-Formerly-Known-SFF-8639-Now-Called-U2 http://www.pcper.com/news/Storage/Connector-Formerly-Known-S... further edit: http://www.intel.com/content/dam/www/public/us/en/documents/product-specifications/ssd-750-spec.pdf http://www.intel.com/content/dam/www/public/us/en/documents/... available; not super cheap, but I'm not sure you'd want the super cheap consumer grade stuff in a server, anyhow. further edit: but I object to the idea of putting SSDs on PCI-e cards for any but disposable "cloud" type servers (unless they are massively more reliable than any that I've seen, which I don't think is the case here.) just because with a U.2 connected hard drive in a U.2 backplane, I can swap a bad drive like I would swap a bad sata drive; an alert goes off and I head off to the co-lo as soon as convenient and I can swap the drive without disturbing users, whereas with a low-profile PCI-e card, I've pretty much gotta shut down the server, de-rack it, then make the swap, which causes downtime that must be scheduled, even if I have enough redundancy that there isn't any data loss.
      - wtallis 11y ago
        Take a look at how much stricter the temperature and airflow requirements are for Intel's 2.5" U.2 drives compared to their add-in card counterparts. (And note that the U.2 drives are twice as thick as most SATA drives.) M.2 has almost no place in the server market. U.2 does and will for the foreseeable future, but I'm not sure that it can serve the high-performance segment for long. It's not clear whether it will reach the limits on capacity, heat, or link speed first, but all of those limits are clearly much closer than for add-in cards.
- djsumdog 11y ago
  I thought this was the point of SATA Expresses, but the last I read, SATA Express actually holds on par and even falls behind M.2.
tw04 11y ago
How exactly does connecting to the PCIe bus obviate the need for RAID? RAID isn't about connecting drives, it's about not losing data. If you don't need RAID for your application today, you can purchase a non-RAID SAS adapter for a couple hundred bucks, or just use the onboard ports that are sure to be there.
- tehwalrus 11y ago
  > RAID isn't about connecting drives, it's about not losing data. RAID-0 is used as a way to get faster performance from spinning disk drives, as you can return parts of each read request from different (striped) drives. You also get better write performance, as your writes are split across the drives.
  - lsc 11y ago
    uh, this shouldn't be voted down. Now, I don't use RAID for performance, I use raid for reliability, like parent said, and I've never actually been in a position where it would make sense, but people do use raid0 to increase performance. It happens, even if it's not nearly as common as using raid to prevent data loss.
    - toomuchtodo 11y ago
      Back in the day, folks would RAID0 AWS EBS volumes to get the level of performance they needed (this was long before SSD EBS volumes).
      - MichaelGG 11y ago
        This is still (AFAIK) the recommended way to get higher IOPS out of Azure SSDs.
      - bigiain 11y ago
        Now you're making me feel old - when "back in the day" and "AWS" get used in the same sentence... :-)
    - 11y ago
derefr 11y ago
If we can connect SSDs directly to the PCIe bus... why are there no cute little Thunderbolt flash drives that run at full NVMe speed?
- wtallis 11y ago
  Windows has had astoundingly bad Thunderbolt hot-plug support given that ExpressCard and CardBus existed. NVMe support for non-Apple SSDs under OS X can be problematic. Thunderbolt PHYs are expensive. USB 3.0 is fast enough for now, and USB 3.1 exists.
something2324 11y ago
This is a bit of a question from my lack of understanding of disk IO: But would this in practice play well with the CPU prefetcher? If you're crunching sequential data can you expect the data in the L1 cache after the initial stall?
- dragontamer 11y ago
  SSDs are still grossly slower than RAM. The fastest SSD I know of is the Intel 750, which is like ~2.0 GigaBYTES/second (or for what is more typical for storage benchmarks: about 16Gbps or so over PCIe x4). Main DDR3 RAM is something like 32GigaBYTES per second, and L1 cache is even further. What I think the poster was talking about, is moving from disk-based databases to SSD-based databases. SSDs are much faster than hard drives. L1 Cache, L2 Cache, L3 Cache, and main memory are all orders of magnitude faster than even the fastest SSDs today. Thinking about the "CPU prefetcher" when we're talking about SSDs or Hard Drives is almost irrelevant due to the magnitudes of speed difference.

wtallis 11y ago

M.2 slots don't make much sense for servers, at least if you're trying to take advantage of the performance benefits possible with a PCIe interface. Current M.2 drives aren't even close to saturating the PCIe 3.0 x4 link but they're severely thermally limited for sustained use and they're restricted in capacity due to lack of PCB area. Server SSDs should stick with the traditional half-height add-in card form factor with nice big heatsinks.

scurvy 11y ago
2.5" NVMe is fine and more scalable than add-in cards (more slots) with increased serviceability. The equivalent add-in card is usually much more expensive than the 2.5" version.
lsc 11y ago
most of the NVMe backplanes I've seen give full 'enterprise 2.5" drive clearance to the thing, so if they are actually as thick as consumer SSDs, as most current SATA 'enterprise SSD' are, there's plenty of room for a heatsink without expanding the slot. The supermicro chassis (and I've only explored the NVMe backplanes from supermicro) usually put a lot of effort into drawing air through the drives, so assuming you put in blanks and stuff, the airflow should be there, if the SSD are setup to take advantage of it.
- wtallis 11y ago
  You need to be more precise with your terminology. There is no such thing as a NVMe backplane. NVMe is a software protocol. The backplanes for 2.5" SSDs to which you refer would be PCIe backplanes using the SFF-8639 aka U.2 connector. None of the above is synonymous with the M.2 connector/form factor standard, which is what I was talking about.
  - lsc 11y ago
    edit: okay, I re-read what you said and yes, these won't support M.2 drives, if I understand what's going on here, and it's possible I still don't. (I have yet to buy any non-sata SSD, though I will soon be making experimental purchases.) I was talking about these: http://www.supermicro.com/products/nfo/NVMe.cfm http://www.supermicro.com/products/nfo/NVMe.cfm Note, though, it looks like if you are willing to pay for a U.2 connected drive, you can get 'em with the giant heatsinks you want: http://www.pcper.com/news/Storage/Connector-Formerly-Known-SFF-8639-Now-Called-U2 http://www.pcper.com/news/Storage/Connector-Formerly-Known-S... further edit: http://www.intel.com/content/dam/www/public/us/en/documents/product-specifications/ssd-750-spec.pdf http://www.intel.com/content/dam/www/public/us/en/documents/... available; not super cheap, but I'm not sure you'd want the super cheap consumer grade stuff in a server, anyhow. further edit: but I object to the idea of putting SSDs on PCI-e cards for any but disposable "cloud" type servers (unless they are massively more reliable than any that I've seen, which I don't think is the case here.) just because with a U.2 connected hard drive in a U.2 backplane, I can swap a bad drive like I would swap a bad sata drive; an alert goes off and I head off to the co-lo as soon as convenient and I can swap the drive without disturbing users, whereas with a low-profile PCI-e card, I've pretty much gotta shut down the server, de-rack it, then make the swap, which causes downtime that must be scheduled, even if I have enough redundancy that there isn't any data loss.
    - wtallis 11y ago
      Take a look at how much stricter the temperature and airflow requirements are for Intel's 2.5" U.2 drives compared to their add-in card counterparts. (And note that the U.2 drives are twice as thick as most SATA drives.) M.2 has almost no place in the server market. U.2 does and will for the foreseeable future, but I'm not sure that it can serve the high-performance segment for long. It's not clear whether it will reach the limits on capacity, heat, or link speed first, but all of those limits are clearly much closer than for add-in cards.
djsumdog 11y ago
I thought this was the point of SATA Expresses, but the last I read, SATA Express actually holds on par and even falls behind M.2.

tw04 11y ago

How exactly does connecting to the PCIe bus obviate the need for RAID? RAID isn't about connecting drives, it's about not losing data. If you don't need RAID for your application today, you can purchase a non-RAID SAS adapter for a couple hundred bucks, or just use the onboard ports that are sure to be there.

tehwalrus 11y ago
> RAID isn't about connecting drives, it's about not losing data. RAID-0 is used as a way to get faster performance from spinning disk drives, as you can return parts of each read request from different (striped) drives. You also get better write performance, as your writes are split across the drives.
- lsc 11y ago
  uh, this shouldn't be voted down. Now, I don't use RAID for performance, I use raid for reliability, like parent said, and I've never actually been in a position where it would make sense, but people do use raid0 to increase performance. It happens, even if it's not nearly as common as using raid to prevent data loss.
  - toomuchtodo 11y ago
    Back in the day, folks would RAID0 AWS EBS volumes to get the level of performance they needed (this was long before SSD EBS volumes).
    - MichaelGG 11y ago
      This is still (AFAIK) the recommended way to get higher IOPS out of Azure SSDs.
    - bigiain 11y ago
      Now you're making me feel old - when "back in the day" and "AWS" get used in the same sentence... :-)
  - 11y ago

derefr 11y ago

If we can connect SSDs directly to the PCIe bus... why are there no cute little Thunderbolt flash drives that run at full NVMe speed?

wtallis 11y ago
Windows has had astoundingly bad Thunderbolt hot-plug support given that ExpressCard and CardBus existed. NVMe support for non-Apple SSDs under OS X can be problematic. Thunderbolt PHYs are expensive. USB 3.0 is fast enough for now, and USB 3.1 exists.

something2324 11y ago

This is a bit of a question from my lack of understanding of disk IO: But would this in practice play well with the CPU prefetcher? If you're crunching sequential data can you expect the data in the L1 cache after the initial stall?

dragontamer 11y ago
SSDs are still grossly slower than RAM. The fastest SSD I know of is the Intel 750, which is like ~2.0 GigaBYTES/second (or for what is more typical for storage benchmarks: about 16Gbps or so over PCIe x4). Main DDR3 RAM is something like 32GigaBYTES per second, and L1 cache is even further. What I think the poster was talking about, is moving from disk-based databases to SSD-based databases. SSDs are much faster than hard drives. L1 Cache, L2 Cache, L3 Cache, and main memory are all orders of magnitude faster than even the fastest SSDs today. Thinking about the "CPU prefetcher" when we're talking about SSDs or Hard Drives is almost irrelevant due to the magnitudes of speed difference.