14 ms·
Bringing SSD Performance to the DIMM form factor
- zitterbewegung 12y agoIs the advantage the low latency because the rest of the specifications seem to be pretty standard for an SSD. Does this require a BIOS patch of some sort?
- brink 12y agoHere's something similar that we've had in the past. http://en.wikipedia.org/wiki/RAM_drive http://en.wikipedia.org/wiki/RAM_drive A bios patch is probably not necessary.
- rplnt 12y agoSo if you don't install drivers it would act as a regular RAM? Thus ruining your performance.. I don't think it will work out of the box on regular systems. Also, isn't pcie better for this?
- k__ 12y agoMaybe it helps if you have much data to process and don't need to move it from disk to RAM anymore. Everything can be done in RAM.
- imanaccount247 12y agoExcept that it is 20,000 times slower than RAM.
- anonova 12y agoBut still faster than swapping to disk. You would also benefit from the increased throughput.
- imanaccount247 12y agoThat doesn't make any sense. If it just showed up as RAM your system would be so slow it would be unusable. There is no benefit to that. Using this requires a supported motherboard specifically because you do not want it to be used as RAM.
- gnufied 12y agoI suppose the main usage will be a RAMDISK mounted partition which will not lose its data on Server reboot, power loss or when program is not running. I don't think this SSD can be accessed through regular SCSI, IDE, AHCI controllers. http://www.sandisk.com.br/enterprise/ulltradimm-ssd/ http://www.sandisk.com.br/enterprise/ulltradimm-ssd/
- diydsp 12y agoYes it would be very cool to have the entire operating system in SSD. Literally flick the computer on and it's running ;) Or flick rapidly between Linux and Windows. That makes me think it might be wise to have a manager to do this. E.g. leave the disks accessible between three+ operating systems.
- icebraining 12y agoYes it would be very cool to have the entire operating system in SSD. Literally flick the computer on and it's running ;) Isn't that what you already get with suspension to memory (S3)? At least on my Debian laptop, it's ready before the lid is fully up.
- sliverstorm 12y agoRight, except: - S3 uses battery power (albeit not much) - NVM provides protection from power faults in datacenters
- sfilipov 12y agoYes, the advantage of the SSD it that it "scales IO performance in a linear fashion while keeping latency constant". [0] [0] http://youtu.be/dGfpdciV9B4?t=1m32s http://youtu.be/dGfpdciV9B4?t=1m32s
- saticmotion 12y agoIf it behaves like RAM, from the POV of the OS, I'd imagine the advantage to be the large size. There's also the fact that it won't lose its data on power loss. Of course, if it behaves like a regular SSD, the read speeds still exceed SATA3.0 speed.
- tfb 12y ago> There's also the fact that it won't lose its data on power loss. Does this mean that for software to take full advantage of this, the software will need to be updated to account for it? For instance, Redis loads data into memory from disk upon starting, but if something like this 400GB SSD is available as memory (and say for instance, 300GB is in use by Redis), wouldn't it make sense for Redis, upon starting, to just "remember" the state of the memory rather than reloading it from disk?
- icebraining 12y agoNot really; we already have suspension (to RAM and to disk), this can work just like it. For the userspace software, it'll be like it never stopped running.
- ars 12y agoNo, that's not correct. This is MUCH MUCH slower than RAM. You would not use this to replace RAM, but rather have a second section of memory for persistent data.
- kalleboo 12y agoIn the hypothetical world where this was as fast as real RAM, it would be cool if the OS was smart enough to read mmap'd data onto it and deal with remembering "this file was mmap'd here and hasn't changed since, next time it's mmap'd again we can just reuse it", and instantly you have support for all the software that just uses mmap (comedy MongoDB option?)
- mrb 12y agoThe RAM slot helps for latency and throughput. SATA 3.0 would restricts this drive's read throughout from 880MB/s to less than 600MB/s, and would increase the write latency (5usec) by at least 1 or 2usec because of the typical CPU->southbridge->PCI express->SATA link it would go through, but here it is CPU->RAM slot. 1-2usec might sound negligible but it's not: it's 20-40% worse latency, so 20-40% fewer write IOPS. SATA 3.2 (2000MB/s) would help for throughput but not for latency. It is also still quite uncommon even amongst latest generation servers, whereas this SSD is compatible with many current and older generation servers.
- NamTaf 12y agoI can't wait to see the programming paradigm of memory vs permanent storage begin to blur in the next 5 or so years. It's going to make some major assumptions about how you program stuff change quite significantly and it's really exciting.
- film42 12y agoI agree! I wonder what will happen when memory and IO are essentially the same speed. Will we still need complex cache stores?
- Xylakant 12y agoI doubt that will happen ever, there's always a tradeoff between cost, speed and space (on the die, ...), otherwise we'd all be running machines with 1TB L1 Cache.
- chongli 12y agoJust because we've had to make tradeoffs so far does not mean these tradeoffs will always exist. Tradeoffs like these are not a natural law.
- pjc50 12y agoYes they are, because the storage isn't arbitrarily small. There is always the latency of distance to overcome. That includes the centimeters to your main memory. There's always going to be a capacity-latency tradeoff.
- petercooper 12y agoI appreciate this might not be the right way to look at it, but from a trivia POV, in terms of raw performance, what era of regular memory would be comparable with it? (i.e. "typical desktop memory in 2004", say.)
- throwaway_yy2Di 12y agoThe read latency is about as slow as the earliest RAM ever used -- tubes of liquid mercury used as acoustic delay lines. (150 us vs. 222 us (microseconds)) https://en.wikipedia.org/wiki/Delay_line_memory#Mercury_delay_lines https://en.wikipedia.org/wiki/Delay_line_memory#Mercury_dela...
- unwind 12y agoAccording to this table: http://en.wikipedia.org/wiki/CAS_latency http://en.wikipedia.org/wiki/CAS_latency its read latency of 150 µs is at least 15,000 times worse than 100 MHz "PC100" SDRAM, which is early 2000s-era.
- zrail 12y agoI don't quite understand. Does this act like a stick of memory or an SSD? Is it just using the memory controller as a super fast parallel bus? If it does act like normal memory, what happens at reboot?
- toyg 12y ago> Does this act like a stick of memory or an SSD? Like an SSD. > Is it just using the memory controller as a super fast parallel bus? Afaik, it's this.
- jpgvm 12y agoTechnically it acts like memory. The dev kit comes with a device driver that makes it look like a block device though, which would be analogous to something like NVMe.
- solarexplorer 12y agoIIRC a single DDR3 slot can address max 8GBytes. So the driver will have to map the 400GB Flash into this window.
- Dylan16807 12y agoDDR3 supports 8gbit chips and you can shove 16 onto one DIMM, so 16GB. But that leaves 3 address lines unused for column select. Theoretically they could make a 64GB or 128GB board. Also LRDIMMs can go up to at least 64GB. DDR4 directly supports 512GB, and also throws away three potential column address bits.
- jhallenworld 12y agoWell at the interface level it has to look like memory since it's attached to a memory bus. But that does not mean it looks like a single large DIMM. There is no way in the DRAM protocol to delay a transaction while data is read from the flash chips into the buffer, so you need a protocol on top of the simple memory model. "nvDIMMs" on the other hand do look like and are as fast as plain memory. Disadvantage is that they are not big.
- mohap 12y agoWhat's the catch here?
- peter303 12y agoIt still costs 10x disk, even though both are fantastically cheap compared to decade ago. Flash still has limited write cycles. Its for data you probably wont write more than once an hour, but not virtual memory.
- anonymousDan 12y agoIs that 10x cost per byte?
- JohnBooty 12y ago> Flash still has limited write cycles. Its for data you probably wont write more than once an hour, but not virtual memory. You can write data to SSDs a lot more than once an hour, even virtual memory. The drive's firmware (assuming it does its job) will do wear-leveling and spread the logical writes across different physical locations on the drive. If you write 20GB of data per day to a modern 480GB SSD you can expect 25 years of service. (Realistically, something else will probably fail before then) Example - http://www.anandtech.com/show/8520/sandisk-ultra-ii-240gb-ssd-review http://www.anandtech.com/show/8520/sandisk-ultra-ii-240gb-ss...
- Andys 12y agoultra high bandwidth, but you can only fit a few in a typical server
- unwind 12y agoCool! And weird. I wonder what the OS support looks like. Also available in a non-Brazilian page at http://www.sandisk.com/enterprise/ulltradimm-ssd/ http://www.sandisk.com/enterprise/ulltradimm-ssd/, of course. :)
- jpgvm 12y agoThis technology is actually developed by a company called Diablo Technologies. They seem to have licensed it to Samsung.
- ctz 12y ago'reliability rate of one unrecoverable error in 1017 (sic) bits read' So about 4 unrecoverable errors per sector. Seems legit. Do people not read their own copy?!
- ccozan 12y agoit's 10^17. The article was probably copied and the <<sup>> got lost.
- rlpb 12y agoWhat OS support exists for this, and how does it work?
- jpgvm 12y agoYou need not just OS support but specific motherboard too. Currently AFAIK this is only supported on SuperMicro dev kits.
- marcosdumay 12y agoWhat's the point of making it DDR3 DIMM compatible if they require a special motherboard to plug in?
- jpgvm 12y agoThe hope is that a standard will be created soon to support this behaviour. These sorts of things are complicated because it requires support of the chipset vendors. i.e Intel and AMD.
- bluedino 12y agoHow much real-world improvement is there using a DIMM slot instead of PCI-E bus?
- marcosdumay 12y agoStandard or not, making the sockets equal just add to the confusion. You best plug it into the right socket or your system won't even work. Using a different socket is best in all cases.
- Artemis2 12y agoWouldn't mounting the RAM stick as a ramdisk be enough though?
- NietTim 12y ago
- deleted 12y ago[deleted]
- fpp 12y agoMore info on the UlltraDimms at: http://www.anandtech.com/show/8396/fms-2014-sandisk-ulltradimm-to-ship-in-supermicro-servers http://www.anandtech.com/show/8396/fms-2014-sandisk-ulltradi... (developed together with Diablo, some benchmark links). and a video interview and demo with a SanDisk manager at a computer fair - http://www.youtube.com/watch?v=jarsTLGXx9c http://www.youtube.com/watch?v=jarsTLGXx9c (currently only available for OEM, requires special bios setup)
- bitL 12y agoWhat would be the advantage of this comparing to PCIe/M.2 PCIe SSDs? Is it all about reduced latency, i.e. very small reads/writes would benefit?
- valarauca1 12y ago>The benefit of using a DDR3 interface instead of SATA/SAS or PCIe is lower latency because the SSDs sit closer to the CPU. The memory interface has also been designed with parallelism in mind and can thus take greater advantage of multiple drives without sacrificing performance or latency. SanDisk claims write latency of less then five microseconds, which is lower than what even PCIe SSDs offer (e.g. Intel SSD DC P3700 is rated at 20µs). Source: http://www.anandtech.com/show/8396/fms-2014-sandisk-ulltradimm-to-ship-in-supermicro-servers http://www.anandtech.com/show/8396/fms-2014-sandisk-ulltradi... TL;DR about 4x faster then PCIe SSD's
- justincormack 12y agoAlthough it is apparently implemented via Sata on the module so there is scope to speed it up more.
- digikata 12y agoBut raw flash writes on the order of 1-2 milliseconds, so the 5 microseconds must be the latency to write to a ram cache inside the stick. I wonder what the latency would be for a write larger than the buffer. For that matter, how large is the buffer?
- valarauca1 12y agoIf you read the source you will see that no independent benchmarks have been done yet.
- digikata 12y agoThe article did update with a benchmark link comparing a couple of models to a fusion io card. http://www.percona.com/blog/2014/08/12/benchmarking-exflash-with-sysbench-fileio/ http://www.percona.com/blog/2014/08/12/benchmarking-exflash-... So the 95%, write latency is 1-1.5 msec for one model, and 2.5-3.5 msec for another model. Maybe representing SLC/MLC flash? 95% read latencies are maybe 350 or 250 usec.
- jgrodziski 12y agoI look forward to the point where 3/4 of business application code will go the trash when we'll have a persistant memory with the latency of actual RAM disk (Memristor !!). Exit all that "copy from RAM to Disk/Network" code. It will definitely change the way we code and look at code. That's why nowadays I think a good interface to your entities, like the Repository pattern, is a must have.
- pjc50 12y agoI work on an application that was originally designed to persist transactions to battery backed SRAM. It's still full of "business logic", because that's where all the value is. Copying data around really doesn't consume that much time and effort; performing logical, semantic or administrative transformation on it does.
- itry 12y agoIt will definitely change the way we code Only on a low level, right? I have not been thinking about ram and disk in ages. I calculate with variables, I store stuff in a DB or in a file, I access the net via http. Nothing would change if ram increases I think.
- tokenizerrr 12y agoWell, if you have plentiful persistent RAM an application could easily "suspend" itself (on shutdown for example), and then resume running at a later point in time. The code would not have to think about this, since the entire state is just snapshotted. It's basically what machines do now when they "hibernate", except more effecient and could even happen if the power got cut. Additionally I suppose your variables could also be persistent automatically. Though the biggest change I see is that software would never really need to completely close anymore when RAM == Disk. For RAM usage it wouldn't really matter if they were running or not, unless they delete data upon shutdown. To save CPU or other IO usage the application could simply be suspended.
- realharo 12y ago
- snake_plissken 12y agoThis is pretty cool, especially since the data is persistent. You could put an entire data warehouse onto one or of a couple of these things, update it once a week and get amazing response time. But I thought the main drawback of SSDs was that eventually the individual memory cells will degrade and lose the ability to write new data. I don't see anything about endurance other than a MTBF of 2.5 million hours and the disk-writes-per-day (DWPD), which I had to look up. I have no idea if these are good or bad. I feel like these things will be great if you didn't write new data a lot, or you have deep pockets to replace them when you need to.
- Filligree 12y ago2.5 Mhours MTBF equates to 285 years. That's a long time. On any kind of large-scale installation you need redundancy, but if those numbers pan out then you could expect any single SSD to outlast your organisation. Take backups anyway.
- tw04 12y agoAt what duty cycle? The MTBF rating means literally nothing without that detail.
- _dark_matter_ 12y agoI just want to point out that is not how MTBF works. MTBF (and MTTF) are used in a group of components, and cannot be used to estimate lifetime of a single component. This does not say that, on average, each component will last 285 years. This says that, given a group of components (if you bought 2.5M of them, for example) there will be a failure every 2.5 Mhours of operating time (so for 2.5M components, every hour one will fail).
- gr3yh47 12y agoFTA: "Reduces total processing time, compared to hard-drive drives (HDDs)." i loled.
- consum3dbyfire 12y agoI couldn't believe that either... it's kind of sad.
- lowlevel 12y agoMind blown.
- userbinator 12y agoThe capacities are very odd - you'd expect something in a DIMM format to have a power-of-2 size. I think the market for this could be much bigger if it behaved like a regular RAM DIMM, only slower and nonvolatile; it somewhat reminds me of old machines that used magnetic core RAM. This could be useful for laptops, like a zero-power suspend-to-(NV)RAM. The only thing that is worrying is the endurance of the flash - especially if it's being treated almost like RAM in this application.
- 0x0 12y agoMaybe they leave off a certain % to handle "bad blocks" and do remapping?
- jdcarter 12y agoOver-provisioning is normal in enterprise SSD. They use extra capacity for re-packing blocks (i.e. garbage-collecting partially erased blocks) and to account for blocks wearing out from use.
- vidarh 12y agoIt's still way too slow to be treated as RAM. The performance they quote is equivalent to a mid to lower range PCIe SSD. For comparison, the stated read performance is 800MB/sec or so. Last time I checked - on ca. 8 year old hardware, we could read files from a RAM disk under Linux at a rate of about 6GB/sec. That's with the added overhead of the benchmark software running and going through a filesystem.
- com2kid 12y agoI wouldn't say that 800MB/s is slow. http://en.wikipedia.org/wiki/PC133 http://en.wikipedia.org/wiki/PC133 I also wonder if that 880 is for just one slot filled up. If it is, then I wonder if you can parallelize to 4 slots for ~3.4GB/s. At that point, I imagine many types of data sets would work A-OK. Though I wonder how that's scaled for cores, given that in the situation I imagined above, you'd be streaming all that data to just one CPU to get that aggregate bandwidth. Hmm. OK what about some other scenarios? Being able to have RAM and Storage on a sliding partition is sort of cool. I wonder if it'd make provisioning VMs any easier. "How much RAM do you want? Set the slider!"
- nine_k 12y agoHow come read latency is 150 µs, while write latency is 5µs, 30 times shorter? Do they mean the latency to start a write operation? IIRC, flash memory is written block by block, with a pretty significant time to write one block.
- joelthelion 12y agoDo OSes already support this? Do they support it as memory, a storage device, or both?
- Fando 12y agoWhat implications does this have for us simple folk?
- mrb 12y agoYou can tell SanDisk's performance numbers do not add up and that they are likely misrepresenting the true performance of their device. (Those red asterisks next to the numbers correspond to a footnote that is conveniently missing from the page...) A "read latency of 150usec" translates to maximum possible read IOPS rate of 1/150e-6 = 6.67K (with one outstanding I/O). But they quote a "random read IOPS of 140K". That would only be possible if their DDR3-based DIMM could process 21 concurrent read I/O operations. But to the best of my knowledge, DDR3 is limited to 8 banks/DIMM, so there could not possibly be more than 8 concurrent read I/O at any one time. Far from 21. So SanDisk is likely quoting a worst case read latency and/or a best case read IOPS. Customers are left to themselves to figure out which of these numbers is most likely to represent the average performance... PS: here is a paper with some more details but that still fails to explain this discrepancy: http://www.snia.org/sites/default/files/SanDisk%20ULLtraDIMM_0.pdf http://www.snia.org/sites/default/files/SanDisk%20ULLtraDIMM...
- nitrogen 12y agoBut to the best of my knowledge, DDR3 is limited to 8 banks/DIMM, so there could not possibly be more than 8 concurrent read I/O at any one time. Maybe you can submit more than 8 queued requests to the controller and have them stream back at the full DDR3 data rate. Maybe it's not actually addressable as RAM, it just uses the DDR3 interface as a communication bus.
- IgorPartola 12y agoThis is sort of the opposite of the RAM-based battery backed drives [1]. I can see this being immediately useful for things like large database servers: instead of re-priming caches on reboots you just have them already warmed up. You can also suddenly have a whole lot more "RAM" at the cost of its speed. I do have a hard time picturing what this will look like if it was as fast as traditional RAM. If I can store everything in RAM, from the OS binaries, to the running processes, it certainly has a kind of elegance to it. Lots of microcontrollers already act this way: all your hardware has an address in a single address space, be at a hardware port, ROM, RAM, NVRAM, etc. However, I wonder how the modern UNIX OS will work with a system like this without block devices at all. I wonder if what it'd do is actually just use RAM disks, and years from now we'll still be doing this the way we do double emulation of the TTY. That'd be kind of sad since it means we can never get away from the concept of old block devices. On the other hand it's damn convenient to be able to just append to a file and not have to worry about reallocating it. This means that perhaps all we'd need is a filesystem that's designed to work over RAM rather than over block devices. [1] http://en.wikipedia.org/wiki/I-RAM http://en.wikipedia.org/wiki/I-RAM
- itchyouch 12y agoWhile block devices can go away, the notion of a file will probably continue to remain. We run clusters of ephemeral linux hosts, and while we use disk, we also have plenty of hosts that run diskless. The disk mount point just becomes a tmpfs mount and the required minimal installation is made on a tmpfs partition which amounts to a trivial amount for hosting busybox and some conf files in /etc. Linux does appear to understand that tmpfs backed filestores are already in memory and do not need to synchronize pages and flush out dirty pages. That whole caching/memory layer for this fake block device is just not used at all.
- IgorPartola 12y agoThat's right, we already have a filesystem backed by RAM: tmpfs. Except now it's no longer so tmp. I suppose then yes we don't have to do much in the way of emulating block devices: just run tmpfs or it's later revisions and know that everything is permanent. In that case, this is not much of a leap for the current OS's to support it, which is great. I wonder if this means that theoretically mmap() can become a one liner now.
- egillie 12y agoSSDs as RAM + projects like Tachyon are the future of big data processing -- current workflows are still way too slow. I wonder how this will affect Spark.
- sixdimensional 12y agoI am not a silver bullet kind of person, but I think this kind of technology plus things like Spark are potentially the holy grail. Ok, let me rephrase - I've often believed that there was a sort of "ultimate architecture" with distributed cluster computing, distributed programming frameworks, high-speed interconnect, and flash based storage. I think we are getting closer and closer each day. I wish I worked in R&D at any of the companies doing this fundamental research now - I have a feeling they have some amazing things cooked up behind closed doors.
- progman 12y ago> SSDs as RAM Not yet. I think first we will have SSD as static RAM plus dynamic DIMM RAM as Cache. I wonder when SSD will compete with DIMM regarding maximum number of write operations.
- imaginenore 12y agoWe have PCIe SSDs approaching 7.2 GB/s. While that's lower than DDR3 speeds, it's not that far off. http://hothardware.com/Reviews/OCZ-CES-2012-Product-Tour-ZDrive-R5-Everest-2-Step-Out/ http://hothardware.com/Reviews/OCZ-CES-2012-Product-Tour-ZDr...
- kcarnold 12y agoFor all of us who are thinking "it's just like RAM, just persistent!", here's some perspectives about what persistent RAM means for OSes and applications: http://lwn.net/Articles/610174/ http://lwn.net/Articles/610174/
- fastest963 12y agoApparently once you hit 3.2TB the IOPs falls to only 200K. Typo? http://cl.ly/image/2J0u182D3229 http://cl.ly/image/2J0u182D3229
- Wildgoose 12y agoI would still say that the real performance bottleneck is ultimately the bandwidth between the CPU and this memory. This suggests that the next stage will be to incorporate heterogenous processors alongside that memory - thus upgrading your computer could then be as simple as plugging in an another combined non-volatile memory/CPU block into a fast inter-connector. Rather reminds me of the the old S100 bus where everything just plugged into the same channel, (which probably dates me quite well).
- robmccoll 12y agoI honestly can't imagine having something in my DIMM slots with this bad of latency and throughput. 150 microsecond reads? Doing a full POST would take forever. What OS and software use - case does this serve?
- vidarh 12y agoYou'd put a filesystem on it and mount it as a "ramdisk", with the result that you can cut out expensive RAID controllers and/or PCIe SSDs, and can potentially reduce form factor. E.g. I can get 1U servers with 32 DDR3 slots and 64 cores, but only space for 3x 3.5" drives, or if you double up (but few chassis will then let you use hot swap caddies), 6x 2.5" drives. You could easily fit 256GB RAM and still have 16 slots free for RAID arrays over those DIMMs.