12 ms·
SandForce SSD controller beats Intel X-25M by compressing data before storing
- wmf 17y agoWhenever compression or dedupe is involved, it matters what data you're writing; unfortunately Anand doesn't specify that so it's hard to tell how realistic these numbers are.
- xtho 17y agoOn the last but one page, there is some discussion of this issue.
- weaksauce 17y agoAnd the conclusion is that it is slower than the original version of the vertex but it is still faster in writes compared to the G2 ~140MB/s vs 109MB/s where the original vertex SSD was able to get 185MB/s. FTA: Presumably the majority of your file writes aren’t going to be compressed files so your performance shouldn’t be gated by this issue, even then I’ve shown that you shouldn’t be any worse off than you would be with Intel’s X25-M.
- seunosewa 17y agoVideo files are compressed, and they are the biggest drive fillers.
- spudlyo 17y agoLast but one = penultimate
- jrockway 17y agoOr "second to last".
- alexgartrell 17y agoTo be fair, there's a good chance I'm missing something here, but 25 gb of data (and not super-redundant low-entropy ascii, a lot of executable machine code) being compressed into 11 gb seems pretty amazing. Have any third parties verified these numbers? Editted to add: Beyond that, it seems like all this tech would make random lookups abhorrently expensive, because a seek(30) doesn't necessarily jump to the (30 * A + B)'th place. But I'm not a file systems/storage guy at all, so I'd love for some education here.
- houseabsolute 17y agoI just compressed the 1.6MB cgo application into 400kB using standard Mac OS X zip. A few other applications I tried compressed similarly well.
- jbellis 17y agostandard zip is far more cpu intensive than anything you could afford to run on something where you (a) have very limited cpu resources and (b) need to add very very little latency.
- houseabsolute 17y agotar+gzip performed similarly well (1.6MB -> 450kB) and is a lot faster than normal zip. GZip was similarly able to compress my Mac's System folder from four gigabytes to two. Anyway, the point stands that executable files are quite compressible. If you'd like me to try it with an algorithm you consider more appropriate, please direct me to the download page.
- wmf 17y agoit seems like all this tech would make random lookups abhorrently expensive SSDs are virtualized (like virtual memory), so there is a mapping table from virtual to physical addresses. This is needed for wear-leveling, so you might as well also use it for thin provisioning (aka trim), dedupe, etc. In the worst case, one read command may require reading several pages of metadata from flash to find the data. Because flash is fundamentally so fast, an SSD can tolerate a lot of overhead and still be faster than a hard disk.
- dpifke 17y agoUnpaginated version: http://www.anandtech.com/printarticle.aspx?i=3702 http://www.anandtech.com/printarticle.aspx?i=3702
- andrewcooke 17y agoit would be interesting to compare that to a file system with transparent encryption on an intel drive. this is the best way i have found to think about what they are doing (it's a while since i read the article, so apologies if i am just repeating what is said, or get something wrong - iirc, anand hints at the below but doesn't state quite as much): given an arbitrary, unreliable storage medium, you need to store both "raw" data and additional information for error recovery. it seems that until now there were technical / historical reasons that made it optimal / normal to store these separately, as i have described above (ie a disk stores a chunk of data and then has a relatively small checksum afterwards). but there's no real reason why that need be optimal in all cases. for example, raid does something different. raid 1 stores two copies (ignore that each has error correction too, for the sake of argument). now raid has certain technical motivations (cheap disks fail, hopefully independently) that make that reasonable. so what is new about ssds compared to spinning disks that is the enabling factor here? one guess is that since you can read from various chips at the same time you can do something like raid. for example, say you have 8 memory chips, then you could use a raid 5 style approach with one chip as parity, losing 12% of your space. or two chips losing 1/4 of the space. my inference (and what i think anand implies at some point) is that when you sit down and do the maths, there's some number of parity chips (say) that allows you to start using cheaper chips (with higher error rates). but that doesn't entail compression. so either i am missing something, or the compression is optional - perhaps it is being used to hide the fact that they are having to use so much space for error correction? or perhaps it is just a marketing gimmick? edit: or perhaps without compression it's actually too slow to sell? i think this may be it. and if so, putting compression on an intel drive will actually beat this. or perhaps there's something about the approach that means the data have to be compressed anyway?
- wmf 17y agoYou're confusing two separate features. The improved error correction allows using cheaper flash. The compression/dedupe is what improves performance and endurance.
- andrewcooke 17y agoi am? i thought that is what i was saying... what's confusing me is that this is being sold as a "high end" drive. it's not. it's a drive engineered to be as cheap as possible, using lower quality components, that's being sold as "fast" only because it has compression. put compression (eg with zfs - i don't know what other file systems have transparent compression) on an intel drive and it should be better. but the review hints that the two are linked. if they are, then perhaps it's not as simple as that.