7 ms·

How does Chuck imagine the increasing popularity (read: affordability) of solid state drives will affect this paradigm? Very interesting post btw. Thanks.

ChuckMcM 14y ago
I had a partner at a VC ask me that too :-) The question you have to ask is "What is Flash?" which sounds really dumb when you ask it but let me explain. So in the storage business everything used to be done on the highest performance, most reliable drives. Typically those were 15K RPM Fibre Channel drives which cost 8x the price of an equivalent SATA drive [1]. In 2003 NetApp created the 'NearStore' line which was filers that used SATA drives. The idea was that tape was way to slow, you could put stuff on SATA drives and for low duty cycles get to it a lot faster, SATA drives were 'nearline' storage which was defined to be "about 99.8% available." This was hugely better than tape but not as good as a "real" disk array. Of course what people discovered was that you could have some pretty huge data sets be reasonably accessible that way, the first folks were the oil and gas folks [2] but lots of people jumped onto the bandwagon, it was cheaper than a storage array, faster than tape. Now lets look at flash. Most people think of flash as a disk replacement because it connects to the computer through a disk interface (even though it doesn't have disk mechanicals). So it seems like a really fast disk. My assertion though is that it isn't a fast disk, its slow memory. It is 'near line' memory. Which is to say it takes a lot longer to access something in flash than it does in memory, but its craploads faster than getting off of spinning rust. Four years ago, I told a guy who claimed to be Intel's key flash architect that if Intel would put flash on the same side of the MMU as the processor we would completely change the way servers are built. Why? Can you imagine that you've got 200GB of address space that is already ready to go when you turn on the processor? Once you start loading page tables you read in the ones in non-volatile flash and blam all your data structures are ready to go, right now. You want to make some sort of logical computation based on the state of a 50GB data structure, and all you do is follow the pointers? The 'driver' for flash attached as memory is var = *((var type *)(0xsome64bitaddress); But we cannot do that because flash doesn't live on the front side bus, it lives, at best, in the PCI Express address space on the other side of the Southbridge. [3]. Bottom line Solid State drives are doomed, you can't make them much denser the physics doesn't work, you don't want to waste your time going through the disk driver and I/O subsystem to get to what is essentially more memory. But as flash moves closer to the CPU its impact will become pretty impressive. [1] This lead people like IBM to propose 'bricks' which were 8 SATA drives in a RAID or Mirrored config pretending to be a single reliable drive. [2] One of my favorite Dave Hitz quotes: "Oil and Gas companies have an awesome compression algorithm, they can take a 600TB data set and compress it to one bit, 'oil' or 'no oil.'" [3] Yes the new Sandy Bridge DDIO architecture helps with this. (http://www.realworldtech.com/sandy-bridge-ep/2/ http://www.realworldtech.com/sandy-bridge-ep/2/)
- spitfire 14y ago
  You get some extra benefits too: You get rid of the split between memory and filesystem address space. In fact, you can kill off the traditional filesystem altogether. I'm excited by that, filesystems are a dirty kludge. But this would require completely changing how modern operating systems work. No doubt applications developers would love it though.
  - ithkuil 14y ago
    do you mean filesystems as the hierarchical way to organize stuff or the filesystem implementation that ensures that data remains persisted also in the event of unflushed writes in case of crash/poweroff etc ?
    - spitfire 14y ago
      Both, but much more the second. disks and filesystems as a weird, separate appendage hanging off the side of a computer are conceptually a weak idea. Once you've unified your address space - stable storage is in the same space as unstable storage, then your entire world changes. The trick then becomes, how do you represent objects (Hint: Others have already tread this ground, see AS/400).
      - ithkuil 14y ago
        Although many of the problems (i.e. seek time) of rotating disks are overcome by solid state persistent memory, you still need special datastructures to ensure that: a) data is buffered during write and that a power failure won't leave the storage in an inconsistent state b) blocks to be overwritten are reallocated and erased in background (you cannot simply overwrite a flash block before erasing it) c) the used datastructures (trees etc) are optimized for the size of the flash block d) gracefully handle block corruption Since it should be possible to remap blocks (either because of (b) or corruption (d)) it's common to have a logical block remapping layer somewhere in the (disk) controller. At this point classical filesystem algorithms and datastructure can easily solve the remaining problems (consistency with journaling and block optimized access like on rotating disks). Of course, there is another approach for solving (b): see http://en.wikipedia.org/wiki/Log-structured_file_system http://en.wikipedia.org/wiki/Log-structured_file_system (actually, log structured filesystem technique can be used to implement the block remapping within the ssd compatibility layer which lies inside the disk controller itself, but that's a detail) or copy on write filesystems like zfs/btrfs etc (although AFAIK they rely on a disk interface which is compatible to a rotating disk). That's to say that solid state storage is not really just persistent ram where you can apply the same datastructures and procedures you would use in core. Future technologies might be different and trim down these differences so that you could really have huge amount of super fast word addressable random memory access, where nothing can be lost even in case of power failure. But, even assuming that this might be possible, it's possible that later there will be yet another technology with even better storage density/ cost but which will introduce back some of the complications. So, it's understandable why you want to keep some abstractions, so that you can easily replace the underlying technologies and get most of the benefits.

ChuckMcM 14y ago

I had a partner at a VC ask me that too :-) The question you have to ask is "What is Flash?" which sounds really dumb when you ask it but let me explain. So in the storage business everything used to be done on the highest performance, most reliable drives. Typically those were 15K RPM Fibre Channel drives which cost 8x the price of an equivalent SATA drive [1]. In 2003 NetApp created the 'NearStore' line which was filers that used SATA drives. The idea was that tape was way to slow, you could put stuff on SATA drives and for low duty cycles get to it a lot faster, SATA drives were 'nearline' storage which was defined to be "about 99.8% available." This was hugely better than tape but not as good as a "real" disk array. Of course what people discovered was that you could have some pretty huge data sets be reasonably accessible that way, the first folks were the oil and gas folks [2] but lots of people jumped onto the bandwagon, it was cheaper than a storage array, faster than tape. Now lets look at flash. Most people think of flash as a disk replacement because it connects to the computer through a disk interface (even though it doesn't have disk mechanicals). So it seems like a really fast disk. My assertion though is that it isn't a fast disk, its slow memory. It is 'near line' memory. Which is to say it takes a lot longer to access something in flash than it does in memory, but its craploads faster than getting off of spinning rust. Four years ago, I told a guy who claimed to be Intel's key flash architect that if Intel would put flash on the same side of the MMU as the processor we would completely change the way servers are built. Why? Can you imagine that you've got 200GB of address space that is already ready to go when you turn on the processor? Once you start loading page tables you read in the ones in non-volatile flash and blam all your data structures are ready to go, right now. You want to make some sort of logical computation based on the state of a 50GB data structure, and all you do is follow the pointers? The 'driver' for flash attached as memory is var = *((var type *)(0xsome64bitaddress); But we cannot do that because flash doesn't live on the front side bus, it lives, at best, in the PCI Express address space on the other side of the Southbridge. [3]. Bottom line Solid State drives are doomed, you can't make them much denser the physics doesn't work, you don't want to waste your time going through the disk driver and I/O subsystem to get to what is essentially more memory. But as flash moves closer to the CPU its impact will become pretty impressive. [1] This lead people like IBM to propose 'bricks' which were 8 SATA drives in a RAID or Mirrored config pretending to be a single reliable drive. [2] One of my favorite Dave Hitz quotes: "Oil and Gas companies have an awesome compression algorithm, they can take a 600TB data set and compress it to one bit, 'oil' or 'no oil.'" [3] Yes the new Sandy Bridge DDIO architecture helps with this. (http://www.realworldtech.com/sandy-bridge-ep/2/ http://www.realworldtech.com/sandy-bridge-ep/2/)

spitfire 14y ago
You get some extra benefits too: You get rid of the split between memory and filesystem address space. In fact, you can kill off the traditional filesystem altogether. I'm excited by that, filesystems are a dirty kludge. But this would require completely changing how modern operating systems work. No doubt applications developers would love it though.
- ithkuil 14y ago
  do you mean filesystems as the hierarchical way to organize stuff or the filesystem implementation that ensures that data remains persisted also in the event of unflushed writes in case of crash/poweroff etc ?
  - spitfire 14y ago
    Both, but much more the second. disks and filesystems as a weird, separate appendage hanging off the side of a computer are conceptually a weak idea. Once you've unified your address space - stable storage is in the same space as unstable storage, then your entire world changes. The trick then becomes, how do you represent objects (Hint: Others have already tread this ground, see AS/400).
    - ithkuil 14y ago
      Although many of the problems (i.e. seek time) of rotating disks are overcome by solid state persistent memory, you still need special datastructures to ensure that: a) data is buffered during write and that a power failure won't leave the storage in an inconsistent state b) blocks to be overwritten are reallocated and erased in background (you cannot simply overwrite a flash block before erasing it) c) the used datastructures (trees etc) are optimized for the size of the flash block d) gracefully handle block corruption Since it should be possible to remap blocks (either because of (b) or corruption (d)) it's common to have a logical block remapping layer somewhere in the (disk) controller. At this point classical filesystem algorithms and datastructure can easily solve the remaining problems (consistency with journaling and block optimized access like on rotating disks). Of course, there is another approach for solving (b): see http://en.wikipedia.org/wiki/Log-structured_file_system http://en.wikipedia.org/wiki/Log-structured_file_system (actually, log structured filesystem technique can be used to implement the block remapping within the ssd compatibility layer which lies inside the disk controller itself, but that's a detail) or copy on write filesystems like zfs/btrfs etc (although AFAIK they rely on a disk interface which is compatible to a rotating disk). That's to say that solid state storage is not really just persistent ram where you can apply the same datastructures and procedures you would use in core. Future technologies might be different and trim down these differences so that you could really have huge amount of super fast word addressable random memory access, where nothing can be lost even in case of power failure. But, even assuming that this might be possible, it's possible that later there will be yet another technology with even better storage density/ cost but which will introduce back some of the complications. So, it's understandable why you want to keep some abstractions, so that you can easily replace the underlying technologies and get most of the benefits.