4 ms·

The core concept that you ship computation to data rather than the other way around is what made Google so impressive when it launched. There are lots of algori

The core concept that you ship computation to data rather than the other way around is what made Google so impressive when it launched. There are lots of algorithms that do well in that model. Back when I was at NetApp I did a design of a system where the "smart storage" essentially labeled blocks with an MD5 hash when you went to store them. That allowed you to rapidly determine if you already had the block stored and could safely toss the one being written[1]. Really fast de-duplication and good storage compression. At Blekko they had taken this concept to the next logical step and built a storage array out of triply replicated blocks (called 'buckets') that were distributed by their hashid. You could then write templated perl code that operated in parallel over hundreds (or thousands) of buckets and giving a composite result. It always surprised me that IBM didn't care about that system when they acquired Blekko, it was pretty cool. If you implemented it in these Samsung drives it would make for a killer data science appliance. That design almost writes itself. Also in the storage space, there was the CMU "Active disk" architecture[2] which was supposed to replace RAID. There was a startup spin-off from this work but I cannot recall its name anymore, sigh. These days it would useful to design a simulator for systems like this and derive a calculus for analyzing their performance with respect to other architectures. Probably a masters thesis and maybe a PhD or two in that work. [1] Yes MD5 hash collisions are a thing but not for identical length documents (aka an 8K block), and yes NetApp got a patent issued for it. [2] https://www.pdl.cmu.edu/PDL-FTP/Active/ActiveDisksBerkeley98.pdf https://www.pdl.cmu.edu/PDL-FTP/Active/ActiveDisksBerkeley98...

2sk21 6y ago
Wasn't this the aim of Hadoop as well? Also, to be fair, IBM wasn't able to do much with any of the companies they acquired in the same timeframe as Blekko. I was working at IBM at that time and witnessed this first hand.
- ChuckMcM 6y ago
  I kind of gathered that IBM had this sort of love/hate relationship with acquisitions. They loved the "new ideas" but many of the "old guard" hated the idea that something from outside of IBM was novel and worth pursuing. I watched first hand as things that had been done by a company that was acquired were buried while the exact same kind of thing done by IBM research was given lots of funding. I understood not having two efforts, but suggested they be blended to get the best of both teams, alas that was shot down. At one time IBM research had a pretty awesome storage group, perhaps they will will have a computational storage fabric offering at some point.
  - 2sk21 6y ago
    I know exactly the IBM Research group you are referring to but won't name names to avoid embarrassment :-) Politics in large organizations can be pretty nasty.
- jeffbee 6y ago
  The way HBase/HDFS does this is perfectly backwards and counterproductive. You don't want to put a program and its data on one local disk, because that one local disk has almost no resources. The later-period thinking at Google is the opposite of shipping the program to the data. Instead, you allocate the data randomly over as many spindles as you can find, then when you read it all back out you very briefly use all of those spindles at once. Really, it's something to behold in action. That's batch processing: mapreduce and flume and whatnot. Search is still very much an exercise in getting the queries out to where the index shards live.
  - Xorlev 6y ago
    Did you mean Hadoop/HDFS? HBase works pretty analogously to Bigtable (though, having run HBase in production at $dayjob-1, I'd take the latter any day).
    - jeffbee 6y ago
      HBase doesn't work analogously to bigtable. HBase intentionally tries to move region files onto the same node as the region server. That is ass-backwards. HBase devs think of this as "good locality" but all you're really getting from it is terrible performance.
  - ithkuil 6y ago
    The secret sauce for that is networking. Once you have a domain with full clos connectivity and very low latencies you can start disaggregating storage from compute again
  - ddorian43 6y ago
    Assuming you have a 1GB file. According to Colossus paper, that gets split into 1MB parts, so 1000. And they probably use erasure-coding for replication. How does it work ? Do they EC each 1000 part separately, or ?
    - jeffbee 6y ago
      According to these slides, each stripe is a separate replication group. Although the figure depicts replicated rather than erasure coded files, I think it's safe to imagine that the idea holds for both. http://www.pdsw.org/pdsw-discs17/slides/PDSW-DISCS-Google-Keynote.pdf http://www.pdsw.org/pdsw-discs17/slides/PDSW-DISCS-Google-Ke...
jpgvm 6y ago
Worth mentioning this model is reversing with Dremel/BigQuery/Pulsar etc looking to disaggregate compute from storage again.
mangamadaiyan 6y ago
> "smart storage" essentially labeled blocks with an MD5 hash when you went to store them Asking out of curiosity - isn't this similar to what Venti (from Plan 9) did? Of course, Venti was content-addressed, and in this case I'm guessing this system sat above WAFL (which is definitely not content-addressed). * http://doc.cat-v.org/plan_9/4th_edition/papers/venti/ http://doc.cat-v.org/plan_9/4th_edition/papers/venti/
- ChuckMcM 6y ago
  Many of the same pieces solving a slightly different problem. In the NetApp case it was pushing the edge of the de-duplication for efficient archival storage problem. EMC had discovered that MD5 hash collisions made deduping on a document level dangerous (you could think you had a document when you didn't). Those collisions came from dissimilar sized documents and indeed you could "attack" MD5 signatures in that way. With a fixed document size, the probabilities went back to the actual MD5 collision probabilities. Those probabilities were acceptably small. On the "fast" part of the archival server, instead of storing 8K blocks you could store 16 byte "block identifiers" (but still using all of the standard WAFL file system layout, it thinks it is using 16 byte blocks. Those could be stored in "fast" storage (think SSD) and the actual data on slow "cold" storage. Your back-end server does do "content addressable" kinds of things in terms of hash->block location services but it is simple, only three operations "does block <x> exist?", "give me block <x>" and "store this block as <x>." For a write mostly fabric attached archival device it had some benefits over the SATA based filers (higher density, lower watts/terabyte, less CPU load on the filer head (spread out to the storage retrieval unit which could have many of the hash->block tranalators) etc. I don't believe NetApp ever built a complete one though. Just "too many degrees off their bow" as an engineer I knew would say.
  - mangamadaiyan 6y ago
    Thank you, this is fascinating to read! This is a piece of NTAP history I was completely unaware of :)
joshu 6y ago
panasas?
- ChuckMcM 6y ago
  Yes, man I could not come up with that.

2sk21 6y ago

Wasn't this the aim of Hadoop as well? Also, to be fair, IBM wasn't able to do much with any of the companies they acquired in the same timeframe as Blekko. I was working at IBM at that time and witnessed this first hand.

ChuckMcM 6y ago
I kind of gathered that IBM had this sort of love/hate relationship with acquisitions. They loved the "new ideas" but many of the "old guard" hated the idea that something from outside of IBM was novel and worth pursuing. I watched first hand as things that had been done by a company that was acquired were buried while the exact same kind of thing done by IBM research was given lots of funding. I understood not having two efforts, but suggested they be blended to get the best of both teams, alas that was shot down. At one time IBM research had a pretty awesome storage group, perhaps they will will have a computational storage fabric offering at some point.
- 2sk21 6y ago
  I know exactly the IBM Research group you are referring to but won't name names to avoid embarrassment :-) Politics in large organizations can be pretty nasty.
jeffbee 6y ago
The way HBase/HDFS does this is perfectly backwards and counterproductive. You don't want to put a program and its data on one local disk, because that one local disk has almost no resources. The later-period thinking at Google is the opposite of shipping the program to the data. Instead, you allocate the data randomly over as many spindles as you can find, then when you read it all back out you very briefly use all of those spindles at once. Really, it's something to behold in action. That's batch processing: mapreduce and flume and whatnot. Search is still very much an exercise in getting the queries out to where the index shards live.
- Xorlev 6y ago
  Did you mean Hadoop/HDFS? HBase works pretty analogously to Bigtable (though, having run HBase in production at $dayjob-1, I'd take the latter any day).
  - jeffbee 6y ago
    HBase doesn't work analogously to bigtable. HBase intentionally tries to move region files onto the same node as the region server. That is ass-backwards. HBase devs think of this as "good locality" but all you're really getting from it is terrible performance.
- ithkuil 6y ago
  The secret sauce for that is networking. Once you have a domain with full clos connectivity and very low latencies you can start disaggregating storage from compute again
- ddorian43 6y ago
  Assuming you have a 1GB file. According to Colossus paper, that gets split into 1MB parts, so 1000. And they probably use erasure-coding for replication. How does it work ? Do they EC each 1000 part separately, or ?
  - jeffbee 6y ago
    According to these slides, each stripe is a separate replication group. Although the figure depicts replicated rather than erasure coded files, I think it's safe to imagine that the idea holds for both. http://www.pdsw.org/pdsw-discs17/slides/PDSW-DISCS-Google-Keynote.pdf http://www.pdsw.org/pdsw-discs17/slides/PDSW-DISCS-Google-Ke...

jpgvm 6y ago

Worth mentioning this model is reversing with Dremel/BigQuery/Pulsar etc looking to disaggregate compute from storage again.

mangamadaiyan 6y ago

> "smart storage" essentially labeled blocks with an MD5 hash when you went to store them Asking out of curiosity - isn't this similar to what Venti (from Plan 9) did? Of course, Venti was content-addressed, and in this case I'm guessing this system sat above WAFL (which is definitely not content-addressed). * http://doc.cat-v.org/plan_9/4th_edition/papers/venti/ http://doc.cat-v.org/plan_9/4th_edition/papers/venti/

ChuckMcM 6y ago
Many of the same pieces solving a slightly different problem. In the NetApp case it was pushing the edge of the de-duplication for efficient archival storage problem. EMC had discovered that MD5 hash collisions made deduping on a document level dangerous (you could think you had a document when you didn't). Those collisions came from dissimilar sized documents and indeed you could "attack" MD5 signatures in that way. With a fixed document size, the probabilities went back to the actual MD5 collision probabilities. Those probabilities were acceptably small. On the "fast" part of the archival server, instead of storing 8K blocks you could store 16 byte "block identifiers" (but still using all of the standard WAFL file system layout, it thinks it is using 16 byte blocks. Those could be stored in "fast" storage (think SSD) and the actual data on slow "cold" storage. Your back-end server does do "content addressable" kinds of things in terms of hash->block location services but it is simple, only three operations "does block <x> exist?", "give me block <x>" and "store this block as <x>." For a write mostly fabric attached archival device it had some benefits over the SATA based filers (higher density, lower watts/terabyte, less CPU load on the filer head (spread out to the storage retrieval unit which could have many of the hash->block tranalators) etc. I don't believe NetApp ever built a complete one though. Just "too many degrees off their bow" as an engineer I knew would say.
- mangamadaiyan 6y ago
  Thank you, this is fascinating to read! This is a piece of NTAP history I was completely unaware of :)

joshu 6y ago

panasas?

ChuckMcM 6y ago
Yes, man I could not come up with that.