4 ms·

First thanks so much for doing this, I really appreciate it. I only did a quick read of your post apologies if it addresses these comments/questions. I recent

First thanks so much for doing this, I really appreciate it. I only did a quick read of your post apologies if it addresses these comments/questions. I recently built a system for image storage and made some similar but different tradeoffs so I'm very curious about some of the motivation behind your decisions. Definitely would be cool to go into them in a follow on post. I'm curious why you choose to what seems like an identical bucket stored on many nodes. I have recently built a specialized system that resembles facebook's haystack for storing several petabytes of images. I think it is way more simple and efficient to precreate all the possible 'slabs' on each machine, and just place the data in a round robin fashion over all open slabs. You can periodically grow these with system calls to allocate blocks of disk to make it efficient, maybe several 100's of megs at a time when running at scale. That way each server makes its own placement decisions, and no master is necessary to figure out data placement. Especially since you went with a mysql architecture, I think this makes a lot of sense. I guess it does make things more complex on the erasure coding side, since each block would then have to be erasure coded individually. It might be worth exploring though, since you could completely eliminate the requirement of a master. Of course my system is considerably simpler since my smallest atomic unit is a whole file (keyed by sha1 hash + length to deduplicate), like facebook's haystack it was designed primarily to be an image store. I also do full replication rather than erasure coding, and only plan to implement that later. One thing you didn't go into very much that would be very interesting would be the on disk format of the bucket. I used a modified version of folly RecordIO that returns the disk offsets and stored my payload using thrift with the sha1 hash and some flags. This allows for both quick indexing to each block and a nice recovery story. I'd also be interested in what sort of index format you are using, I store mine in both mysql and in a google sparsehash table by record token. Storing the index locally has great benefits if the mysql server goes down, I'd be curious if your system does that as well.

james_cowling 10y ago
Technically the different copies of the buckets aren't "identical" since they may get populated in unserialized ways (they may store data in different orders) but yes they all store the same set of blocks. Note that I just described the replicated data storage in the blog post. Approximately 24 hours later these volumes get packaged up and erasure coded into a much more efficient storage representation, so the initial replicated writes are just there for high durability and efficient writes while the buckets are mutable. I'm not sure I understand exactly what your question is but I think it's asking why we use 1GB buckets allocated all over the place rather than just picking a set of disks and filling them up RAID-style? This is a pretty important point: If we picked X nodes and stored all the same data on them then if we lost one of those nodes we'd only have X-1 other nodes to re-replicate from. This sets up a hotspot in the system and limits how quickly we can re-replicate. This can be a pretty big deal if we lose a machine storing 1PB of data - we need to amortize that reconstruction cost over a large number of NICs and disk spindles (this is covered in more detail in the blog post). The other advantage of having "small" buckets is that they're a manageable unit for combining into larger volumes when erasure coding, or for tracking partial repair operations, or for garbage collection and compaction when data is deleted. Seems like it might be worth going into more detail on erasure coding in a future blog post.
- jhartmann 10y ago
  My main questions was why always store the same blocks together, and you definitely answered that. Interesting approach. It definitely achieves your goal of spreading out the load. I think this can be achieved without making sure you store the blocks always together though, and it might help move towards a less coordinated system. Here is how I solved the problem: You can arrange each instance (you call it cell in your post) so it has different pools of storage. Maybe something that might make sense is each machine at the top of each rack is one pool, all the next machines down is another pool, and so on. All replication is always done across a pool. All the machines in a pool are stored to in a round robin fashion at the client. The end user can choose how many pools they want to replicate over. Abstract each of the machines as a storage unit. Each storage unit actually contains a series of 'slabs' which are largish files (32 gb in size generally), that grow in some large increment and close out when they hit a high watermark. Each write to a machine goes to a different writable slab in a round robin fashion. Each server knows the path and the contents of each slab, and on disk failure all the individual pieces are replicated to another set of open slabs just like the blocks came in fresh (with an exclusion of the slabs that already have a copy). Since all requests are balanced across the pools, and each client has a different round robin list it manages locally the individual pieces are evenly distributed over the other machines in the other pools. I tried really hard though to optimize for a zero master situation, so my clients handle contacting all the different machines and doing caching and retrying. So it sounds like I had different goals than you guys did, so I'm not surprised your have different trade offs.
  - james_cowling 10y ago
    Yep looks like we may have had slightly different goals here. Another one of our design requirements was that we wanted to be able to move data around within the system (repair operations, rebalancing, encoding, compaction etc) without dealing with the metadata overhead of addressing hundreds of billions of blocks. Statically mapping a block to a 1GB bucket allows us to manage placement decisions on the granularity of buckets rather than orders of magnitude more individual files/blocks.
    - 10y ago

james_cowling 10y ago

Technically the different copies of the buckets aren't "identical" since they may get populated in unserialized ways (they may store data in different orders) but yes they all store the same set of blocks. Note that I just described the replicated data storage in the blog post. Approximately 24 hours later these volumes get packaged up and erasure coded into a much more efficient storage representation, so the initial replicated writes are just there for high durability and efficient writes while the buckets are mutable. I'm not sure I understand exactly what your question is but I think it's asking why we use 1GB buckets allocated all over the place rather than just picking a set of disks and filling them up RAID-style? This is a pretty important point: If we picked X nodes and stored all the same data on them then if we lost one of those nodes we'd only have X-1 other nodes to re-replicate from. This sets up a hotspot in the system and limits how quickly we can re-replicate. This can be a pretty big deal if we lose a machine storing 1PB of data - we need to amortize that reconstruction cost over a large number of NICs and disk spindles (this is covered in more detail in the blog post). The other advantage of having "small" buckets is that they're a manageable unit for combining into larger volumes when erasure coding, or for tracking partial repair operations, or for garbage collection and compaction when data is deleted. Seems like it might be worth going into more detail on erasure coding in a future blog post.

jhartmann 10y ago
My main questions was why always store the same blocks together, and you definitely answered that. Interesting approach. It definitely achieves your goal of spreading out the load. I think this can be achieved without making sure you store the blocks always together though, and it might help move towards a less coordinated system. Here is how I solved the problem: You can arrange each instance (you call it cell in your post) so it has different pools of storage. Maybe something that might make sense is each machine at the top of each rack is one pool, all the next machines down is another pool, and so on. All replication is always done across a pool. All the machines in a pool are stored to in a round robin fashion at the client. The end user can choose how many pools they want to replicate over. Abstract each of the machines as a storage unit. Each storage unit actually contains a series of 'slabs' which are largish files (32 gb in size generally), that grow in some large increment and close out when they hit a high watermark. Each write to a machine goes to a different writable slab in a round robin fashion. Each server knows the path and the contents of each slab, and on disk failure all the individual pieces are replicated to another set of open slabs just like the blocks came in fresh (with an exclusion of the slabs that already have a copy). Since all requests are balanced across the pools, and each client has a different round robin list it manages locally the individual pieces are evenly distributed over the other machines in the other pools. I tried really hard though to optimize for a zero master situation, so my clients handle contacting all the different machines and doing caching and retrying. So it sounds like I had different goals than you guys did, so I'm not surprised your have different trade offs.
- james_cowling 10y ago
  Yep looks like we may have had slightly different goals here. Another one of our design requirements was that we wanted to be able to move data around within the system (repair operations, rebalancing, encoding, compaction etc) without dealing with the metadata overhead of addressing hundreds of billions of blocks. Statically mapping a block to a 1GB bucket allows us to manage placement decisions on the granularity of buckets rather than orders of magnitude more individual files/blocks.
  - 10y ago