Thank you for asking this question! The most relevant systems to Shard Manager (SM) are Google Slicer (GS), Azure Service Fabric (https://dl.acm.org/doi/abs/10.

Thank you for asking this question! The most relevant systems to Shard Manager (SM) are Google Slicer (GS), Azure Service Fabric (https://dl.acm.org/doi/abs/10.1145/3190508.3190546 https://dl.acm.org/doi/abs/10.1145/3190508.3190546), and open-source Apache Helix (helix.apache.org). Previously, we had in-person discussions with the developers of these systems based on public information. SM started in 2011. To our knowledge, SM probably predated the other systems, but we were so excited in 2016 when noticing the publication of the GS paper. We learned a lot from that paper about both the similarity and difference, which will be further explained below. With SM’s hundreds of diverse applications running on hundreds of thousands of servers, we speculate that SM might be the largest deployment in the industry according to published numbers. One advantage of GS is that it does not divide the key space into fixed “shards” ahead of time. In other words, the “shards” are transparent to applications. Internally, GS can dynamically merge or split “shards” and move “shards” around to balance the load as needed. This is more flexible than SM’s approach, which divides the key space into fixed shards. SM can move shards around to balance the load, but currently does not merge or split shards. On the other hand, to enable shard merge or split, GS hashes application keys into its internal 64-bit random key space, which is then internally divided into “shards”. This approach means that it is hard to support certain constraints, e.g., preference for key locality, which are important for some of our user-facing global applications. In addition, to our knowledge, SM supports a wider range of load-balancing constraints by leveraging a generic constraint solver, which allows us to more easily introduce new constraints or change the load balancing algorithm. Facebook infrastructure takes a layered but fully integrated approach, which enables SM, as one layer of an integrated stack, to offer more out-of-the-box functionalities. For example, SM is integrated with our cluster management system called Twine via TaskControl; see details in https://engineering.fb.com/data-center-engineering/twine/ https://engineering.fb.com/data-center-engineering/twine/. This allows SM to better manage shard availability in the face of planned and unplanned infra churns, including container rolling upgrades, maintenance operations, and hardware failures. There are certainly more similarities or differences among SM, GS, Azure Service Fabric, and Apache Helix, beyond what we can describe here. We hope to publish a paper in the future to dive more deeply into these topics.