3 ms·

Open sourcing Dicer: Databricks's auto-sharder

85 points by vivek-jain 9mo ago

vivek-jain 9mo ago
[dead]
ayf 9mo ago
Does anyone else have something similar? What are some use cases that you found are useful?
- vivek-jain 9mo ago
  Sharded in-memory caching turns out to be rather useful at scale :) Some of the key examples highlighted on our blog are Unity Catalog, which is essentially the metadata layer for Databricks, our Query Orchestration Engine, and our distributed remote cache. See the blog post for more!
- louis-paul 9mo ago
  Sounds related to Google Slicer: https://research.google/pubs/slicer-auto-sharding-for-datacenter-applications/ https://research.google/pubs/slicer-auto-sharding-for-datace...
  - atuladya 9mo ago
    It is similar to Slicer in terms of the abstraction (I built Slicer at Google) but the architecture, implementation and algorithms have a lot of differences
    - bigwheels 9mo ago
      Did you also work on this databricks dicery?
      - hiyer 9mo ago
        Yes he did. I attended a talk from him on the same, so that's how I know.
- WookieRushing 9mo ago
  These show up once you have a certain scale where it is either cost inefficient or the hot spots are very dynamic. They also try to avoid latency by being eventually consistent sidecars instead of proxies. I’ve seen them used for traffic routing, storage system metadata systems, distributed cache etc
khaki54 9mo ago
Seems weird to call it sharding since it's not sharding indexed datasets or anything like that. Is this just a tool to mitigate Databricks’ internal service-scaling challenges?
- atuladya 9mo ago
  Right - this is not about sharding data/datasets. This is for sharding in-memory state that a service might have. The problem of building services at low cost, high scale, low latency and high throughput is common in many environments including our services at Databricks, and Dicer helps with that.
deleted 9mo ago
[deleted]
charleshn 9mo ago
> Application pods learn the current assignment through a library called the Slicelet (S for server side). The Slicelet maintains a local cache of the latest assignment by fetching it from the Dicer service and watching for updates. When it receives an updated assignment, the Slicelet notifies the application via a listener API. For a critical control plane component like this, I tend to prefer a constant work pattern [0], to avoid metastable failures [1], e.g. periodically pull the data instead of relying on notifications. [0] https://aws.amazon.com/builders-library/reliability-and-constant-work/ https://aws.amazon.com/builders-library/reliability-and-cons... [1] https://brooker.co.za/blog/2021/05/24/metastable.html https://brooker.co.za/blog/2021/05/24/metastable.html
- jdellithorpe 9mo ago
  The Dicer Slicelet supports such a pattern, you can poll the assignment directly on the Slicelet: https://github.com/databricks/dicer/blob/master/dicer/external/src/Slicelet.scala#L59 https://github.com/databricks/dicer/blob/master/dicer/extern... (btw the notification mechanism itself does not deliver the assignment to the application, only notifies the application that the assignment has changed: https://github.com/databricks/dicer/blob/master/dicer/external/src/SliceletListener.scala#L10-L12 https://github.com/databricks/dicer/blob/master/dicer/extern...)
yomartin 9mo ago
[dead]

85 points by vivek-jain 9mo ago

vivek-jain 9mo ago

[dead]

ayf 9mo ago

Does anyone else have something similar? What are some use cases that you found are useful?

vivek-jain 9mo ago
Sharded in-memory caching turns out to be rather useful at scale :) Some of the key examples highlighted on our blog are Unity Catalog, which is essentially the metadata layer for Databricks, our Query Orchestration Engine, and our distributed remote cache. See the blog post for more!
louis-paul 9mo ago
Sounds related to Google Slicer: https://research.google/pubs/slicer-auto-sharding-for-datacenter-applications/ https://research.google/pubs/slicer-auto-sharding-for-datace...
- atuladya 9mo ago
  It is similar to Slicer in terms of the abstraction (I built Slicer at Google) but the architecture, implementation and algorithms have a lot of differences
  - bigwheels 9mo ago
    Did you also work on this databricks dicery?
    - hiyer 9mo ago
      Yes he did. I attended a talk from him on the same, so that's how I know.
WookieRushing 9mo ago
These show up once you have a certain scale where it is either cost inefficient or the hot spots are very dynamic. They also try to avoid latency by being eventually consistent sidecars instead of proxies. I’ve seen them used for traffic routing, storage system metadata systems, distributed cache etc

khaki54 9mo ago

Seems weird to call it sharding since it's not sharding indexed datasets or anything like that. Is this just a tool to mitigate Databricks’ internal service-scaling challenges?

atuladya 9mo ago
Right - this is not about sharding data/datasets. This is for sharding in-memory state that a service might have. The problem of building services at low cost, high scale, low latency and high throughput is common in many environments including our services at Databricks, and Dicer helps with that.

deleted 9mo ago

[deleted]

charleshn 9mo ago

> Application pods learn the current assignment through a library called the Slicelet (S for server side). The Slicelet maintains a local cache of the latest assignment by fetching it from the Dicer service and watching for updates. When it receives an updated assignment, the Slicelet notifies the application via a listener API. For a critical control plane component like this, I tend to prefer a constant work pattern [0], to avoid metastable failures [1], e.g. periodically pull the data instead of relying on notifications. [0] https://aws.amazon.com/builders-library/reliability-and-constant-work/ https://aws.amazon.com/builders-library/reliability-and-cons... [1] https://brooker.co.za/blog/2021/05/24/metastable.html https://brooker.co.za/blog/2021/05/24/metastable.html

jdellithorpe 9mo ago
The Dicer Slicelet supports such a pattern, you can poll the assignment directly on the Slicelet: https://github.com/databricks/dicer/blob/master/dicer/external/src/Slicelet.scala#L59 https://github.com/databricks/dicer/blob/master/dicer/extern... (btw the notification mechanism itself does not deliver the assignment to the application, only notifies the application that the assignment has changed: https://github.com/databricks/dicer/blob/master/dicer/external/src/SliceletListener.scala#L10-L12 https://github.com/databricks/dicer/blob/master/dicer/extern...)

yomartin 9mo ago

[dead]