3 ms·

Sharding sucks and we really need to get away from it. I wonder if there's any databases like these that use a hierarchy to distribute and manage access of data

Sharding sucks and we really need to get away from it. I wonder if there's any databases like these that use a hierarchy to distribute and manage access of data rather than with a flat topology of arbitrary chunks of data. Currently there's a lot of shitty hacks, like memory-resident databases in front of disk-backed, proxies, etc. We can do better than that if we build hierarchical data and node management into the network.

elvinyung 9y ago
I mean, isn't that still just sharding? What does hierarchy give us? Look at Bigtable [1], for example. Tablets are organized in a three level B-tree-like hierarchy, where data is only stored at the lowest level. Is this like what you're talking about? [1] https://research.google.com/archive/bigtable-osdi06.pdf https://research.google.com/archive/bigtable-osdi06.pdf
- peterwwillis 9y ago
  Not necessarily. It's good to use a hierarchy, but they're still sharding. And the client has to contact all the tablets directly. And the client has to use certain key naming conventions to benefit from the sharding. And there's a seemingly unnecessary amount of round trips. And a lot of other extra complexity, to the point where a bigtable can perform very efficiently, or it can perform many times poorer, depending on many factors. It sounds like a pet rather than cattle. The downside of sharding is operationally it's a pain, and operational pain translates into pain either in your wallet or for your employees or customers, none of which you want. Hierarchy is supposed to give you management, access, and performance gains, but if it's too specific you have to use the data exactly how the database wants you to and then you can't take advantage of general interfaces. It shouldn't get in your way, it should just give you free performance boosts with known guarantees and be as simple as possible while also surviving scaling. I'm not saying this is easy, or even feasible, but it's where we should be headed if we want dev, ops, and the people in suits to be happy.
  - elvinyung 9y ago
    > The downside of sharding is operationally it's a pain I am really curious about this premise. Isn't it only a pain if there's no direct system support? (i.e. you have to manually rebalance shards, split up queries at the application layer, etc.) Isn't this the whole thing that NewSQL systems are trying to solve? Moreover, I don't quite understand what benefits a hierarchical structure provides that a flat (or mostly-flat) partitioning scheme doesn't, ceteris paribus in terms of system support. Especially: doesn't a hierarchical structure mean it takes O(log n) round trips to resolve a lookup rather than just O(1) as in a flat partitioning?
vikiomega9 9y ago
I've always thought about sharding with a caveat of some notion of an implicit index either via timestamps or composite keys.

elvinyung 9y ago

I mean, isn't that still just sharding? What does hierarchy give us? Look at Bigtable [1], for example. Tablets are organized in a three level B-tree-like hierarchy, where data is only stored at the lowest level. Is this like what you're talking about? [1] https://research.google.com/archive/bigtable-osdi06.pdf https://research.google.com/archive/bigtable-osdi06.pdf

peterwwillis 9y ago
Not necessarily. It's good to use a hierarchy, but they're still sharding. And the client has to contact all the tablets directly. And the client has to use certain key naming conventions to benefit from the sharding. And there's a seemingly unnecessary amount of round trips. And a lot of other extra complexity, to the point where a bigtable can perform very efficiently, or it can perform many times poorer, depending on many factors. It sounds like a pet rather than cattle. The downside of sharding is operationally it's a pain, and operational pain translates into pain either in your wallet or for your employees or customers, none of which you want. Hierarchy is supposed to give you management, access, and performance gains, but if it's too specific you have to use the data exactly how the database wants you to and then you can't take advantage of general interfaces. It shouldn't get in your way, it should just give you free performance boosts with known guarantees and be as simple as possible while also surviving scaling. I'm not saying this is easy, or even feasible, but it's where we should be headed if we want dev, ops, and the people in suits to be happy.
- elvinyung 9y ago
  > The downside of sharding is operationally it's a pain I am really curious about this premise. Isn't it only a pain if there's no direct system support? (i.e. you have to manually rebalance shards, split up queries at the application layer, etc.) Isn't this the whole thing that NewSQL systems are trying to solve? Moreover, I don't quite understand what benefits a hierarchical structure provides that a flat (or mostly-flat) partitioning scheme doesn't, ceteris paribus in terms of system support. Especially: doesn't a hierarchical structure mean it takes O(log n) round trips to resolve a lookup rather than just O(1) as in a flat partitioning?

vikiomega9 9y ago

I've always thought about sharding with a caveat of some notion of an implicit index either via timestamps or composite keys.