4 ms·

Databases = Frameworks for Distributed Systems

91 points by laplab 4y ago

gslin 4y ago
I would say the author just tried to redefined "distributed", and ignored those real distributed systems, e.g. Paxos/Raft-based ones.
- quietbritishjim 4y ago
  You are thinking of fully decentralised systems. A system where there is a centralised leader is still a distributed system so long as at least some of the processing takes place on other nodes.
- codelorado 4y ago
  As per Wikipedia, a distributed system is: "a system whose components are located on different networked computers, which communicate and coordinate their actions by passing messages to one another from any system." You're describing consensus algorithms.
ianbutler 4y ago
I was watching a video by one of Amazon’s distinguished engineers a while ago and for the life of me I can’t find it now but the thing that stuck with me from it is, “There are only three modes for a distributed system - it implements paxos/raft itself, it relies on a data store that implements paxos/raft or it’s wrong.” Obviously a bit of an over simplification since there have been some advancements in that area but not far off. This article introduces an approach that falls under the second mode and I think it’s a good way to build a distributed system without having to re invent the wheel especially if you can just rely on the data store for the consensus and make your brokers and nodes relatively dumb. Look at k8s and etcd for a very widely used example of this approach as well. I’m also currently building a distributed search engine using etcd for the service registry, broker peer announcements, and worker queue and it’s been a good experience so far.
- hardwaresofton 4y ago
  Well in some situations you can also get away with CRDTs or Gossip (SWIM), but yeah in the general sense Paxos (Raft is a Paxos-family algo). There are lots of paxos family algos to choose from[0]. > I’m also currently building a distributed search engine using etcd for the service registry, broker peer announcements, and worker queue and it’s been a good experience so far. As a random stranger on the internet, please build an abstraction layer around etcd. Even if there's only one implementation, I've found that so many distributed projects that just never reach the scale that etcd is built for would benefit from the option of writing their own drivers (you don't need a full plugin system just a regular abstraction layer). One example is Kine[0] for k8s -- if k8s had a built in option for writing/reading from something like Postgres from the beginning it would have been a better project for it, IMO. If you do build a plugin system though, you can pass off the work of maintaining the other implementations! [0]: https://vadosware.io/post/paxosmon-gotta-concensus-them-all https://vadosware.io/post/paxosmon-gotta-concensus-them-all [1]: https://github.com/k3s-io/kine https://github.com/k3s-io/kine
  - ianbutler 4y ago
    I think that's a great idea, I already have the beginnings of a plugin system since local non distributed is a mode you can run in and I had to abstract etcd away to do that. Will add it to the list!
    - hardwaresofton 4y ago
      Thanks! Also I’m pretty curious where I can find your project…
      - ianbutler 4y ago
        Hey at the moment it’s not public but I’m getting to the point where it will be. Happy to ping you when it is, also planning to do a show HN post here as well with it.
        4y ago
scott_s 4y ago
Some of academia’s preeminent database researchers have made a similar hypothesis: DBOS: A DBMS-oriented Operating System, https://vldb.org/pvldb/vol15/p21-skiadopoulos.pdf https://vldb.org/pvldb/vol15/p21-skiadopoulos.pdf and https://dbos-project.github.io/ https://dbos-project.github.io/
janandonly 4y ago
This article was originally posted on dev.to, but it turns out that HackerNews banned this website, so I decided to create my own. HN "blocks" dev.to ? Why ?
- spijdar 4y ago
  Dang answered that here: https://news.ycombinator.com/item?id=27110093 https://news.ycombinator.com/item?id=27110093
  - janandonly 4y ago
    Interesting. It never occurred to me that some domains may stuff a lot of spam/adverts in between good articles. Sadly, it does happen.
jandrewrogers 4y ago
While databases having been converging on composability for a long time, the model presented here is not the correct one and causes problems for many important and emerging use cases. As an elementary observation, isolating the designs of storage, compute, and networking as separate modules is a reliable way to achieve poor performance and resource efficiency. Some of the most important scalability and performance optimizations in modern database architecture are schedule-driven, which requires tight coupling across the design of storage, compute, and networking. Ignoring this class of optimizations produces very large reductions in real-world workload performance, and feeds a narrative that the software is wasteful and not environmentally friendly. Heterogeneous and specialized resource collections with complex topologies are becoming more common for a diverse set of reasons. This breaks most horizontal resource abstractions. The practice of treating every thread as a complete monolithic database kernel connected together with protocols that are good at equilibrium finding lends itself to this environment very well, and the edge (for which database technology is broadly broken), in addition to being ideal for the boring data center case where the topology is highly regular and every node has uniform resources. Most of our distributed systems are modeled as giant distributed file systems, and that fits okay with the idea of horizontally partitioning compute, storage, and network. This is easy for developers to reason about because they are familiar with file systems. However, it is difficult to express important database-y operations in this model. For example, if you want to do ad hoc joins in a scale-out environment, you need mechanics like decentralized parallel orchestration which isn't really a concept in a "distributed file system" model of "distributed". Databases have been slowly converging on latency-hiding HPC software architectures (which has no implication of HPC hardware), which look very different than a distributed file system, can express more powerful capabilities, and are generally more scalable. To the extent there is a general principle and framework behind databases, it is more this than traditional distributed systems.
erikerikson 4y ago
This post is on to an important dynamic. We've scaled databases then broken them up and our applications are distributed databases now. The tools mentioned seem a bit random but... There does seem to be space for database tooling that melds into our applications but everyone wants to bring you into their bubble instead.
_vvhw 4y ago
We've been designing TigerBeetle [1], a distributed financial accounting database, from the beginning with the ultimate goal that the whole architecture of: * global consensus layer (Viewstamped Replication) [2], and the * local storage engine (a high-performance storage fault-tolerant LSM-tree over io_uring) [3], * which can all be deterministically simulated and fuzzed, can eventually be used as a distributed library that gives you not only consensus but storage, all in one, kind of like “a DistSys Iron Man suit”—you put in your own state machine and business logic [4] inside it, and get all the replication and fault-tolerance around that, out of the box. TigerBeetle is based on some of the latest research around storage faults, how to recover safely and maximize high availability, and we've been heavily inspired by the deterministic simulation testing of FoundationDB [5]. [1] https://github.com/coilhq/tigerbeetle https://github.com/coilhq/tigerbeetle [2] https://github.com/coilhq/tigerbeetle/blob/main/src/vsr/replica.zig#L624-L647 https://github.com/coilhq/tigerbeetle/blob/main/src/vsr/repl... [3] https://www.hytradboi.com/2022/tigerbeetles-lsm-forest https://www.hytradboi.com/2022/tigerbeetles-lsm-forest [4] https://github.com/coilhq/tigerbeetle/blob/main/src/state_machine.zig#L277-L303 https://github.com/coilhq/tigerbeetle/blob/main/src/state_ma... [5] https://www.youtube.com/watch?v=OJb8A6h9jQQ https://www.youtube.com/watch?v=OJb8A6h9jQQ — “How I Learned to Stop Worrying and Trust the Database”
buildbot 4y ago
At My first programming job, as an intern we went with his approach, using rethinkDB amd python “microservices” that were all separate python processes reading from their own rethinkDB change feed lambda - the system was essentially built as a distributed reactive microservice pipeline, with rethinkDB as the source of truth for every other process - which existed on 100-1000 machines depending!
brewkode 4y ago
A few years back, a colleague and me built a distributed system toolkit in an effort to understand the blocks/pieces of a distributed system and also to make it easier to build one for our needs. This article seems to hit on the same set of notes that we had in mind when we started out building suuchi. https://github.com/ashwanthkumar/suuchi https://github.com/ashwanthkumar/suuchi.

91 points by laplab 4y ago

I would say the author just tried to redefined "distributed", and ignored those real distributed systems, e.g. Paxos/Raft-based ones.

quietbritishjim 4y ago
You are thinking of fully decentralised systems. A system where there is a centralised leader is still a distributed system so long as at least some of the processing takes place on other nodes.
codelorado 4y ago
As per Wikipedia, a distributed system is: "a system whose components are located on different networked computers, which communicate and coordinate their actions by passing messages to one another from any system." You're describing consensus algorithms.

ianbutler 4y ago

I was watching a video by one of Amazon’s distinguished engineers a while ago and for the life of me I can’t find it now but the thing that stuck with me from it is, “There are only three modes for a distributed system - it implements paxos/raft itself, it relies on a data store that implements paxos/raft or it’s wrong.” Obviously a bit of an over simplification since there have been some advancements in that area but not far off. This article introduces an approach that falls under the second mode and I think it’s a good way to build a distributed system without having to re invent the wheel especially if you can just rely on the data store for the consensus and make your brokers and nodes relatively dumb. Look at k8s and etcd for a very widely used example of this approach as well. I’m also currently building a distributed search engine using etcd for the service registry, broker peer announcements, and worker queue and it’s been a good experience so far.

hardwaresofton 4y ago
Well in some situations you can also get away with CRDTs or Gossip (SWIM), but yeah in the general sense Paxos (Raft is a Paxos-family algo). There are lots of paxos family algos to choose from[0]. > I’m also currently building a distributed search engine using etcd for the service registry, broker peer announcements, and worker queue and it’s been a good experience so far. As a random stranger on the internet, please build an abstraction layer around etcd. Even if there's only one implementation, I've found that so many distributed projects that just never reach the scale that etcd is built for would benefit from the option of writing their own drivers (you don't need a full plugin system just a regular abstraction layer). One example is Kine[0] for k8s -- if k8s had a built in option for writing/reading from something like Postgres from the beginning it would have been a better project for it, IMO. If you do build a plugin system though, you can pass off the work of maintaining the other implementations! [0]: https://vadosware.io/post/paxosmon-gotta-concensus-them-all https://vadosware.io/post/paxosmon-gotta-concensus-them-all [1]: https://github.com/k3s-io/kine https://github.com/k3s-io/kine
- ianbutler 4y ago
  I think that's a great idea, I already have the beginnings of a plugin system since local non distributed is a mode you can run in and I had to abstract etcd away to do that. Will add it to the list!
  - hardwaresofton 4y ago
    Thanks! Also I’m pretty curious where I can find your project…
    - ianbutler 4y ago
      Hey at the moment it’s not public but I’m getting to the point where it will be. Happy to ping you when it is, also planning to do a show HN post here as well with it.
      - 4y ago

scott_s 4y ago

Some of academia’s preeminent database researchers have made a similar hypothesis: DBOS: A DBMS-oriented Operating System, https://vldb.org/pvldb/vol15/p21-skiadopoulos.pdf https://vldb.org/pvldb/vol15/p21-skiadopoulos.pdf and https://dbos-project.github.io/ https://dbos-project.github.io/

janandonly 4y ago

This article was originally posted on dev.to, but it turns out that HackerNews banned this website, so I decided to create my own. HN "blocks" dev.to ? Why ?

spijdar 4y ago
Dang answered that here: https://news.ycombinator.com/item?id=27110093 https://news.ycombinator.com/item?id=27110093
- janandonly 4y ago
  Interesting. It never occurred to me that some domains may stuff a lot of spam/adverts in between good articles. Sadly, it does happen.

jandrewrogers 4y ago

While databases having been converging on composability for a long time, the model presented here is not the correct one and causes problems for many important and emerging use cases. As an elementary observation, isolating the designs of storage, compute, and networking as separate modules is a reliable way to achieve poor performance and resource efficiency. Some of the most important scalability and performance optimizations in modern database architecture are schedule-driven, which requires tight coupling across the design of storage, compute, and networking. Ignoring this class of optimizations produces very large reductions in real-world workload performance, and feeds a narrative that the software is wasteful and not environmentally friendly. Heterogeneous and specialized resource collections with complex topologies are becoming more common for a diverse set of reasons. This breaks most horizontal resource abstractions. The practice of treating every thread as a complete monolithic database kernel connected together with protocols that are good at equilibrium finding lends itself to this environment very well, and the edge (for which database technology is broadly broken), in addition to being ideal for the boring data center case where the topology is highly regular and every node has uniform resources. Most of our distributed systems are modeled as giant distributed file systems, and that fits okay with the idea of horizontally partitioning compute, storage, and network. This is easy for developers to reason about because they are familiar with file systems. However, it is difficult to express important database-y operations in this model. For example, if you want to do ad hoc joins in a scale-out environment, you need mechanics like decentralized parallel orchestration which isn't really a concept in a "distributed file system" model of "distributed". Databases have been slowly converging on latency-hiding HPC software architectures (which has no implication of HPC hardware), which look very different than a distributed file system, can express more powerful capabilities, and are generally more scalable. To the extent there is a general principle and framework behind databases, it is more this than traditional distributed systems.

erikerikson 4y ago

This post is on to an important dynamic. We've scaled databases then broken them up and our applications are distributed databases now. The tools mentioned seem a bit random but... There does seem to be space for database tooling that melds into our applications but everyone wants to bring you into their bubble instead.

_vvhw 4y ago

We've been designing TigerBeetle [1], a distributed financial accounting database, from the beginning with the ultimate goal that the whole architecture of: * global consensus layer (Viewstamped Replication) [2], and the * local storage engine (a high-performance storage fault-tolerant LSM-tree over io_uring) [3], * which can all be deterministically simulated and fuzzed, can eventually be used as a distributed library that gives you not only consensus but storage, all in one, kind of like “a DistSys Iron Man suit”—you put in your own state machine and business logic [4] inside it, and get all the replication and fault-tolerance around that, out of the box. TigerBeetle is based on some of the latest research around storage faults, how to recover safely and maximize high availability, and we've been heavily inspired by the deterministic simulation testing of FoundationDB [5]. [1] https://github.com/coilhq/tigerbeetle https://github.com/coilhq/tigerbeetle [2] https://github.com/coilhq/tigerbeetle/blob/main/src/vsr/replica.zig#L624-L647 https://github.com/coilhq/tigerbeetle/blob/main/src/vsr/repl... [3] https://www.hytradboi.com/2022/tigerbeetles-lsm-forest https://www.hytradboi.com/2022/tigerbeetles-lsm-forest [4] https://github.com/coilhq/tigerbeetle/blob/main/src/state_machine.zig#L277-L303 https://github.com/coilhq/tigerbeetle/blob/main/src/state_ma... [5] https://www.youtube.com/watch?v=OJb8A6h9jQQ https://www.youtube.com/watch?v=OJb8A6h9jQQ — “How I Learned to Stop Worrying and Trust the Database”

buildbot 4y ago

At My first programming job, as an intern we went with his approach, using rethinkDB amd python “microservices” that were all separate python processes reading from their own rethinkDB change feed lambda - the system was essentially built as a distributed reactive microservice pipeline, with rethinkDB as the source of truth for every other process - which existed on 100-1000 machines depending!

brewkode 4y ago

A few years back, a colleague and me built a distributed system toolkit in an effort to understand the blocks/pieces of a distributed system and also to make it easier to build one for our needs. This article seems to hit on the same set of notes that we had in mind when we started out building suuchi. https://github.com/ashwanthkumar/suuchi https://github.com/ashwanthkumar/suuchi.