5 ms·

In "The Future of Data Systems", the author imagines a system where the application writes events to a Kafka-like distributed log. Consumers of the log do work

In "The Future of Data Systems", the author imagines a system where the application writes events to a Kafka-like distributed log. Consumers of the log do work like de-duping, committing the data to a RDBMS, invalidating caches, updating search indexes, etc. The application might read state directly from log updates, or have a system to sync w/ some sort of atomic state (e.g. RethingDB changefeeds). The architecture seems to solve two big problems * Scaling RDBMS (there are solutions like Cloud Spanner but they rely heavily on Google's proprietary network for low latency and are expensive as balls) * Keeping downstream systems in sync. A lot of companies have gone with a "Tail the RDBMS" of some kind, e.g. writing the MySQL binlog to a Kafka queue and having downstream consumers read from that, but this seems like a more elegant solution. Are there any examples or experiences of people working with systems like this? What are some downsides, challenges, and actual benefits?

criloz2 9y ago
HI I am working in one is called omniql, I am lone developer, so probably it will take time, but there are much of the what they said in the book planned for omniql, is based in observable pattern so it will have live queries by default, it will use system like kafka, nats, Redis or any message system, for communicate the partitions of the distributed databases and do computation that can be merged hierarchically, the computation are done by `serverless function` in any language, is a model that can be extended to ui interface to and create a framework very similar to mobx , but more lightwave, and ui components that can automatically query their data requirement to the server just when they are observed, I am creating a binary protocol of communication for this called delta https://github.com/nebtex/delta https://github.com/nebtex/delta, that is perfect for this system an incremental computing, is a really a big project, but I hope to have something working for the next 6 months.
lobster_johnson 9y ago
We're building an open source database like this. It's document-oriented and relies on a transaction log core (currently using Postgres, but it's pluggable), that feeds into a query subsystem (currently layered on top of Elasticsearch, but also pluggable). The transaction log encourages small logical "patches" (set a field, increment a number, replace a substring, move an array element, etc.) that are applied in sequence but can be disentangled by clients to generate a consistent UI, and also used to resolve conflicts between multiple distributed writers. You can also follow the log through the gRPC and HTTP APIs, and you can register "watches" on queries that cause matching changes to trigger an event. While the transaction log is the underlying data model, we also maintain a consistent view of the current version so that you can use it as a document database with classical CRUD operations. So on the surface it's a lot like Firebase or CouchDB, except you get things like joins and schemas. Drop me an email (see profile) and I can send you some links.
- HodGreeley 9y ago
  Curious how this compares to Couchbase + N1QL, since that gets you a document db that also supports full SQL.
  - lobster_johnson 9y ago
    In our case, we wanted something more compact, so a query looks more like GraphQL (it's technically a superset of JSON, but it usually doesn't look like JSON). Joins, for example, are just nested declarations that list which attributes on the joined collections to fetch. Here's a query that shows many of the features: *[_type == "blogpost" && published == true] { _id, title, "bodyExcerpt": regexReplace(body, "(.+?)(\. |$)", "\\1"), author -> { name, category }, "sectionNames": sections -> name } | order(createdAt desc)[0..20] I don't know much about N1QL, I will have to read more about it.
- criloz2 9y ago
  this looks very similar what I am doing, the difference appears to be that in delta a number of operations is just a subset that can be done in your system. for example it only allows the set operation on scalars/string fields and push operation on vectors, also the delete operations are placed in a way in which automatically let the merge algorithm to know if it needs to check past version in order to compute the current version of the resource, it also allows to compact all the history of a resource in distributed way.
  - lobster_johnson 9y ago
    Not sure what you mean by subset. In our case, a single transaction contains one or more document operations (create, delete, etc.), one of which is a "patch" operation that applies a fine-grained transformation. The transformations use an extended version of JSONPath in order to be able to target deep tree nodes as well as apply transformations to multiple fields (e.g. authors[0].publications[*].title). Operations such as set, increment etc. are specific to the data type of the value being transformed and will fail if the data type does not support the transformation function.
jamesblonde 9y ago
We have built something like this on top of a distributed in-memory database. The changelog of the distributed in-memory database is the 'source of truth' that downstream clients would like to consume. The research problem is that the changelog is batches of transactions that complete within a given epoch (time). Transactions may execute in parallel on different data nodes within an epoch and there is no global ordering over them (no global time, only logical time). Downstream clients, however, may require an ordering over the transactions, so we worked on a solution to this problem when the database stores metadata for a filesystem. Our solution ensures we don't have to serialize the transactions to ensure a global ordering, and provides strong eventual consistency to clients.
- erwan 9y ago
  Can you elaborate on the solution? Do you use an aggregator node, or have you massaged real-time reqs to allow for an aggregation period on the client side? I'm super curious to hear how you proceeded!
  - jamesblonde 9y ago
    No, we're using a table as a 'queue' in the db. Client are decoupled and if our middleware is offline, it can restart and catch up by draining the table. Transactions over the table ensure the consistency and integrity of the 'queue'. We provide at-least-once semantics to downstream apps, by exploiting transactions in the DB. Actually, we have one sink in the DB itself and we get exactly once semantics for that. The work in under submission with anonymous reviewing, so can't elaborate massively on everything else. Performance numbers are good, though, a sigle server can forward more than 10k ops/sec from the database changelog to a downstream db used for freetext search.
slackingoff2017 9y ago
I have been involved in using a system built like this. All I can say is... It feels like you're building a database out of an event stream. A shitty one at that... Basically the write log part, only without a way to apply that state reliably like a real database. So you have to keep the log around basically forever. It's like you're in the middle of a DB recovery all the time. After insane amounts of research and deep thought my personal opinion is that this is the wrong way to do scalable systems. Event sourcing and eventual consistency are taking industry for a ride in the wrong direction. In my quest to find a better way I found some research/leaks/opinions of Googlers, and I think they're right. Even Netflix admits that using eventual consistency means they have to build systems that go around and "fixup" data that ends up in bad states. Ew. Service RPC loops in any such systems are Pandora's box. Are these calls getting the most recently updated version of the data? Nobody knows. Even replaying the event log can't save you, the log may be strongly ordered but the data state between services that call each other is party determined by timing. Undefined behavior. You'll notice that LinkedIn/Netflix/Uber etc all seem to be building their systems using this pattern. Who is conspicuously absent? Google. The father of containers, VM's, and highly distributed systems is mum. Researching Google's systems gives some fascinating answers to the problem of distributed consistency, a solution I'm stunned hasn't seen more attention. Google decided as early as 2005 that eventually consistent systems were too hard to use and manage. All of their databases, BigTable, MegaStore, Spanner, F1... They're all strongly consistent in certain ways. How does Google do it? They make the database the source of truth. Service RPC calls either fail or succeed immediately. Service call loops, while bad for performance, produce consistent results. Failures are easy to find because data updates either succeed or fail immediately, not in some unbouded future time. The rest of the industry is missing the point of microservices IMO. Google's massively distributed systems are enabled largely by their innovative database designs. The rest of the industry is trying to replicate the topography of Google's internal systems without understanding what makes them work well. For microservices to be realistically usable for most use cases we need someone to come up with decent competition to Google's database systems. When you have a transactional distributed database all the problems with data spread across multiple services goes away. HBase was a good attempt but doesn't get enough love. A point missed in the creation of HBase, that becomes clear when reading the papers about MegaStore and Spanner, is that it wasn't designed to be used as a data store by itself. Instead, it has the minimal features needed to build a MegaStore on top of it. The weirder features of HBase/BigTable (like keeping around 3 copies of changed data, and row level atomicity without transactions) are clearly designed to make it possible to build a database on top of it. Unfortunately nobody thus far has taken up that challenge, and outside Google were all stuck with shitty databases that Google tossed away a decade ago.
ChemicalWarfare 9y ago
From my personal experience (which probably has to do with working in industries where incoming data has a wide array of clients who need it ASAP) - the biggest challenge with any distributed DB system is the problem of "reading your own writes" and how the system approaches it. Not to be confused with tx isolation levels. It's a balancing act between two extremes - locking everything down and ensuring the tx has been committed on all nodes/propagated to all consumers on one hand and sending an "ack" back to the client with a loose promise of eventual consistency on the other.
doh 9y ago
We built a kind of hybrid infrastructure around this idea. Any component sends its data into kafka-esque (we're using combination of NATS and PubSub) pipeline where series of workers read, process and write data into our RDBMS which is the ultimate source of the truth. This allowed us to run a double scale system, where all components are running on their own pace and the RDBMS is running on its own. There is some inherited delay in the data propagation, but it works for service like our (search engine) that doesn't require real-time exposure of newly acquired data. This design also allows for a frugality as the RDBMS cluster is only scaled based on long term trends and now short term bursts. We also are able to buy committed usage for the cluster as we've great predictability in its growth.

criloz2 9y ago

HI I am working in one is called omniql, I am lone developer, so probably it will take time, but there are much of the what they said in the book planned for omniql, is based in observable pattern so it will have live queries by default, it will use system like kafka, nats, Redis or any message system, for communicate the partitions of the distributed databases and do computation that can be merged hierarchically, the computation are done by `serverless function` in any language, is a model that can be extended to ui interface to and create a framework very similar to mobx , but more lightwave, and ui components that can automatically query their data requirement to the server just when they are observed, I am creating a binary protocol of communication for this called delta https://github.com/nebtex/delta https://github.com/nebtex/delta, that is perfect for this system an incremental computing, is a really a big project, but I hope to have something working for the next 6 months.

lobster_johnson 9y ago

We're building an open source database like this. It's document-oriented and relies on a transaction log core (currently using Postgres, but it's pluggable), that feeds into a query subsystem (currently layered on top of Elasticsearch, but also pluggable). The transaction log encourages small logical "patches" (set a field, increment a number, replace a substring, move an array element, etc.) that are applied in sequence but can be disentangled by clients to generate a consistent UI, and also used to resolve conflicts between multiple distributed writers. You can also follow the log through the gRPC and HTTP APIs, and you can register "watches" on queries that cause matching changes to trigger an event. While the transaction log is the underlying data model, we also maintain a consistent view of the current version so that you can use it as a document database with classical CRUD operations. So on the surface it's a lot like Firebase or CouchDB, except you get things like joins and schemas. Drop me an email (see profile) and I can send you some links.

HodGreeley 9y ago
Curious how this compares to Couchbase + N1QL, since that gets you a document db that also supports full SQL.
- lobster_johnson 9y ago
  In our case, we wanted something more compact, so a query looks more like GraphQL (it's technically a superset of JSON, but it usually doesn't look like JSON). Joins, for example, are just nested declarations that list which attributes on the joined collections to fetch. Here's a query that shows many of the features: *[_type == "blogpost" && published == true] { _id, title, "bodyExcerpt": regexReplace(body, "(.+?)(\. |$)", "\\1"), author -> { name, category }, "sectionNames": sections -> name } | order(createdAt desc)[0..20] I don't know much about N1QL, I will have to read more about it.
criloz2 9y ago
this looks very similar what I am doing, the difference appears to be that in delta a number of operations is just a subset that can be done in your system. for example it only allows the set operation on scalars/string fields and push operation on vectors, also the delete operations are placed in a way in which automatically let the merge algorithm to know if it needs to check past version in order to compute the current version of the resource, it also allows to compact all the history of a resource in distributed way.
- lobster_johnson 9y ago
  Not sure what you mean by subset. In our case, a single transaction contains one or more document operations (create, delete, etc.), one of which is a "patch" operation that applies a fine-grained transformation. The transformations use an extended version of JSONPath in order to be able to target deep tree nodes as well as apply transformations to multiple fields (e.g. authors[0].publications[*].title). Operations such as set, increment etc. are specific to the data type of the value being transformed and will fail if the data type does not support the transformation function.

jamesblonde 9y ago

We have built something like this on top of a distributed in-memory database. The changelog of the distributed in-memory database is the 'source of truth' that downstream clients would like to consume. The research problem is that the changelog is batches of transactions that complete within a given epoch (time). Transactions may execute in parallel on different data nodes within an epoch and there is no global ordering over them (no global time, only logical time). Downstream clients, however, may require an ordering over the transactions, so we worked on a solution to this problem when the database stores metadata for a filesystem. Our solution ensures we don't have to serialize the transactions to ensure a global ordering, and provides strong eventual consistency to clients.

erwan 9y ago
Can you elaborate on the solution? Do you use an aggregator node, or have you massaged real-time reqs to allow for an aggregation period on the client side? I'm super curious to hear how you proceeded!
- jamesblonde 9y ago
  No, we're using a table as a 'queue' in the db. Client are decoupled and if our middleware is offline, it can restart and catch up by draining the table. Transactions over the table ensure the consistency and integrity of the 'queue'. We provide at-least-once semantics to downstream apps, by exploiting transactions in the DB. Actually, we have one sink in the DB itself and we get exactly once semantics for that. The work in under submission with anonymous reviewing, so can't elaborate massively on everything else. Performance numbers are good, though, a sigle server can forward more than 10k ops/sec from the database changelog to a downstream db used for freetext search.

slackingoff2017 9y ago

I have been involved in using a system built like this. All I can say is... It feels like you're building a database out of an event stream. A shitty one at that... Basically the write log part, only without a way to apply that state reliably like a real database. So you have to keep the log around basically forever. It's like you're in the middle of a DB recovery all the time. After insane amounts of research and deep thought my personal opinion is that this is the wrong way to do scalable systems. Event sourcing and eventual consistency are taking industry for a ride in the wrong direction. In my quest to find a better way I found some research/leaks/opinions of Googlers, and I think they're right. Even Netflix admits that using eventual consistency means they have to build systems that go around and "fixup" data that ends up in bad states. Ew. Service RPC loops in any such systems are Pandora's box. Are these calls getting the most recently updated version of the data? Nobody knows. Even replaying the event log can't save you, the log may be strongly ordered but the data state between services that call each other is party determined by timing. Undefined behavior. You'll notice that LinkedIn/Netflix/Uber etc all seem to be building their systems using this pattern. Who is conspicuously absent? Google. The father of containers, VM's, and highly distributed systems is mum. Researching Google's systems gives some fascinating answers to the problem of distributed consistency, a solution I'm stunned hasn't seen more attention. Google decided as early as 2005 that eventually consistent systems were too hard to use and manage. All of their databases, BigTable, MegaStore, Spanner, F1... They're all strongly consistent in certain ways. How does Google do it? They make the database the source of truth. Service RPC calls either fail or succeed immediately. Service call loops, while bad for performance, produce consistent results. Failures are easy to find because data updates either succeed or fail immediately, not in some unbouded future time. The rest of the industry is missing the point of microservices IMO. Google's massively distributed systems are enabled largely by their innovative database designs. The rest of the industry is trying to replicate the topography of Google's internal systems without understanding what makes them work well. For microservices to be realistically usable for most use cases we need someone to come up with decent competition to Google's database systems. When you have a transactional distributed database all the problems with data spread across multiple services goes away. HBase was a good attempt but doesn't get enough love. A point missed in the creation of HBase, that becomes clear when reading the papers about MegaStore and Spanner, is that it wasn't designed to be used as a data store by itself. Instead, it has the minimal features needed to build a MegaStore on top of it. The weirder features of HBase/BigTable (like keeping around 3 copies of changed data, and row level atomicity without transactions) are clearly designed to make it possible to build a database on top of it. Unfortunately nobody thus far has taken up that challenge, and outside Google were all stuck with shitty databases that Google tossed away a decade ago.

ChemicalWarfare 9y ago

From my personal experience (which probably has to do with working in industries where incoming data has a wide array of clients who need it ASAP) - the biggest challenge with any distributed DB system is the problem of "reading your own writes" and how the system approaches it. Not to be confused with tx isolation levels. It's a balancing act between two extremes - locking everything down and ensuring the tx has been committed on all nodes/propagated to all consumers on one hand and sending an "ack" back to the client with a loose promise of eventual consistency on the other.

doh 9y ago

We built a kind of hybrid infrastructure around this idea. Any component sends its data into kafka-esque (we're using combination of NATS and PubSub) pipeline where series of workers read, process and write data into our RDBMS which is the ultimate source of the truth. This allowed us to run a double scale system, where all components are running on their own pace and the RDBMS is running on its own. There is some inherited delay in the data propagation, but it works for service like our (search engine) that doesn't require real-time exposure of newly acquired data. This design also allows for a frugality as the RDBMS cluster is only scaled based on long term trends and now short term bursts. We also are able to buy committed usage for the cluster as we've great predictability in its growth.