7 ms·

Why is it so important for transactions to be interactive? I feel like this is not a feature that I use frequently as an application developer and it makes dist

Why is it so important for transactions to be interactive? I feel like this is not a feature that I use frequently as an application developer and it makes distributed transactions so much harder. It seems like something that accidentally came along from SQL, and is imposing a large cost.

zaphar 7y ago
Interactive may be a little misleading here. Consider a common case in any java application at my current place of employment. 1. A controller starts a transaction. 2. Some code looks up a record in a database. 3. Based on the results of that lookup we run 1 of two possible writes to a table in that database. Knowing what kind of write you will do in the transaction requires knowing the result of the lookup which itself must be run in the transaction. Now take this small case and imagine it in the real world scenario where there might many such lookup specific write logic all wrapped in a single transaction and the space of possible combinations of statements you will want to execute in the transaction is large. None of this is interactive in the sense of someone working in a shell. But it is interactive in a way that makes the transaction useful in the real word.
- aboodman 7y ago
  Exactly. What Calvin (and FaunaDB, I think) does is allow you to run an arbitrary function which takes input on the database. That function can do whatever it wants, including any number of reads, writes, and arbitrary logic based on the reads. But critically, it can't talk back to the calling client, except for to send the final result. This allows you to implement the pattern you describe, which I agree is common, but in a dramatically simpler way. Having the database, which is not really a single thing, but a swarm of computers spread across the globe separated by unreliable links who are trying to stay in consensus, pause their work to hear back from the client just seems ... well it seems miraculous that it can work at all. The Calvin way is so much easier, it seems like there must be some very good reason that it's not what CockroachDB does. But I've never heard what that reason is.
  - foota 7y ago
    Wouldn't you have to implement a DSL and all the parts related to it for that to work? Also things like serialization cost. What if I have a large local in memory structure I want to base my query results off of? Funny enough this is kind of similar to the problem solved by things like apache beam.
    - aboodman 7y ago
      No you'd need a complete programming language. But SQL is basically one already, most variants are already turing-complete. You'd also have to provide the programming environment with concepts of cursors and so on so they could page through data efficiently.
      - Pxtl 7y ago
        Every procedural layer I've ever used bolted onto SQL (pl/SQL, t-sql) has been absolute goddamned agony to use. SQL is a good (if dated) language for relational access and manipulation, but awful for procedural scripting.
  - evanweaver 7y ago
    Nobody in industry understood how to apply Calvin until we did at Fauna. That is the only reason; the rest is engineering path dependence.
    - aboodman 7y ago
      if you don’t mind sharing, I’m curious to learn more what was the main challenge was applying Calvin. Thanks!
      - evanweaver 7y ago
        See @freels’ reply above. I think it is fair to say that the Calvin paper is visibly incomplete and expresses some constraints in a way that makes them seem insurmountable when they are not; specifically, they are only constraints within the log, but do not constrain the database experience overall. Applying Calvin to traditional RDBMS workloads was a very unlikely creative exercise because it required questioning these explicit constraints. The Spanner paper also leaves a lot unexplained, but it is less obvious until you are too far down the path to turn back. After all, it worked for Google. Calvin did not have that real-world proof. Combine that with the pessimism of the paper itself and nobody was willing to pick it up.
BubRoss 7y ago
This article implies that it reduces latency.

zaphar 7y ago

Interactive may be a little misleading here. Consider a common case in any java application at my current place of employment. 1. A controller starts a transaction. 2. Some code looks up a record in a database. 3. Based on the results of that lookup we run 1 of two possible writes to a table in that database. Knowing what kind of write you will do in the transaction requires knowing the result of the lookup which itself must be run in the transaction. Now take this small case and imagine it in the real world scenario where there might many such lookup specific write logic all wrapped in a single transaction and the space of possible combinations of statements you will want to execute in the transaction is large. None of this is interactive in the sense of someone working in a shell. But it is interactive in a way that makes the transaction useful in the real word.

aboodman 7y ago
Exactly. What Calvin (and FaunaDB, I think) does is allow you to run an arbitrary function which takes input on the database. That function can do whatever it wants, including any number of reads, writes, and arbitrary logic based on the reads. But critically, it can't talk back to the calling client, except for to send the final result. This allows you to implement the pattern you describe, which I agree is common, but in a dramatically simpler way. Having the database, which is not really a single thing, but a swarm of computers spread across the globe separated by unreliable links who are trying to stay in consensus, pause their work to hear back from the client just seems ... well it seems miraculous that it can work at all. The Calvin way is so much easier, it seems like there must be some very good reason that it's not what CockroachDB does. But I've never heard what that reason is.
- foota 7y ago
  Wouldn't you have to implement a DSL and all the parts related to it for that to work? Also things like serialization cost. What if I have a large local in memory structure I want to base my query results off of? Funny enough this is kind of similar to the problem solved by things like apache beam.
  - aboodman 7y ago
    No you'd need a complete programming language. But SQL is basically one already, most variants are already turing-complete. You'd also have to provide the programming environment with concepts of cursors and so on so they could page through data efficiently.
    - Pxtl 7y ago
      Every procedural layer I've ever used bolted onto SQL (pl/SQL, t-sql) has been absolute goddamned agony to use. SQL is a good (if dated) language for relational access and manipulation, but awful for procedural scripting.
- evanweaver 7y ago
  Nobody in industry understood how to apply Calvin until we did at Fauna. That is the only reason; the rest is engineering path dependence.
  - aboodman 7y ago
    if you don’t mind sharing, I’m curious to learn more what was the main challenge was applying Calvin. Thanks!
    - evanweaver 7y ago
      See @freels’ reply above. I think it is fair to say that the Calvin paper is visibly incomplete and expresses some constraints in a way that makes them seem insurmountable when they are not; specifically, they are only constraints within the log, but do not constrain the database experience overall. Applying Calvin to traditional RDBMS workloads was a very unlikely creative exercise because it required questioning these explicit constraints. The Spanner paper also leaves a lot unexplained, but it is less obvious until you are too far down the path to turn back. After all, it worked for Google. Calvin did not have that real-world proof. Combine that with the pessimism of the paper itself and nobody was willing to pick it up.

BubRoss 7y ago

This article implies that it reduces latency.