3 ms·

I didn't notice a mention of transactions in the article, nor of constraints. It's all fine to claim that you can compose arbitrary event source domains togeth

I didn't notice a mention of transactions in the article, nor of constraints. It's all fine to claim that you can compose arbitrary event source domains together and query them but IMHO the biggest power of RDBMS are transactions and constraints for data integrity. Maybe Rama comes with amazing composability features that ensure cross-domain constraints, but I would be really surprised if they can maintain globally consistent real-time transactions. I've worked on huge ETL pipelines with materialized views (Photon, Ubiq, Mesa) and the business logic in Ubiq to materialize the view updates for Mesa was immense. None of it was transactional; everything was for aggregate statistics and so it worked well. Ads-DB and Payments used Spanner for very good reasons.

nathanmarz 3y ago
Rama supports very strong transactions actually. Besides being able to do arbitrary atomic updates across multiple indexes on a single partition, you can also do distributed transactions. Our bank transfer example in rama-demo-gallery demonstrates this. https://github.com/redplanetlabs/rama-demo-gallery https://github.com/redplanetlabs/rama-demo-gallery
- photon_lines 3y ago
  Sorry, but can you elaborate on what type of transaction isolation-levels your solution supports (i.e. read committed, repeatable read, serializable, read uncommitted)?
  - nathanmarz 3y ago
    Those concepts are relevant to databases like RDBMS's where you have multiple writes happening concurrently to the same index. Rama doesn't work like that – you get parallelism by having multiple partitions, writes on a single partition are strictly ordered, and Rama gets performance by batching the flushing of writes. For a foreign query in Rama, you only ever get back committed data. So that would be analogous to "read committed". For ETL code, an event has atomic access to all indexes on its task thread. Any data written to those indexes before the event would have "repeatable read" consistency. Any writes done by the event before it finishes have read-after-write consistency, which would be analogous to "read uncommitted". This is correct during ETL execution because no acknowledgement of completion has been given back to clients yet, and uncommitted writes are never visible to client queries.
    - photon_lines 3y ago
      Thank you for the clarification. I'm still a bit confused on how this replaces databases. Your write-up claims that this solution is a replacement for modern DB systems, but I've seen a need for all 4 transaction isolation levels among various systems that I've implemented for different clients. In essence, your solution provides a 'one size fits all' scenario and different isolation levels for different components which I don't see replacing databases...I'd say your solution more-in is a great and extremely efficient implementation of event-sourcing. Either way - good luck with your company and thanks once again for the clarification :)
      - nathanmarz 3y ago
        You need all four in the context of databases and the architectures in which they live, where computation is separate from storage and concurrent updates are possible. In the context of Rama it's different, where computation is colocated with storage and you have tight control over the ordering of events. It's probably easier to understand if you share a specific example you've dealt with in the past that needed a particular isolation level. Then I can explain how that would work with Rama.
- benlivengood 3y ago
  If I understand that example correctly it takes careful up-front design of event-sources to achieve the desired constraints. E.g. if, for some reason, transactions between users needed to now be initiated from the receiving user (toUser asks for $amt from fromUser) the microbatching would need to be paused and switched to a new microbatch that feeds from the new event source. Does Rama natively support sequencing of these kinds of schema changes if, e.g. you had to replay all the events from scratch to restore the Pstates from scratch. Traditionally, an event-id or timestamp would be used as a demarcation of different aggregation semantics before and after a sentinel value, and I suppose each microbatch could check against sentinel value before proceeding. In short, though, I struggle to see lower complexity for transactional workloads in Rama v.s. traditional RDBMS. Do you plan to provide a higher-level abstraction like SQL that internally compiles to correct Rama implementations of Pstate generation? I think combining ETL and OTP in a single system has a lot of merit, as several RDBMS systems have been adding materialized views to support ETL workflows for that reason. I'm not convinced that low-level index generation is the right solution. I could be convinced that SQL-style high-level schema declarations parsed into malleable low-level operations that can be modified as necessary with solid support for the semantics of point-in-time schema changes could win over traditional RDBMS. For most use-cases the high-level definitions would be enough but reaching under the hood in a reliable and ops-friendly way would avoid the need for an explosion of custom and mutually-vendor-incompatible RDBMS extensions to SQL.
  - nathanmarz 3y ago
    The transactionality comes from the fact that the depot append is of the entire transaction. So it atomically captures both the debit and credit that happen as a materialization of that record into PStates by the ETL. If you need a different kind of transaction, then that can be done with a different depot or a different event type within the same depot. And the ETL code would be updated to handle that. We won't be providing a SQL -> dataflow compiler for Rama ETLs anytime soon. SQL is way more limited than a general purpose programming language, even with extensions.
blagie 3y ago
Constraints are good, but personally, I consider transactions harmful. Most of the need for transactions comes from either: - The inability to do things atomically (e.g. computation bouncing back-and-forth between the backend and the database). I should be able to ship off a query which does everything I need atomically. - The inability to merge states (e.g. you send a changeset, I send a changeset, and the two combine). Transactions lead to transaction /failures/, which need to be dealt with properly. That introduces a very high cost to transactions, and is very rarely handled correctly. By the time you deal with failures correctly (which is hard), models without transactions become simpler.
- benlivengood 3y ago
  Payment/accounting systems are the typical example of where transactions are critical. The sending and receiving accounts need to be updated in the same transaction that correctly fails if either account would become negative as a result of the operation.
  - blagie 3y ago
    I disagree. Transactions are the wrong way to handle this. The right way to do this is to send a query which says: "If sending account has more than $10: Decrease sending account by $10. Increase receiving account by $10. Return what happened." The database should be able to do the above (or any other query) atomically. Under normal circumstances (e.g. barring a network failure or similar), this should never fail. The reason a transaction is necessary is because the above would typically be handled by 2-3 separate queries: - Start transaction - Check sending balance - [Client-side code] - Increase one account - [Client-side code] - Decrease the other - Finish transaction Having this back-and-forth can cause this to fail, and network latency means that it often DOES fail. A database could implement this internally with transactions and retries (in which case, the application programmer doesn't need to think about it and therefore have bugs there), but that's probably the wrong way to do it. There are many more reasonable ways, such as locks in rows + ordering operations, rearranging data, or otherwise. From the perspective of the programmer, this SHOULD be atomic. There is no reason why everyone writing a financial system needs to be competent at managing these sorts of parallel consistency issues. The current model introduces potential for a lot of consistency bugs, performance issues, and other problems. Transactions also introduce a lot of unnecessary complexity to the store itself.
vidarh 3y ago
Their example doesn't require transactions at all, which presumably is one of the reasons why they picked it. It's certainly an example that is ideally suited to showcase this because it's trivial - most references between entities are "weak" in that you can't really be sure what will arrive or whether you'll be able to re-fetch it from its origin. Put another way: The totally naive "homegrown" re-implementation of their Mastodon architecture would: * 1) Log somewhere, to whatever. * 2) Replicate those logs * 3) Run "something" that auto-create or migrates tables for any "P-stores" that have changed or are not present and streams updates into them. * 4) A query frontend that queries multiple P-store partitions with fairly restricted query syntax (because, if you need anything complex, in this architecture you should create a new "P-store") and merges the result. Their use of Mastodon and Twitter as an example is very much because both the original Twitter architecture and the current Mastodon (the software, not the aggregate network) architecture punted on the above. Which is fine - you can migrate towards an architecture like that at any time by simply breaking out a "P-store" and letting a worker populate it from your canonical data store. Their "problem" is that you need a large scale before this architecture needs something more than 1 database plus a few triggers and user-defined functions, and an even more massive scale before it needs something more than a few databases with replication and a few triggers and user-defined functions and foreign data wrappers, and most systems never get to that scale. And even at that scale there are then plenty of solutions both on the "depot" side and the "p-store" side for scaling further already and has been pretty much "forever". So they badly need this to be pleasant enough for systems below the scale where people need something like this that people will pick it despite not needing the scale, or there'll be few incentives to use it.

nathanmarz 3y ago

Rama supports very strong transactions actually. Besides being able to do arbitrary atomic updates across multiple indexes on a single partition, you can also do distributed transactions. Our bank transfer example in rama-demo-gallery demonstrates this. https://github.com/redplanetlabs/rama-demo-gallery https://github.com/redplanetlabs/rama-demo-gallery

photon_lines 3y ago
Sorry, but can you elaborate on what type of transaction isolation-levels your solution supports (i.e. read committed, repeatable read, serializable, read uncommitted)?
- nathanmarz 3y ago
  Those concepts are relevant to databases like RDBMS's where you have multiple writes happening concurrently to the same index. Rama doesn't work like that – you get parallelism by having multiple partitions, writes on a single partition are strictly ordered, and Rama gets performance by batching the flushing of writes. For a foreign query in Rama, you only ever get back committed data. So that would be analogous to "read committed". For ETL code, an event has atomic access to all indexes on its task thread. Any data written to those indexes before the event would have "repeatable read" consistency. Any writes done by the event before it finishes have read-after-write consistency, which would be analogous to "read uncommitted". This is correct during ETL execution because no acknowledgement of completion has been given back to clients yet, and uncommitted writes are never visible to client queries.
  - photon_lines 3y ago
    Thank you for the clarification. I'm still a bit confused on how this replaces databases. Your write-up claims that this solution is a replacement for modern DB systems, but I've seen a need for all 4 transaction isolation levels among various systems that I've implemented for different clients. In essence, your solution provides a 'one size fits all' scenario and different isolation levels for different components which I don't see replacing databases...I'd say your solution more-in is a great and extremely efficient implementation of event-sourcing. Either way - good luck with your company and thanks once again for the clarification :)
    - nathanmarz 3y ago
      You need all four in the context of databases and the architectures in which they live, where computation is separate from storage and concurrent updates are possible. In the context of Rama it's different, where computation is colocated with storage and you have tight control over the ordering of events. It's probably easier to understand if you share a specific example you've dealt with in the past that needed a particular isolation level. Then I can explain how that would work with Rama.
benlivengood 3y ago
If I understand that example correctly it takes careful up-front design of event-sources to achieve the desired constraints. E.g. if, for some reason, transactions between users needed to now be initiated from the receiving user (toUser asks for $amt from fromUser) the microbatching would need to be paused and switched to a new microbatch that feeds from the new event source. Does Rama natively support sequencing of these kinds of schema changes if, e.g. you had to replay all the events from scratch to restore the Pstates from scratch. Traditionally, an event-id or timestamp would be used as a demarcation of different aggregation semantics before and after a sentinel value, and I suppose each microbatch could check against sentinel value before proceeding. In short, though, I struggle to see lower complexity for transactional workloads in Rama v.s. traditional RDBMS. Do you plan to provide a higher-level abstraction like SQL that internally compiles to correct Rama implementations of Pstate generation? I think combining ETL and OTP in a single system has a lot of merit, as several RDBMS systems have been adding materialized views to support ETL workflows for that reason. I'm not convinced that low-level index generation is the right solution. I could be convinced that SQL-style high-level schema declarations parsed into malleable low-level operations that can be modified as necessary with solid support for the semantics of point-in-time schema changes could win over traditional RDBMS. For most use-cases the high-level definitions would be enough but reaching under the hood in a reliable and ops-friendly way would avoid the need for an explosion of custom and mutually-vendor-incompatible RDBMS extensions to SQL.
- nathanmarz 3y ago
  The transactionality comes from the fact that the depot append is of the entire transaction. So it atomically captures both the debit and credit that happen as a materialization of that record into PStates by the ETL. If you need a different kind of transaction, then that can be done with a different depot or a different event type within the same depot. And the ETL code would be updated to handle that. We won't be providing a SQL -> dataflow compiler for Rama ETLs anytime soon. SQL is way more limited than a general purpose programming language, even with extensions.

blagie 3y ago

Constraints are good, but personally, I consider transactions harmful. Most of the need for transactions comes from either: - The inability to do things atomically (e.g. computation bouncing back-and-forth between the backend and the database). I should be able to ship off a query which does everything I need atomically. - The inability to merge states (e.g. you send a changeset, I send a changeset, and the two combine). Transactions lead to transaction /failures/, which need to be dealt with properly. That introduces a very high cost to transactions, and is very rarely handled correctly. By the time you deal with failures correctly (which is hard), models without transactions become simpler.

benlivengood 3y ago
Payment/accounting systems are the typical example of where transactions are critical. The sending and receiving accounts need to be updated in the same transaction that correctly fails if either account would become negative as a result of the operation.
- blagie 3y ago
  I disagree. Transactions are the wrong way to handle this. The right way to do this is to send a query which says: "If sending account has more than $10: Decrease sending account by $10. Increase receiving account by $10. Return what happened." The database should be able to do the above (or any other query) atomically. Under normal circumstances (e.g. barring a network failure or similar), this should never fail. The reason a transaction is necessary is because the above would typically be handled by 2-3 separate queries: - Start transaction - Check sending balance - [Client-side code] - Increase one account - [Client-side code] - Decrease the other - Finish transaction Having this back-and-forth can cause this to fail, and network latency means that it often DOES fail. A database could implement this internally with transactions and retries (in which case, the application programmer doesn't need to think about it and therefore have bugs there), but that's probably the wrong way to do it. There are many more reasonable ways, such as locks in rows + ordering operations, rearranging data, or otherwise. From the perspective of the programmer, this SHOULD be atomic. There is no reason why everyone writing a financial system needs to be competent at managing these sorts of parallel consistency issues. The current model introduces potential for a lot of consistency bugs, performance issues, and other problems. Transactions also introduce a lot of unnecessary complexity to the store itself.

vidarh 3y ago

Their example doesn't require transactions at all, which presumably is one of the reasons why they picked it. It's certainly an example that is ideally suited to showcase this because it's trivial - most references between entities are "weak" in that you can't really be sure what will arrive or whether you'll be able to re-fetch it from its origin. Put another way: The totally naive "homegrown" re-implementation of their Mastodon architecture would: * 1) Log somewhere, to whatever. * 2) Replicate those logs * 3) Run "something" that auto-create or migrates tables for any "P-stores" that have changed or are not present and streams updates into them. * 4) A query frontend that queries multiple P-store partitions with fairly restricted query syntax (because, if you need anything complex, in this architecture you should create a new "P-store") and merges the result. Their use of Mastodon and Twitter as an example is very much because both the original Twitter architecture and the current Mastodon (the software, not the aggregate network) architecture punted on the above. Which is fine - you can migrate towards an architecture like that at any time by simply breaking out a "P-store" and letting a worker populate it from your canonical data store. Their "problem" is that you need a large scale before this architecture needs something more than 1 database plus a few triggers and user-defined functions, and an even more massive scale before it needs something more than a few databases with replication and a few triggers and user-defined functions and foreign data wrappers, and most systems never get to that scale. And even at that scale there are then plenty of solutions both on the "depot" side and the "p-store" side for scaling further already and has been pretty much "forever". So they badly need this to be pleasant enough for systems below the scale where people need something like this that people will pick it despite not needing the scale, or there'll be few incentives to use it.