2 ms·

Rama supports very strong transactions actually. Besides being able to do arbitrary atomic updates across multiple indexes on a single partition, you can also d

Rama supports very strong transactions actually. Besides being able to do arbitrary atomic updates across multiple indexes on a single partition, you can also do distributed transactions. Our bank transfer example in rama-demo-gallery demonstrates this. https://github.com/redplanetlabs/rama-demo-gallery https://github.com/redplanetlabs/rama-demo-gallery

photon_lines 3y ago
Sorry, but can you elaborate on what type of transaction isolation-levels your solution supports (i.e. read committed, repeatable read, serializable, read uncommitted)?
- nathanmarz 3y ago
  Those concepts are relevant to databases like RDBMS's where you have multiple writes happening concurrently to the same index. Rama doesn't work like that – you get parallelism by having multiple partitions, writes on a single partition are strictly ordered, and Rama gets performance by batching the flushing of writes. For a foreign query in Rama, you only ever get back committed data. So that would be analogous to "read committed". For ETL code, an event has atomic access to all indexes on its task thread. Any data written to those indexes before the event would have "repeatable read" consistency. Any writes done by the event before it finishes have read-after-write consistency, which would be analogous to "read uncommitted". This is correct during ETL execution because no acknowledgement of completion has been given back to clients yet, and uncommitted writes are never visible to client queries.
  - photon_lines 3y ago
    Thank you for the clarification. I'm still a bit confused on how this replaces databases. Your write-up claims that this solution is a replacement for modern DB systems, but I've seen a need for all 4 transaction isolation levels among various systems that I've implemented for different clients. In essence, your solution provides a 'one size fits all' scenario and different isolation levels for different components which I don't see replacing databases...I'd say your solution more-in is a great and extremely efficient implementation of event-sourcing. Either way - good luck with your company and thanks once again for the clarification :)
    - nathanmarz 3y ago
      You need all four in the context of databases and the architectures in which they live, where computation is separate from storage and concurrent updates are possible. In the context of Rama it's different, where computation is colocated with storage and you have tight control over the ordering of events. It's probably easier to understand if you share a specific example you've dealt with in the past that needed a particular isolation level. Then I can explain how that would work with Rama.
benlivengood 3y ago
If I understand that example correctly it takes careful up-front design of event-sources to achieve the desired constraints. E.g. if, for some reason, transactions between users needed to now be initiated from the receiving user (toUser asks for $amt from fromUser) the microbatching would need to be paused and switched to a new microbatch that feeds from the new event source. Does Rama natively support sequencing of these kinds of schema changes if, e.g. you had to replay all the events from scratch to restore the Pstates from scratch. Traditionally, an event-id or timestamp would be used as a demarcation of different aggregation semantics before and after a sentinel value, and I suppose each microbatch could check against sentinel value before proceeding. In short, though, I struggle to see lower complexity for transactional workloads in Rama v.s. traditional RDBMS. Do you plan to provide a higher-level abstraction like SQL that internally compiles to correct Rama implementations of Pstate generation? I think combining ETL and OTP in a single system has a lot of merit, as several RDBMS systems have been adding materialized views to support ETL workflows for that reason. I'm not convinced that low-level index generation is the right solution. I could be convinced that SQL-style high-level schema declarations parsed into malleable low-level operations that can be modified as necessary with solid support for the semantics of point-in-time schema changes could win over traditional RDBMS. For most use-cases the high-level definitions would be enough but reaching under the hood in a reliable and ops-friendly way would avoid the need for an explosion of custom and mutually-vendor-incompatible RDBMS extensions to SQL.
- nathanmarz 3y ago
  The transactionality comes from the fact that the depot append is of the entire transaction. So it atomically captures both the debit and credit that happen as a materialization of that record into PStates by the ETL. If you need a different kind of transaction, then that can be done with a different depot or a different event type within the same depot. And the ETL code would be updated to handle that. We won't be providing a SQL -> dataflow compiler for Rama ETLs anytime soon. SQL is way more limited than a general purpose programming language, even with extensions.

photon_lines 3y ago

Sorry, but can you elaborate on what type of transaction isolation-levels your solution supports (i.e. read committed, repeatable read, serializable, read uncommitted)?

nathanmarz 3y ago
Those concepts are relevant to databases like RDBMS's where you have multiple writes happening concurrently to the same index. Rama doesn't work like that – you get parallelism by having multiple partitions, writes on a single partition are strictly ordered, and Rama gets performance by batching the flushing of writes. For a foreign query in Rama, you only ever get back committed data. So that would be analogous to "read committed". For ETL code, an event has atomic access to all indexes on its task thread. Any data written to those indexes before the event would have "repeatable read" consistency. Any writes done by the event before it finishes have read-after-write consistency, which would be analogous to "read uncommitted". This is correct during ETL execution because no acknowledgement of completion has been given back to clients yet, and uncommitted writes are never visible to client queries.
- photon_lines 3y ago
  Thank you for the clarification. I'm still a bit confused on how this replaces databases. Your write-up claims that this solution is a replacement for modern DB systems, but I've seen a need for all 4 transaction isolation levels among various systems that I've implemented for different clients. In essence, your solution provides a 'one size fits all' scenario and different isolation levels for different components which I don't see replacing databases...I'd say your solution more-in is a great and extremely efficient implementation of event-sourcing. Either way - good luck with your company and thanks once again for the clarification :)
  - nathanmarz 3y ago
    You need all four in the context of databases and the architectures in which they live, where computation is separate from storage and concurrent updates are possible. In the context of Rama it's different, where computation is colocated with storage and you have tight control over the ordering of events. It's probably easier to understand if you share a specific example you've dealt with in the past that needed a particular isolation level. Then I can explain how that would work with Rama.

benlivengood 3y ago

If I understand that example correctly it takes careful up-front design of event-sources to achieve the desired constraints. E.g. if, for some reason, transactions between users needed to now be initiated from the receiving user (toUser asks for $amt from fromUser) the microbatching would need to be paused and switched to a new microbatch that feeds from the new event source. Does Rama natively support sequencing of these kinds of schema changes if, e.g. you had to replay all the events from scratch to restore the Pstates from scratch. Traditionally, an event-id or timestamp would be used as a demarcation of different aggregation semantics before and after a sentinel value, and I suppose each microbatch could check against sentinel value before proceeding. In short, though, I struggle to see lower complexity for transactional workloads in Rama v.s. traditional RDBMS. Do you plan to provide a higher-level abstraction like SQL that internally compiles to correct Rama implementations of Pstate generation? I think combining ETL and OTP in a single system has a lot of merit, as several RDBMS systems have been adding materialized views to support ETL workflows for that reason. I'm not convinced that low-level index generation is the right solution. I could be convinced that SQL-style high-level schema declarations parsed into malleable low-level operations that can be modified as necessary with solid support for the semantics of point-in-time schema changes could win over traditional RDBMS. For most use-cases the high-level definitions would be enough but reaching under the hood in a reliable and ops-friendly way would avoid the need for an explosion of custom and mutually-vendor-incompatible RDBMS extensions to SQL.

nathanmarz 3y ago
The transactionality comes from the fact that the depot append is of the entire transaction. So it atomically captures both the debit and credit that happen as a materialization of that record into PStates by the ETL. If you need a different kind of transaction, then that can be done with a different depot or a different event type within the same depot. And the ETL code would be updated to handle that. We won't be providing a SQL -> dataflow compiler for Rama ETLs anytime soon. SQL is way more limited than a general purpose programming language, even with extensions.