7 ms·
How to get transactions between almost any data stores
- FrameworkFred 3y agoI think I followed that pretty well (I'll maybe need to read it again to be sure)...very nice, well done!
- dang 3y agoPaper: http://petereliaskraft.net/res/p2732-kraft.pdf http://petereliaskraft.net/res/p2732-kraft.pdf
- jimbokun 3y agoDoes this mean Epoxy must execute all the calls to the underlying services? If so, how is the data to be stored in each db communicated to Epoxy?
- jiaaro 3y agoDoes Epoxy use postgres' own transaction ids across datastores? If not, does it implement its own "transaction id horizon" like postgres has so that you can assume all transactions before a certain counter are committed?
- KraftyOne 3y agoYeah, our implementation uses Postgres transaction IDs across data stores.
- samsquire 3y agoThank you for submitting this and thank you to the authors. I am curious how this kind of thing can scale. What I'm inspired by this article is the idea of applying multiversion semantics to datastores that aren't multiversioned. Sounds like if you have a transactionally sound store of "versions" and a way to query every datastore with a version field, you could implement point in time consistency, because the current version would only change when all datastores are updated. You still have to wait for all datastores to be updated, so that part is similar to 2 phase commit. My understanding is that two phase commit does not scale and is slow because of all the round trips, but we still want linearizability and strong consistency in systems. I tried to implement an asynchronously replicating data protocol but it is eventually consistent, not linearizable (it fails the Jepsen linearizability test) https://github.com/samsquire/eventually-consistent-mesh https://github.com/samsquire/eventually-consistent-mesh I'm a beginner in this area but I have a toy multiversion concurrency control implementation that uses Java threads. (See MVCC.java and TransactionC.java) the key statements of code is in commit() and shouldRestart(). https://github.com/samsquire/multiversion-concurrency-control https://github.com/samsquire/multiversion-concurrency-contro... Dropbox does something 2 phase commit with their cross shard replication which is interesting https://dropbox.tech/infrastructure/cross-shard-transactions-at-10-million-requests-per-second https://dropbox.tech/infrastructure/cross-shard-transactions...
- KraftyOne 3y agoWhile we haven't tested this, I suspect the main scaling bottleneck in practice is the primary database, as every Epoxy transaction goes through it. If you wanted to run Epoxy at large scale, you'd need a primary that supports very high transaction throughput.
- yawaramin 3y agoWould 4M qps be enough? :-) SQLite can achieve this with some help: https://blog.expensify.com/2018/01/08/scaling-sqlite-to-4m-qps-on-a-single-server/ https://blog.expensify.com/2018/01/08/scaling-sqlite-to-4m-q... More seriously, I suspect that plain SQLite by itself, running in process in the main service, writing to a ':memory:' database for ultimate speed or perhaps to a local file with WAL mode for slightly more durability, could achieve pretty nice efficiency and work as the main transaction manager database.
- PaulHoule 3y agoDistributed transactions have been around for a while.
- cratermoon 3y agoacross heterogeneous data stores?
- PaulHoule 3y agoYep. https://en.wikipedia.org/wiki/Distributed_transaction https://en.wikipedia.org/wiki/Distributed_transaction
- cratermoon 3y agoTIL
- gregw2 3y agoHow heterogeneous do you mean? But basically, "yes". IBM had CICS transactions back in the day for some definition of heterogeneity. Tuxedo did this in the 80s across broader platforms. In the 90s (when I heard of it) there was an open standard created for it: https://en.wikipedia.org/wiki/X/Open_XA https://en.wikipedia.org/wiki/X/Open_XA so that Unix-y type systems had some agreement on how to do it without buying mainframe/tuxedo-y type components. And around or a bit after then, Microsoft created its own "Distributed Transaction Coordinator". This is not my area of expertise but I've heard about it throughout the last 30 years. The NoSQL side of heterogeneous is newer than the 90s but even there, I found in 2 minutes a paper from 2006, 17 years ago, on the topic: https://www.researchgate.net/profile/Akon-Dey/publication/282156834_Scalable_Distributed_Transactions_across_Heterogeneous_Stores/links/56058b9608ae5e8e3f32b98d/Scalable-Distributed-Transactions-across-Heterogeneous-Stores.pdf https://www.researchgate.net/profile/Akon-Dey/publication/28... This is definitely a wheel that gets reinvented.
- cratermoon 3y agoI'd understood distributed transactions to be a vendor-specific thing. I'd never have realized I could have IMS/DB, DB2, and MSSQL all in the same txn.
- numbsafari 3y agoAgain with the straw man example of a bank transfer. Another article you didn’t need to read because it talks about a problem you likely don’t have because the author can’t be bothered to come up with actual, real world examples.
- marktani 3y agocan you share a more helpful real world example? as someone who hasn't implemented transactions in microservices before, I have only seen the bank transfer example and it seems adequate and easy to understand - I wouldn't know what limitations it has
- numbsafari 3y agoIt doesn’t exist. This isn’t how banks transfer funds between accounts. It’s fake. As a result, you can’t actually evaluate the engineering trade offs of the proposed solution. Here’s the thing: do you even need transactions (in the sense of ACID transactions that are resolved heuristically, as described in this article) between your microservices? Likely not. And what are the implications of having to wait around for PGSQL and Mongo negotiate a transaction between a third data store? Probably blocking and latency and all the associated problems which will lead to … not use transaction across service boundaries.
- jen20 3y agoThe trick is to work out what you would do if you had to implement the entire system on paper in a 50s style office with triplicate carbon paper. Then do that, but with computers.
- yathaid 3y agoThat is not a trick, it is a false analogy. Two humans in eyesight line can be assured that they have exchanged messages successfully. Two computers cannot due to recursing acknowledgements (byzantine generals).
- zabzonk 3y agouse a tp monitor? such things have been around for a very long time. i remember writing code to interface with one in the 90s, and a bit later. but things like cics and tuxedo were there long before that.
- deleted 3y ago[deleted]
- _m8fo 3y agoAn interesting idea, but if I’m understanding the problem trying to be solved - might be better suited by durable execution (two examples being Azure’s durable functions, and Temporal.io). In practice transactions between arbitrary data stores would result in potentially boundless and unpredictable latency, no? Also, is Postgres strongly consistent and linearizable)? One alternative would be using a database with stronger consistency guarantees (Spanner is but not open source, FoundationDB is but has limitations on transactions unless you implement mvcc yourself, which to be fair you are).
- jeremyjh 3y agoConsistency is a property of distributed databases. Stock Postgres is not distributed, and thus gets strong consistency for free. There is still a concept of (transaction) isolation levels, and the ANSI SQL standard defines a transaction mode READ UNCOMMITTED that could give you inconsistent results, but Postgres ignores that and treats it as READ COMMITTED.
- _m8fo 3y agoYeah you are right - I thought the primary in this case was distributed since most of the shims were (CouchDB, Mongo, etc).
- qaq 3y agoin terms of ACID "Consistency ensures that a transaction can only bring the database from one consistent state to another, preserving database invariants: any data written to the database must be valid according to all defined rules, including constraints, cascades, triggers, and any combination thereof. This prevents database corruption by an illegal transaction. Referential integrity guarantees the primary key–foreign key relationship." So no it's not free
- jeremyjh 3y agoThat is an entirely different meaning of consistency. GP was talking about CAP theory, as they acknowledged in a follow-up comment.
- banq 3y ago[dead]
- deleted 3y ago[deleted]
- hakunin 3y agoIn most cases microservices are an anti-pattern, because they are trying to solve problems that can be solved without incurring their complexity and sacrifices. However, I remember that Sagas originally were proposed to deal with multiple 3rd party APIs. The example was "book flight, hotel, rental car". If you're dealing with necessarily separate systems, then there's probably a place for sagas.
- nextaccountic 3y agoThe author talks about this too > As soon as we start doing separation of domains according to data governance, we may quickly realize that in the vast majority of cases, microservices look a lot like traditional monoliths and the domain they should handle is big. Or realize that traditional monoliths are, in fact, microservices. This happens because most organizations have only a very limited number of truly independent domains. Most often — one.
- paulddraper 3y agoHow so? If I want to pay for and book a flight and hotel..... You have to implement distributed transactions. (Either that or YOLO.)
- numbsafari 3y agoThis is not true, and not YOLO, either. You have four separate businesses (at least): a bank, an airline, a hotel, and booking agency. There's no way that some random booking agency is getting an XA transaction (or other distributed transaction protocol) implemented between the disparate systems it is orchestrating. It has to maintain conversational state with each of those systems in the domain/model of the exposed APIs of each in order to coordinate a resolution. Even if you could do that with an distributed transaction coordinated between those disparate entities, the possible failure states of each are such that you likely wouldn't want to. For example: go ahead and book the flight and pay for it if you can, but don't rollback the whole thing if the hotel ends up needing to decline for some reason. These business models exist today as real world systems, and they aren't implemented using distributed transactions.
- jongjong 3y agoI'm a bit confused why OP says that because two-phase commits are not natively supported by various databases, that it's not a viable strategy. It's possible to implement 'two-phase commits' manually through a simple algorithm. You just need an extra field to keep track of whether or note the record is 'pending' or 'settled' and also some additional fields for creation date and settlement date. The main process just copies over all the records as 'pending' state then you have another parallel process/task which goes over all the pending records in order and changes them to 'settled' if some condition (e.g. some related records in different tables/collections also exist) is met. Records need to be settled in a particular order to protect from failure of the settlement process itself but it's not particularly difficult to implement. Also, you need good indexing for fast/efficient cross-table/collection lookups (you especially need indexing on date fields in order to quickly look up and settle records in the same order as they were inserted starting from the last settled record). I did something like this a while ago; some operations in my system inserted multiple records into different collections (into a database which did not support atomic transactions) and I needed to guarantee that when a record existed in a particular collection in the 'settled' state that it would guarantee the existence of a matching record in a different collection. It was very reliable. I believe I didn't even end up needing to launch a separate process for the settlement logic since I was using Node.js, I just used an async task scheduled to run on an interval. The front end would just ignore records until they were in the settled state (but you could also show them as pending). Another thing which really helps is using UUIDs are IDs as it guarantees idempotence so you don't have to worry about double-insertion in case of process failure/restart. You can just re-process some records that you may have already processed with no side effects; you just need to check that the error is an insertion conflict and continue.
- marklar423 3y agoHow did you handle cleaning up aborted / rolled back transactions after binaries restarted? Or did you just let the pending records hang around forever.
- unmole 3y agoWhen a node rejoins after a restart, have all other nodes drop any pending transactions initiated by the node in question. I implemented something similar for a distributed filesystem and it worked pretty well.
- hcarvalhoalves 3y ago> The canonical example of a transaction is transferring money between bank accounts: to transfer $100 from me to you, the bank runs a transaction which withdraws $100 from my account and deposits $100 in your account. By running both operations in one transaction … I know it’s canon at this point, but please stop using this example people. No bank ever worked this way, and often outsiders who hear about this example get the wrong idea.
- maxfan8 3y agoI'm curious -- how do banks actually do this, then?
- eximius 3y agoLedgers. They record a list of transactions, they don't just plus or minus the balance.
- yawaramin 3y agoBut to record a transfer with a list of transactions, you need at least two transactions, and you need to write both transactions atomically in the ledger. Otherwise you end up with the same problem.
- pstuart 3y agohttps://en.wikipedia.org/wiki/Double-entry_bookkeeping https://en.wikipedia.org/wiki/Double-entry_bookkeeping
- hakunin 3y agoThis is a checksum-like trick for manual record keeping, and isn't specifically relevant to the grandparent. It isn't required in digital world. The grandparent talks about having to write 2 separate transactions in 2 accounts in such a way that either both are successfully written, or none are written. This means that there must be transactionality between 2 services, regardless of balance being cached or not.
- stevefan1999 3y agoSo what you essentially did is to add a virtual arbiter...
- mh-cx 3y ago> We then ask the primary database for a snapshot, a list of all past primary database transactions whose effects are visible to our new transaction. I wonder what happens for records with many changes during their lifetime. Wouldn't the size of such a snapshot grow infinetely for them?
- robertkoss 3y agoInteresting to see Matei's name on the paper. Maybe Epoxy ends up at Databricks to enable updates across, e.g., Delta Lake & a traditional DWH simultaneously, for, let's say, migrations. Who knows :)
- pharmakom 3y agoAFAICT the approach is to use existing transactional store (say postgres) for coordination and use the non transactional store like dumb storage.