10 ms·
Modern Transactional Stack
- ninepoints 3y agoI'm sorry but there was a lot of jargon thrown around and circumnavigating standard problems in data consistency but no real takeaway or upshot. Felt like a chatgpt article
- geodel 3y ago> Felt like a chatgpt article I think its even worse than that.
- ly3xqhl8g9 3y agoThe takeaway seems to be "you must pay for every query and for every bit and byte we hold and transact for you", here's a list of companies you probably never heard of before (Upstash, Momento, Stellate, Xata), some with unsearchable names (Convex, Temporal, Trigger), pay them to have your database locked arbitrarily or whenever they somehow reach unicorn status and will be too big to reply to your emails. Accessing the website of any of the listed companies, the Pricing page in the header is predictably there: everyone is looking for a rent. Just to be a contrarian, because I just can't believe all these companies are making significant money, perhaps it's time for a return to the pay-for-binary model in such an Everything as a (payable) Service world: here's the blob for a nice database (or whatever), pay $1,000 (or maybe even $10,000, $100,000) and the current version is yours forever.
- creamyhorror 3y agoBut if you don't get your well-funded company to pony up for these SaaSes, highly paid dev positions at companies like these are going to dwindle, reducing labour market demand, and dev wages will fall!
- eropple 3y agoI think you're mostly right, but I will drop an asterisk: Temporal is self-hostable and is incredibly good. Like, I am pretty convinced that for the classes of problems it addresses around offline/the-customer-isn't-waiting workflow management, it is the Correct Way To Do Things. I'd never have them white-label AWS to host my stuff, but it's great.
- evntdrvn 3y agoIndeed.
- deleted 3y ago[deleted]
- JumpCrisscross 3y agoMinor quibble: i.e. stands for id est, or “that is.” It’s a re-statement or clarification. Exempli gratia (e.g.) means for example. In the diagramme, every i.e. should be an e.g.. Put another way, if you can’t put an et cetera at the end, it’s an i.e.. Wouldn’t ordinarily rise to the level of commenting. But it’s an Andreessen post, and so I’m perhaps primed for the purpose being promotion versus informing.
- Unearned5161 3y agoi always remember the difference by thinking, "(eg)sample" and "(i)n (e)other words"
- bjornsing 3y agoIt’s quite telling that not even one of the world’s top companies, dedicated to communication, has command over the English language anymore…
- paxys 3y ago> one of the world’s top companies, dedicated to communication Is there another Andreessen Horowitz that I don't know about? Because otherwise a $35B fund that invests in tech startups is very far from that description.
- glenngillen 3y agoI suspect the OP might be alluding to the fact that a16z is basically a content machine, that happens to also invest. Their content (or more charitably: their highly visible external communication) is a large part of what launched them onto the scene so quickly and had them in consideration as a top tier firm despite having no track record as a fund (GPs reputation as founders notwithstanding)
- JumpCrisscross 3y ago> content (or more charitably: their highly visible external communication) is a large part of what launched them onto the scene so quickly and had them in consideration as a top tier firm The VC predates the blog. It used to be a serious name, being an early backer of Twitter and Y Combinator. It started to go dogshit about ten years ago and criminal after 2019. They remain a fundraising powerhouse. Though the market seems to be noticing their flagging performance, with secondary funds trading at strong double-digit discounts to NAV.
- matt-attack 3y agoThe problem is very clear to me, but I'm not sure I understand the solution.
- wpnx 3y agoI was pleasantly surprised the solution wasn’t “crypto” given this is an a16z article
- politician 3y agoTLDR: ALTP is backend-as-a-service. Write your front-end application and consume from a BaaS that provides durability and strong consistency as well as business logic.
- zacksiri 3y agoI implemented something similar to the approach in this article. I was building a payment transaction system and ran into the issue of having locks and transactions in the database causing higher and higher latency as the queue got larger. I never coined a term, but I implemented an “ALTP” using in memory cache library in elixir. The library is called nebulex. It was so simple to implement. Since the library supported distribution out of the box. I had a distributed in memory transactional guarantee that worked well in a short amount of time. The cool thing was I also wrapped the database lock under the distributed in memory lock so in the case of a net split the database’s lock’s got my back. In normal circumstances it made things scalable because the in memory lock would prevent the client from connecting to the database until the in memory lock was let go. In elixir doing these things are really intuitive and enjoyable and without requiring an external dependency. I think it took me only a couple of days work design / implement and solved the problem.
- AtlasBarfed 3y agoA couple of days? For a distributed partition tolerant double wrapped locck?
- zacksiri 3y agoYeah. Couple of days. To be fair Elixir and Nebulex library did most of the heavy lifting. I had been using Nebulex for a long time before implementing this solution. But as our system scaled we ran into more and more problems so I decided to try it. Check out this video https://youtu.be/v-n8L-DiPL4 https://youtu.be/v-n8L-DiPL4
- avinassh 3y ago> The cool thing was I also wrapped the database lock under the distributed in memory lock so in the case of a net split the database’s lock’s got my back. In normal circumstances it made things scalable because the in memory lock would prevent the client from connecting to the database until the in memory lock was let go. how did you coordinate this, since both are different systems altogether? edit: I went through the video at 1.5x speed and I understand it now. This is the grand idea: 1. Use a separate distributed lock manager, get lock on the wallet. This lock is in-memory 2. Start transaction on DB and make changes 3. Release the lock 4. Other transactions wait at #1, since they can't get the lock. Thus this reduces the overall load on the DB. This is cool though!
- seanhunter 3y agoI'm guessing the authors are making an internal pitch that a16z should invest in upstash[1]? I'm not feeling it personally - the whole hypothesis seems definitely not aligned with what I've seen. I've never seen people trying to build transaction guarantees into their whole microservices stack using Fancy New Startup Tech[tm]. What I have seen a lot is people building transaction guarantees into their stack using something like event sourcing and cqrs. You pay some complexity but what you get in return is a transaction guarantee when events are processed which guarantees consistency when you need it (eg financial transactions etc), and the view part is eventually consistent but is always accurate as of a point in time. [1] https://upstash.com/about https://upstash.com/about doesn't show a16z as an investor right now.
- swyx 3y ago> I'm guessing the authors are making an internal pitch that a16z should invest in upstash thats not how it works, they already made their bets long before writing something like this (+ validated by talking to a ton of pple than can fit into this post... i know bc i've spent time with the authors haha) > What I have seen a lot is people building transaction guarantees into their stack using something like event sourcing and cqrs. You pay some complexity but what you get in return is a transaction guarantee when events are processed which guarantees consistency when you need it (eg financial transactions etc), and the view part is eventually consistent but is always accurate as of a point in time. correct. what's happening here is the rise of well tested frameworks that hopefully reduce the cost of implementation while preserving/extending the benefits (which can be very large if you go as far as temporal have done)
- JumpCrisscross 3y ago> thats not how it works, they already made their bets long before writing something like this Correct. Andreessen is unusual in that its partners have a habit of plugging their books irrespective of the underlying companies’ health. So when e.g. PG says “look, I funded this company and they’re being badass,” people listen. When these guys write this tripe, it makes me wonder who’s running out of cash on that list.
- ftlio 3y agoI figured a modern stack would involve modern tools and not be a head first dive into enterprise SaaS contracts and integration hell, but what do I know. At this point I’m diversified enough in my investments to just start agreeing. “Yes, you’re right, we’ll all be drinking our meals and everyone who uses Postgres will be fired”
- geodel 3y agoMost likely this time A16z got invested in some bottom-of-the-heap Dev tools SAAS businesses. > And a giant hairball of queues and retry logic is too brittle. To deal with this, we’ve seen, over the last few years, the emergence of new solutions that bring sanity back to transactional logic. They can be roughly categorized as either workflow-centric approaches or database-centric approaches. Ok, using databases and queues is bad. So what is good then: > we call this the application logic transactional platform (ALTP) approach because, ultimately, it extends OLTP transactions into the application Really how much I will love to get few of my brain farts funded on being brilliant insights for typical enterprise application.
- Mandatum 3y agoLike GraphQL, we just moved the integration and interface design complexity to the clients. Just publish your fucking read-only SQL credentials at this point.
- theK 3y agoTo be fair, GraphQL manages to reign in the ludicrussness that is exporting a read only SQL connection to levels that are merely unpredictable instead of outright dangerous. Still probably not the tech one needs. EDIT: add last paragraph
- Kwpolska 3y agoGraphQL does not prevent you or the GraphQL→DB translation layer from doing stupid things. Depending on the capabilities exposed by the GraphQL schema and the implementation of the translation layer, you can write a GraphQL query that is translated to 20 JOINs or N+1 hell or other bad things.
- hackandthink 3y agoGraphQL flexibility is fine but you need some discipline. We have been using GraphQL for 4 years and limit ourselves to queries with simple or medium complexity (no load the world, no problems).
- klysm 3y agoI’ll stick to Postgres
- KingOfCoders 3y ago;-) https://www.amazingcto.com/postgres-for-everything/ https://www.amazingcto.com/postgres-for-everything/
- KingOfCoders 3y agoThe prediction of the future is tough. Basically no-one predicted the AI iphone moment, everyone was suprised, the landscape changed forever in a very quick way. I usually look at the authors, because if a partner at a VC had great tech insights, they would found a startup themselves, and what I see from my coachees, most VCs just follow the herd ;-) (yes, some few have) One of the authors was a CTO though.
- swyx 3y agomartin casado is an industry legend and, yes, did found nicira :)
- KingOfCoders 3y agoYes.
- davidshepherd7 3y agoThe article was too vague for me to really understand, but I think it's suggesting something like Brandur's deeply-integrated idempotency keys design [1] except that the infra is provided by a saas company? Also somehow it supports rollbacks? Does anyone know of any other more concrete explanation of how these kinds of systems could work? [1] https://brandur.org/idempotency-keys https://brandur.org/idempotency-keys
- swyx 3y agoi attempted to do so in my high level Temporal explainer: https://www.swyx.io/why-temporal https://www.swyx.io/why-temporal and more recently they also did one at Strange Loop: https://www.youtube.com/watch?v=V_5WeVmyhzg https://www.youtube.com/watch?v=V_5WeVmyhzg
- popcorncowboy 3y agoIt's so hard to read this VC babble as anything other than lukewarm also-ran puff. The massive disclaimer at the end includes a write-off that the whole thing "may not be reliable" and "we haven't actually checked any of this", and "this might just be an advert" and if anything is wrong with it, well it's not a16z's views, it's just one of our ICs, we stand by none of it. All presented as authoritative "industry" commentary. Only to be revealed as diluted waffle, regurgitating some investment thesis as a thinly veiled advert. Thanks for the 2c a16z.
- gitfan86 3y agoTheir target audience isn't engineers but decision makers at large organizations. They want ALDC (or whatever acronym they are pushing) to become a standard part of the stack so that everyone will have to buy ALDC Ops Service or ALDC Security service. And what do you know A16Z has investments in companies that offer those SaaS products.
- SinParadise 3y agoAlso plays into the trend of "lets hire fullstack devs AKA frontend devs that knows how to call APIs and have a vague understanding of transactions and its pitfalls" and offload all backend work to SaaS products. If the startup takes off? Great! If it doesn't? At least those SaaS products offered free trials.
- swyx 3y agoseeing a lot of ad hominem comments on a16z and think that's a pretty low quality response - let's try to respect HN guidelines and discuss the substance of the article? this is an emerging space and we all lack the language to adequately describe it, so cut them a little slack there if the chosen terminology doesnt connect at first. as someone who has worked in this space (https://www.swyx.io/why-temporal https://www.swyx.io/why-temporal), here's a few quick responses: > "We’re seeing the emergence of systems that extend strong transactional guarantees beyond the database, into the distributed apps themselves." yes absolutely. but its not just transactions, but also the ability to route/distribute work, and "self provision" (https://www.swyx.io/self-provisioning-runtime https://www.swyx.io/self-provisioning-runtime) queues and timers and new application states (unconstrained by preset DAG) as you code > a giant hairball of queues and retry logic is too brittle my favorite img on how this looks (on a real diagram, from a real aws blogpost) https://twitter.com/swyx/status/1423025792783568899 https://twitter.com/swyx/status/1423025792783568899 > (temporal) will ensure the code runs to completion, even during a failure. i'd correct to "transient failure" (eg due to network or uptime issues). retries cant solve application logic issues and mostly fatten the tail of timeout failures that still need to be dealt with. fault tolerant code is a much better default, but there's no totally-free lunch here. > The downside is that it (temporal) often requires a lot of integration work and wrapper code to expose useful and safe interfaces to application developers. yes (example https://temporal.io/blog/why-rust-powers-core-sdk https://temporal.io/blog/why-rust-powers-core-sdk) - but there are lots of higher level systems built atop temporal (eg https://news.ycombinator.com/item?id=34686216 https://news.ycombinator.com/item?id=34686216 and https://www.youtube.com/watch?v=M25p1cgjM2U&list=PLl9kRkvFJrlSM-s-s-8WbgjN-R3Y4L7WG&index=6&pp=iAQB https://www.youtube.com/watch?v=M25p1cgjM2U&list=PLl9kRkvFJr...) but this is also how netflix/stripe/etc are able to customize it for their stack and encode their opinions in a way that the other alternatives mentioned would not be able to > The other approach, which is more popular with application developers (particularly Typescript/Javascript) is for the workflow engine to serve as an orchestrator of async functions (e.g. Inngest, Defer, and Trigger)... (truncated) lest i sound like a pure temporal shill, i also agree that this is an attractive model for fullstack web developers (as opposed to say backend/systems devs). this is why i got excited about Defer enough to review it (https://www.youtube.com/watch?v=iGccpHaB1hA https://www.youtube.com/watch?v=iGccpHaB1hA) > Database-centric approaches start with a database, but extend it to support arbitrary code execution to allow for workflows alongside data management. i've always used the analogy that workflow systems are "stored procedures on steroids". glad to see some signals that Convex (and Supabase?) have plans to make this available to their users. i think there'll be happy users of either approach, but personally am doubtful that the DB-centric approach will take significant share simply because 1) this stuff is not existential for them while it is for the workflow companies, so they will probably lag behind in features, investment, product/GTM focus, and 2), workflow systems easily bridge multiple systems and databases while the db-centric approach is necessarily limited to the system that db lives in. if you have a real microservices setup instead of a distributed monolith, its pretty obvious which approach will win here
- theK 3y agoIsn't the whole problem area of distributed transactions a bit of a conceptional issue? Looking at it from an event driven, eventually consistent view it seems to me that one can achieve considerable complexity gains by explicitly avoiding distributed transactions. In the past I had often the discussion that "Oh, no! Our business requirement X forces us to have atomic transactions across these 74 services". I cannot talk in absolutes but all cases I have personally seen actually didn't need distributed transactions and could (in most cases easily) be translated into event driven architectures. In the not easy cases it usually amounted to services being split badly or the IT org being too strongly present in the deployed architecture.
- yuvadam 3y agoI see the word 'distributed' being thrown around way too much in TFA without any justification as to why apps should be so distributed in the first place.
- fulafel 3y agoThis sounds a bit historyless.. We've had PostgREST since ~2015, they only indirectly acknowledge it in the form of the Supabase mention.
- Myrmornis 3y agoIt's a difficult subject, and it seems to me that the authors got confused / overreached themselves in the second half. For example, one of their diagrams shows Temporal/Cadence being used together with Zeebe/Camunda in the same backend. This would be an extremely complicated and redundant bad idea and I believe no-one with expertise in the area would consider it for a moment.
- fogzen 3y agoVC discovers transaction logs. Someone tell A16Z about Datomic.
- avinassh 3y agowhat are transaction logs?
- vinay_ys 3y agoWhoa, the lock-in is crazy with these companies. I wonder why any reasonably sized company would use such SaaS solutions.
- jatins 3y agoJust a word salad to name drop some of their portfolio companies
- twic 3y agoWow, so now we need to do what we might call "distributed transaction processing", we need some kind of "extended architecture". Wonder if there's a snappy abbreviation for that?
- chevman 3y agoI will give you a warning from BigCo land, having watched this pattern play out in various spaces/tech/industries over the past decade. When folks run out of ways to make money, they productize smaller and smaller chunks of the overall stack to attempt to extract more value/rent/payment out of their customer base. They will bring in ever more increasingly crazy sounding architecture experts to justify this (often times these folks get planted as FTEs in the very orgs they are selling to!). When you find yourself asking 'is this complexity for complexities sake', know that you are being sold vaporware :)
- SinParadise 3y agoThis is trading architectural complexity with integration complexity. You can learn to architecture better, but you can never get better at duck taping together the capricious and arbitrary APIs between hottest new techs.
- chiph 3y agoI'm sure I don't know, but why doesn't two-phase commit work across a distributed system?
- Mawr 3y ago> However, microservices and other modern application architectures introduced new complexities into application design: Developers needed to manage data across different services and ensure consistency between them, which forced them to build complex data synchronization and processing mechanisms in-house. I always wonder how many of these "modern" deployments could be replaced by a monolith + DB running on a ~single server. What are we up to these days for off-the-shelf hardware? 128 cores, 6TB+ of RAM, x NVMe SSDs in RAID? Application on one server, cache-backed DB on another. Failover to a cloned setup. Is anybody here running something like this on latest hardware and can provide rough performance metrics?