11 ms·

Moving from relational data to events

240 points by alexzeitler 3y ago

alecco 3y ago
This article is not good. Event Sourcing and the Relational Model are orthogonal. SQL:2011 added a lot of temporal features. Datomic is based on Datalog which even though is not relational it's kind of the same, and has temporal support. [2] [1] https://en.wikipedia.org/wiki/SQL:2011 https://en.wikipedia.org/wiki/SQL:2011 [2] https://vvvvalvalval.github.io/posts/2018-11-12-datomic-event-sourcing-without-the-hassle.html https://vvvvalvalval.github.io/posts/2018-11-12-datomic-even... (BTW 24 points at the top of HN and no comments? Hmm)
- nomoreusernames 3y ago
  [dead]
- ithkuil 3y ago
  Datalog is relational in the original sense of relational algebra.
  - alecco 3y ago
    Sure. But the point of the article is for Document or Object-Oriented. And the specific points would apply to Datalog, too. My point was you can do the equivalent of Event Sourcing with SQL and even better with Datomic. > In Event Sourcing Church, we’re not doing that anymore. We’re not losing business data; we keep them. We keep them as events.
- cmrdporcupine 3y ago
  Datalog is definitely relational. More so than SQL. In terms of temporal data handling & relational & Datalog, it's worth looking at Differential Datalog: https://github.com/vmware/differential-datalog https://github.com/vmware/differential-datalog
- refset 3y ago
  > Datomic [...] has temporal support Note it only supports "transaction time" (or "system time" per SQL:2011) but not "valid time" (~"event time") which is needed for a bitemporal data model. [1] [1] https://vvvvalvalval.github.io/posts/2017-07-08-Datomic-this-is-not-the-history-youre-looking-for.html https://vvvvalvalval.github.io/posts/2017-07-08-Datomic-this...
- hot_gril 3y ago
  You don't need specific temporal support for an event-oriented DB, nor would I want it. I usually design a vanilla relational schema around events regardless of which DBMS I'm using. E.g. instead of an "order" table with multiple states updated in-place, I'd have "order_placed" and "order_filled" where each row is an event, insert-only.
- deleted 3y ago
  [deleted]
ekorz 3y ago
I’ve never before seen a post hit #1 here after 30 minutes with zero comments. Is that normal?
- deleted 3y ago
  [deleted]
- alexzeitler 3y ago
  Funny, I had the same thought
- rockwotj 3y ago
  Someone got around the upvote ring detection? I know there are lots of folks that claim to know how to get around it, but agreed it seems sketchy
- jddj 3y ago
  To give the benefit of the doubt, maybe it was an intriguing title? I didn't personally think that the style of the writing or the content itself was particularly good, but event sourcing has an almost narcotic appeal. ES (nosql did too) has this wild ability to slip past scepticism. It feels right, I think because it milks some dopamine by feeling both simpler in a rewarding abstract sense and sufficiently (for Devs) complex in a day to day operations sense.
- Tao3300 3y ago
  I've got nothing smart to say about databases on a Saturday
- alecco 3y ago
  And at 1 hour it's still at the top in spite of all the negative comments. I'm assuming it got flagged by enough of us and yet it prevails. Odd.
- deleted 3y ago
  [deleted]
candiddevmike 3y ago
Where's the hitchhikers guide to moving back to relational data after our whiz bang dev made us event driven and left for a new opportunity?
- tarruda 3y ago
  This reminds me of https://youtu.be/b2F-DItXtZs https://youtu.be/b2F-DItXtZs
- jtwebman 3y ago
  He tried at the beginning to say only when the DB gets too big should you even think about this but there are things you can do there as well.
  - Nextgrid 3y ago
    The way I see it, either your business domain requires querying over a large amount of data, or it doesn't. If an application allows someone to be able to enter let's say an order number from anywhere in the world from the last 10 years and be able to find the order, there is no magic - some server out there is going to have to scan a huge amount of data to find a match. Tricks such as indexes, partitioned tables, etc can be employed, but those tricks have nothing to do with event-sourcing and are independent of it.
    - mrkeen 3y ago
      > Tricks such as indexes, partitioned tables, etc can be employed, but those tricks have nothing to do with event-sourcing and are independent of it. You might want to use different tricks in different situations. Different situations means different services, and different tricks means different storage/query technologies. So how do you get your data into three systems - and more crucially - keep them in sync? Webhooks? Triggers? Some bidirectional sync magic app that claims to beat CAP? Just use event-sourcing (append-only, disallow modification) and the multiple systems will stay in sync as long as they know how to process one more message.
colonwqbang 3y ago
The concept sounds interesting, but the article doesn't do a great job of explaining how it works. How do I efficiently reconstruct the current state from the event stream? How would the event stream be modelled in the database?
- alexzeitler 3y ago
  There are several talks by the author: https://www.youtube.com/watch?v=gG6DGmYKk4I https://www.youtube.com/watch?v=gG6DGmYKk4I https://www.youtube.com/watch?v=jnDchr5eabI https://www.youtube.com/watch?v=jnDchr5eabI https://www.youtube.com/watch?v=ArcypYS5XBQ https://www.youtube.com/watch?v=ArcypYS5XBQ https://www.youtube.com/watch?v=uODSwR2CIV4 https://www.youtube.com/watch?v=uODSwR2CIV4 He also maintains samples on GitHub: https://github.com/oskardudycz/EventSourcing.NetCore https://github.com/oskardudycz/EventSourcing.NetCore https://github.com/oskardudycz/EventSourcing.NodeJS https://github.com/oskardudycz/EventSourcing.NodeJS https://github.com/oskardudycz/EventSourcing.JVM https://github.com/oskardudycz/EventSourcing.JVM
- corethree 3y ago
  Two ways to do it. 1. Use a database designed for this stuff. Google big query, Amazon redshift, clickhouse..etc. all current data is essentially a type of aggregation. Or in other words it's equivalent to a group-by query on an event database. It makes sense right? With events I can technically rebuild the current state or the past state of the data through some aggregation query. 2. Rename your relational storage and call it a caching layer that lives next to the event system. It's functionally the same thing but won't trigger any red flags in people who are obsessed with making everything event driven. The architecture he describes exists. It's just massively complicated so services that utilize it usually do very targeted things. Think Google analytics, data dog, splunk... etc. Etc.
- mrkeen 3y ago
  > How do I efficiently reconstruct the current state from the event stream? There isn't one 'current state'. That thinking comes from centralising everything in one DB. You create different states in different systems according to different requirements. If you're building a shopping system, with Purchases and Customers, one service could read events and produce a relational table for finance purposes. Another service could read events and produce a key-value store of customer data. A third service could power an OpenSearch service for searching over products. > How would the event stream be modelled in the database? It's a list. If you're using something fit-for-purpose like Kafka, then it's multiple lists (topics, partitions, etc.).
2023throwawayy 3y ago
> Once you distinguish all events you’re fine with and want to migrate your relational data, don’t try to cheat; don’t put your events as small and granular. Relational data is flattened; if you try to retrofit what happened from the final state, you will likely fail or not be precise at best. .. what?
- mrkeen 3y ago
  I think I follow. The article is aimed at people who already have relational data, and want to build an event-driven system (whose events will eventually end up as relational data again downstream.) Your system A might look like: | Name | Balance | | Michael | $3.03 | You might design your system B to have the events AmountCredited{name, amount} and AmountDebited{name, amount}. You don't know how system A ended up at its current state. That's what's meant by "flattened". When you want to "migrate the relational data", i.e. convert system A's relations into events, it's tempting to use the obvious AmountCredited{"Michael", $3.03} because you know it will result in Michael having the correct balance in the final system. But it's not good to reuse AmountCredited, _because no such event actually happened_, which is why it could be called "cheating". If future-you looks at historical data, $3.03 won't correspond to any real transaction that happened. Instead you should instead make a special event like AmountImported{name, amount}.
  - lelanthran 3y ago
    For this example the convention in accounting is to use 'balance brought forward'. It's a real transaction that happened, and everyone knows what it means i.e. the previous ledger with this account was closed off and the new one has the balance that was there when th book was closed. Using 'imported' describes what you did, but not what the intention was.
- deleted 3y ago
  [deleted]
bananaowl 3y ago
The architecture from hell. Hard to debug. Why didn't my service pick up that event? Error handling. What happens with the state if a service throws an exception? Resource hog. How do I map/reduce all these events into a state? I do like events. They go into my elk stack where I can look at pretty graphs which gives me a story of how my system behaves over time.
- deleted 3y ago
  [deleted]
- mrkeen 3y ago
  > Why didn't my service pick up that event? What event? If you're not event-sourcing, you can't even ask that question. Instead, some user interacted with the system and got a 500. Maybe you got a stack trace in the logs, but the user's data is gone (or maybe half of it was stored). Persistent events mean you get to fix the problem and try again - no data lost. > What happens with the state if a service throws an exception? Whatever you programmed it to do. Same as any other service. > How do I map/reduce all these events into a state? There's no one way to store them like there is in an RDBMS. UserService listens to User events. TransactionService listens to Transaction events. SearchService listens to both. All three have different schemas. Those schemas can be thrown away and redesigned at any time without data loss. Do a bad job of map/reducing them into a state today. Do it better tomorrow.
xwowsersx 3y ago
I was looking forward to reading this based solely on the title, but I find the writing style and tone to be quite unbearable. The forced attempt at being relatable and light-hearted comes across as patronizing and distracts from the intended message or points being conveyed.
- deleted 3y ago
  [deleted]
- hot_gril 3y ago
  I don't take offense to the tone, it's just too much text and too little substance.
move-on-by 3y ago
I was a team once that strongly considered event sourcing. To me, it seemed like a solution looking for a problem. It could have worked for us, but we ended up passing on it as the benefits were not immediately clear and the risk of doing something new and the lessons learned that would come with it just didn’t seem in the best interest of the project/company. Maybe that makes us tools for passing up a learning opportunity, but I don’t regret getting into that rabbit hole without a fox chasing us down it.
- Nextgrid 3y ago
  A boring, conventional system that works is a threat to a bloated engineering team who don't have any work to do & polish their resumes with and might feel at threat of redundancy. That is the "problem" this solution solves.
- alecco 3y ago
  Temporal databases make a lot of sense for financial data, for example. But in most cases you can just have a normal database and store the historic changes in auxiliary tables. So the main database is kind of a materialized view.
  - GartzenDeHaes 3y ago
    The traditional way to handle this is to use a GAAP style transaction/journal and a roll-up summary table. The current state can be reconstructed from the read only transaction table and you don't need any complex event processing system.
    - theteapot 3y ago
      > The current state can be reconstructed from the read only transaction table .. Isn't that called event-sourcing?
- devjab 3y ago
  Almost every piece of data we store in SQL would be better on a document database, but since nobody is familiar with those we keep on trucking. I don’t mind too much, I don’t even think we made the wrong choice, but it does cause us some issues with how we have to handle data model changes. I think most data storage didn’t really keep pace with how a lot of software is being build now though, and things like events and queues are what we build on top of what we have because we need it. For the most part a lot of the data relations happen outside of our databases today through various services, because that’s just how the modern IT landscape looks in many organisations. You’ll have internal master data that supports different teams in the business and interacts with 300+ different IT systems and applications in order to streamline things. With micro services it’s easy to keep the business logic and data models clean, but then you need to manage events, queues and data states as well as reliant storage. Which is just so complicated right now. I do like SQL but these days, the systems we’re building could frankly be put in a SQLite and be perfectly fine, well almost.
zabzonk 3y ago
all the comments are negative, but the post at this time 64 upvotes - why? i've seen this so often on HN, but i really don't understand it.
- simonbarker87 3y ago
  Hate the article enough to leave a negative comment, want to see the fall out of it and have it not drop off the front page, so stick an upvote on it as well would be my guess. I have commented on this but haven’t upvoted as it clearly a bonkers article.
- jupp0r 3y ago
  Because you can't downvote submissions, you can only go in and write a negative comment. Makes complete sense to me to see the effects of that with this article.
  - zabzonk 3y ago
    yes, i'm not denying the comments, it's the upvotes that i don't understand.
    - jupp0r 3y ago
      There are probably some people who find the article interesting and well written or who upvote based solely on the headline without reading it.
- lolinder 3y ago
  Certain topics get a lot of interest (positive and negative) based on the title alone. Event-driven isn't quite peak hype any more, but it still gets a lot of instinctive love from a certain group of people, and a lot of instinctive hate from another. So you get a whole bunch of upvotes (but they don't have anything substantial to say about it), then a whole bunch of negative reactions in the comments based on the title alone. And then in this case, you get a whole bunch of negative reactions from people who tried to read the article and couldn't get past the weird tone.
simonbarker87 3y ago
What on earth is this article trying to accomplish? The tone is bizarre and the underlying concept sounds horrendous to work with if you truly want to replace your static data store with it. By all means add a formal event layer on top of your existing data store but to replace it sounds madness. If that’s not what the article is proposing then for once I’m going to say it’s not a failure of my intelligence, it’s the articles fault here.
- AtlasBarfed 3y ago
  Four years ago I heard "Kafka IS your database". I thought maybe these insane people (probably parroting some tech company enterprise penetration propaganda e.g. Confluent) would have a better story, but... no. Anyway, yeah, sure, keep logs. But a lot of that article about commands and events is something that only exists if you had one ubiquitous language, system, and OS. You know, the almost literal "seamless" where there aren't any seams. Sure that will probably plug into some enterprise bus and enterprise integration and enterprise ... anyway. Competent developers will understand what events to preserve and log and possibly allow retry/repeats. Anyone who has looked at a Splunk bill will realize that just storing all the logs everywhere is very expensive, which is another way of saying "wasteful". But any generic enterprisey event system will basically start and end with splunk-level log aggregation and kinda-analysis.
  - dgellow 3y ago
    Dealt with something like this when I joined a previous job years ago, Kafka was the main data store, microservices had their state in memory they would build at startup by processing their whole history of events (that was ridiculously slow for some of them). Then GDPR came in place and the whole “keep everything in Kafka forever” had to face the reality of “not allowed to keep PII for longer than 30 days” :) (Before someone suggests it, no messages weren’t encrypted, just throwing the key away wasn’t an option)
    - jabradoodle 3y ago
      Would be expensive but you can do a copy and replace to keep the data you need, on a new topic. If you don't need the data, then you don't need it.
freecodyx 3y ago
Sounds like a junior work
nivertech 3y ago
It's top-down vs bottom-up, or custom vs generic. Top-down vs bottom-up: Top-down: starting from the business domain, and then mapping an implementation onto available technologies, tools, and vendors. Bottom-up: starting from the available technologies, tools, and vendors, and thinking how to bolt up a working solution out of them. Custom vs generic: Custom: DDD, CQRS/ES, Sagas, TBUI (Task-based/driven UI), GraphQL, Algebraic Data Types, etc. Generic: RDBMS, CRUD, REST, ACID transactions, CDC, generic admin UIs, nocode/lowcode, limited/generic types, etc.
lngnmn2 3y ago
[dead]
asah 3y ago
2c: if you need PostgreSQL elsewhere in your app anyway, then store your event data in PostgreSQL + FOSS reporting tools (apache superset, metabase, etc) until you hit ~2TB. After that, decide if you need 2TB online or just need daily/hourly summaries - if so, stick with PostgreSQL forever[1]. I have one client with 10TB+ and 1500 events per sec @ 600 bytes/rec (80GB/day before indexing), 2 days of detail online and the rest summarized and details moved to S3 where they can still query via Athena SQL[2]. They're paying <$2K for everything, including a reporting portal for their clients. AWS RDS multi-AZ with auto-failover (db.m7g.2xlarge) serving both inserts and reporting queries at <2% load. One engineer spends <5 hours per MONTH maintaining everything, in part because the business team builds their own charts/graphs. Sure, with proprietary tools you get a dozen charts "out of the box" but with pgsql, your data is one place, there's one system to learn, one system to keep online/replicate/backup/restore, one system to secure, one system to scale, one vendor (vendor-equivalent) to manage and millions of engineers who know the system. Building a dozen charts takes an hour in systems like preset or metabase, and non-technical people can do it. Note: I'm biased, but over 2 decades I've seen databases and reporting systems come & go, and good ol' PostgreSQL just gets better every year. https://instances.vantage.sh/aws/rds/db.m7g.2xlarge?region=us-east-1&os=PostgreSQL&cost_duration=monthly&reserved_term=Standard.partialUpfront https://instances.vantage.sh/aws/rds/db.m7g.2xlarge?region=u... [1] if you really need, there's PostgreSQL-compatible systems for additional scaling: Aurora for another 3-5x scaling, TimescaleDB for 10x, CitusDB for 10x+. With each, there's tradeoffs for being slightly-non-standard and thus I don't recommend using them until you really need. [2] customer reporting dashboards require sub-second response, which is provided by PostgreSQL queries to indexed summary tables; Athena delivers in 1-2 sec via parallel scans.
- datadeft 3y ago
  When a comment on HN has more merit than the article. The only problem with postgres is that inserting has some interesting scaling problems. Putting a queue between the event sources and the db is usually recommended.
  - withinboredom 3y ago
    > Putting a queue between the event sources and the db is usually recommended. Emphasis on _usually_. If your db is at 2% CPU utilization with/without queues ... you probably don't need a queue.
ChicagoDave 3y ago
I'm on board with event-based architectures, but this article struggles to get its point across. I would focus on the difference between data relations and business behaviors. Once you start thinking in terms of behaviors and business activities, the move away from operational relational data stores becomes much more obvious.
- 3abiton 3y ago
  On an abstract level, events can be modelled as relations.
wrestlingmonkey 3y ago
Lot of negative comments so thought I should counter as I quite liked it. The article style isn’t great but I get the impression it was only ever meant to be a toe dip into one aspect of thinking about events. I’ve worked at a couple of places that have had a lot of success with EDA, DDD and microservices. I’m sure these patterns are considered by many as buzzwords but they’ve been around for a long, long time now and they can be very effective. There’s no such thing as a silver bullet but the problems these ideas look to solve I’ve experienced very often, so I don’t buy into it being a solution in search of a problem.
- deleted 3y ago
  [deleted]
CodeCompost 3y ago
Didn't read the article but I'm in the process of eradicating Event Sourcing from a codebase and returning to the classical ACID database model. The boneheaded decisions made by our predecessors is staggering and choosing to use Event Sourcing for everything is the dumbest of them all.
- politician 3y ago
  What are the major deficiencies of ES in that particular codebase? If you could drop some specifics in bullets that would be really helpful for me. I promise not to ambush you with apologetics.
  - CodeCompost 3y ago
    This system was developed from 2008-2013, a very different time when hardware was "cheap" and the Cloud was not not a thing. Event Sourcing dictates that Events are never deleted which means that the data volume keeps growing and growing. There is - in this system - no way to delete old events. When I brought this up, the response was "Just add another hard drive". In the modern Cloud era, adding a hard drive is extremely expensive. The system uses CQRS and all events generate reports that are stored in Elasticsearch. Data is never deleted, only an extra event gets added /saying/ it's deleted. The data is still cached in Elasticsearch. All of it, all the data back to 2013. Added an extra 32GB of RAM just to keep up with it is ludicrously expensive. We're in Europe. Guess what a system like this does to GDPR. Can you tell me which events I need to delete when somebody says they want to be forgotten? Yeah. I can't delete old data. It's impossible without collapsing the entire system like a house of cards. Finally, and this is the piece de resistance, the developers decided to develop a relational database structure ON TOP OF EVENT SOURCING. We're talking primary keys, foreign keys, cascading and non-cascading deletes. Importing an Excel sheet of 10000 rows takes TWO WEEKS because it generates hundreds of events per Excel cell that is being read. We brought it down to 10 minutes and there is plenty of room for improvement it's just that we have other priorities right now. Currently, simple flat "tables" with no foreign keys take several seconds to import (just like in a regular RDMS). Oh yeah note that this is a system that is used by 4-5 users at a time, not hundreds or thousands of users.
revskill 3y ago
No, what you need is a command queue, command event is not domain event.
ulrischa 3y ago
I made a php demo for a idea: a event based observer based modelling system. I.e. for game of life: https://github.com/ulrischa/OCell https://github.com/ulrischa/OCell
audnaun252 3y ago
Modelling domain events is useful for describing the problem your trying to solve with the domain experts, and it should probably be left in the documentation when planning a solution. For actually implementing a system that provides an audit trail of long-lived state machines, you're probably better off using something like Temporal.io/durable functions which uses event sourcing internally for their persistence, and has a programming model which forces you to think about deduplication/idempotency by adding different constraints for the code that orchestrates the functionality (workflows), vs the code that actually interacts with the real world (activities)
- DenisM 3y ago
  Durable functions suffer from lack of Observability tho. I’d love to hear suggestions on overcoming this issue.
  - mfateev 3y ago
    temporal.io just released .NET SDK. The observability and scalability of the platform is really good. Disclaimer: I'm one of the founders of the project.
    - DenisM 3y ago
      What I meant specifically is that the current state of a workflow is stored in a format that’s opaque to any component other than the workflow itself. E.g. if I have a “shopping cart checkout” workflow and the user is not making progress, how can I can I tell which step of the workflow the user is stuck at?
      - audnaun252 3y ago
        The function's event data and current state is all stored in table storage, so you could query that - I'd expect you'd need to query an event-store-based solution in a similar way?
      - mfateev 3y ago
tiku 3y ago
Yeah, ehh I'm just going to stick to good old fashioned relational data.
- agentultra 3y ago
  Good, do it until you can’t. Don’t use a hammer on a screw.
kinddevil 3y ago
Question has been asked every time when someone wants to use event sourcing -- why do you want to redo the stuff that databases already did? "Ensure your design works if scale changes by 10X or 20X, not 100x".
- baq 3y ago
  because the database is inconsistent and incorrect is usually the answer.
onion-soup 3y ago
IT is doomed
asimpletune 3y ago
Something that might be missing from these discussions is when event driven architecture is even appropriate. The short answer is if your customer did something and expects a response it's not even driven, that's just request/response. Event driven is when something happens out of band. E.g. you push your code to GH, which triggered a build. In this example, you reloading the page to see that your updated code is request/response, however that CI build that was enqueued is event driven. Hope that helps.
- berkes 3y ago
  It's not that Simple. Request-response isn't a factor to choose ES or ED architectures on. You can have request-response, inline, blocking, cycles with ES or ED. And you can have async without ES or ED just fine too (e.g. workers, queues, actors, multithreaded etc)
  - asimpletune 3y ago
    > It's not that Simple. It can be as complicated as you want to make it. My original point was to supply the missing context that explains 95% of use cases. > Request-response isn't a factor to choose ES or ED architectures on. One's business use case and how it translates into an appropriate messaging pattern is arguably the most important factor to base an architecture on. > You can have request-response, inline, blocking, cycles with ES or ED [...] Yes, of course. And it's fair to say, at any point in your life, that one can do anything that one wants. Like, one could have rollerblades for hands, or one could implement request/response functionality with an event driven system. But once we've moved past that, probably the next order of business is to justify the extremely good reason for overriding all prevailing common sense.
    - berkes 3y ago
      No need to be condescending. ES and ED architectures have good use-cases. And CRUD has good use-cases, and RDMSes and time series and document databases. Non will ever fit perfectly and non are a silver bullet. Choosing the right architecture is an an art. Experience helps. And being able to look into the future even more. Point being: a silly parameter as "wether you need the response to a request to have the data" is certainly not the only metric to base architectural choices on. It's not even a significant metric. It's at most a technical requirements. And one that can be fulfilled with many architectures. Like event sourced architectures even.
jstummbillig 3y ago
I was not aware of event driven design until recently, but coincidentally concluded something like it, after considering the optimal data structure in an AI powered world. While it's clear how event driven design might have been worth the trade off (assuming you were able manage the complexities and actually made use of the data) being able to query an AI with knowledge of every event that happened to your business, with will make it ubiquitous over the coming years.
erikpukinskis 3y ago
Event sourcing has a lot of nice properties, so I’m intrigued. But don’t you still need relations? And then how do you implement those? If the answer is “they’re all implicit in the application layer code” then that’s not really acceptable. I still need some way to query for relations, or keep relation views up to date, or something like that. I don’t mind if relations are not core to your persistence model, but they have to be implemented _somewhere_ in your data layer, and I’m not seeing any mention of that here. I have the same issue with Firestore, everyone does relations _somehow_ but it’s all just spaghetti application code which isn’t scalable.
- mdm12 3y ago
  In event sourced systems, you project the event stream into read models, of which there can be many (relational, time series, etc.) If you're familiar with functional programming, it is essentially a fold operation over the stream of events into a single state. Having worked with event sourced systems in the past, there are benefits in having a persisted explicit event history, but there is much added complexity (how do those read models actually get generated? how do you version the model? do you have snapshots of your read models?). In my experience, the additional complexity was not worth it for most contexts in which the pattern was applied...

240 points by alexzeitler 3y ago

This article is not good. Event Sourcing and the Relational Model are orthogonal. SQL:2011 added a lot of temporal features. Datomic is based on Datalog which even though is not relational it's kind of the same, and has temporal support. [2] [1] https://en.wikipedia.org/wiki/SQL:2011 https://en.wikipedia.org/wiki/SQL:2011 [2] https://vvvvalvalval.github.io/posts/2018-11-12-datomic-event-sourcing-without-the-hassle.html https://vvvvalvalval.github.io/posts/2018-11-12-datomic-even... (BTW 24 points at the top of HN and no comments? Hmm)

nomoreusernames 3y ago
[dead]
ithkuil 3y ago
Datalog is relational in the original sense of relational algebra.
- alecco 3y ago
  Sure. But the point of the article is for Document or Object-Oriented. And the specific points would apply to Datalog, too. My point was you can do the equivalent of Event Sourcing with SQL and even better with Datomic. > In Event Sourcing Church, we’re not doing that anymore. We’re not losing business data; we keep them. We keep them as events.
cmrdporcupine 3y ago
Datalog is definitely relational. More so than SQL. In terms of temporal data handling & relational & Datalog, it's worth looking at Differential Datalog: https://github.com/vmware/differential-datalog https://github.com/vmware/differential-datalog
refset 3y ago
> Datomic [...] has temporal support Note it only supports "transaction time" (or "system time" per SQL:2011) but not "valid time" (~"event time") which is needed for a bitemporal data model. [1] [1] https://vvvvalvalval.github.io/posts/2017-07-08-Datomic-this-is-not-the-history-youre-looking-for.html https://vvvvalvalval.github.io/posts/2017-07-08-Datomic-this...
hot_gril 3y ago
You don't need specific temporal support for an event-oriented DB, nor would I want it. I usually design a vanilla relational schema around events regardless of which DBMS I'm using. E.g. instead of an "order" table with multiple states updated in-place, I'd have "order_placed" and "order_filled" where each row is an event, insert-only.
deleted 3y ago
[deleted]

ekorz 3y ago

I’ve never before seen a post hit #1 here after 30 minutes with zero comments. Is that normal?

deleted 3y ago
[deleted]
alexzeitler 3y ago
Funny, I had the same thought
rockwotj 3y ago
Someone got around the upvote ring detection? I know there are lots of folks that claim to know how to get around it, but agreed it seems sketchy
jddj 3y ago
To give the benefit of the doubt, maybe it was an intriguing title? I didn't personally think that the style of the writing or the content itself was particularly good, but event sourcing has an almost narcotic appeal. ES (nosql did too) has this wild ability to slip past scepticism. It feels right, I think because it milks some dopamine by feeling both simpler in a rewarding abstract sense and sufficiently (for Devs) complex in a day to day operations sense.
Tao3300 3y ago
I've got nothing smart to say about databases on a Saturday
alecco 3y ago
And at 1 hour it's still at the top in spite of all the negative comments. I'm assuming it got flagged by enough of us and yet it prevails. Odd.
deleted 3y ago
[deleted]

candiddevmike 3y ago

Where's the hitchhikers guide to moving back to relational data after our whiz bang dev made us event driven and left for a new opportunity?

tarruda 3y ago
This reminds me of https://youtu.be/b2F-DItXtZs https://youtu.be/b2F-DItXtZs
jtwebman 3y ago
He tried at the beginning to say only when the DB gets too big should you even think about this but there are things you can do there as well.
- Nextgrid 3y ago
  The way I see it, either your business domain requires querying over a large amount of data, or it doesn't. If an application allows someone to be able to enter let's say an order number from anywhere in the world from the last 10 years and be able to find the order, there is no magic - some server out there is going to have to scan a huge amount of data to find a match. Tricks such as indexes, partitioned tables, etc can be employed, but those tricks have nothing to do with event-sourcing and are independent of it.
  - mrkeen 3y ago
    > Tricks such as indexes, partitioned tables, etc can be employed, but those tricks have nothing to do with event-sourcing and are independent of it. You might want to use different tricks in different situations. Different situations means different services, and different tricks means different storage/query technologies. So how do you get your data into three systems - and more crucially - keep them in sync? Webhooks? Triggers? Some bidirectional sync magic app that claims to beat CAP? Just use event-sourcing (append-only, disallow modification) and the multiple systems will stay in sync as long as they know how to process one more message.

colonwqbang 3y ago

The concept sounds interesting, but the article doesn't do a great job of explaining how it works. How do I efficiently reconstruct the current state from the event stream? How would the event stream be modelled in the database?

alexzeitler 3y ago
There are several talks by the author: https://www.youtube.com/watch?v=gG6DGmYKk4I https://www.youtube.com/watch?v=gG6DGmYKk4I https://www.youtube.com/watch?v=jnDchr5eabI https://www.youtube.com/watch?v=jnDchr5eabI https://www.youtube.com/watch?v=ArcypYS5XBQ https://www.youtube.com/watch?v=ArcypYS5XBQ https://www.youtube.com/watch?v=uODSwR2CIV4 https://www.youtube.com/watch?v=uODSwR2CIV4 He also maintains samples on GitHub: https://github.com/oskardudycz/EventSourcing.NetCore https://github.com/oskardudycz/EventSourcing.NetCore https://github.com/oskardudycz/EventSourcing.NodeJS https://github.com/oskardudycz/EventSourcing.NodeJS https://github.com/oskardudycz/EventSourcing.JVM https://github.com/oskardudycz/EventSourcing.JVM
corethree 3y ago
Two ways to do it. 1. Use a database designed for this stuff. Google big query, Amazon redshift, clickhouse..etc. all current data is essentially a type of aggregation. Or in other words it's equivalent to a group-by query on an event database. It makes sense right? With events I can technically rebuild the current state or the past state of the data through some aggregation query. 2. Rename your relational storage and call it a caching layer that lives next to the event system. It's functionally the same thing but won't trigger any red flags in people who are obsessed with making everything event driven. The architecture he describes exists. It's just massively complicated so services that utilize it usually do very targeted things. Think Google analytics, data dog, splunk... etc. Etc.
mrkeen 3y ago
> How do I efficiently reconstruct the current state from the event stream? There isn't one 'current state'. That thinking comes from centralising everything in one DB. You create different states in different systems according to different requirements. If you're building a shopping system, with Purchases and Customers, one service could read events and produce a relational table for finance purposes. Another service could read events and produce a key-value store of customer data. A third service could power an OpenSearch service for searching over products. > How would the event stream be modelled in the database? It's a list. If you're using something fit-for-purpose like Kafka, then it's multiple lists (topics, partitions, etc.).

2023throwawayy 3y ago

> Once you distinguish all events you’re fine with and want to migrate your relational data, don’t try to cheat; don’t put your events as small and granular. Relational data is flattened; if you try to retrofit what happened from the final state, you will likely fail or not be precise at best. .. what?

mrkeen 3y ago
I think I follow. The article is aimed at people who already have relational data, and want to build an event-driven system (whose events will eventually end up as relational data again downstream.) Your system A might look like: | Name | Balance | | Michael | $3.03 | You might design your system B to have the events AmountCredited{name, amount} and AmountDebited{name, amount}. You don't know how system A ended up at its current state. That's what's meant by "flattened". When you want to "migrate the relational data", i.e. convert system A's relations into events, it's tempting to use the obvious AmountCredited{"Michael", $3.03} because you know it will result in Michael having the correct balance in the final system. But it's not good to reuse AmountCredited, _because no such event actually happened_, which is why it could be called "cheating". If future-you looks at historical data, $3.03 won't correspond to any real transaction that happened. Instead you should instead make a special event like AmountImported{name, amount}.
- lelanthran 3y ago
  For this example the convention in accounting is to use 'balance brought forward'. It's a real transaction that happened, and everyone knows what it means i.e. the previous ledger with this account was closed off and the new one has the balance that was there when th book was closed. Using 'imported' describes what you did, but not what the intention was.
deleted 3y ago
[deleted]

bananaowl 3y ago

The architecture from hell. Hard to debug. Why didn't my service pick up that event? Error handling. What happens with the state if a service throws an exception? Resource hog. How do I map/reduce all these events into a state? I do like events. They go into my elk stack where I can look at pretty graphs which gives me a story of how my system behaves over time.

deleted 3y ago
[deleted]
mrkeen 3y ago
> Why didn't my service pick up that event? What event? If you're not event-sourcing, you can't even ask that question. Instead, some user interacted with the system and got a 500. Maybe you got a stack trace in the logs, but the user's data is gone (or maybe half of it was stored). Persistent events mean you get to fix the problem and try again - no data lost. > What happens with the state if a service throws an exception? Whatever you programmed it to do. Same as any other service. > How do I map/reduce all these events into a state? There's no one way to store them like there is in an RDBMS. UserService listens to User events. TransactionService listens to Transaction events. SearchService listens to both. All three have different schemas. Those schemas can be thrown away and redesigned at any time without data loss. Do a bad job of map/reducing them into a state today. Do it better tomorrow.

xwowsersx 3y ago

I was looking forward to reading this based solely on the title, but I find the writing style and tone to be quite unbearable. The forced attempt at being relatable and light-hearted comes across as patronizing and distracts from the intended message or points being conveyed.

deleted 3y ago
[deleted]
hot_gril 3y ago
I don't take offense to the tone, it's just too much text and too little substance.

move-on-by 3y ago

I was a team once that strongly considered event sourcing. To me, it seemed like a solution looking for a problem. It could have worked for us, but we ended up passing on it as the benefits were not immediately clear and the risk of doing something new and the lessons learned that would come with it just didn’t seem in the best interest of the project/company. Maybe that makes us tools for passing up a learning opportunity, but I don’t regret getting into that rabbit hole without a fox chasing us down it.

Nextgrid 3y ago
A boring, conventional system that works is a threat to a bloated engineering team who don't have any work to do & polish their resumes with and might feel at threat of redundancy. That is the "problem" this solution solves.
alecco 3y ago
Temporal databases make a lot of sense for financial data, for example. But in most cases you can just have a normal database and store the historic changes in auxiliary tables. So the main database is kind of a materialized view.
- GartzenDeHaes 3y ago
  The traditional way to handle this is to use a GAAP style transaction/journal and a roll-up summary table. The current state can be reconstructed from the read only transaction table and you don't need any complex event processing system.
  - theteapot 3y ago
    > The current state can be reconstructed from the read only transaction table .. Isn't that called event-sourcing?
devjab 3y ago
Almost every piece of data we store in SQL would be better on a document database, but since nobody is familiar with those we keep on trucking. I don’t mind too much, I don’t even think we made the wrong choice, but it does cause us some issues with how we have to handle data model changes. I think most data storage didn’t really keep pace with how a lot of software is being build now though, and things like events and queues are what we build on top of what we have because we need it. For the most part a lot of the data relations happen outside of our databases today through various services, because that’s just how the modern IT landscape looks in many organisations. You’ll have internal master data that supports different teams in the business and interacts with 300+ different IT systems and applications in order to streamline things. With micro services it’s easy to keep the business logic and data models clean, but then you need to manage events, queues and data states as well as reliant storage. Which is just so complicated right now. I do like SQL but these days, the systems we’re building could frankly be put in a SQLite and be perfectly fine, well almost.

zabzonk 3y ago

all the comments are negative, but the post at this time 64 upvotes - why? i've seen this so often on HN, but i really don't understand it.

simonbarker87 3y ago
Hate the article enough to leave a negative comment, want to see the fall out of it and have it not drop off the front page, so stick an upvote on it as well would be my guess. I have commented on this but haven’t upvoted as it clearly a bonkers article.
jupp0r 3y ago
Because you can't downvote submissions, you can only go in and write a negative comment. Makes complete sense to me to see the effects of that with this article.
- zabzonk 3y ago
  yes, i'm not denying the comments, it's the upvotes that i don't understand.
  - jupp0r 3y ago
    There are probably some people who find the article interesting and well written or who upvote based solely on the headline without reading it.
lolinder 3y ago
Certain topics get a lot of interest (positive and negative) based on the title alone. Event-driven isn't quite peak hype any more, but it still gets a lot of instinctive love from a certain group of people, and a lot of instinctive hate from another. So you get a whole bunch of upvotes (but they don't have anything substantial to say about it), then a whole bunch of negative reactions in the comments based on the title alone. And then in this case, you get a whole bunch of negative reactions from people who tried to read the article and couldn't get past the weird tone.

simonbarker87 3y ago

What on earth is this article trying to accomplish? The tone is bizarre and the underlying concept sounds horrendous to work with if you truly want to replace your static data store with it. By all means add a formal event layer on top of your existing data store but to replace it sounds madness. If that’s not what the article is proposing then for once I’m going to say it’s not a failure of my intelligence, it’s the articles fault here.

AtlasBarfed 3y ago
Four years ago I heard "Kafka IS your database". I thought maybe these insane people (probably parroting some tech company enterprise penetration propaganda e.g. Confluent) would have a better story, but... no. Anyway, yeah, sure, keep logs. But a lot of that article about commands and events is something that only exists if you had one ubiquitous language, system, and OS. You know, the almost literal "seamless" where there aren't any seams. Sure that will probably plug into some enterprise bus and enterprise integration and enterprise ... anyway. Competent developers will understand what events to preserve and log and possibly allow retry/repeats. Anyone who has looked at a Splunk bill will realize that just storing all the logs everywhere is very expensive, which is another way of saying "wasteful". But any generic enterprisey event system will basically start and end with splunk-level log aggregation and kinda-analysis.
- dgellow 3y ago
  Dealt with something like this when I joined a previous job years ago, Kafka was the main data store, microservices had their state in memory they would build at startup by processing their whole history of events (that was ridiculously slow for some of them). Then GDPR came in place and the whole “keep everything in Kafka forever” had to face the reality of “not allowed to keep PII for longer than 30 days” :) (Before someone suggests it, no messages weren’t encrypted, just throwing the key away wasn’t an option)
  - jabradoodle 3y ago
    Would be expensive but you can do a copy and replace to keep the data you need, on a new topic. If you don't need the data, then you don't need it.

freecodyx 3y ago

Sounds like a junior work

nivertech 3y ago

It's top-down vs bottom-up, or custom vs generic. Top-down vs bottom-up: Top-down: starting from the business domain, and then mapping an implementation onto available technologies, tools, and vendors. Bottom-up: starting from the available technologies, tools, and vendors, and thinking how to bolt up a working solution out of them. Custom vs generic: Custom: DDD, CQRS/ES, Sagas, TBUI (Task-based/driven UI), GraphQL, Algebraic Data Types, etc. Generic: RDBMS, CRUD, REST, ACID transactions, CDC, generic admin UIs, nocode/lowcode, limited/generic types, etc.

lngnmn2 3y ago

[dead]

asah 3y ago

2c: if you need PostgreSQL elsewhere in your app anyway, then store your event data in PostgreSQL + FOSS reporting tools (apache superset, metabase, etc) until you hit ~2TB. After that, decide if you need 2TB online or just need daily/hourly summaries - if so, stick with PostgreSQL forever[1]. I have one client with 10TB+ and 1500 events per sec @ 600 bytes/rec (80GB/day before indexing), 2 days of detail online and the rest summarized and details moved to S3 where they can still query via Athena SQL[2]. They're paying <$2K for everything, including a reporting portal for their clients. AWS RDS multi-AZ with auto-failover (db.m7g.2xlarge) serving both inserts and reporting queries at <2% load. One engineer spends <5 hours per MONTH maintaining everything, in part because the business team builds their own charts/graphs. Sure, with proprietary tools you get a dozen charts "out of the box" but with pgsql, your data is one place, there's one system to learn, one system to keep online/replicate/backup/restore, one system to secure, one system to scale, one vendor (vendor-equivalent) to manage and millions of engineers who know the system. Building a dozen charts takes an hour in systems like preset or metabase, and non-technical people can do it. Note: I'm biased, but over 2 decades I've seen databases and reporting systems come & go, and good ol' PostgreSQL just gets better every year. https://instances.vantage.sh/aws/rds/db.m7g.2xlarge?region=us-east-1&os=PostgreSQL&cost_duration=monthly&reserved_term=Standard.partialUpfront https://instances.vantage.sh/aws/rds/db.m7g.2xlarge?region=u... [1] if you really need, there's PostgreSQL-compatible systems for additional scaling: Aurora for another 3-5x scaling, TimescaleDB for 10x, CitusDB for 10x+. With each, there's tradeoffs for being slightly-non-standard and thus I don't recommend using them until you really need. [2] customer reporting dashboards require sub-second response, which is provided by PostgreSQL queries to indexed summary tables; Athena delivers in 1-2 sec via parallel scans.

datadeft 3y ago
When a comment on HN has more merit than the article. The only problem with postgres is that inserting has some interesting scaling problems. Putting a queue between the event sources and the db is usually recommended.
- withinboredom 3y ago
  > Putting a queue between the event sources and the db is usually recommended. Emphasis on _usually_. If your db is at 2% CPU utilization with/without queues ... you probably don't need a queue.

ChicagoDave 3y ago

I'm on board with event-based architectures, but this article struggles to get its point across. I would focus on the difference between data relations and business behaviors. Once you start thinking in terms of behaviors and business activities, the move away from operational relational data stores becomes much more obvious.

3abiton 3y ago
On an abstract level, events can be modelled as relations.

wrestlingmonkey 3y ago

Lot of negative comments so thought I should counter as I quite liked it. The article style isn’t great but I get the impression it was only ever meant to be a toe dip into one aspect of thinking about events. I’ve worked at a couple of places that have had a lot of success with EDA, DDD and microservices. I’m sure these patterns are considered by many as buzzwords but they’ve been around for a long, long time now and they can be very effective. There’s no such thing as a silver bullet but the problems these ideas look to solve I’ve experienced very often, so I don’t buy into it being a solution in search of a problem.

deleted 3y ago
[deleted]

CodeCompost 3y ago

Didn't read the article but I'm in the process of eradicating Event Sourcing from a codebase and returning to the classical ACID database model. The boneheaded decisions made by our predecessors is staggering and choosing to use Event Sourcing for everything is the dumbest of them all.

politician 3y ago
What are the major deficiencies of ES in that particular codebase? If you could drop some specifics in bullets that would be really helpful for me. I promise not to ambush you with apologetics.
- CodeCompost 3y ago
  This system was developed from 2008-2013, a very different time when hardware was "cheap" and the Cloud was not not a thing. Event Sourcing dictates that Events are never deleted which means that the data volume keeps growing and growing. There is - in this system - no way to delete old events. When I brought this up, the response was "Just add another hard drive". In the modern Cloud era, adding a hard drive is extremely expensive. The system uses CQRS and all events generate reports that are stored in Elasticsearch. Data is never deleted, only an extra event gets added /saying/ it's deleted. The data is still cached in Elasticsearch. All of it, all the data back to 2013. Added an extra 32GB of RAM just to keep up with it is ludicrously expensive. We're in Europe. Guess what a system like this does to GDPR. Can you tell me which events I need to delete when somebody says they want to be forgotten? Yeah. I can't delete old data. It's impossible without collapsing the entire system like a house of cards. Finally, and this is the piece de resistance, the developers decided to develop a relational database structure ON TOP OF EVENT SOURCING. We're talking primary keys, foreign keys, cascading and non-cascading deletes. Importing an Excel sheet of 10000 rows takes TWO WEEKS because it generates hundreds of events per Excel cell that is being read. We brought it down to 10 minutes and there is plenty of room for improvement it's just that we have other priorities right now. Currently, simple flat "tables" with no foreign keys take several seconds to import (just like in a regular RDMS). Oh yeah note that this is a system that is used by 4-5 users at a time, not hundreds or thousands of users.

revskill 3y ago

No, what you need is a command queue, command event is not domain event.

ulrischa 3y ago

I made a php demo for a idea: a event based observer based modelling system. I.e. for game of life: https://github.com/ulrischa/OCell https://github.com/ulrischa/OCell

audnaun252 3y ago

Modelling domain events is useful for describing the problem your trying to solve with the domain experts, and it should probably be left in the documentation when planning a solution. For actually implementing a system that provides an audit trail of long-lived state machines, you're probably better off using something like Temporal.io/durable functions which uses event sourcing internally for their persistence, and has a programming model which forces you to think about deduplication/idempotency by adding different constraints for the code that orchestrates the functionality (workflows), vs the code that actually interacts with the real world (activities)

DenisM 3y ago
Durable functions suffer from lack of Observability tho. I’d love to hear suggestions on overcoming this issue.
- mfateev 3y ago
  temporal.io just released .NET SDK. The observability and scalability of the platform is really good. Disclaimer: I'm one of the founders of the project.
  - DenisM 3y ago
    What I meant specifically is that the current state of a workflow is stored in a format that’s opaque to any component other than the workflow itself. E.g. if I have a “shopping cart checkout” workflow and the user is not making progress, how can I can I tell which step of the workflow the user is stuck at?
    - audnaun252 3y ago
      The function's event data and current state is all stored in table storage, so you could query that - I'd expect you'd need to query an event-store-based solution in a similar way?
    - mfateev 3y ago

tiku 3y ago

Yeah, ehh I'm just going to stick to good old fashioned relational data.

agentultra 3y ago
Good, do it until you can’t. Don’t use a hammer on a screw.

kinddevil 3y ago

Question has been asked every time when someone wants to use event sourcing -- why do you want to redo the stuff that databases already did? "Ensure your design works if scale changes by 10X or 20X, not 100x".

baq 3y ago
because the database is inconsistent and incorrect is usually the answer.

onion-soup 3y ago

IT is doomed

asimpletune 3y ago

Something that might be missing from these discussions is when event driven architecture is even appropriate. The short answer is if your customer did something and expects a response it's not even driven, that's just request/response. Event driven is when something happens out of band. E.g. you push your code to GH, which triggered a build. In this example, you reloading the page to see that your updated code is request/response, however that CI build that was enqueued is event driven. Hope that helps.

berkes 3y ago
It's not that Simple. Request-response isn't a factor to choose ES or ED architectures on. You can have request-response, inline, blocking, cycles with ES or ED. And you can have async without ES or ED just fine too (e.g. workers, queues, actors, multithreaded etc)
- asimpletune 3y ago
  > It's not that Simple. It can be as complicated as you want to make it. My original point was to supply the missing context that explains 95% of use cases. > Request-response isn't a factor to choose ES or ED architectures on. One's business use case and how it translates into an appropriate messaging pattern is arguably the most important factor to base an architecture on. > You can have request-response, inline, blocking, cycles with ES or ED [...] Yes, of course. And it's fair to say, at any point in your life, that one can do anything that one wants. Like, one could have rollerblades for hands, or one could implement request/response functionality with an event driven system. But once we've moved past that, probably the next order of business is to justify the extremely good reason for overriding all prevailing common sense.
  - berkes 3y ago
    No need to be condescending. ES and ED architectures have good use-cases. And CRUD has good use-cases, and RDMSes and time series and document databases. Non will ever fit perfectly and non are a silver bullet. Choosing the right architecture is an an art. Experience helps. And being able to look into the future even more. Point being: a silly parameter as "wether you need the response to a request to have the data" is certainly not the only metric to base architectural choices on. It's not even a significant metric. It's at most a technical requirements. And one that can be fulfilled with many architectures. Like event sourced architectures even.

jstummbillig 3y ago

I was not aware of event driven design until recently, but coincidentally concluded something like it, after considering the optimal data structure in an AI powered world. While it's clear how event driven design might have been worth the trade off (assuming you were able manage the complexities and actually made use of the data) being able to query an AI with knowledge of every event that happened to your business, with will make it ubiquitous over the coming years.

erikpukinskis 3y ago

Event sourcing has a lot of nice properties, so I’m intrigued. But don’t you still need relations? And then how do you implement those? If the answer is “they’re all implicit in the application layer code” then that’s not really acceptable. I still need some way to query for relations, or keep relation views up to date, or something like that. I don’t mind if relations are not core to your persistence model, but they have to be implemented _somewhere_ in your data layer, and I’m not seeing any mention of that here. I have the same issue with Firestore, everyone does relations _somehow_ but it’s all just spaghetti application code which isn’t scalable.

mdm12 3y ago
In event sourced systems, you project the event stream into read models, of which there can be many (relational, time series, etc.) If you're familiar with functional programming, it is essentially a fold operation over the stream of events into a single state. Having worked with event sourced systems in the past, there are benefits in having a persisted explicit event history, but there is much added complexity (how do those read models actually get generated? how do you version the model? do you have snapshots of your read models?). In my experience, the additional complexity was not worth it for most contexts in which the pattern was applied...