8 ms·

EventStore: Open-Source, Functional Database with Complex Event Processing in JS

122 points by juancampa 8y ago

mattdeboard 8y ago
What's it good at? What is this the right tool for?
- prophesi 8y ago
  It's kind of weird that it's linked to the repo, and not the documentation. https://eventstore.org/docs/event-sourcing-basics/index.html https://eventstore.org/docs/event-sourcing-basics/index.html
- mrdoops 8y ago
  Persisting a log of events. Used with CQRS and event-sourcing to decouple the Read model(s) from the Command layer for benefits like: multiple read models (projections), time-travel and audit-ability (rebuild the read model to certain point in time), and decoupled contexts. Event-sourcing can be a really useful tool in many domains, but especially where having a state-of-the-art audit log is helpful.
  - slowmovintarget 8y ago
    In the parlance of Domain-Driven Design, every aggregate (think object instance as a loose equivalence) is its own stream of events. Loading aggregates involves replaying events until you are caught up. The key design requirement here is to deeply understand the reads your application will need to perform. The downsides are you have to push the data somewhere else to digest and report on it. Relations are tough to model, too, as events that happen to more than one aggregate essentially have to repeat the data in each stream in the form appropriate to that domain entity. (You can often model this as one event causing another. Projections work less well for this circumstance.) This can make it hard to draw data from the underlying store. Instead you must "hydrate" it into objects in memory to ask questions of it, though projections can help. EventStore is definitely not the right choice if you need ad hoc query capability. (I used EventStore in production for 2 years with F#.) I'd classify this as an exotic data store. Use it if you have a really strong need for event-sourcing. Me... I'd likely choose Datomic instead.
kyleperik 8y ago
"Functional Database" is contradictory. Honestly this sounds like a bunch of buzzwords concatenated together. Is there an example or a purpose for this?
- derefr 8y ago
  Personally, I've heard of it in the context of https://github.com/commanded/commanded https://github.com/commanded/commanded, a CQRS framework for Elixir that supports EventStore as a storage backend. I think the idea is that it's something simpler than a database. It's more like an append-only file that has on-insert triggers/durable running queries. (Sort of like https://www.pipelinedb.com/ https://www.pipelinedb.com/ does, or like blockchain nodes do.) Or, you could think of it as message queue like Kafka, with permanent durability of all "messages", and a single fixed subscriber bolted onto the queue server, where that subscriber is exposed through the queue server's API, allowing users to reprogram it arbitrarily.
- lojack 8y ago
  It isn’t that contradictory. You can have data in a functional language. One of the common features of a functional language that separates it from imperative is data immutability. In this case it’s a database of immutable data. I personally understand exactly what they’re talking about when they call it a functional database.
  - kyleperik 8y ago
    I don't understand how this is useful. The purpose of functional programming isn't to make everything immutable, but to use immutability to prevent unnecessary change of state which can result in flaky code. So where's the usefulness in a functional Database? Also this still isn't functional. Is adding records to a set not mutation?
    - fulafel 8y ago
      I don't know anything about EventStore, but immutability and databases go surprisingly nicely together. Mash together versioned data and persistent data structures, and you get to store history efficiently (and can have separate operations for excising history). See eg Datomic.
circleoncircles 8y ago
It's not a new concept.
- servercobra 8y ago
  Few things are new concepts. Not being a new concept doesn't mean this is interesting or useful.
- topspin 8y ago
  Was some claim to a new concept made somewhere? I haven't seen this. A new realization of an existing concept can be notable.
- ouro 8y ago
  To be fair I have been talking about event sourcing for over a decade. The concepts have been around far longer than that. In fact most databases "event source" their own internal structures. There is however benefit to modelling a system in that way and not just an internal transaction log. I have done many talks on the subject and you should be able to pull one up on youtube etc fairly quickly (its a bit much to go through in a comment).
megakid 8y ago
We use this heavily in production and to good effect. It has its downsides (doesn't every product) but we have found it works well once you've grasped them. If you have a primarily .NET focused team, have plenty of event-driven applications and want to do CQRS / Event Sourcing for some of your applications I would recommend it as a starter step towards that. It gives you a lot 'out of the box' which is great when starting out but over-reliance on some of its features can make it hard to replace. Make sure you have a good handle on retention and an archival strategy. Storage maybe financially cheap but it's certainly not maintenance friendly once your database grows beyond certain sizes (we're currently running our own fork to solve some of these issues whilst the PR process is going on).
- rjbwork 8y ago
  We just wrote our own on top of Sql Server. Runs fine for thousands of accounts with many events per day. We currently host it on the smallest azure sql scale level. We have virtually unlimited scaling with azure, and should have no problem sharding it in future because we include extra metadata (event type, accountid, etc.) so we could choose to shard on any of those dimensions.
fulafel 8y ago
What does functional mean here?
twic 8y ago
Some ex-colleagues were considering using this a few years ago. They were getting interesting in CQRS, got Greg Young in to give a talk, and came away filled with enthusiasm. I think they ended up doing something incremental on top of the MySQL they already had, rather than switching over to this. Is there a particular reason this is suddenly exciting again now?
tobr 8y ago
I've played around recently with a Redux inspired frontend architecture where the complex, nested reducers are replaced with a single one that simply adds every action to an immutable linked list (i.e, `(state, action) => ({state, action})`. This means all the business logic that would normally live in reducers can be moved over to memoized selectors, which fixes the awkward difference in how reducers and selectors are composed that tend to happen in a typical Redux app, and gets all the declarative benefits of selectors. It makes it easy to load code for new selectors asynchronously (for example when navigating to a new route), and because the state is a long list of every state your app has ever been in, they can "rewind" as far back as they need to sync up correctly with the rest of the app. What I've realised recently is, this is basically event sourcing. Replace actions with events, selectors with projections, and memoization with snapshots. So this submission is certainly timely and of interest to me. Thanks!
- jmcqk6 8y ago
  This mirrors exactly the way I've been approaching redux recently as well following the same realization. I feel like this is much cleaner and easier to reason about.
  - acemarke 8y ago
    Hi, I'm a Redux maintainer. Got any examples of this? Also, how do you feel it's better than a typical Redux app setup?
    - mattdeboard 8y ago
      It seems like the main difference with a reducer is that the reducer just returns `state`, not `(state, action)`. I'm reading this for the first time but that's how I'm reading it. But y'all must be storing this `(point-in-time-state, action-that-produced-state)` somewhere, because the Redux Chrome plugin displays this data. Was I wrong in thinking the Chrome plugin was just visualizing data already stored by Redux?
      - 8y ago
Yuioup 8y ago
I've had to break it to my client that had built a CRUD application using CQRS and Event Sourcing that they have wasted a huge amount of resources on a misguided architecture. A rewrite is pending.
- root_axis 8y ago
  > A rewrite is pending. So another huge amount of resources wasted.
  - honestlyidk 8y ago
    rewrites are really the on thing all programers enjoy doing. If it works dont touch it, should have caught it sooner. Make the next project better and move on.
- maskull 8y ago
  Is it being rebuilt as pure CRUD? I've been working on an app lately where the major entities in the relational db won't be updated. They'll will just be superseded by the next record.
- BillinghamJ 8y ago
  Why doesn’t it work for the use case?
- Ezku 8y ago
  I would gladly hear more about this scenario. Why did the initial approach not work? How is the follow-up going to improve on that? And why was a rewrite the best way to fix things? Thanks for any insight you can share.
platz 8y ago
Event Sourcing seems like the ultimate YAGNI architecture. You make an architecture that optimizes for future features at the expense of a coherent flow for the current features that you know you have. it's like if you made pub-sub the core architecture for your system. very actor-system like, but also probably harder to understand the runtime behavior - control flow.
- hudo 8y ago
  Try to solve complex architectures that can handle race conditions without some kind of event sourcing, good luck with that
- pgwhalen 8y ago
  With good tooling, it is arguably a lot easier to understand than a synchronous microservices architecture, because of the persistence of events. I’ll admit it’s probably harder to debug than a monolith, but that’s true of any distributed system.
- Fire-Dragon-DoL 8y ago
  It's not like you are paying that price for long. Usually you'll start getting benefits the next month, as soon as someone wants new numbers in the dashboard. Or as soon as a new feature needs to be added
TheLudd 8y ago
I do event sourcing in my company and we had a look at this when we started out 4 years ago. What I don't understand is why build a database? Why not build something in the application layer that uses your fav database as a storage mechanism instead? Aren't the existing databases more mature and better to use?
- stabbles 8y ago
  I've used a MySQL table `event_store` with fields `uuid`, `playhead`, `payload` and `recorded_on`; works fine.
  - root_axis 8y ago
    How did you manage horizontal scaling with this approach?
    - timClicks 8y ago
      You could shard based on uuid, so that each shard has its set of objects that it manages. The easiest way would be to cast uuid as a 64bit unsigned int, then mod by the number of shards. If the number of shards is dynamic, then use consistent hashing.
    - manigandham 8y ago
      If you need total system order then that will always be your bottleneck, although you can make it very fast by scoping it to just a sequence number generator and doing the actual work in separate processes. Otherwise, most event sourcing uses different "streams" of events for different application functions, so you can shard by stream in whatever way works for you.
  - remontoire 8y ago
    Good to know. Was thinking of this for myself in a simple usecase.
- BillinghamJ 8y ago
  There are a lot of non-trivial aspects to scaling event sourcing systems. Particularly if you want atomicity or asynchronous two-phase committing. Having implemented them ~5 different ways for different services at my startup, I’d definitely welcome a reliable DB-level abstraction.
Nelkins 8y ago
Good article on how EventStore can be used in production: https://medium.com/@eulerfx/scaling-event-sourcing-at-jet-9c873cac33b8 https://medium.com/@eulerfx/scaling-event-sourcing-at-jet-9c...

122 points by juancampa 8y ago

mattdeboard 8y ago

What's it good at? What is this the right tool for?

prophesi 8y ago
It's kind of weird that it's linked to the repo, and not the documentation. https://eventstore.org/docs/event-sourcing-basics/index.html https://eventstore.org/docs/event-sourcing-basics/index.html
mrdoops 8y ago
Persisting a log of events. Used with CQRS and event-sourcing to decouple the Read model(s) from the Command layer for benefits like: multiple read models (projections), time-travel and audit-ability (rebuild the read model to certain point in time), and decoupled contexts. Event-sourcing can be a really useful tool in many domains, but especially where having a state-of-the-art audit log is helpful.
- slowmovintarget 8y ago
  In the parlance of Domain-Driven Design, every aggregate (think object instance as a loose equivalence) is its own stream of events. Loading aggregates involves replaying events until you are caught up. The key design requirement here is to deeply understand the reads your application will need to perform. The downsides are you have to push the data somewhere else to digest and report on it. Relations are tough to model, too, as events that happen to more than one aggregate essentially have to repeat the data in each stream in the form appropriate to that domain entity. (You can often model this as one event causing another. Projections work less well for this circumstance.) This can make it hard to draw data from the underlying store. Instead you must "hydrate" it into objects in memory to ask questions of it, though projections can help. EventStore is definitely not the right choice if you need ad hoc query capability. (I used EventStore in production for 2 years with F#.) I'd classify this as an exotic data store. Use it if you have a really strong need for event-sourcing. Me... I'd likely choose Datomic instead.

kyleperik 8y ago

"Functional Database" is contradictory. Honestly this sounds like a bunch of buzzwords concatenated together. Is there an example or a purpose for this?

derefr 8y ago
Personally, I've heard of it in the context of https://github.com/commanded/commanded https://github.com/commanded/commanded, a CQRS framework for Elixir that supports EventStore as a storage backend. I think the idea is that it's something simpler than a database. It's more like an append-only file that has on-insert triggers/durable running queries. (Sort of like https://www.pipelinedb.com/ https://www.pipelinedb.com/ does, or like blockchain nodes do.) Or, you could think of it as message queue like Kafka, with permanent durability of all "messages", and a single fixed subscriber bolted onto the queue server, where that subscriber is exposed through the queue server's API, allowing users to reprogram it arbitrarily.
lojack 8y ago
It isn’t that contradictory. You can have data in a functional language. One of the common features of a functional language that separates it from imperative is data immutability. In this case it’s a database of immutable data. I personally understand exactly what they’re talking about when they call it a functional database.
- kyleperik 8y ago
  I don't understand how this is useful. The purpose of functional programming isn't to make everything immutable, but to use immutability to prevent unnecessary change of state which can result in flaky code. So where's the usefulness in a functional Database? Also this still isn't functional. Is adding records to a set not mutation?
  - fulafel 8y ago
    I don't know anything about EventStore, but immutability and databases go surprisingly nicely together. Mash together versioned data and persistent data structures, and you get to store history efficiently (and can have separate operations for excising history). See eg Datomic.

circleoncircles 8y ago

It's not a new concept.

servercobra 8y ago
Few things are new concepts. Not being a new concept doesn't mean this is interesting or useful.
topspin 8y ago
Was some claim to a new concept made somewhere? I haven't seen this. A new realization of an existing concept can be notable.
ouro 8y ago
To be fair I have been talking about event sourcing for over a decade. The concepts have been around far longer than that. In fact most databases "event source" their own internal structures. There is however benefit to modelling a system in that way and not just an internal transaction log. I have done many talks on the subject and you should be able to pull one up on youtube etc fairly quickly (its a bit much to go through in a comment).

megakid 8y ago

We use this heavily in production and to good effect. It has its downsides (doesn't every product) but we have found it works well once you've grasped them. If you have a primarily .NET focused team, have plenty of event-driven applications and want to do CQRS / Event Sourcing for some of your applications I would recommend it as a starter step towards that. It gives you a lot 'out of the box' which is great when starting out but over-reliance on some of its features can make it hard to replace. Make sure you have a good handle on retention and an archival strategy. Storage maybe financially cheap but it's certainly not maintenance friendly once your database grows beyond certain sizes (we're currently running our own fork to solve some of these issues whilst the PR process is going on).

rjbwork 8y ago
We just wrote our own on top of Sql Server. Runs fine for thousands of accounts with many events per day. We currently host it on the smallest azure sql scale level. We have virtually unlimited scaling with azure, and should have no problem sharding it in future because we include extra metadata (event type, accountid, etc.) so we could choose to shard on any of those dimensions.

fulafel 8y ago

What does functional mean here?

twic 8y ago

Some ex-colleagues were considering using this a few years ago. They were getting interesting in CQRS, got Greg Young in to give a talk, and came away filled with enthusiasm. I think they ended up doing something incremental on top of the MySQL they already had, rather than switching over to this. Is there a particular reason this is suddenly exciting again now?

tobr 8y ago

I've played around recently with a Redux inspired frontend architecture where the complex, nested reducers are replaced with a single one that simply adds every action to an immutable linked list (i.e, `(state, action) => ({state, action})`. This means all the business logic that would normally live in reducers can be moved over to memoized selectors, which fixes the awkward difference in how reducers and selectors are composed that tend to happen in a typical Redux app, and gets all the declarative benefits of selectors. It makes it easy to load code for new selectors asynchronously (for example when navigating to a new route), and because the state is a long list of every state your app has ever been in, they can "rewind" as far back as they need to sync up correctly with the rest of the app. What I've realised recently is, this is basically event sourcing. Replace actions with events, selectors with projections, and memoization with snapshots. So this submission is certainly timely and of interest to me. Thanks!

jmcqk6 8y ago
This mirrors exactly the way I've been approaching redux recently as well following the same realization. I feel like this is much cleaner and easier to reason about.
- acemarke 8y ago
  Hi, I'm a Redux maintainer. Got any examples of this? Also, how do you feel it's better than a typical Redux app setup?
  - mattdeboard 8y ago
    It seems like the main difference with a reducer is that the reducer just returns `state`, not `(state, action)`. I'm reading this for the first time but that's how I'm reading it. But y'all must be storing this `(point-in-time-state, action-that-produced-state)` somewhere, because the Redux Chrome plugin displays this data. Was I wrong in thinking the Chrome plugin was just visualizing data already stored by Redux?
    - 8y ago

Yuioup 8y ago

I've had to break it to my client that had built a CRUD application using CQRS and Event Sourcing that they have wasted a huge amount of resources on a misguided architecture. A rewrite is pending.

root_axis 8y ago
> A rewrite is pending. So another huge amount of resources wasted.
- honestlyidk 8y ago
  rewrites are really the on thing all programers enjoy doing. If it works dont touch it, should have caught it sooner. Make the next project better and move on.
maskull 8y ago
Is it being rebuilt as pure CRUD? I've been working on an app lately where the major entities in the relational db won't be updated. They'll will just be superseded by the next record.
BillinghamJ 8y ago
Why doesn’t it work for the use case?
Ezku 8y ago
I would gladly hear more about this scenario. Why did the initial approach not work? How is the follow-up going to improve on that? And why was a rewrite the best way to fix things? Thanks for any insight you can share.

platz 8y ago

Event Sourcing seems like the ultimate YAGNI architecture. You make an architecture that optimizes for future features at the expense of a coherent flow for the current features that you know you have. it's like if you made pub-sub the core architecture for your system. very actor-system like, but also probably harder to understand the runtime behavior - control flow.

hudo 8y ago
Try to solve complex architectures that can handle race conditions without some kind of event sourcing, good luck with that
pgwhalen 8y ago
With good tooling, it is arguably a lot easier to understand than a synchronous microservices architecture, because of the persistence of events. I’ll admit it’s probably harder to debug than a monolith, but that’s true of any distributed system.
Fire-Dragon-DoL 8y ago
It's not like you are paying that price for long. Usually you'll start getting benefits the next month, as soon as someone wants new numbers in the dashboard. Or as soon as a new feature needs to be added

TheLudd 8y ago

I do event sourcing in my company and we had a look at this when we started out 4 years ago. What I don't understand is why build a database? Why not build something in the application layer that uses your fav database as a storage mechanism instead? Aren't the existing databases more mature and better to use?

stabbles 8y ago
I've used a MySQL table `event_store` with fields `uuid`, `playhead`, `payload` and `recorded_on`; works fine.
- root_axis 8y ago
  How did you manage horizontal scaling with this approach?
  - timClicks 8y ago
    You could shard based on uuid, so that each shard has its set of objects that it manages. The easiest way would be to cast uuid as a 64bit unsigned int, then mod by the number of shards. If the number of shards is dynamic, then use consistent hashing.
  - manigandham 8y ago
    If you need total system order then that will always be your bottleneck, although you can make it very fast by scoping it to just a sequence number generator and doing the actual work in separate processes. Otherwise, most event sourcing uses different "streams" of events for different application functions, so you can shard by stream in whatever way works for you.
- remontoire 8y ago
  Good to know. Was thinking of this for myself in a simple usecase.
BillinghamJ 8y ago
There are a lot of non-trivial aspects to scaling event sourcing systems. Particularly if you want atomicity or asynchronous two-phase committing. Having implemented them ~5 different ways for different services at my startup, I’d definitely welcome a reliable DB-level abstraction.

Nelkins 8y ago

Good article on how EventStore can be used in production: https://medium.com/@eulerfx/scaling-event-sourcing-at-jet-9c873cac33b8 https://medium.com/@eulerfx/scaling-event-sourcing-at-jet-9c...