4 ms·

Another approach that is often used for these things is now generally referred to as Event Sourcing. You'll see it mostly associated with Domain Driven Design (

• hesdeadjim 11y ago
  I can't upvote this enough. The game backend I've been leading uses event sourcing for almost every "object" in our system. It's proven flexible enough to model both normal CRUD style access patterns and other activity feed-like features that may or may not have an aggregate state. Some huge advantages we've identified: * Modeling our objects as state changes has resulted in less code and improved readability. The effects of a state change are much more obvious than a series of CRUD-style updates. This is anecdotal of course. * Audit trail, we have every mutation for every object in our system. We can go back in time and identify bugs and fix them by just overlaying new change events. We can also feed this data into any number analytics systems and create materialized views without having to know in advance what kind of data we wish to collect. * Optimal performance characteristics for our chosen DBs (Redis+DynamoDB). We use Redis for storing incoming events, and DynamoDB to store our periodic snapshots that compress the aggregate state for an object (and keep the event history). This allows us to keep our Redis space usage bounded, as compressions flush the queue for each object. We optimize for DynamoDB pricing/performance by only needing eventually consistent reads for obtaining a snapshot. We keep our write capacity low because the only writes are when we write a new snapshot -- this happens much less frequently and is allowed to fail since a failure only means we continue using the previous snapshot. * Consistent point in time snapshots. To obtain a backup of the objects, we simply need to snapshot our Redis databases. If we restore, all objects are consistent because the per-object snapshots we store in DynamoDB are immutable and will always exist with any given state of the Redis data. Another consequence of this architecture is that we can take an old snapshot and "go back in time" to the real state of the data. * Highly concurrent without need for optimistic or pessimistic locking. Redis is obviously super performant and all we use it for is to push event items to the back of lists. For concurrency and conflicts, we can resolve the state of an object by de-duping events that may be invalid or duplicate from the point-in-time view of the aggregate object when the event was pushed. Obviously this won't work for every type of object state, but in our experience it has been extremely resilient. A few disadvantages: * Views on the objects must be eventually consistent and must be materialized externally to the storage of the objects. * More CPU required to create the aggregate state when getting the object. Hasn't been an issue for us and can be mitigated by the frequency of compaction into snapshots. * Space usage for snapshots. Eventually long lived objects will have a significant cumulative historical size. It's trivial though to traverse the snapshot history and trim the space usage by simply discarding snapshots after a certain point in time. The object will then be constructed from the last snapshot's aggregate state and events that happen after the purge.
  • numbsafari 11y ago
    Is your project a product or OSS? I've been tinkering with a simple game back-end for building up turn-based, multi-player games (an engine for ripping off board games) mostly as a demo/proving ground for ideas and not as a serious project since this is a pretty active space with established projects.
    • hesdeadjim 11y ago
      Unfortunately its proprietary for our studio's games. I've been hoping I could release some parts of the infrastructure as OSS, but I might have to settle for a presentation at some point.
      • belenos46 11y ago
        Are you part of the team building Underworld Ascendant? Because that sounds very much their speed.
        • hesdeadjim 11y ago
          Oh man I wish I was. Those games were practically life changing as a child. At one point my parents moved our computer into the bedroom to stop us from playing all hours of the night. They then woke up on a Saturday morning to a friend and I sitting on the edge of their bed playing. That did not go over well -- but they did move it out of their room after that!
• platz 11y ago
  Yes, another way to think about this same topic is Append-Only data structures which is why Apache Kafka is popular for this. Also, the transaction log files that RDBMS are built on, are append-only, and have this same Event Sourcing semantics.
  • numbsafari 11y ago
    +1 for mentioning RDBMS transaction logs, which are the canonical "prior art" in this area.
• kodablah 11y ago
  While I use event sourcing w/ Akka persistence, I find it is not a panacea for the problems that the blog post may reference. I find that oftentimes people store the resulting state in memory which may become unwieldy. And if you're storing it on DB/disk (not to be confused w/ storing your events/snapshots) you're basically back to the way traditional CRUD works except you have a stream of events if you need a new data structure from scratch. Also, many times event sourcing offers eventual consistency which, while it is obviously preferred to develop your app that way for scalability reasons, not all apps can do that and need atomic state.
  • hesdeadjim 11y ago
    For atomic updates we implement this via CAS-like semantics. A caller who obtains a point-in-time aggregate view of an object (equivalent to the 'R' in CRUD) is also given the sequence number of the latest event. They can then conditionally push new events into the object's queue and request a rejection if the current sequence id is not equal to the one they have. Since we use Redis for the queue, the event pushes have a consistent view of the queue at the time of the attempted push.
    • kodablah 11y ago
      While scalable, this basically turns it to bidirectional by adding a side-channel for conflict resolution which can add system complexity. This is why many event sourcing approaches recommend making the event represent the mutation (e.g. count + 1) instead of the result (e.g. count = 15)...when it does have to be atomic, I just give up and use a centralized place like you have with Redis. Such is life with atomic, distributed systems...they can only be so distributed and still reject conflicts in real time.
      • hesdeadjim 11y ago
        Most of our events do in fact represent the state change as the mutation. However I see no complexity issue with how we implement CAS semantics. Its 100% equivalent to any other implementation with all the implications -- reduced concurrency being the primary issue. We avoid this at all costs for most objects and, like you suggest, resolve conflicts by de-duping events or combining state changes to the final aggregate.
• emehrkay 11y ago
  I've been playing around with a Graph database OGM (G is for graph) https://github.com/emehrkay/Gizmo https://github.com/emehrkay/Gizmo and its event sourcing is a very simple implementation of what Fowler described in a post a while back. Graphs are simple which made this exercise even easier: entity(vertex) -> edge($RELATIONSHP) -> entity(vertex) vertices are simple, unstructured maps, so in my library you simply add a source entity when saving and all of the changes will be tied to that source. Pseudo: user = get_user('emehrkay') reply = get_reply($THIS_CONTENT) user_reply = connect(user, reply, 'posted') save = save(user_reply, source=user) This would create a graph like this user('emehrkay') -> posted -> reply($THIS_CONTENT) reply($THIS_CONTENT) -> event -> event("reply's changes") event("reply's changes") -> source -> user('emehrkay') "emehrkay made these changes to the reply" Very fun/cool stuff. I haven't had the need to query against the history yet, so I don't know how simple it would be or if it were possible to set a time and query the whole graph against that time (like in the example posted in another comment)
• tmd83 11y ago
  The basic idea looks simple when you have a few entities/event and the numbers of rows in the table are reasonable. It seems to me that the overhead and complexity is somewhat manageable when thats the case. But what do you do or how do you manage the complexity when you have 100s of entities you have tables with rows in the 10s of millions and lots of complicated SELECTS and JOINS to do? I'm not saying that it's impossible but how does people manage complexity at that scale and also the performance overhead.
  • numbsafari 11y ago
    With reasonable abstractions. Someone else mentioned DB transaction logs, but you can also look at Kafka and GetEventStore for additional reference points. That said, you don't necessarily want/need to apply ES to every single entity in your model. Typically, you want to apply it to those entities that have complex lifecycles or that are highly dynamic and might involve a lot of complexity. ES actually often makes it much easier to reason about complex lifecycles because you can isolate the transition logic into discrete units of code. So, it's actually tool for managing complexity. In terms of performance overhead, ES usually results in significant write-side performance improvements because you can either remove or dramatically simplify your transaction coordination overhead (increase concurrency, reduced latency). To address read-side performance, people generally use a cached or materialized view that is lazily updated or implemented with the events as the "write-behind" log for the cached view. This read-side detail is why ES is most often associated with CQRS or lambda-style architectures. As an example, in banking scenarios, you would determine "current balance" by adding up all of the past transactions on an account. Rather than doing this computation every single time you want the current balance, you typically settle the account in a nightly job and store the "nightly balance" and then compute the "current balance" by only considering new transactions for the day.
    • tmd83 11y ago
      I understand the basic architecture (or so I think) and its again the question of what happens when you have lots of complex entities with complex workflow/significance. For example in banking the transactions is one core thing while the rest of customer information is not so much, same goes for a the shipping example (or at least it looks that way from the outside). In a health scenario on the other hand I can see most information being required being the type that require an ES model. What I gather from Kafka style architecture is the storing/persistent works through the abstraction the processing itself you have to do separately for each type of entity and if you have many of them and your query requirement are complex not sure how to abstract that. I might be missing something. Again the idea of caching, lazy updating, horizontal scaling those resonates with me but I don't see how to write good abstractions that allows to apply that. For most complex entities it seems I have to do this case by case basis and I can't figure out ways to contain/manage that complexity. And while ES adds for write scalability transactional integrity seems to become much more a complicated questions and hard to abstract about, doesn't it?