It's definitely a write-up I've wanted to make before. The short version of event sourcing is that instead of looking at your data model in standard CRUD terms

It's definitely a write-up I've wanted to make before. The short version of event sourcing is that instead of looking at your data model in standard CRUD terms, represent your data as an ordered list of changes. Updating an address field on a user model would become something like ChangeEvent{address=...}, rather than a full read/write of a user blob. To derive the current state of the object, you would replay and apply all events to a blank object. Obviously over time the list of changes could become enormous, so what you do to optimize is periodically create a snapshot of the current full state of the object and only replay events that happened after it was made. For our persistence layer we used Redis as the input queue for all events. Each object we stored required a list to push changes into, an integer counter to represent the sequence ID of the most recent event in the list, and a string value containing the location in S3 of the most recent snapshot (if it exists). Performance was incredible for writes because a) it's Redis and b) RedisLabs supports semi-transparent sharding. We used S3 to store the snapshots. We called the creation process a "compaction" because besides storing the most recent full state of the object, we would also store off the list of changes used to make it and purge those events from Redis. This kept our Redis storage usage tightly bounded and our costs down. This compaction process was also 100% safe with no chance of race conditions or data loss. Our algorithm would fetch the list of changes from Redis, create and serialize the current state of the object as a snapshot, and then write it and the list of changes to S3. Only upon success of those S3 writes would it then issue a command to Redis to purge that object's list of changes. To prevent race conditions on this part of the operation, we would send the sequence ID we are compacting from and in a Redis script we would atomically check to make sure no more-recent compaction has happened, update the snapshot pointer, and flush the list of changes only up to the sequence number we compacted from (changes were free to be added to the list while a compaction process was in progress). The killer part of this whole setup was that those snapshots and change lists stored in S3 were completely immutable and formed a linked list of the entire history of changes for an object. We could visualize and inspect the entire change history of an object, find bugs and retroactively fix them by changing how an event is applied, and if needed for read performance, cache the S3 snapshots in memcache without any concern for needing to invalidate them. Concurrent writes to an object became less of an issue as well. Instead of needing to do a CAS write or a transaction, you would just push the change you wanted to make to the object, read it back, and verify it was actually applied. No locking needed. The major downside to this architecture is that any querying you need to do will always be weakly consistent and require an external index. We had a small Elastisearch cluster for some of our objects that had fields needing indexing. One last major advantage of this setup was that it was trivial to "clone" production to your local dev machine. Because snapshots were immutable, we set up a tiered database layer that would first try and load snapshots locally, and failing that, would use a read-only IAM user to access S3 to pull down a snapshot from production. All you would then do is load a backup from a RedisLabs database locally and boom, you have an exact copy of production that you can freely modify because any new writes all happen locally. Believe it or not, the most unexpected cost from this architecture was PUT requests to S3. Depending upon the frequency of your compactions it can start to add up. However, if usage declines, that cost also declines with no impact on the availability of your data.