4 ms·

No matter how skilled I become as a developer, there is always something lurking around the corner to make me feel more naive than ever. As I was reading this

No matter how skilled I become as a developer, there is always something lurking around the corner to make me feel more naive than ever. As I was reading this article, I realized that my whole career and knowledge about the way applications work is based around the one core idea that when non-binary data needs to be persisted, you use a database. The idea that you can reliably use event sourcing in memory to persist your data is as foreign to me as it is impressive. Is anyone familiar with major applications (web apps, ideally) that use this method for their data persistence?

dpark 15y ago
It seems like email maps onto this model fairly well. An event (mail) comes in and gets written to the log (mbox). Starting the server (session) entails reading the events from file. It's not a perfect fit, of course. The big players are backing their services with custom data stores, though.
wpietri 15y ago
You're already familiar with a couple of things that can be built this way: word processors and multiplayer game servers. In both cases SQL databases are too slow and too awkward. Financial trading is another area where databases are too slow. I know of one place that uses this approach to keep pricing data hot in RAM for their financial models. And Fowler previously documented using this for a financial exchange: http://martinfowler.com/articles/lmax.html http://martinfowler.com/articles/lmax.html
- silverbax88 15y ago
  I wonder about this concept. The reality is that having enough RAM to power NASDAQ, and then being able to accurately reproduce the state of the data following a crash based on input being kept in a durable store - which effectively is IO to the disk, which is the same as, well, just writing to a DB to begin with. Of course, Fowler talks about 'snapshotting' the data, which, again, makes me wonder if playing with all of this resident memory and the systems needed to make that happen haven't already been solved by...um...databases.
dwolfson20 15y ago
This is the field we choose. Expand your thinking in the abstract about what a database is. As 71104 mentions, a file system is also a database. What you are thinking of as "a database" is really a specific type of key-value store that is located on disk. But the fact most DBs are on disk has nothing to do with the concept itself.
jkkramer 15y ago
I don't know if it uses "event sourcing" per se, but doesn't HN use in-memory & serialized Lisp data structures instead of a DB?
- philwelch 15y ago
  As did Viaweb.

dpark 15y ago

It seems like email maps onto this model fairly well. An event (mail) comes in and gets written to the log (mbox). Starting the server (session) entails reading the events from file. It's not a perfect fit, of course. The big players are backing their services with custom data stores, though.

wpietri 15y ago

You're already familiar with a couple of things that can be built this way: word processors and multiplayer game servers. In both cases SQL databases are too slow and too awkward. Financial trading is another area where databases are too slow. I know of one place that uses this approach to keep pricing data hot in RAM for their financial models. And Fowler previously documented using this for a financial exchange: http://martinfowler.com/articles/lmax.html http://martinfowler.com/articles/lmax.html

silverbax88 15y ago
I wonder about this concept. The reality is that having enough RAM to power NASDAQ, and then being able to accurately reproduce the state of the data following a crash based on input being kept in a durable store - which effectively is IO to the disk, which is the same as, well, just writing to a DB to begin with. Of course, Fowler talks about 'snapshotting' the data, which, again, makes me wonder if playing with all of this resident memory and the systems needed to make that happen haven't already been solved by...um...databases.

dwolfson20 15y ago

This is the field we choose. Expand your thinking in the abstract about what a database is. As 71104 mentions, a file system is also a database. What you are thinking of as "a database" is really a specific type of key-value store that is located on disk. But the fact most DBs are on disk has nothing to do with the concept itself.

jkkramer 15y ago

I don't know if it uses "event sourcing" per se, but doesn't HN use in-memory & serialized Lisp data structures instead of a DB?

philwelch 15y ago
As did Viaweb.