4 ms·

I admittedly don't have your level experience, especially with such huge scale. However, I've worked with smaller systems and have found a "sweet-spot" that wor

I admittedly don't have your level experience, especially with such huge scale. However, I've worked with smaller systems and have found a "sweet-spot" that worked well in my cases: 1. Keep a traditional RDBM system. Place in here everything that needs consistency (e.g., the bank's accounts, balances and transactions) or has to be stored indefinitely. 2. That part of the system generates events. Those events are used to maintain Queryable stores that are better structured for your required queries. For instance, you could store per-client transaction lists in here. Or you could fill in an OLAP database, or whatever. 3. For each query-able store, implement a process that can populate it from the RDBMS data. This serves several purposes: a) You can rebuild any such stores at any time. This removes the durability requirement from these stores, so they may be simpler and more efficient. b) You can compare a rebuilt store against the current one. If there are any differences, there's either a bug in your event tracking code or a bug in the populator. c) This consistency-check procedure is really useful during development and testing. When you make a change, it is really hard to get both the event-sourced store and the populator wrong but consistent with each other. This only happened to me due to mis-specified or mis-understood specifications, completely fencing out pure programming mistakes. Of course, this system only works so long as your write volume can be ingested by your RDBMS. Luckily, this is the case on all of my current projects (and I would argue most projects out there). Notice that scaling reads should be much easier! Finally, this architecture just accepts that read-errors may happen (e.g., a client doesn't see a transaction in their transactions list because some event got lost). This hasn't been a huge problem for us, since the mistake will be repaired on the next "reconciliation" process (along with a warning to get devs to investigate what happened). These reconciliations can be run as frequently or as sparsely as desired, or even on some user-initiated action (e.g.: the support guy has a button to "force reconciliation now" and instructions to press it whenever a customer complains about missing transactions).

ninjakeyboard 8y ago
Yeah I'm using kafka right now with small queues between processes (generally implemented as elixir/OTP gen_server). With a blend of RDBMS (postgres which is now a swiss army knife), Redis (also a swiss army knife), and zookeeper (mostly because I know it well and it's there with kafka. It's very useful for co-ordination across processes.) Because the events coalesce into kafka, you can use a mechanism like logstash to spew this log data into eg elastic search and then use that for your queries. Or write to the db here (in process) or there (cqrs style.) There are different architectural approach used, and they all yield good results, of very high reliability in processing, if slightly "eventual." It works well. We do use rdbms too, but a lot of the time we avoid reading from it after initialization (or after encountering an exception causing an actor to restart). Depends on the data though. Some places I read on every command because I know the risk front is small and it's easier for a jr to get in there and understand what's happening. A good rule is that a single process should own that data so you don't have to worry much about consistency related concerns. In microservices good practices, each service should really own its own data completely but I believe that it's fair enough to say a bounded context owns its own data (even if in a shared rdbms) if building at a smaller scale. In terms of code, I'm usually building with some variation of onion architecture these days, where all context is build on the outside (eg receiving a request, getting info from state or db), and pure domain logic 100% effect free exists on the inside that receives all of the data from the outside and turns out some commands or events in response. This makes the important logic very testable, and easy to reason about. Core domain logic has no effects - not even logging - but only generates events/commands in response to commands and events. You never mock that stuff - it's the star of the show. Nor do you ever have to. The generated commands and events are later "applied" by the outer layer, logging, shipping them off to kafka, and/or writing some stuff to a database. That's where I'm at today. I am working in realtime systems so it's a good fit for the approaches. I'm finding the software is turning out very well. We're working fairly quickly, the software is very encapsulated to a bounded context, the domain logic is clear in the core, easy to test and read, and change. Or even rewrite if needed.

ninjakeyboard 8y ago

Yeah I'm using kafka right now with small queues between processes (generally implemented as elixir/OTP gen_server). With a blend of RDBMS (postgres which is now a swiss army knife), Redis (also a swiss army knife), and zookeeper (mostly because I know it well and it's there with kafka. It's very useful for co-ordination across processes.) Because the events coalesce into kafka, you can use a mechanism like logstash to spew this log data into eg elastic search and then use that for your queries. Or write to the db here (in process) or there (cqrs style.) There are different architectural approach used, and they all yield good results, of very high reliability in processing, if slightly "eventual." It works well. We do use rdbms too, but a lot of the time we avoid reading from it after initialization (or after encountering an exception causing an actor to restart). Depends on the data though. Some places I read on every command because I know the risk front is small and it's easier for a jr to get in there and understand what's happening. A good rule is that a single process should own that data so you don't have to worry much about consistency related concerns. In microservices good practices, each service should really own its own data completely but I believe that it's fair enough to say a bounded context owns its own data (even if in a shared rdbms) if building at a smaller scale. In terms of code, I'm usually building with some variation of onion architecture these days, where all context is build on the outside (eg receiving a request, getting info from state or db), and pure domain logic 100% effect free exists on the inside that receives all of the data from the outside and turns out some commands or events in response. This makes the important logic very testable, and easy to reason about. Core domain logic has no effects - not even logging - but only generates events/commands in response to commands and events. You never mock that stuff - it's the star of the show. Nor do you ever have to. The generated commands and events are later "applied" by the outer layer, logging, shipping them off to kafka, and/or writing some stuff to a database. That's where I'm at today. I am working in realtime systems so it's a good fit for the approaches. I'm finding the software is turning out very well. We're working fairly quickly, the software is very encapsulated to a bounded context, the domain logic is clear in the core, easy to test and read, and change. Or even rewrite if needed.