4 ms·
I guess the pattern is pretty inevitable if you're building on a centralised distributed log. You write events to a log which consumers read and create custom d
by usgroup 10y ago
I guess the pattern is pretty inevitable if you're building on a centralised distributed log. You write events to a log which consumers read and create custom databases , which are the more convenient means to your ends.
Fine but the semantic of your events will be exceptionally important as will versioning if you'd like a consistent way of making your views re-buildable. This is easy to mess up.
Kafka + YARN + Hadoop + DBs + code. To me this is neither elegant nor simple but perhaps necessary complexity if you're into something in realtime and on scale.
- josephg 10y agoI'm currently in the process of rewriting a simple version of this stack with some nice versioning properties. It doesn't look simple from the outside, but thats how complex any modern database is. The only difference with samza is that you can decide how much of that complexity to buy into. And because you have access to the "internals" of how data flows through the system, you can extend that view through your application. (Eg by making a standalone access control filter which only allows some users access to some documents. Or wrapping your react static HTML rendering code as a database view that you can, for free get a change feed to.) For me the complex part that I can't quite stomach is having to buy into hadoop and zookeeper. But thats a tooling problem, not an architecture problem. We need lighter implementations of all these tools.