4 ms·

What he proposes here seems to be: Service -> Messaging Broker (Kafka) -> Consumers (Samza) -> View, where writes go to service and reads read from a view. H

by sbpayne 10y ago

What he proposes here seems to be: Service -> Messaging Broker (Kafka) -> Consumers (Samza) -> View, where writes go to service and reads read from a view. However, I more often see a CDC (change data capture) approach used in practice: Service -> DB -> CDC -> Messaging Broker -> Consumers -> View My understanding is that there were three reasons that this was the case: 1) Many already had some conventional DB in place, so it was easier to slap a CDC + Messaging Broker to get a lot of the benefits with less infra changes. 2) You require strong consistency 3) Messaging broker of choice might not be durable enough to entrust your writes to. My understanding is that Kafka is durable (as this article states), so aside from (1) and (2) is there a strong reason to prefer a CDC approach?

weixiyen 10y ago
One potential downside is disk space. If you actually want to retain all records, that's going to be an order of magnitude more space if your workload is update-heavy. If it's primarily inserts not a big deal. The other potential downside I can see is if lost your views for any reason. If your views are not tolerant to partitions, then you face a long time to rebuild it the further out you go, so you may have to snapshot your views at certain intervals and sync with the specific record of the last kafka insert during snapshot, otherwise you have the same problem that people have when they clone a huge long-running git repo from the beginning. (edit: Kafka has log compaction so it may help alleviate some of the rebuilding process)
- fintler 10y ago
  > Kafka has log compaction so it may help alleviate some of the rebuilding process I've seen jobs spend 6+ hours rebuilding from Kafka when the local store is blown away for all partitions (using log compaction) -- if it's just a partition or 10 (single node failure), it takes way less time. When it comes to production, Samza's host affinity support is probably the most important thing to have working for when a job fails. https://samza.apache.org/learn/documentation/latest/yarn/yarn-host-affinity.html https://samza.apache.org/learn/documentation/latest/yarn/yar...
josephg 10y ago
From a theoretical point of view both models are pretty much equivalent. Either way you need to pick somewhere to accept transactions as the ground truth / root store. That root store is the arbiter of which transactions get accepted and which are rejected due to conflicts. (With Kafka you can figure this out through the event stream's order). With Samza the root store is Kafka, and with CDC its your database. Its obviously easy to scale everything "downstream" of your root source of truth because everything there can easily be redeployed and trivially scaled for redundancy. But your root store needs to worry about partition tolerance. That said, PostgresQL and Kafka probably scale fine for the needs of 99% of users. You can do queries either way. Using CDC you can run normal queries directly from your database, or another database downstream if you want to distribute the load. Using Samza you can run a query from any of your views.

by sbpayne 10y ago

weixiyen 10y ago

One potential downside is disk space. If you actually want to retain all records, that's going to be an order of magnitude more space if your workload is update-heavy. If it's primarily inserts not a big deal. The other potential downside I can see is if lost your views for any reason. If your views are not tolerant to partitions, then you face a long time to rebuild it the further out you go, so you may have to snapshot your views at certain intervals and sync with the specific record of the last kafka insert during snapshot, otherwise you have the same problem that people have when they clone a huge long-running git repo from the beginning. (edit: Kafka has log compaction so it may help alleviate some of the rebuilding process)

fintler 10y ago
> Kafka has log compaction so it may help alleviate some of the rebuilding process I've seen jobs spend 6+ hours rebuilding from Kafka when the local store is blown away for all partitions (using log compaction) -- if it's just a partition or 10 (single node failure), it takes way less time. When it comes to production, Samza's host affinity support is probably the most important thing to have working for when a job fails. https://samza.apache.org/learn/documentation/latest/yarn/yarn-host-affinity.html https://samza.apache.org/learn/documentation/latest/yarn/yar...

josephg 10y ago

From a theoretical point of view both models are pretty much equivalent. Either way you need to pick somewhere to accept transactions as the ground truth / root store. That root store is the arbiter of which transactions get accepted and which are rejected due to conflicts. (With Kafka you can figure this out through the event stream's order). With Samza the root store is Kafka, and with CDC its your database. Its obviously easy to scale everything "downstream" of your root source of truth because everything there can easily be redeployed and trivially scaled for redundancy. But your root store needs to worry about partition tolerance. That said, PostgresQL and Kafka probably scale fine for the needs of 99% of users. You can do queries either way. Using CDC you can run normal queries directly from your database, or another database downstream if you want to distribute the load. Using Samza you can run a query from any of your views.