4 ms·

This is what Kafka is for. You put Kafka on top of your database to expose data and events. Now BI can take the events put them into their system as they want.

BrentOzar 7y ago
> You put Kafka on top of your database to expose data and events. Now BI can take the events put them into their system as they want. Right, that's the data warehouse method that I described. "Put them into their system" is a lot harder work than just typing in "put Kafka on top of your database."
- jsjohnst 7y ago
  Things like Maxwell[0] help a lot with the getting stuff into Kafka from the DB, but I agree with your point entirely. Kafka is not something I’d recommend unless you’ve got a team dedicated to maintaining it. Thankfully Confluent is rising up to fill that need and reasonably priced. Confluent also has an offering for streaming binlogs into Kafka too. [0] http://maxwells-daemon.io/ http://maxwells-daemon.io/
skywhopper 7y ago
Worth pointing out that running Kafka is itself no small thing. So now you've added BI and also Kafka, which itself requires a bundle of services to operate. And you have to keep BI and Kafka in sync with all your various other data stores' schemas, and with each other. Which all gets back to the point of the OP.
bfung 7y ago
Without tone of voice over text, unsure if the suggestion is in sincerity or sarcasm. The article talks about micro services being split up due to fad as opposed to deliberate, researched reasons. Putting Kafka over the database also makes the data distributed when in most cases, it’s not necessary!
gunnarmorling 7y ago
Shameless plug: feeding data from operational databases to DWH is one main use case of change data capture as e.g. implemented by Debezium (https://debezium.io/ https://debezium.io/) for MySQL, Postgres, MongoDB, and more, using Apache Kafka as the messaging layer. Disclaimer: working on Debezium
- Yeroc 7y ago
  Curious how support is for commercial databases like SQL Server and Oracle? In the larger, older, enterprises these databases are rampant...
- chrisjc 7y ago
  Data Warehouses are (usually) not optimized for individual inserts, updates, deletes (DMLs). Loading data into DWHs is usually done through copy/ingestion commands where aggregated/batched data is copied from blob storage (s3, azure, etc...) to a staging table in the DWH. In a non-append-only scenario, Debezium tracks each source operation (insert, update, delete) from the replication log (oplog, binlog, etc) as an individual operation that's emitted into a kafka topic. How does one efficiently replicate this to a Data Warehouse in an efficient manner? I have not been able to use Debezium as way to replicate to a Data Warehouse for this very reason. At least not without having to resort to very complicated data warehouse merge strategies. Note, there exist Data Warehouses that allow tables to be created in either OLAP or OLTP flavor. I understand that Debezium could easily replicate to an OLTP staging table. But are there any solutions if this isn't an option?
zomglings 7y ago
> You put Kafka on top of your database Your casual tone is at odds with what I've seen when teams run Kafka clusters in production. Not a decision I would take so lightly.

BrentOzar 7y ago

> You put Kafka on top of your database to expose data and events. Now BI can take the events put them into their system as they want. Right, that's the data warehouse method that I described. "Put them into their system" is a lot harder work than just typing in "put Kafka on top of your database."

jsjohnst 7y ago
Things like Maxwell[0] help a lot with the getting stuff into Kafka from the DB, but I agree with your point entirely. Kafka is not something I’d recommend unless you’ve got a team dedicated to maintaining it. Thankfully Confluent is rising up to fill that need and reasonably priced. Confluent also has an offering for streaming binlogs into Kafka too. [0] http://maxwells-daemon.io/ http://maxwells-daemon.io/

skywhopper 7y ago

Worth pointing out that running Kafka is itself no small thing. So now you've added BI and also Kafka, which itself requires a bundle of services to operate. And you have to keep BI and Kafka in sync with all your various other data stores' schemas, and with each other. Which all gets back to the point of the OP.

bfung 7y ago

Without tone of voice over text, unsure if the suggestion is in sincerity or sarcasm. The article talks about micro services being split up due to fad as opposed to deliberate, researched reasons. Putting Kafka over the database also makes the data distributed when in most cases, it’s not necessary!

gunnarmorling 7y ago

Shameless plug: feeding data from operational databases to DWH is one main use case of change data capture as e.g. implemented by Debezium (https://debezium.io/ https://debezium.io/) for MySQL, Postgres, MongoDB, and more, using Apache Kafka as the messaging layer. Disclaimer: working on Debezium

Yeroc 7y ago
Curious how support is for commercial databases like SQL Server and Oracle? In the larger, older, enterprises these databases are rampant...
chrisjc 7y ago
Data Warehouses are (usually) not optimized for individual inserts, updates, deletes (DMLs). Loading data into DWHs is usually done through copy/ingestion commands where aggregated/batched data is copied from blob storage (s3, azure, etc...) to a staging table in the DWH. In a non-append-only scenario, Debezium tracks each source operation (insert, update, delete) from the replication log (oplog, binlog, etc) as an individual operation that's emitted into a kafka topic. How does one efficiently replicate this to a Data Warehouse in an efficient manner? I have not been able to use Debezium as way to replicate to a Data Warehouse for this very reason. At least not without having to resort to very complicated data warehouse merge strategies. Note, there exist Data Warehouses that allow tables to be created in either OLAP or OLTP flavor. I understand that Debezium could easily replicate to an OLTP staging table. But are there any solutions if this isn't an option?

zomglings 7y ago

> You put Kafka on top of your database Your casual tone is at odds with what I've seen when teams run Kafka clusters in production. Not a decision I would take so lightly.