3 ms·

This is a solved problem, "Data Engineering" teams solve this by building a data pipeline. It's not for all orgs, but for a large org, this is worth doing righ

This is a solved problem, "Data Engineering" teams solve this by building a data pipeline. It's not for all orgs, but for a large org, this is worth doing right.

derision 7y ago
"data pipeline" is just the new trendy phrase for ETL, which the GP mentioned. just because a solution exists, does not make it a solved problem. it's not the right solution for everyone
- segmondy 7y ago
  no it's not. ETL is really database focused and batch focused , Extract, Transform, Load. Data pipeline, is a combination of streams and batch. For example, you can implement a Data Capture using something like https://debezium.io/ https://debezium.io/ Here's how Netflix solves it https://netflixtechblog.com/dblog-a-generic-change-data-capture-framework-69351fb9099b https://netflixtechblog.com/dblog-a-generic-change-data-capt... Overview "Change-Data-Capture (CDC) allows capturing committed changes from a database in real-time and propagating those changes to downstream consumers [1][2]. CDC is becoming increasingly popular for use cases that require keeping multiple heterogeneous datastores in sync (like MySQL and ElasticSearch) and addresses challenges that exist with traditional techniques like dual-writes and distributed transactions [3][4]."
  - cleong 7y ago
    What your saying applys more to ELT, which is an efficient method when using batches. There's no reason why you can't stream ETL.
eganist 7y ago
> It's not for all orgs Then it doesn't seem to be solved. Seems like teams operating at a lean scale would have an issue with this, especially teams with lopsided business:engineering ratios
timClicks 7y ago
The remark wasn't that there weren't solutions, but that it is still a problem that needs a solution. Storing all data within a single database is a much simpler way to get started if you want to run analytic queries. You spin up a read slave that is tuned for expensive queries, rather than OLAP ones.

This is a solved problem, "Data Engineering" teams solve this by building a data pipeline. It's not for all orgs, but for a large org, this is worth doing right.

derision 7y ago

"data pipeline" is just the new trendy phrase for ETL, which the GP mentioned. just because a solution exists, does not make it a solved problem. it's not the right solution for everyone

segmondy 7y ago
no it's not. ETL is really database focused and batch focused , Extract, Transform, Load. Data pipeline, is a combination of streams and batch. For example, you can implement a Data Capture using something like https://debezium.io/ https://debezium.io/ Here's how Netflix solves it https://netflixtechblog.com/dblog-a-generic-change-data-capture-framework-69351fb9099b https://netflixtechblog.com/dblog-a-generic-change-data-capt... Overview "Change-Data-Capture (CDC) allows capturing committed changes from a database in real-time and propagating those changes to downstream consumers [1][2]. CDC is becoming increasingly popular for use cases that require keeping multiple heterogeneous datastores in sync (like MySQL and ElasticSearch) and addresses challenges that exist with traditional techniques like dual-writes and distributed transactions [3][4]."
- cleong 7y ago
  What your saying applys more to ELT, which is an efficient method when using batches. There's no reason why you can't stream ETL.

eganist 7y ago

> It's not for all orgs Then it doesn't seem to be solved. Seems like teams operating at a lean scale would have an issue with this, especially teams with lopsided business:engineering ratios

timClicks 7y ago

The remark wasn't that there weren't solutions, but that it is still a problem that needs a solution. Storing all data within a single database is a much simpler way to get started if you want to run analytic queries. You spin up a read slave that is tuned for expensive queries, rather than OLAP ones.