4 ms·

"data pipeline" is just the new trendy phrase for ETL, which the GP mentioned. just because a solution exists, does not make it a solved problem. it's not the r

• segmondy 7y ago
  no it's not. ETL is really database focused and batch focused , Extract, Transform, Load. Data pipeline, is a combination of streams and batch. For example, you can implement a Data Capture using something like https://debezium.io/ https://debezium.io/ Here's how Netflix solves it https://netflixtechblog.com/dblog-a-generic-change-data-capture-framework-69351fb9099b https://netflixtechblog.com/dblog-a-generic-change-data-capt... Overview "Change-Data-Capture (CDC) allows capturing committed changes from a database in real-time and propagating those changes to downstream consumers [1][2]. CDC is becoming increasingly popular for use cases that require keeping multiple heterogeneous datastores in sync (like MySQL and ElasticSearch) and addresses challenges that exist with traditional techniques like dual-writes and distributed transactions [3][4]."
  • cleong 7y ago
    What your saying applys more to ELT, which is an efficient method when using batches. There's no reason why you can't stream ETL.