3 ms·
no it's not. ETL is really database focused and batch focused , Extract, Transform, Load. Data pipeline, is a combination of streams and batch. For example,
by segmondy 7y ago
no it's not.
ETL is really database focused and batch focused , Extract, Transform, Load.
Data pipeline, is a combination of streams and batch. For example, you can implement a Data Capture using something like https://debezium.io/ https://debezium.io/
Here's how Netflix solves it https://netflixtechblog.com/dblog-a-generic-change-data-capture-framework-69351fb9099b https://netflixtechblog.com/dblog-a-generic-change-data-capt...
Overview
"Change-Data-Capture (CDC) allows capturing committed changes from a database in real-time and propagating those changes to downstream consumers [1][2]. CDC is becoming increasingly popular for use cases that require keeping multiple heterogeneous datastores in sync (like MySQL and ElasticSearch) and addresses challenges that exist with traditional techniques like dual-writes and distributed transactions [3][4]."
- cleong 7y agoWhat your saying applys more to ELT, which is an efficient method when using batches. There's no reason why you can't stream ETL.