4 ms·
As someone that doesn't know much about ETL, I don't really see why it's worth mentioning in this context. Yes, what the OP is doing can be called extract/tran
by blub 8y ago
As someone that doesn't know much about ETL, I don't really see why it's worth mentioning in this context.
Yes, what the OP is doing can be called extract/transform/load, but that's an extremely general set of operations. Do you have some specific tool in mind that would make solving their problem a breeze? Is there a specific process that one should follow for ETL which was ignored?
- camel_Snake 8y agoNote: I'm just a backend dev, but I've been looking into data engineering a bit recently so I'm not terribly experienced. Happy to be corrected Problem here is they are using a system designed for the typical webapp use-case (OLTP). It's optimized to find a particular row or rows and then maybe update them with low latency. ETL tools are more optimized towards this sort of bulk data processing (OLAP) where we want to scan all the rows for a particular column and run some sort of analysis. A simple optimization some databases make for this is to orient the data so that each whole column is stored together rather than storing each row together. This makes it much quicker to run queries over an entire column or two in a table. Generally modern ETL is optimized towards working with LARGE amounts of data so there are many tools that are optimized for this.