Note: I'm just a backend dev, but I've been looking into data engineering a bit recently so I'm not terribly experienced. Happy to be corrected Problem here i

Note: I'm just a backend dev, but I've been looking into data engineering a bit recently so I'm not terribly experienced. Happy to be corrected Problem here is they are using a system designed for the typical webapp use-case (OLTP). It's optimized to find a particular row or rows and then maybe update them with low latency. ETL tools are more optimized towards this sort of bulk data processing (OLAP) where we want to scan all the rows for a particular column and run some sort of analysis. A simple optimization some databases make for this is to orient the data so that each whole column is stored together rather than storing each row together. This makes it much quicker to run queries over an entire column or two in a table. Generally modern ETL is optimized towards working with LARGE amounts of data so there are many tools that are optimized for this.