3 ms·

This has been called "ETL" for as long as I can remember. Could someone explain why they're using "ELT"?

by kbd 6y ago

visch 6y ago
"Transform" step is happening after the load to data warehouse here. https://blog.getdbt.com/what--exactly--is-dbt-/ https://blog.getdbt.com/what--exactly--is-dbt-/ is a pretty good overview.
tayloramurphy 6y ago
This article[0] from Census is a good primer on the difference and what's driving it. But the TL;DR is that storage is cheap and it's easier to change SQL-based transformations using something like dbt[1] than to re-extract the data when the business logic changes. [0] https://blog.getcensus.com/dbt-the-etl-elt-disrupter/ https://blog.getcensus.com/dbt-the-etl-elt-disrupter/ [1] https://www.getdbt.com/ https://www.getdbt.com/
JPKab 6y ago
While there are technical differences with transforming the data within it's storage location/database, I've interacted with my former employer's board of directors enough to know there's a bigger, rather obnoxious reason for this: VC valuations in Silicon Valley for traditional "ETL" tools isn't great. The term "ETL" has baggage of Talend and other big enterprise tools that have an aura of "not being user friendly". "ELT" is intended to signal that a tool is trying really hard to not be like those sad, difficult to use, dirty "ETL" tools. That's the real reason. There really isn't an actual difference in how the tools work. Traditional ETL tools are perfectly capable of loading the data and initiating the transform in the database it's being stored in. This has been done for decades. I have been repeatedly lectured by senior strategy team executives about using the proper vernacular to ensure perception of value is maximized in the trendy, buzzword driven VC world. So fucking stupid.
nojito 6y ago
Marketing. There's been a huge marketing push of tools that designed to automate the SQL portion of data pipeline. Doesn't make much sense to me, because anyone who's been in the data space quickly sees the rampant marketing of different tools that all do the same thing but simply change the order of the pipeline.
kpierce 6y ago
Keeping a raw version of the data make it easier to change in the future. If you use s3 as a data lake you can transform later into the shape you want it. Also can use tools like presto where schema is on read instead of hardcoded so transform step is moved to another time.
snidane 6y ago
ELT means: let's focus on moving data between from system A to system B and skip the transform part, which is a business of whoever has access to the target system. Which is usually a sql database so you can then do arbitrary data transforms there. In other words just get the data in house into a queryable form and worry about the T part later.

This has been called "ETL" for as long as I can remember. Could someone explain why they're using "ELT"?

by kbd 6y ago

This has been called "ETL" for as long as I can remember. Could someone explain why they're using "ELT"?

"Transform" step is happening after the load to data warehouse here. https://blog.getdbt.com/what--exactly--is-dbt-/ https://blog.getdbt.com/what--exactly--is-dbt-/ is a pretty good overview.

tayloramurphy 6y ago

This article[0] from Census is a good primer on the difference and what's driving it. But the TL;DR is that storage is cheap and it's easier to change SQL-based transformations using something like dbt[1] than to re-extract the data when the business logic changes. [0] https://blog.getcensus.com/dbt-the-etl-elt-disrupter/ https://blog.getcensus.com/dbt-the-etl-elt-disrupter/ [1] https://www.getdbt.com/ https://www.getdbt.com/

JPKab 6y ago

While there are technical differences with transforming the data within it's storage location/database, I've interacted with my former employer's board of directors enough to know there's a bigger, rather obnoxious reason for this: VC valuations in Silicon Valley for traditional "ETL" tools isn't great. The term "ETL" has baggage of Talend and other big enterprise tools that have an aura of "not being user friendly". "ELT" is intended to signal that a tool is trying really hard to not be like those sad, difficult to use, dirty "ETL" tools. That's the real reason. There really isn't an actual difference in how the tools work. Traditional ETL tools are perfectly capable of loading the data and initiating the transform in the database it's being stored in. This has been done for decades. I have been repeatedly lectured by senior strategy team executives about using the proper vernacular to ensure perception of value is maximized in the trendy, buzzword driven VC world. So fucking stupid.

nojito 6y ago

Marketing. There's been a huge marketing push of tools that designed to automate the SQL portion of data pipeline. Doesn't make much sense to me, because anyone who's been in the data space quickly sees the rampant marketing of different tools that all do the same thing but simply change the order of the pipeline.

kpierce 6y ago

Keeping a raw version of the data make it easier to change in the future. If you use s3 as a data lake you can transform later into the shape you want it. Also can use tools like presto where schema is on read instead of hardcoded so transform step is moved to another time.

snidane 6y ago

ELT means: let's focus on moving data between from system A to system B and skip the transform part, which is a business of whoever has access to the target system. Which is usually a sql database so you can then do arbitrary data transforms there. In other words just get the data in house into a queryable form and worry about the T part later.