4 ms·

What exactly IS an "ETL framework"? I looked at both this project and Apache Airflow, and I'm not quite sure I understand...

by notamy 8y ago

Diederich 8y ago
https://en.wikipedia.org/wiki/Extract,_transform,_load https://en.wikipedia.org/wiki/Extract,_transform,_load
abd12 8y ago
To me, it's easiest to think of it as "Make, for data". Your ETL pipelines are often a complex graph of dependencies. If some step halfway down the chain fails, you don't want to restart at the very beginning -- you want to restart where it failed and keep moving. The ETL frameworks (Airflow, Luigi, now Mara) help with this, allowing you to build dependency graphs in code, determine which dependencies are already satisfied, and process those which are not. They'll usually contain helper code for common ETL tasks, such as interacting with a database, writing to/reading from S3, or running shell scripts.
- martin_loetzsch 8y ago
  (author here). Mara data integration is indeed a glorified version of Make (with cost based scheduling and lots of visualizations)
shadowmint 8y ago
Basically software for batch processing a tonne of data. eg. importing a SAP feed into a database, or loading a bunch of csv files, or like processing a bunch of images... ...anything where you have to convert data from some source through a series of steps (typically a DAG) into some useful output. However, its often misused. For example, if you have a trivial amount of data, or trivial process an ETL is over engineered cruft where a simple script would do. So there are many (rubbish) ‘simple ETL frameworks’ which offer zero value and technical complexity for no benefit. Unless you need multiple servers processing data through multiple steps and you need the auditing and process control... you probably don’t need an ETL, just a simple script.
- cosmie 8y ago
  > you probably don’t need an ETL, just a simple script. +1 > Unless you need multiple servers processing data through multiple steps and you need the auditing and process control I'll stress the "multiple servers" part. You can add in a substantial amount of multiple, sequential steps and auditing and process control in a simple script. The part that adds orders of magnitude worth of complexity and operational overhead and points of failure is being able to distribute it to multiple servers. Distributed architectures are operationally and architecturally expensive. And far more often than not, completely unnecessary for a given use case.
  - bduerst 8y ago
    Some ETLs these days are simple scripts - e.g. Spark, Dataflow - with their config files.
    - cosmie 8y ago
      Being able to define an ETL workload within a simple script is not the same as your ETL system itself being a simple script. While I love both Spark and Dataflow, both of them are incredibly complex distributed systems with very high operational costs. Someone, somewhere is paying a lot of money to have an operational resource maintain that complexity. Whether you have an internal devops resource doing so or you're using a managed service, you're paying for that complexity somehow. And, for a lot of workloads, you aren't actually getting any more value than you would from standing up a ~$50/month standard Debian/Ubuntu server and a set of simple scripts on it.
      - Mironor 8y ago
        You don't have to have a cluster to run spark scripts, setting master to `local` (and running it on one machine) is often enough for small anounts of data.
      - bduerst 8y ago
        They don't have high operational costs - you can run them as a script on your local machine. You're making them out to be more complex than they really are.
collyw 8y ago
Extract Transform Load. Getting data from one place to another - usually the end place being a structured database (or it ought to be). If you have been in the industry over 10 years you probably used Perl or Python scripts.
slenk 8y ago
See the other comments also, but ETL stands for "Extract, Transform, Load"
thibaut_barrere 8y ago
(I’m the author of a popular Ruby ETL framework named Kiba ETL) An « ETL » framework is for me something that will allow you to express and run maintainable data pipelines, as a coder. Kiba is exactly that, in the sense that it does not include built-in sources/transforms/destinations, but rather defines a set of guidelines you can follow to easily implement reusable ETL components with high quality & unit tests. More information in the README: https://github.com/thbar/kiba/blob/master/README.md https://github.com/thbar/kiba/blob/master/README.md
- me_bx 8y ago
  +1 for the maintainability of an ETL framework brings, compared to "handmade" scripting...
KirinDave 8y ago
Let me give you a quippy answer: Cron is awful for cloud-based deployments trying to keep data from multiple sources in sync and compute business-specific metrics over them. ETL frameworks seek to address this pain by building a large stack of software around scheduling the running of and unifying the writing of these data munging tasks. As such, they tend to be as awful as the sum of their dependencies minus some small constant quality factor. So, more awful than any one part but less awful than the storm you'd be howling into if you didn't take some sort of infrastructural solution.

What exactly IS an "ETL framework"? I looked at both this project and Apache Airflow, and I'm not quite sure I understand...

by notamy 8y ago

What exactly IS an "ETL framework"? I looked at both this project and Apache Airflow, and I'm not quite sure I understand...

Diederich 8y ago

https://en.wikipedia.org/wiki/Extract,_transform,_load https://en.wikipedia.org/wiki/Extract,_transform,_load

abd12 8y ago

To me, it's easiest to think of it as "Make, for data". Your ETL pipelines are often a complex graph of dependencies. If some step halfway down the chain fails, you don't want to restart at the very beginning -- you want to restart where it failed and keep moving. The ETL frameworks (Airflow, Luigi, now Mara) help with this, allowing you to build dependency graphs in code, determine which dependencies are already satisfied, and process those which are not. They'll usually contain helper code for common ETL tasks, such as interacting with a database, writing to/reading from S3, or running shell scripts.

martin_loetzsch 8y ago
(author here). Mara data integration is indeed a glorified version of Make (with cost based scheduling and lots of visualizations)

shadowmint 8y ago

Basically software for batch processing a tonne of data. eg. importing a SAP feed into a database, or loading a bunch of csv files, or like processing a bunch of images... ...anything where you have to convert data from some source through a series of steps (typically a DAG) into some useful output. However, its often misused. For example, if you have a trivial amount of data, or trivial process an ETL is over engineered cruft where a simple script would do. So there are many (rubbish) ‘simple ETL frameworks’ which offer zero value and technical complexity for no benefit. Unless you need multiple servers processing data through multiple steps and you need the auditing and process control... you probably don’t need an ETL, just a simple script.

cosmie 8y ago
> you probably don’t need an ETL, just a simple script. +1 > Unless you need multiple servers processing data through multiple steps and you need the auditing and process control I'll stress the "multiple servers" part. You can add in a substantial amount of multiple, sequential steps and auditing and process control in a simple script. The part that adds orders of magnitude worth of complexity and operational overhead and points of failure is being able to distribute it to multiple servers. Distributed architectures are operationally and architecturally expensive. And far more often than not, completely unnecessary for a given use case.
- bduerst 8y ago
  Some ETLs these days are simple scripts - e.g. Spark, Dataflow - with their config files.
  - cosmie 8y ago
    Being able to define an ETL workload within a simple script is not the same as your ETL system itself being a simple script. While I love both Spark and Dataflow, both of them are incredibly complex distributed systems with very high operational costs. Someone, somewhere is paying a lot of money to have an operational resource maintain that complexity. Whether you have an internal devops resource doing so or you're using a managed service, you're paying for that complexity somehow. And, for a lot of workloads, you aren't actually getting any more value than you would from standing up a ~$50/month standard Debian/Ubuntu server and a set of simple scripts on it.
    - Mironor 8y ago
      You don't have to have a cluster to run spark scripts, setting master to `local` (and running it on one machine) is often enough for small anounts of data.
    - bduerst 8y ago
      They don't have high operational costs - you can run them as a script on your local machine. You're making them out to be more complex than they really are.

collyw 8y ago

Extract Transform Load. Getting data from one place to another - usually the end place being a structured database (or it ought to be). If you have been in the industry over 10 years you probably used Perl or Python scripts.

slenk 8y ago

See the other comments also, but ETL stands for "Extract, Transform, Load"

thibaut_barrere 8y ago

(I’m the author of a popular Ruby ETL framework named Kiba ETL) An « ETL » framework is for me something that will allow you to express and run maintainable data pipelines, as a coder. Kiba is exactly that, in the sense that it does not include built-in sources/transforms/destinations, but rather defines a set of guidelines you can follow to easily implement reusable ETL components with high quality & unit tests. More information in the README: https://github.com/thbar/kiba/blob/master/README.md https://github.com/thbar/kiba/blob/master/README.md

me_bx 8y ago
+1 for the maintainability of an ETL framework brings, compared to "handmade" scripting...

KirinDave 8y ago

Let me give you a quippy answer: Cron is awful for cloud-based deployments trying to keep data from multiple sources in sync and compute business-specific metrics over them. ETL frameworks seek to address this pain by building a large stack of software around scheduling the running of and unifying the writing of these data munging tasks. As such, they tend to be as awful as the sum of their dependencies minus some small constant quality factor. So, more awful than any one part but less awful than the storm you'd be howling into if you didn't take some sort of infrastructural solution.