3 ms·

After being frustrated with building 'traditional' ETL (Extract-Transform-Load) pipelines - and around the same time as the famous 'Engineers Shouldn’t Write ET

After being frustrated with building 'traditional' ETL (Extract-Transform-Load) pipelines - and around the same time as the famous 'Engineers Shouldn’t Write ETL' blog post - we started building a framework/toolkit to allow Technical Business Analysts to be able to build reliable data pipelines without much developer support: Arc. This has been implemented as a Jupyter Notebooks extension. Arc is declarative and currently targets the Apache Spark execution engine but the abstracted API allows replacing execution engines without having to rewrite the logic or intent of the pipeline in future. It supports parameterized notebooks to build complex pipelines which can be executed in CICD environments for safe deployment. We would be interested to hear your feedback.

armanboyaci 6y ago
We also inspired from the same blog post ('Engineers Shouldn’t Write ETL') and built our own internal ETL tools. Our primary design goal was the system to be self-service for data scientists. Since our data scientists use pandas dataframes and jupyter notebooks all the time, we built the system around these two: (1) We have a library (that we call pype) acting an interface between the database and python dataframes (similar to .to_csv method), so there is no SQL queries in ETL scripts, (2) schedule (parametrized) notebooks using some special keywords. We have a demo screencast: https://drive.google.com/file/d/1SVTduaIH_3IsJ-QoGI4mLYZE8JvCXThA/view?usp=sharing https://drive.google.com/file/d/1SVTduaIH_3IsJ-QoGI4mLYZE8Jv...
- seddonm1 6y ago
  Looks good. It is nice to see how much influence the 'Engineers Shouldn't Write ETL' post had! With Apache Arrow (https://arrow.apache.org/ https://arrow.apache.org/) I think the future looks very bright for both of our projects. It is important to have standard open source libraries and my early experiments have shown very good performance results.
moreati 6y ago
> the famous 'Engineers Shouldn’t Write ETL' blog post for anyone else wondering, this appears to be Engineers Shouldn’t Write ETL: A Guide to Building a High Functioning Data Science Department, by Jeff Magnusson in 2016 https://multithreaded.stitchfix.com/blog/2016/03/16/engineers-shouldnt-write-etl/ https://multithreaded.stitchfix.com/blog/2016/03/16/engineer....
tuyguntn 6y ago
> repeatable in that if a job is executed multiple times it will produce the same result can you please elaborate more on this statement, does it mean you dump the data first time job is executed? Because if data is evolving, at some point execution should produce different results
- seddonm1 6y ago
  The better statement would be that this facilitates the development of idempotent jobs and aims to minimise side-effects.
johnx123-up 6y ago
What is your opinion about https://min.io/ https://min.io/ (It also has S3 compatibility AFAIK)
- tyingq 6y ago
  I think you replied to the wrong thread. Perhaps you meant here: https://news.ycombinator.com/item?id=26577176 https://news.ycombinator.com/item?id=26577176
  - johnx123-up 6y ago
    Sorry, you're right.

armanboyaci 6y ago

We also inspired from the same blog post ('Engineers Shouldn’t Write ETL') and built our own internal ETL tools. Our primary design goal was the system to be self-service for data scientists. Since our data scientists use pandas dataframes and jupyter notebooks all the time, we built the system around these two: (1) We have a library (that we call pype) acting an interface between the database and python dataframes (similar to .to_csv method), so there is no SQL queries in ETL scripts, (2) schedule (parametrized) notebooks using some special keywords. We have a demo screencast: https://drive.google.com/file/d/1SVTduaIH_3IsJ-QoGI4mLYZE8JvCXThA/view?usp=sharing https://drive.google.com/file/d/1SVTduaIH_3IsJ-QoGI4mLYZE8Jv...

seddonm1 6y ago
Looks good. It is nice to see how much influence the 'Engineers Shouldn't Write ETL' post had! With Apache Arrow (https://arrow.apache.org/ https://arrow.apache.org/) I think the future looks very bright for both of our projects. It is important to have standard open source libraries and my early experiments have shown very good performance results.

moreati 6y ago

> the famous 'Engineers Shouldn’t Write ETL' blog post for anyone else wondering, this appears to be Engineers Shouldn’t Write ETL: A Guide to Building a High Functioning Data Science Department, by Jeff Magnusson in 2016 https://multithreaded.stitchfix.com/blog/2016/03/16/engineers-shouldnt-write-etl/ https://multithreaded.stitchfix.com/blog/2016/03/16/engineer....

tuyguntn 6y ago

> repeatable in that if a job is executed multiple times it will produce the same result can you please elaborate more on this statement, does it mean you dump the data first time job is executed? Because if data is evolving, at some point execution should produce different results

seddonm1 6y ago
The better statement would be that this facilitates the development of idempotent jobs and aims to minimise side-effects.

johnx123-up 6y ago

What is your opinion about https://min.io/ https://min.io/ (It also has S3 compatibility AFAIK)

tyingq 6y ago
I think you replied to the wrong thread. Perhaps you meant here: https://news.ycombinator.com/item?id=26577176 https://news.ycombinator.com/item?id=26577176
- johnx123-up 6y ago
  Sorry, you're right.