3 ms·
From their docs: > Koheesio is a Python library that simplifies the development of data engineering pipelines. It provides a structured way I think this pretty
by tiew9Vii 2y ago
From their docs:
> Koheesio is a Python library that simplifies the development of data engineering pipelines. It provides a structured way
I think this pretty much sums it up, "a structured way".
It's looks to be a thin wrapper around spark to provide a consistent way to structure ETL jobs. They've implemented a mini dsl defining jobs as a datastructure on top of Spark.
I've seen several companies build stuff similar to this internally, defining jobs as a data structure. It all amounts to each company having their own internal conventions, their own view of what is easier for their devs and creating a framework for it. Nike have just decided to make theirs public.
You can do all this simply with simple spark scripts. Personally I'd use simple spark scripts. Big companies with lots of people love making these tools as companies love conventions, their conventions, their style guides, deal with staff churn/on-boarding frequently so believe these kind of things make that easier.
Probably makes sense in Nike as a way of organizing their ETL jobs but that looks to be all it brings. A way to structure/define your simple spark jobs the way Nike devs believe it should be done.