3 ms·

I used to work a little with ETLs, Spark, Storm, etc and I honestly don't understand the value proposition of this library. I'm no data engineer expert by any m

I used to work a little with ETLs, Spark, Storm, etc and I honestly don't understand the value proposition of this library. I'm no data engineer expert by any means (it was like 2 years working on data eng stuff about 30% of the time 5+ years ago), but I expected that at least I'd get what this is useful for.

ubercore 2y ago
This, at a glance, seems pretty simplistic. A neat project, but not something I would have expected on HN front page.
- whoiscroberts 2y ago
  After decades of working on overly abstracted clever applications the only place I see elegance is in simplicity. I’d like to see more libraries like this on the front page.
tiew9Vii 2y ago
From their docs: > Koheesio is a Python library that simplifies the development of data engineering pipelines. It provides a structured way I think this pretty much sums it up, "a structured way". It's looks to be a thin wrapper around spark to provide a consistent way to structure ETL jobs. They've implemented a mini dsl defining jobs as a datastructure on top of Spark. I've seen several companies build stuff similar to this internally, defining jobs as a data structure. It all amounts to each company having their own internal conventions, their own view of what is easier for their devs and creating a framework for it. Nike have just decided to make theirs public. You can do all this simply with simple spark scripts. Personally I'd use simple spark scripts. Big companies with lots of people love making these tools as companies love conventions, their conventions, their style guides, deal with staff churn/on-boarding frequently so believe these kind of things make that easier. Probably makes sense in Nike as a way of organizing their ETL jobs but that looks to be all it brings. A way to structure/define your simple spark jobs the way Nike devs believe it should be done.
anentropic 2y ago
it looks like a layer of sugar over PySpark seems to be Spark-only AFAICT?

ubercore 2y ago

This, at a glance, seems pretty simplistic. A neat project, but not something I would have expected on HN front page.

whoiscroberts 2y ago
After decades of working on overly abstracted clever applications the only place I see elegance is in simplicity. I’d like to see more libraries like this on the front page.

tiew9Vii 2y ago

From their docs: > Koheesio is a Python library that simplifies the development of data engineering pipelines. It provides a structured way I think this pretty much sums it up, "a structured way". It's looks to be a thin wrapper around spark to provide a consistent way to structure ETL jobs. They've implemented a mini dsl defining jobs as a datastructure on top of Spark. I've seen several companies build stuff similar to this internally, defining jobs as a data structure. It all amounts to each company having their own internal conventions, their own view of what is easier for their devs and creating a framework for it. Nike have just decided to make theirs public. You can do all this simply with simple spark scripts. Personally I'd use simple spark scripts. Big companies with lots of people love making these tools as companies love conventions, their conventions, their style guides, deal with staff churn/on-boarding frequently so believe these kind of things make that easier. Probably makes sense in Nike as a way of organizing their ETL jobs but that looks to be all it brings. A way to structure/define your simple spark jobs the way Nike devs believe it should be done.

anentropic 2y ago

it looks like a layer of sugar over PySpark seems to be Spark-only AFAICT?