4 ms·

> Koheesio is not in competition with other libraries. Yes, it is, because nobody wants to run multiple orchestrators, and the "What sets Koheesio apart from o

• zaptheimpaler 2y ago
  This is the kind of attitude that makes people, companies, researchers hesitant to publish code online. Its free code for everyone to see, they don't owe you anything. Its not necessarily a "product" for your consumption, its just a repo.
  • esafak 2y ago
    I don't think so; this is a corporate project, not an enthusiast's. They probably had to internally justify building it over using an existing solution, so they could simply share their rationale. I am not trying to rain on their parade.
    • fforflo 2y ago
      It's a library written by some devs who thought it might be useful to others, too, and/or are proud enough to share their work. It's not that they'll publish it in an SEC filing. For every n-th solution in the market, n-1 existing ones could have been used, but they weren't for (many times) good reasons. And looking at their Makefile and pyproject.toml I can see that they knew what they wanted.
      • klohto 2y ago
        yes, Nike is a corp where everyone sings kumbaya and the engineers have unlimited time to work on and publish libraries they are proud of, they don’t have to deliver specific values at all.
        • blitzar 2y ago
          the kumbaya and coding is only allowed in the 5 minute lunchbreak at the nike sweatshops
  • muspimerol 2y ago
    I think this is a fair criticism. You chose to share this on hackernews, which invites feedback (including constructive criticism). I don't see the problem ¯\_(ツ)_/¯ Do you expect people to only voice praise for open source projects?
    • mb5 2y ago
      Why do you think the poster is connected with the repo? I couldn’t see any link.
      • muspimerol 2y ago
        I don't. Regardless who posted it, discussion and constructive criticism is warranted in the comments.
        • yunohn 2y ago
          /s Like it or hate it, it’s how social media works. Everyone posts other people’s things, and then more people get together to rip it to shreds.
  • koliber 2y ago
    It's a fair set of questions, posed fairly directly, without any sugarcoating. If you read it as a ruthless critique, it stings. If you read it as constructive feedback, it can bring the author a lot of value. If the author can clearly show the value proposition of their library, it will get more adoption, and the community and the author will get value. If the author realizes they coded something that is already a well-solved problem, or a poorly-constructed alternative, the author and Nike could gain by throwing this away and going with a better alternative. Personally, I wasted too many hours of my life creating a solution that did not solve any problems. Had I done some thinking and/or market research ahead of time, it would have saved me ton's of time to work on more worthwhile endeavors. Good feedback is worth its weight in gold, even if it hurts a bit to hear it.
  • deleted 2y ago
    [deleted]
• bt1a 2y ago
  I'll agree that the pages about Koheesio hosted on nike's website are strangely empty of persuasive specifics (seems like generative AI used by someone who was in charge of writing it up lol..), but maybe the authors believe that someone who's seriously considering a data pipeline library would want to look at the actual code (which by the way, is structured quite well). Python doesn't "inspire confidence"? Think of the target audience. The frameworks claims to take typing and tests quite seriously, so perhaps your confidence is gained elsewhere. I didn't go through the codebase because I'm not interested, but your dismissive attitude doesn't radiate good faith
• vincnetas 2y ago
  Regarding comments saying that parent comment lacks good faith, and code is free so take it as is, should also consider that parent has all the rights to express his concerns and opinions. If you don't like criticism or inconvenient questions, just skip it or ignore. For example parent insights were interesting to me. Good questions that i can learn from how critically evaluate things.
• boffinAudio 2y ago
  >nobody wants to run multiple orchestrators Straw man argument. >I would expect someone who's seriously writing a new orchestrator in 2024 to cite the alternatives, their shortcomings, and how you intend to address them. Bonus points if you make a neat little table. Did you even try to read the docs before you launched this critical diatribe? From the docs (https://engineering.nike.com/koheesio/latest/tutorials/onboarding.html#transforming-to-koheesio https://engineering.nike.com/koheesio/latest/tutorials/onboa...): Advantages of Koheesio Using Koheesio instead of raw Spark has several advantages: Modularity: Each step in the pipeline (reading, transformation, writing) is encapsulated in its own class, making the code easier to understand and maintain. Reusability: Steps can be reused across different tasks, reducing code duplication. Testability: Each step can be tested independently, making it easier to write unit tests. Flexibility: The behavior of a task can be customized using a Context class. Consistency: Koheesio enforces a consistent structure for data processing tasks, making it easier for new developers to understand the codebase. Error Handling: Koheesio provides a consistent way to handle errors and exceptions in data processing tasks. Logging: Koheesio provides a consistent way to log information and errors in data processing tasks. In contrast, using the plain PySpark API for transformations can lead to more verbose and less structured code, which can be harder to understand, maintain, and test. It also doesn't provide the same level of error handling, logging, and flexibility as the Koheesio Transform class. It took me less than 15 seconds to find the solution to the problem you propose. How long did it take you to formulate your critique? Do you perhaps just have a prejudice against Nike (corporate haze), or is it an investment in a 'competing orchestrator' that is clouding your judgement?
  • esafak 2y ago
    Every modern workflow orchestrator does those things you quote, and more. You make it sound like they're innovations when they're table stakes. Why wouldn't you just use Flyte or Kubeflow? Also, the fact that they say that the alternative is "raw Spark" tells me either that they're confused, or not very good at explaining. Spark is used to execute tasks in a pipeline, not to orchestrate it.
    • thenaturalist 2y ago
• serial_dev 2y ago
  > Workflow orchestration is a mature category, as evidenced by the length of this list (...) Or could it be an evidence that the existing tools all have their flaws and any reasonably sized organization will hit these flaws pretty early, so many orgs come to the conclusion that the best approach is to just roll their own that fits their case relatively well?
  • thenaturalist 2y ago
    Are you at all familiar with the data integration / ETL framework space? If so, I think you would recognize how unreasonable the presumption of your comment is. It certainly strikes me as such. Most if not all the big OSS frameworks and commercial offerings originated in one big corp and subsequently moved into the Apache direction (Airflow) or spun out into their own companies.
  • whimsicalism 2y ago
    no offense, but you clearly don’t know much about orchestrators
• gonzo41 2y ago
  Every task orchestration tool kinda has a crappy security model. Sure they're a ton of them but when you start putting it all over the place it's just a hectic to get right. That is a feel the space someone could make gains in a big way.
• whimsicalism 2y ago
  yeah they need to explain why this is better than flyte
• rockostrich 2y ago
  > Workflow orchestration is a mature category As far as I can tell from the docs, this is not an orchestrator at all. It is a Python-wrapper for certain runtimes like PySpark. I don't see anything in the docs that mentions DAGs, dependency definitions, scheduling, or deployment.