4 ms·

This is an ML framework (I'm not actually sure what it does) made by McKinsey. I don't know if it was always open sourced but now it has been "donated to the co

This is an ML framework (I'm not actually sure what it does) made by McKinsey. I don't know if it was always open sourced but now it has been "donated to the community" which I think means abandoned? I'd be very curious to hear if anyone has used it and what impressions are on it's value. I've only heard some businessy talking points about it, and am hopefully understandably skeptical. If someone not from McKinsey had used it, I'd love to hear impressions.

waylonwalker 5y ago
I use it quite heavily at my day job. The core pieces of kedro are indispensable on large projects (Pipeline, Catalog, DataSets, Runner, hooks). The marketing has a lot of focus on the template, It's a great template, but not the key feature that brings me back to it. Really like that I never have to worry about io, the catalog abstracts this away. I don't worry about long runs to get started with my work. It saves all intermediate steps automatically, and the Pipeline (DAG) lets me run just the parts that I need for my task. Before using kedro I would need to scaffold out a way to save all of these intermediate steps, or suffer from long run times to get started on my task. Lastly the DAG is indispensible during environment migrations, it can quickly tell me all of its edges that I need to be concerned with picking up and moving with me. Unlike similar projects kedro is just a python framework. It lets me build, deploy, and orchestrate however works best for my team.
- barefeg 5y ago
  Could you point me to the component responsible for storing the intermediate data of the DAG run? I was looking for this but couldn’t find it from quickly scanning the docs
  - joelschw 5y ago
    Here you go - https://kedro.readthedocs.io/en/0.17.6/09_development/03_commands_reference.html#modifying-a-kedro-run https://kedro.readthedocs.io/en/0.17.6/09_development/03_com...
  - waylonwalker 5y ago
    By adding a catalog entry for each of your datasets with a filepath argument, it will save them to that place. All their datasets use fsspec under the hood so this path can be an s3 bucket or a local path and it all just works.
- kjkjadksj 5y ago
  Is this different from snakemake?
  - joelschw 5y ago
    Some similar concepts, but Kedro is specialised on the ML engineering workflow and team collaboration
oneoffmk999 5y ago
It’s a framework to organize data science code into pipelines. E.g. if your default might be a series of Python files connected by a master script and some config files, Kedro organizes them into an explicit Pipeline object. Think loading, cleaning, feature gen, model training, mode predictions, etc.
holocen 5y ago
Am from McK not involved with Kedro, it’s been open source for awhile now and I don’t think we are abandoning it, but it was apparently donated to the Linux foundation. As far as impressions of it as a tool go it’s really just an opinionated way to build a pipeline and structure a project. Which is pretty useful when you have many. If you’re doing something that you know will be one off it is a bit overkill. Like using airflow when a makefile will do. There are some pretty nice plug-ins though if you want to try and have an opinionated ecosystem.
joelschw 5y ago
Hello maintainer here - hopefully some of our open source users chime in, but my favourite will always be NASA: https://github.com/nasa/ML-airport-configuration https://github.com/nasa/ML-airport-configuration Donation is the normal term for proejcts joining the foundation - in doing so you establish a steering committee where members are entitled to voting rights. To graduate as an incubation project, 5 organisations need to join your board and the project is thus the priorities of the project will no longer be driven by just one organisation. In the short term there is still a full time internal team staffed and maintaining the project, but excitingly we now have a mechanism for new collaborators to properly come onboard.
- joelschw 5y ago
  I should clarify by short term - the foreseeable future is a more appropriate description
ekianjo 5y ago
Worked with McK on a modeling tool built with a data pipeline in Kedro - it's very opinionated as to how the pipeline has to be built, and I was not super convinced that it would be useful to use it in any other context (as in, outside of a collaboration with McK). Also, YMMV, but I have never heard of any company doing ML or data pipelines using Kedro on their own.

waylonwalker 5y ago

I use it quite heavily at my day job. The core pieces of kedro are indispensable on large projects (Pipeline, Catalog, DataSets, Runner, hooks). The marketing has a lot of focus on the template, It's a great template, but not the key feature that brings me back to it. Really like that I never have to worry about io, the catalog abstracts this away. I don't worry about long runs to get started with my work. It saves all intermediate steps automatically, and the Pipeline (DAG) lets me run just the parts that I need for my task. Before using kedro I would need to scaffold out a way to save all of these intermediate steps, or suffer from long run times to get started on my task. Lastly the DAG is indispensible during environment migrations, it can quickly tell me all of its edges that I need to be concerned with picking up and moving with me. Unlike similar projects kedro is just a python framework. It lets me build, deploy, and orchestrate however works best for my team.

barefeg 5y ago
Could you point me to the component responsible for storing the intermediate data of the DAG run? I was looking for this but couldn’t find it from quickly scanning the docs
- joelschw 5y ago
  Here you go - https://kedro.readthedocs.io/en/0.17.6/09_development/03_commands_reference.html#modifying-a-kedro-run https://kedro.readthedocs.io/en/0.17.6/09_development/03_com...
- waylonwalker 5y ago
  By adding a catalog entry for each of your datasets with a filepath argument, it will save them to that place. All their datasets use fsspec under the hood so this path can be an s3 bucket or a local path and it all just works.
kjkjadksj 5y ago
Is this different from snakemake?
- joelschw 5y ago
  Some similar concepts, but Kedro is specialised on the ML engineering workflow and team collaboration

oneoffmk999 5y ago

It’s a framework to organize data science code into pipelines. E.g. if your default might be a series of Python files connected by a master script and some config files, Kedro organizes them into an explicit Pipeline object. Think loading, cleaning, feature gen, model training, mode predictions, etc.

holocen 5y ago

Am from McK not involved with Kedro, it’s been open source for awhile now and I don’t think we are abandoning it, but it was apparently donated to the Linux foundation. As far as impressions of it as a tool go it’s really just an opinionated way to build a pipeline and structure a project. Which is pretty useful when you have many. If you’re doing something that you know will be one off it is a bit overkill. Like using airflow when a makefile will do. There are some pretty nice plug-ins though if you want to try and have an opinionated ecosystem.

joelschw 5y ago

Hello maintainer here - hopefully some of our open source users chime in, but my favourite will always be NASA: https://github.com/nasa/ML-airport-configuration https://github.com/nasa/ML-airport-configuration Donation is the normal term for proejcts joining the foundation - in doing so you establish a steering committee where members are entitled to voting rights. To graduate as an incubation project, 5 organisations need to join your board and the project is thus the priorities of the project will no longer be driven by just one organisation. In the short term there is still a full time internal team staffed and maintaining the project, but excitingly we now have a mechanism for new collaborators to properly come onboard.

joelschw 5y ago
I should clarify by short term - the foreseeable future is a more appropriate description

ekianjo 5y ago

Worked with McK on a modeling tool built with a data pipeline in Kedro - it's very opinionated as to how the pipeline has to be built, and I was not super convinced that it would be useful to use it in any other context (as in, outside of a collaboration with McK). Also, YMMV, but I have never heard of any company doing ML or data pipelines using Kedro on their own.