10 ms·

Kedro – Creating reproducible, maintainable and modular data science code

96 points by maydemir 5y ago

rladd 5y ago
Anything similar for R?
- joelschw 5y ago
  Some users have reported the Targets library is similar for that ecosystem
- idomi 5y ago
  We've built the Ploomber to work with R as well comparing to others who believe only python is for ML. Ploomber is an open source tool that can integrate with R and R studio out of the box. Check it out! https://github.com/ploomber/ploomber https://github.com/ploomber/ploomber
- kjkjadksj 5y ago
  I use snakemake to call up r scripts in pipelines
tomrod 5y ago
Good to see McKinsey donating something potentially beneficial to the world. Kudos to the team that developed this.
- Proven 5y ago
lordgroff 5y ago
I love this project, not from McK. I mean I guess it's "opinionated" but having worked in this field for some time, ML code and pipelines are often laughably immature, stuck with spit, glue and sellotape. Kedro forces people into a bit of a straight jacket for sure, but man does it make things more readable. Plus points, it's very extensible so if there's a design element you're missing or one of its opinions you just can't abide by, then you can extend it. I think it deserves more exposure.
- bshipp 5y ago
  It might be opinionated but it's no less accurate. Data science is many researchers first foray into programming in any language and, speaking from my own path, was a relatively self-directed discipline in the early days. I often longed for a 'standard approach' but many of the projects that suggested them were packed full of boilerplate that made code even harder to read and write, especially if your background wasn't heavily into OOP.
  - waylonwalker 5y ago
    I feel ya. Before using kedro I was using a version of the cookiecutter for data science template that I modified to have something kind of like their data catalog. Before that it was so wild west, every project was its own unique thing.
- waylonwalker 5y ago
  Maybe a better word is that it comes with an opinionated template? The core parts of Kedro can all be used individually as you wish. Only want a DAG, you can use just the pipeline, Want a catalog setup so you can store intermediate steps, use the Catalog. You can even make kedro project using all the features in a single file with surprisingly little boilerplate.
troiskaer 5y ago
How does Kedro compare to MLFlow and Metaflow?
- stevesimmons 5y ago
  And Prefect
- joelschw 5y ago
  Kedro sort of fits into a niche where it just overlaps somewhat with 'orchestrators' like Prefect, Metaflow, Dagster, Airflow and others. What makes it slightly different is that it it is focused on the rapid development journey to production, providing guardrails for teams to co-develop ML projects in a way that nudges software engineering best practice and clean code. The 'finished article' in many cases should be deployed in production in one of those tools which provide specialised bells and whistles like scheduling, monitoring and observability. Regarding MLFlow, there is also a slight overlap in terms of experimentation, but not things like model serving. Kedro has a mechanism to track experiments, but it's more designed to give users with zero infrastructure something for free out of the box. It's been built in a way that it can be repurposed for more dedicated experiment tracking tools - the folks at neptune.ai built their own plug-in for this purpose: https://docs.neptune.ai/integrations-and-supported-tools/automation-pipelines/kedro https://docs.neptune.ai/integrations-and-supported-tools/aut...
  - troiskaer 5y ago
    Seems like Kedro has a similar thesis to Metaflow - I will look into it.
    - vtuulos 5y ago
      Yep, Kedro and Metaflow are more similar to each other than to other generic DAG orchestrators like Airflow. Kedro and Metaflow make it easier to develop robust ML projects where orchestration plays an important role but it is not everything. They are two separate projects, so the way how they approach the problem differs greatly in details.
oneoffmk999 5y ago
I am from McKinsey. I do not work on Kedro, but am a data scientist who has used it. I have mixed feelings on Kedro. Pros: * Forces data scientists to produce an end product that is not poorly organized Jupyter notebooks. * Data Catalog is good for well structured systems * Pipeline visualization stack is great (Kedro viz) * Config options are pretty good * Seems stable. Dev team is pretty good on this and avoiding breaking changes. Cons: * Data catalog is kind of bad for any non structured setup with flat file data with manual file movement (which is bad to begin with but sometimes that’s life) * Productivity of making brand new data science code seems to drop when data scientists leave notebooks and * Most of the time I get brought into a client context because the client doesn’t know anything about data scientist. A lot of data scientists, from both parties, come from academic backgrounds and aren’t great at code. The nice thing about notebooks is that they run. Kedro requires you to create pipeline and node objects to wrap around your code before it runs. It requires some familiarity with Kedro to understand, run, or modify. This makes it seem like a bad idea to dump on a novice client. If the data scientist on their side inheriting it doesn’t really get it, or leaves, there’s unlikely to be enough internal knowledge to maintain it. I try to avoid pushing any new tech stacks on my clients where I can for this reason. So… I like it but don’t love it for consulting work, which is ironic.
- sails 5y ago
  No pros or cons on deploying models. Seems like that is a significant part of the functional value?
  - oneoffmk999 5y ago
    Not my area of expertise. Can’t comment on it either way. But I would tend imagine it’s a strength of Kedro.
  - joelschw 5y ago
    Will point to some experiences open source users have had on this journey https://medium.com/hacking-talent/production-code-for-data-science-and-our-experience-with-kedro-60bb69934d1f https://medium.com/hacking-talent/production-code-for-data-s... https://medium.com/google-cloud/migrate-kedro-pipeline-on-vertex-ai-fa3f2c6f7aad https://medium.com/google-cloud/migrate-kedro-pipeline-on-ve...
bshipp 5y ago
I've apparently already starred this repo in the past and have no recollection of it. I really need to find a mechanism to sort and categorize my starred repositories; this is getting out of hand.
- pjot 5y ago
  You should be able to do that natively in GitHub with lists! https://docs.github.com/en/get-started/exploring-projects-on-github/saving-repositories-with-stars#organizing-starred-repositories-with-lists https://docs.github.com/en/get-started/exploring-projects-on...
  - bshipp 5y ago
    Oooh, is that new? I have used Astral[0] in the past which works well but required me to both define all my individual categories (no social/group suggested tags) and remember to use it every time I went on a starring spree. This looks really interesting! [0]: https://github.com/astralapp/astral https://github.com/astralapp/astral
arnaldog12 5y ago
I am from Wildlife Studios (a startup game company) and we have been constructing an ML Platform. Kedro has an important role on that, being one of the core tools. In my opinion, the main advantages of Kedro are: * standardize the development of ML pipelines * easy to extend with plugins * easy to share pipelines components between different projects * data catalog (also easy to create custom datasets) * kedro-viz Today, we have a cookiecutter template with a custom Kedro pipeline that is used by new projects. It has the most common steps of ML pipelines developed here in Wildlife. Therefore, speeding up the development of new models. Moreover, it has the Kedro MLFlow plugin for automatic experiment tracking. Finally, we also have a catalog of pipeline steps that can be easily added to a Kedro project.
version_five 5y ago
This is an ML framework (I'm not actually sure what it does) made by McKinsey. I don't know if it was always open sourced but now it has been "donated to the community" which I think means abandoned? I'd be very curious to hear if anyone has used it and what impressions are on it's value. I've only heard some businessy talking points about it, and am hopefully understandably skeptical. If someone not from McKinsey had used it, I'd love to hear impressions.
- waylonwalker 5y ago
  I use it quite heavily at my day job. The core pieces of kedro are indispensable on large projects (Pipeline, Catalog, DataSets, Runner, hooks). The marketing has a lot of focus on the template, It's a great template, but not the key feature that brings me back to it. Really like that I never have to worry about io, the catalog abstracts this away. I don't worry about long runs to get started with my work. It saves all intermediate steps automatically, and the Pipeline (DAG) lets me run just the parts that I need for my task. Before using kedro I would need to scaffold out a way to save all of these intermediate steps, or suffer from long run times to get started on my task. Lastly the DAG is indispensible during environment migrations, it can quickly tell me all of its edges that I need to be concerned with picking up and moving with me. Unlike similar projects kedro is just a python framework. It lets me build, deploy, and orchestrate however works best for my team.
  - barefeg 5y ago
    Could you point me to the component responsible for storing the intermediate data of the DAG run? I was looking for this but couldn’t find it from quickly scanning the docs
    - joelschw 5y ago
      Here you go - https://kedro.readthedocs.io/en/0.17.6/09_development/03_commands_reference.html#modifying-a-kedro-run https://kedro.readthedocs.io/en/0.17.6/09_development/03_com...
    - 5y ago

96 points by maydemir 5y ago

rladd 5y ago

Anything similar for R?

joelschw 5y ago
Some users have reported the Targets library is similar for that ecosystem
idomi 5y ago
We've built the Ploomber to work with R as well comparing to others who believe only python is for ML. Ploomber is an open source tool that can integrate with R and R studio out of the box. Check it out! https://github.com/ploomber/ploomber https://github.com/ploomber/ploomber
kjkjadksj 5y ago
I use snakemake to call up r scripts in pipelines

tomrod 5y ago

Good to see McKinsey donating something potentially beneficial to the world. Kudos to the team that developed this.

Proven 5y ago

lordgroff 5y ago

I love this project, not from McK. I mean I guess it's "opinionated" but having worked in this field for some time, ML code and pipelines are often laughably immature, stuck with spit, glue and sellotape. Kedro forces people into a bit of a straight jacket for sure, but man does it make things more readable. Plus points, it's very extensible so if there's a design element you're missing or one of its opinions you just can't abide by, then you can extend it. I think it deserves more exposure.

bshipp 5y ago
It might be opinionated but it's no less accurate. Data science is many researchers first foray into programming in any language and, speaking from my own path, was a relatively self-directed discipline in the early days. I often longed for a 'standard approach' but many of the projects that suggested them were packed full of boilerplate that made code even harder to read and write, especially if your background wasn't heavily into OOP.
- waylonwalker 5y ago
  I feel ya. Before using kedro I was using a version of the cookiecutter for data science template that I modified to have something kind of like their data catalog. Before that it was so wild west, every project was its own unique thing.
waylonwalker 5y ago
Maybe a better word is that it comes with an opinionated template? The core parts of Kedro can all be used individually as you wish. Only want a DAG, you can use just the pipeline, Want a catalog setup so you can store intermediate steps, use the Catalog. You can even make kedro project using all the features in a single file with surprisingly little boilerplate.

troiskaer 5y ago

How does Kedro compare to MLFlow and Metaflow?

stevesimmons 5y ago
And Prefect
joelschw 5y ago
Kedro sort of fits into a niche where it just overlaps somewhat with 'orchestrators' like Prefect, Metaflow, Dagster, Airflow and others. What makes it slightly different is that it it is focused on the rapid development journey to production, providing guardrails for teams to co-develop ML projects in a way that nudges software engineering best practice and clean code. The 'finished article' in many cases should be deployed in production in one of those tools which provide specialised bells and whistles like scheduling, monitoring and observability. Regarding MLFlow, there is also a slight overlap in terms of experimentation, but not things like model serving. Kedro has a mechanism to track experiments, but it's more designed to give users with zero infrastructure something for free out of the box. It's been built in a way that it can be repurposed for more dedicated experiment tracking tools - the folks at neptune.ai built their own plug-in for this purpose: https://docs.neptune.ai/integrations-and-supported-tools/automation-pipelines/kedro https://docs.neptune.ai/integrations-and-supported-tools/aut...
- troiskaer 5y ago
  Seems like Kedro has a similar thesis to Metaflow - I will look into it.
  - vtuulos 5y ago
    Yep, Kedro and Metaflow are more similar to each other than to other generic DAG orchestrators like Airflow. Kedro and Metaflow make it easier to develop robust ML projects where orchestration plays an important role but it is not everything. They are two separate projects, so the way how they approach the problem differs greatly in details.

oneoffmk999 5y ago

I am from McKinsey. I do not work on Kedro, but am a data scientist who has used it. I have mixed feelings on Kedro. Pros: * Forces data scientists to produce an end product that is not poorly organized Jupyter notebooks. * Data Catalog is good for well structured systems * Pipeline visualization stack is great (Kedro viz) * Config options are pretty good * Seems stable. Dev team is pretty good on this and avoiding breaking changes. Cons: * Data catalog is kind of bad for any non structured setup with flat file data with manual file movement (which is bad to begin with but sometimes that’s life) * Productivity of making brand new data science code seems to drop when data scientists leave notebooks and * Most of the time I get brought into a client context because the client doesn’t know anything about data scientist. A lot of data scientists, from both parties, come from academic backgrounds and aren’t great at code. The nice thing about notebooks is that they run. Kedro requires you to create pipeline and node objects to wrap around your code before it runs. It requires some familiarity with Kedro to understand, run, or modify. This makes it seem like a bad idea to dump on a novice client. If the data scientist on their side inheriting it doesn’t really get it, or leaves, there’s unlikely to be enough internal knowledge to maintain it. I try to avoid pushing any new tech stacks on my clients where I can for this reason. So… I like it but don’t love it for consulting work, which is ironic.

sails 5y ago
No pros or cons on deploying models. Seems like that is a significant part of the functional value?
- oneoffmk999 5y ago
  Not my area of expertise. Can’t comment on it either way. But I would tend imagine it’s a strength of Kedro.
- joelschw 5y ago
  Will point to some experiences open source users have had on this journey https://medium.com/hacking-talent/production-code-for-data-science-and-our-experience-with-kedro-60bb69934d1f https://medium.com/hacking-talent/production-code-for-data-s... https://medium.com/google-cloud/migrate-kedro-pipeline-on-vertex-ai-fa3f2c6f7aad https://medium.com/google-cloud/migrate-kedro-pipeline-on-ve...

bshipp 5y ago

I've apparently already starred this repo in the past and have no recollection of it. I really need to find a mechanism to sort and categorize my starred repositories; this is getting out of hand.

pjot 5y ago
You should be able to do that natively in GitHub with lists! https://docs.github.com/en/get-started/exploring-projects-on-github/saving-repositories-with-stars#organizing-starred-repositories-with-lists https://docs.github.com/en/get-started/exploring-projects-on...
- bshipp 5y ago
  Oooh, is that new? I have used Astral[0] in the past which works well but required me to both define all my individual categories (no social/group suggested tags) and remember to use it every time I went on a starring spree. This looks really interesting! [0]: https://github.com/astralapp/astral https://github.com/astralapp/astral

arnaldog12 5y ago

I am from Wildlife Studios (a startup game company) and we have been constructing an ML Platform. Kedro has an important role on that, being one of the core tools. In my opinion, the main advantages of Kedro are: * standardize the development of ML pipelines * easy to extend with plugins * easy to share pipelines components between different projects * data catalog (also easy to create custom datasets) * kedro-viz Today, we have a cookiecutter template with a custom Kedro pipeline that is used by new projects. It has the most common steps of ML pipelines developed here in Wildlife. Therefore, speeding up the development of new models. Moreover, it has the Kedro MLFlow plugin for automatic experiment tracking. Finally, we also have a catalog of pipeline steps that can be easily added to a Kedro project.

version_five 5y ago

This is an ML framework (I'm not actually sure what it does) made by McKinsey. I don't know if it was always open sourced but now it has been "donated to the community" which I think means abandoned? I'd be very curious to hear if anyone has used it and what impressions are on it's value. I've only heard some businessy talking points about it, and am hopefully understandably skeptical. If someone not from McKinsey had used it, I'd love to hear impressions.

waylonwalker 5y ago
I use it quite heavily at my day job. The core pieces of kedro are indispensable on large projects (Pipeline, Catalog, DataSets, Runner, hooks). The marketing has a lot of focus on the template, It's a great template, but not the key feature that brings me back to it. Really like that I never have to worry about io, the catalog abstracts this away. I don't worry about long runs to get started with my work. It saves all intermediate steps automatically, and the Pipeline (DAG) lets me run just the parts that I need for my task. Before using kedro I would need to scaffold out a way to save all of these intermediate steps, or suffer from long run times to get started on my task. Lastly the DAG is indispensible during environment migrations, it can quickly tell me all of its edges that I need to be concerned with picking up and moving with me. Unlike similar projects kedro is just a python framework. It lets me build, deploy, and orchestrate however works best for my team.
- barefeg 5y ago
  Could you point me to the component responsible for storing the intermediate data of the DAG run? I was looking for this but couldn’t find it from quickly scanning the docs
  - joelschw 5y ago
    Here you go - https://kedro.readthedocs.io/en/0.17.6/09_development/03_commands_reference.html#modifying-a-kedro-run https://kedro.readthedocs.io/en/0.17.6/09_development/03_com...
  - 5y ago