6 ms·

Kedro: open-source library for production-ready machine learning code

74 points by ereli1 7y ago

prepend 7y ago
I really like how they implemented the data catalog [0] so that it’s yaml-based and also has a paths-style cascading method of files that can be common across or within teams as well as personal for individual projects. I think this makes it easy to build up with tools for meta analysis (how many data sets are used, etc) and even viz using a variety of tools rather than having the metadata management tied to a system or product. Are there other techniques for data catalogs that are file based or at least open standard based that scale all the way up from developer? [0] https://kedro.readthedocs.io/en/latest/04_user_guide/04_data_catalog.html https://kedro.readthedocs.io/en/latest/04_user_guide/04_data...
- infinite8s 7y ago
  There's the intake project from the Anaconda folks.
domenicrosati 7y ago
Conjecture: production quality of ml code has mostly to do with how heuristics are designed and battle tested and almost nothing to do with how the training/inference pipeline is constructed.
- stichers 7y ago
  Just because the challenge is relatively trivial to solve, doesn't make it any less important though. Experiment management, and the transition to production, is recognised as having potentially high impact to successful delivery. My understanding is that this takes care of details, which can otherwise get forgotten in the race for the best model. But YMMV.
wokwokwok 7y ago
tldr, if you really dig past the marketing (from the FAQ (1)): > We see Airflow and Luigi as complementary frameworks: Airflow and Luigi are tools that handle deployment, scheduling, monitoring and alerting. Kedro is the worker that should execute a series of tasks, and report to the Airflow and Luigi managers. > Create the data transformation steps as pure Python functions Personally, I feel mystified why you would use something like this rather than a more mature product like say, Spark, that natively supports clustering, etc, which is what I would really like to see in the FAQ. Is it a processing solution? Not really, since it suggests you can offload the heavy lifting to an engine, eg. spark. An orchestrator? Apparently not, because that's a complementary product. So... it's like, a configuration management tool? Pretty hard to see the use case to me. 1. https://kedro.readthedocs.io/en/latest/06_resources/01_faq.html#how-does-kedro-compare-to-other-projects https://kedro.readthedocs.io/en/latest/06_resources/01_faq.h...
- whoevercares 7y ago
  It makes a lot of sense to me, who struggled to work with a group consist of mostly Computer Vision scientists. The pushback to use anything heavier than pure python + s3 is amazing. Spark is still considered too heavy for us and people need to do in memory analysis and friction free experiments with convoluted dependencies. Most of the time it’s not their pipeline that goes to production but rather their trained model. They need to do most steps on their laptop and easy lift it to cloud. No engineers want to do that lift job trust me. Netflix has Metaflow for this which seems a more fully-fledged product. But it’s not open sourced.
- FridgeSeal 7y ago
  Because running Spark to do anything that doesn’t actually require a whole cluster is like using earthmoving equipment to assemble a series of small ikea tables?
  - wokwokwok 7y ago
    If you're doing something that trivial, you don't need anything more complicated than airflow.
FridgeSeal 7y ago
> Machine learning models which can be deployed effortlessly and operate unattended are far more likely to achieve commercial objectives. Likeliness of achieving commercial objectives is tied to the commercial usefulness and accuracy of your analysis and predictions, not the ease of deployment, or-even more curiously-ability to be left unattended.
- joelschw 7y ago
  This is a wider point for anyone looking to take advantage of machine learning, but reproducibility is also a problem which needs to be catered for.
- IanCal 7y ago
  It's surely not a particularly contentious point that hard to deploy systems that require lots of attention to keep running are less likely to achieve commercial objectives. Just like your website being stable and easy to update helps your business use it to make money. Of course it also needs to be tied to commercial usefulness.
coverman 7y ago
Starting to see a lot of these frameworks pop up to simplify deployment of machine learning models. I’m really hoping one or two start to stand out...but it doesn’t feel like this one.
bserial 7y ago
I’m curious as to if anyone can say how this compares to dagster since both libraries seems to rely on deploying to engines like Airflow?
- Peteris 7y ago
  Kedro puts emphasis on seamless transition to prod without jeopardizing work in experimentation stage: - pipeline syntax is absolutely minimal (even supporting lambdas for simple transitions), inspired by the Clojure library core.graph https://github.com/plumatic/plumbing https://github.com/plumatic/plumbing - sequential and parallel runners are built-in (don't have to rely on Airflow) - io provides wrappers for existing familiar data sources, but directly borrows arguments from Pandas, Spark APIs so no new API to learn - flexibility in the sense you could rip out anything, for example, the whole Data Catalog replacing with another mechanism for data access like Haxl - there's a project template which serves as a framework with built-in conventions from 50+ analytics engagements

74 points by ereli1 7y ago

I really like how they implemented the data catalog [0] so that it’s yaml-based and also has a paths-style cascading method of files that can be common across or within teams as well as personal for individual projects. I think this makes it easy to build up with tools for meta analysis (how many data sets are used, etc) and even viz using a variety of tools rather than having the metadata management tied to a system or product. Are there other techniques for data catalogs that are file based or at least open standard based that scale all the way up from developer? [0] https://kedro.readthedocs.io/en/latest/04_user_guide/04_data_catalog.html https://kedro.readthedocs.io/en/latest/04_user_guide/04_data...

infinite8s 7y ago
There's the intake project from the Anaconda folks.

domenicrosati 7y ago

Conjecture: production quality of ml code has mostly to do with how heuristics are designed and battle tested and almost nothing to do with how the training/inference pipeline is constructed.

stichers 7y ago
Just because the challenge is relatively trivial to solve, doesn't make it any less important though. Experiment management, and the transition to production, is recognised as having potentially high impact to successful delivery. My understanding is that this takes care of details, which can otherwise get forgotten in the race for the best model. But YMMV.

wokwokwok 7y ago

tldr, if you really dig past the marketing (from the FAQ (1)): > We see Airflow and Luigi as complementary frameworks: Airflow and Luigi are tools that handle deployment, scheduling, monitoring and alerting. Kedro is the worker that should execute a series of tasks, and report to the Airflow and Luigi managers. > Create the data transformation steps as pure Python functions Personally, I feel mystified why you would use something like this rather than a more mature product like say, Spark, that natively supports clustering, etc, which is what I would really like to see in the FAQ. Is it a processing solution? Not really, since it suggests you can offload the heavy lifting to an engine, eg. spark. An orchestrator? Apparently not, because that's a complementary product. So... it's like, a configuration management tool? Pretty hard to see the use case to me. 1. https://kedro.readthedocs.io/en/latest/06_resources/01_faq.html#how-does-kedro-compare-to-other-projects https://kedro.readthedocs.io/en/latest/06_resources/01_faq.h...

whoevercares 7y ago
It makes a lot of sense to me, who struggled to work with a group consist of mostly Computer Vision scientists. The pushback to use anything heavier than pure python + s3 is amazing. Spark is still considered too heavy for us and people need to do in memory analysis and friction free experiments with convoluted dependencies. Most of the time it’s not their pipeline that goes to production but rather their trained model. They need to do most steps on their laptop and easy lift it to cloud. No engineers want to do that lift job trust me. Netflix has Metaflow for this which seems a more fully-fledged product. But it’s not open sourced.
FridgeSeal 7y ago
Because running Spark to do anything that doesn’t actually require a whole cluster is like using earthmoving equipment to assemble a series of small ikea tables?
- wokwokwok 7y ago
  If you're doing something that trivial, you don't need anything more complicated than airflow.

FridgeSeal 7y ago

> Machine learning models which can be deployed effortlessly and operate unattended are far more likely to achieve commercial objectives. Likeliness of achieving commercial objectives is tied to the commercial usefulness and accuracy of your analysis and predictions, not the ease of deployment, or-even more curiously-ability to be left unattended.

joelschw 7y ago
This is a wider point for anyone looking to take advantage of machine learning, but reproducibility is also a problem which needs to be catered for.
IanCal 7y ago
It's surely not a particularly contentious point that hard to deploy systems that require lots of attention to keep running are less likely to achieve commercial objectives. Just like your website being stable and easy to update helps your business use it to make money. Of course it also needs to be tied to commercial usefulness.

coverman 7y ago

Starting to see a lot of these frameworks pop up to simplify deployment of machine learning models. I’m really hoping one or two start to stand out...but it doesn’t feel like this one.

bserial 7y ago

I’m curious as to if anyone can say how this compares to dagster since both libraries seems to rely on deploying to engines like Airflow?

Peteris 7y ago
Kedro puts emphasis on seamless transition to prod without jeopardizing work in experimentation stage: - pipeline syntax is absolutely minimal (even supporting lambdas for simple transitions), inspired by the Clojure library core.graph https://github.com/plumatic/plumbing https://github.com/plumatic/plumbing - sequential and parallel runners are built-in (don't have to rely on Airflow) - io provides wrappers for existing familiar data sources, but directly borrows arguments from Pandas, Spark APIs so no new API to learn - flexibility in the sense you could rip out anything, for example, the whole Data Catalog replacing with another mechanism for data access like Haxl - there's a project template which serves as a framework with built-in conventions from 50+ analytics engagements