3 ms·

love the transparency this brings. Any 3rd party tools with plans to integrate with it? (e.g. analytics layer companies?)

• elijahbenizzy 4y ago
  Thanks! Lots of plans in the works. Two directions we're moving: 1. Integrate with orchestration systems (E.G. run Hamilton pipelines on different orchestration platforms). You could imagine compiling to airflow pipelines, running hamilton on metaflow, compiling to a vertex pipeline etc... 2. Adapters to load data from/save data to external providers. E.G. logging data quality to whylogs, loading data up from snowflake, saving a model to mlflow... We're actively working on building this out though -- have some partners who hare helping build out use-cases!
  • rubenfiszel 4y ago
    Ola, I'm the founder of windmill [1] which is an OSS orchestration platform for scripts, including python, and I'm pretty sure one could compile an hamilton dag to a windmill openflow very simply (we use a mounted folder in `./shared` to share heavy data across steps). Right now, one can do ETL using polars on windmill but i'd love to have an even more structure code framework to recommend. We should chat! [1]: https://github.com/windmill-labs/windmill https://github.com/windmill-labs/windmill
    • krawczstef 4y ago
      Nice! Yep that's part of our vision to "compile onto", i.e. generate code for, frameworks such as yours. Happy to sketch something out - want to join our slack to chat https://join.slack.com/t/hamilton-opensource/shared_invite/zt-1bjs72asx-wcUTgH7q7QX1igiQ5bbdcg https://join.slack.com/t/hamilton-opensource/shared_invite/z...?