4 ms·
I think MLflow is a good idea (very) badly executed. I would like to have a library that combines: - simple logging of (simple) metrics during and after traini
by LeanderK 4y ago
I think MLflow is a good idea (very) badly executed. I would like to have a library that combines:
- simple logging of (simple) metrics during and after training
- simple logging of all arguments the model was created with
- simple logging of a textual representation of the model
- simple logging of general architecture details (number of parameters, regularisation hyperparameters, learning rate, number of epochs etc.)
- and of course checkpoints
- simple archiving of the model (and relevant data)
and all that without much (coding) overhead and only using a shared filesystem (!) And with an easy notebook integration. MLflow just has way to many unnecessary features and is unreliable and complicated. When it doesn't work it's so frustrating, it's also quite often super slow. But I always end up creating something like MLflow when working on an architecture for a long time.
EDIT: having written this...I fell like trying to write my own simple library after finishing the paper. A few ideas have already accumulated in my notes that would make my life easier.
EDIT2: I actually remember trying to use SQLite to manage my models! But the server I worked on was locked down and going through the process to get somebody to install me SQLite was just not worth it. It's also was not available on the cluster for big experiments, where it would be even more work to get it, so I gave up on the idea of trying SQLite.
- pletnes 4y agoSqlite is in python’s stdlib, so how can this be an issue? Was there no local filesystem whatsoever?
- tekknolagi 4y agosqlite bindings are in the stdlib but not the library itself.
- imachine1980_ 4y agoim asking from ignorance, what the difference in effect this context of not having the library itself?
- funklute 4y agoUsing the bindings is only possible if the library itself is already installed (since the bindings directly make use of the library, under the hood).
- simonw 4y agoI've never encountered a Python installation on any operating system where `import sqlite3` worked but the underlying libraries were not available. I imagine this is because SQLite is VERY easy to bundle with Python itself. So on some platforms the OS SQLite is used, but on others it gets shipped as part of the Python installation itself. It even works in WebAssembly via Pyodide!
- idomi 4y ago+1 I also think it's faster that way on both environment setup and ad hoc rapid experiments, from my experience using the library in a team doesn't scale well, it becomes pretty slow.
- edublancas 4y agoI'm happy to collaborate with you, let's build the best experiment tracker out there! Feel free to ping me at eduardo@ploomber.io
- Fiahil 4y ago> I think MLFlow is a good idea (very) badly executed. Oh yes, I'm glad to see other with similar opinion.
- smehta73 4y agohave you used comet? it basically does everything you are asking and lot more user-friendly than MLFlow
- nerdponx 4y agoIsn't Comet a proprietary SaaS? I like MLFlow because I can run it on my own computer if I want to.
- tomrod 4y agoCheck out flyte and union.ml. No personal affiliation, just good projects in the vein of airflow/prefect/mlflow/kubeflow
- YetAnotherNick 4y agoI really like guild.ai. The best thing is that their developers assumed people to be lazy and automatically makes flag for global variables and track them.
- kumare3 4y ago@tomrod, thank you for the callout. By the way we are integrating mlflow into Flyte in a way that you do not need to start the web server to view the logs. They are available Locally and statically in Flyte Ui. Ofcourse you cal also use mlflow server
- tomrod 4y agoI'm having a fanboy moment. This is like tweeting about Mark Hamil or Ryan Reynolds and they tweet back. Thanks for the great open source libraries yall!
- shmoejoe 4y agoHow about this? https://github.com/capitalone/rubicon-ml https://github.com/capitalone/rubicon-ml
- GeneralMayhem 4y agoYep - totally agree. I respect the attempt to introduce something that is basically an opinionated CRUD app as a central place to put your model metadata. But it's not really ready for large-scale production, or for use by teams bigger than about 5. It's kind of flaky and slow. It doesn't have namespacing. It's overly opinionated on the workflow (the way that states work, with a model version being in exactly one of dev, staging, prod is super hard to work with). But beyond that, the biggest problem I have with MLFlow is what I call the "part of this complete breakfast" problem, which the ML/data-science arena is particularly susceptible to these days: the marketing talks a lot about what problems can be solved using the product, but not a lot about what parts of the problem the product actually solves. This is often because an honest answer to the latter question would be "not much". In the case of MLFlow, that would be totally fine, because honestly an opinionated CRUD app is a very useful thing. But it should be a lot more honest about what it does. It's not a system for automatically tracking model metrics, it's a database into which you can write model metrics with a known key structure.
- dataisamazing 4y agoAutologging literally does log your model metrics automatically in most cases.