3 ms·
I have seen my fair share of ML Python codebases. Distribution is a mess, onboarding new people is a mess. The thing I would says just works is OS level config
by wqtz 2y ago
I have seen my fair share of ML Python codebases. Distribution is a mess, onboarding new people is a mess. The thing I would says just works is OS level configuration things like Kubernetes or NixOS are proven technology that works and there are enough resources for issues that can be self-debugged instead of opening tickets/ gh issues or reaching out to support. But as these are much complicated technology, you need domain experts and should not pressure ML engineers or data scientists to figure this out. I have seen Python packaging to be such a mess it is easier to teah to Python engineers ML or DS, then ML engineers proper package handling and distribution. The very existence dozens of packaging solutions show that engineer would rather create something from scrath rather work with existing tools.
- mardifoufs 2y agoI mean, I completely agree with that. I'm a MLE and I absolutely, utterly hate how much of a mess it can be and how much time is spent just helping interns getting their env set up reliably (we now have a pretty reliable setup/docs but that was after a few painful onboardings). I just think that using another language for some of what python does would be even more painful, just not on the packaging side of it