4 ms·
Docker is the best thing to ever happen to scientific computing yet there is so much griping from the old guard. Trying to make a hermetic, CI/CD, within-machin
by kortex 6y ago
Docker is the best thing to ever happen to scientific computing yet there is so much griping from the old guard. Trying to make a hermetic, CI/CD, within-machine-epsilon reproducible computer vision pipeline opened my eyes to just how un-reproducible your average machine learning project really is. Especially those coming from academia.
"Oh the PhD pinned to a nightly version of some dep and the AUC score plummets when you try to upgrade and customer wants this latest model in production on airgapped machines last week? Aight, stick their Conda env in a container, call it with `docker run`, and sort it out later"
Horrifying? Yes. Solving the problem on short notice? Also yes.
Plenty of more pedestrian uses but that was an actual situation where docker's reproducibility saved my bacon.
- leephillips 6y agoYes. When your calculation uses dozens of files and a change in any of them might change the results, I don’t see how there is any alternative to some form of containerization. Even if you could generate a statically linked binary, if it’s science you want to distribute the sources for everything as well. I notice that most people seem to reach for Docker automatically, though. There are other technologies, such as systemd containers, that might be better in some situations.