3 ms·

/var gets its own disk on my machine! Yay academic machine learning docker containers. What's a few 30GB docker images between friends? So yeah /var gets to liv

by kortex 6y ago

/var gets its own disk on my machine! Yay academic machine learning docker containers. What's a few 30GB docker images between friends? So yeah /var gets to live with training data on SSD which gets cold swapped onto spinny platters as needed. /home is another, on the "main sdd". /usr isn't a bad choice, either.

sillysaurusx 6y ago
Yay academic machine learning docker containers. What's a few 30GB docker images between friends? :) Hopefully docker will lose the fight for mindshare in the ML space.
- kortex 6y ago
  What do you mean, "lose mindshare"? Docker is one of the absolute best things to happen to ML. My work is largely algo integration - I take academics' ML code and bundle it for API consumption. Before this, I was a bench chemist, so I kind of have a thing for reproducing experiments. The academic ML scene has a reproducibility crisis that makes other science reproducibility crises look like Phys 100 labs. These things depend on someone's conda env with versions pinned to nightly builds, inscrutable code written by a postdoc that has since left, datasets which have been mutated since the paper was published, on and on. Docker gives me a fighting chance to actually get reproducible results without going stark raving mad. Should these researchers have better software process to avoid this situation in the first place? Heck yes. But these are people doing things like `os.system("rm " + filepath)` and git committing entire models, they are just really green and most don't know any better yet and academia isn't really known for its mandatory Practical Software Engineering classes for CS majors.
  - sillysaurusx 6y ago
    Ah, you're a docker zealot. I misread the joke. You had a flash of self-awareness. Docker containers take massive amounts of disk space, and you need arcane knowledge just to use them. I'd rather focus on doing ML, not learning docker. In other words, "I'm getting to old for this." (33 is up there.) It bugged me in the game industry when some old programmer said "shaders are a young person's game," implying that they wouldn't even look into how they worked. Now that I am that older programmer, or getting there, I see what they mean. Docker arcana is a young person’s game. If Docker solved the reproducibility crisis, you might have a point. But it doesn't. Most of the crisis is the fact that (a) datasets are trapped behind institutions that won't make them available, (b) the models themselves aren't made available (OpenAI), and (c) the code itself isn't available (also OpenAI). Those three things are the main problem. Forcing everyone to use a 30GB docker container just to do basic ML isn't going to do anything but waste time and turn newbies away from ML. You're a fine debator though. It was an enjoyable read; have an upvote.

by kortex 6y ago

sillysaurusx 6y ago

Yay academic machine learning docker containers. What's a few 30GB docker images between friends? :) Hopefully docker will lose the fight for mindshare in the ML space.

kortex 6y ago
What do you mean, "lose mindshare"? Docker is one of the absolute best things to happen to ML. My work is largely algo integration - I take academics' ML code and bundle it for API consumption. Before this, I was a bench chemist, so I kind of have a thing for reproducing experiments. The academic ML scene has a reproducibility crisis that makes other science reproducibility crises look like Phys 100 labs. These things depend on someone's conda env with versions pinned to nightly builds, inscrutable code written by a postdoc that has since left, datasets which have been mutated since the paper was published, on and on. Docker gives me a fighting chance to actually get reproducible results without going stark raving mad. Should these researchers have better software process to avoid this situation in the first place? Heck yes. But these are people doing things like `os.system("rm " + filepath)` and git committing entire models, they are just really green and most don't know any better yet and academia isn't really known for its mandatory Practical Software Engineering classes for CS majors.
- sillysaurusx 6y ago
  Ah, you're a docker zealot. I misread the joke. You had a flash of self-awareness. Docker containers take massive amounts of disk space, and you need arcane knowledge just to use them. I'd rather focus on doing ML, not learning docker. In other words, "I'm getting to old for this." (33 is up there.) It bugged me in the game industry when some old programmer said "shaders are a young person's game," implying that they wouldn't even look into how they worked. Now that I am that older programmer, or getting there, I see what they mean. Docker arcana is a young person’s game. If Docker solved the reproducibility crisis, you might have a point. But it doesn't. Most of the crisis is the fact that (a) datasets are trapped behind institutions that won't make them available, (b) the models themselves aren't made available (OpenAI), and (c) the code itself isn't available (also OpenAI). Those three things are the main problem. Forcing everyone to use a 30GB docker container just to do basic ML isn't going to do anything but waste time and turn newbies away from ML. You're a fine debator though. It was an enjoyable read; have an upvote.