4 ms·

Been using LXC for years via Proxmox. Just dipping my toes into docker to explore ways to distribute CPU bound workloads. Quite different and different use ca

Been using LXC for years via Proxmox. Just dipping my toes into docker to explore ways to distribute CPU bound workloads. Quite different and different use cases. I do like how the docker files provide a rock solid definition of the environment, that should come in very handy in the future when I have completely forgotten how something works and need to update it. I usually put a config.txt in the root of my LXC containers for a similar purpose, but the Dockerfiles are more explicit. I like to hotpatch my code and I hate complicated built processes, so I don't see going all in with Docker, but it is quite an elegant way to package an application. I always had the impression that it was overly complex, but after actually trying it I am surprised by all the things I like. My original intention was to deploy a Gaussian Process Regression model built with PyMC on AWS Lambda. I need to build and run a few thousand of these each morning. Building the Docker image was fun and it worked great, but AWS Lambda was extremely disappointing. Even when configured with 10GB of RAM, it was pretty slow and running my predictions would cost hundreds of dollars per day. I can spin up a 128 core bare metal EPYC server on vultr for an hour at a cost of $8 that can finish the job faster. That's still more expensive than I had envisioned so if anyone can suggest any smaller providers that can run CPU bound workloads in ephemeral docker containers, let me know.

mkesper 3y ago
Well, Dockerfile is as solid as a bunch of bash scripts apt-get installing and wgetting things from the internet, that means not at all. Maybe nix could provide a real solution for that.
- Jnr 3y ago
  Yes, they are, but once it is built, you can store it so you do not have to run it every time and in every environment.
- deleted 3y ago
  [deleted]
WirelessGigabit 3y ago
I use VS Code to hotpatch hotpatchable code. Half of the stuff I write is in Rust and I'm not smart enough to go and change bits of an executable. But for Python it works out great, except for when the container doesn't start. If it doesn't start you can't connect VS Code to it. The trick then is to use `docker cp` to pull out the file that breaks the python process, fix it, `cp` it back and try and start the container. If possible I like to change the EXEC method to `/bin/bash`, the COMMAND to `tail -f /dev/null`, and execute the python process in an interactive terminal. That mitigates the container not starting issue... unless `tail -f /dev/null` fails. But if that's the case I've got bigger fish to fry.

mkesper 3y ago

Well, Dockerfile is as solid as a bunch of bash scripts apt-get installing and wgetting things from the internet, that means not at all. Maybe nix could provide a real solution for that.

Jnr 3y ago
Yes, they are, but once it is built, you can store it so you do not have to run it every time and in every environment.
deleted 3y ago
[deleted]

WirelessGigabit 3y ago

I use VS Code to hotpatch hotpatchable code. Half of the stuff I write is in Rust and I'm not smart enough to go and change bits of an executable. But for Python it works out great, except for when the container doesn't start. If it doesn't start you can't connect VS Code to it. The trick then is to use `docker cp` to pull out the file that breaks the python process, fix it, `cp` it back and try and start the container. If possible I like to change the EXEC method to `/bin/bash`, the COMMAND to `tail -f /dev/null`, and execute the python process in an interactive terminal. That mitigates the container not starting issue... unless `tail -f /dev/null` fails. But if that's the case I've got bigger fish to fry.