5 ms·

The problem we have with that at my workplace is that it's hard to do things like run IDEs/set breakpoints, run gdb or strace, get a REPL (with useful REPL tool

by geofft 7y ago

The problem we have with that at my workplace is that it's hard to do things like run IDEs/set breakpoints, run gdb or strace, get a REPL (with useful REPL tooling), etc. inside a production-style container. I understand the arguments that you don't want to do this in actual production, you want to kill the offending container and figure out the problem from logs - but once you've got an idea of where the problem lies, for local dev you do actually want to debug problems this way. Even editing a random config file or editing source code to add a printf is more difficult inside a production-style container than in a local checkout. Have you found good approaches for this?

crooked-v 7y ago
For me, the other big complication is that we do backend stuff in Java... and God help you if you want a self-contained image that can intelligently and quickly reload Java code for anything that's not a very specific arrangement of frameworks. It's night and day compared to the Javascript world, where it's nearly a given that the bare minimum should be watchers that recompile code and restart a server before you finish switching window focus.
- geofft 7y ago
  Yeah, I wonder to what extent "Just put it all in Docker" works for specific language communities, and if you're not in those communities the advice falls flat. (My employer is pretty polyglot, but primarily Java; my team is mostly Python; we also build our own C code as well as third-party C code and need to debug it every so often.)
  - crdrost 7y ago
    I mean, what Docker does is, it turns a network of executables and scripts into a single executable with no outside dependencies. It works extremely well for language communities that need this, in particular it caught on because you would build something and test it locally and it would work just fine, but then you are running the wrong version of Python or pip installed the wrong version of a library, or you have the wrong version of PHP or Node.js relative to Prod. For Java and Go, I believe they have already settled these problems, no? You might shell out to other commands and have a weak dependency on your runtime environment that way, but if you stay within your lane, then you already have an executable with all of its dependencies bundled in. They just have a lot less to gain from Docker. I can't see why they would have more to lose.
    - geofft 7y ago
      OK, but, even for Python that solves the problem once you're in the container, but it doesn't really address the problem of whether my local Python installation, the one my IDE uses, is using the same version and the same libraries - unless you either decide I don't get to use an IDE or that my IDE goes into the container. Right?
      - sebastos 7y ago
        Yes, you are exactly right. I mentioned it further down, but my team and I are having preliminary success by integrating with VSCode's "Remote: Docker" extension. They've got it rigged so that all language support can run as remote servers inside the container. They give you a special .json file to control how you'd like to start the image, and you can supply a project-specific dockerfile that takes any additional IDE-specific steps that you want to add in as layers when opening up the project's dev container. It still has some rough edges - for some reason, you can't supply Build-time arguments to the devcontainer Dockerfile I mentioned above, which makes it hard to get permissions to be non-annoying if you're worried your users will have odd UIDs. And there are a couple other random ticky tack things. But it's by far the closest to actually "developing out of a container" as you and I are imagining it, with all of the attendant benefits. When you want a new dev to start working on the project, they basically do this: 1. git clone newproject 2. Open that project in vscode 3. Respond to a pop-up that yes, I do want to open this in a container 4. <wait for the dev-specific layers to be built> 5. Do a normal project build 6. Enjoy full intellisense, syntax highlighting, import resolution, etc. in exactly the environment that's going to run. In addition, by including the Dockerfile that builds the base image (the one that the dev container FROM's) into the source, developers can now change library versions and dependencies as a git commit, and then coerce other developers into automatically updating their environments by pushing a new image with a new tag to your gitlab / w/e, and then updating the tag mentioned in the FROM in your .devcontainer/Dockerfile. It's like how you would expect development to feel in 2020.
        hcrisp 7y ago
        I just figured out a similar workflow using PyCharm Pro. I set up a Deployment mapping from my Windows laptop to my Linux workstation which deploys code whenever I save. Then the Linux folder is volume-mounted into the Docker container. I set up a Remote Debugging configuration, and then after the Docker container is brought up, it will attach via SSH to the Python instance running in the remote Docker container. (Your code needs only have the pydevd_pycharm library installed.) From there you can breakpoint, watch variables, interact at the console, etc. It was a bit of bear to set up, but now I find it very tractable to run and immensely useful.
  - deleted 7y ago
    [deleted]
- taleodor 7y ago
  Fat jars is the way - works fine with containers; java ee is another story - but should be considered legacy these days.
- dclusin 7y ago
  I solve this problem by having everything testable. I'm not saying this is a replacement for running your code as a assembled app before committing... But like, you should know that the code is going to work as intended before you incur the cost of running the application. For exploratory programming I used to just enable remote debugging and set break points. This should be going away in favor of the REPL introduced in Java 9 tho... Haven't met lots of developers that have integrated it into their gradle builds yet tho...
  - geofft 7y ago
    Are you testing your code inside or outside of Docker? How do you do exploratory programming in languages/environments that don't give you a remote REPL? (For instance, how do you do it in C?)
    - dclusin 7y ago
      The common setups I've worked with are Docker instances for the automated CI tests and then running tests on your dev machine w/o a container. If I had to do exploratory programming without a REPL I'd probably leverage my test environment to write small bits of code, step through them with a debugger, and then use the results to structure a larger program.
gravypod 7y ago
I've done some initial investigation a while ago. You should be able to attach gdb to a process in a different cgroup. To more quickly edit configs just make it so your containers build and start quickly. This is very easy to do if you are aware of the docker layer cache. Hot reload is basically `while (true) build() && restart()`
- geofft 7y ago
  Oh, you can definitely attach gdb from outside the container to processes in the container. But I need to load debug symbols, libraries, etc. I'd like gdb to be able to see the source code I'm editing. I've done it and it doesn't quite work (I have stories about gdb's strange attempts at container-awareness... it spawns a thread and tries to stick that thread in the container so it can find files). Also, now your gdb setup isn't containerized, and that is genuinely part of your "dev environment" (it's a thing you want to share with other developers), so you lose out on many of the advantages of Dockerizing your dev environments. Note that you can't gdb/strace across containers because you can't see pids in one container from another container. (Similarly, a Kubernetes sidecar with gdb/strace isn't really going to work.) Not all of my software restarts quickly, and it's not because I don't know how to make containers do things - it's because the underlying build system is slow even on the local machine, or because they do a lot of processing at startup, or whatever. (Very few of the things I work on are webapps!)
  - amazingman 7y ago
    > Note that you can't gdb/strace across containers because you can't see pids in one container from another container. (Similarly, a Kubernetes sidecar with gdb/strace isn't really going to work.) In Kubernetes, you can configure a Pod to share a single process namespace among its containers. https://kubernetes.io/docs/tasks/configure-pod-container/share-process-namespace/ https://kubernetes.io/docs/tasks/configure-pod-container/sha...
    - geofft 7y ago
      Wow, I totally missed this - good to know!
dclusin 7y ago
JVM has remote debugging as well. Works great with Intellij and Eclipse.

by geofft 7y ago

crooked-v 7y ago

For me, the other big complication is that we do backend stuff in Java... and God help you if you want a self-contained image that can intelligently and quickly reload Java code for anything that's not a very specific arrangement of frameworks. It's night and day compared to the Javascript world, where it's nearly a given that the bare minimum should be watchers that recompile code and restart a server before you finish switching window focus.

geofft 7y ago
Yeah, I wonder to what extent "Just put it all in Docker" works for specific language communities, and if you're not in those communities the advice falls flat. (My employer is pretty polyglot, but primarily Java; my team is mostly Python; we also build our own C code as well as third-party C code and need to debug it every so often.)
- crdrost 7y ago
  I mean, what Docker does is, it turns a network of executables and scripts into a single executable with no outside dependencies. It works extremely well for language communities that need this, in particular it caught on because you would build something and test it locally and it would work just fine, but then you are running the wrong version of Python or pip installed the wrong version of a library, or you have the wrong version of PHP or Node.js relative to Prod. For Java and Go, I believe they have already settled these problems, no? You might shell out to other commands and have a weak dependency on your runtime environment that way, but if you stay within your lane, then you already have an executable with all of its dependencies bundled in. They just have a lot less to gain from Docker. I can't see why they would have more to lose.
  - geofft 7y ago
    OK, but, even for Python that solves the problem once you're in the container, but it doesn't really address the problem of whether my local Python installation, the one my IDE uses, is using the same version and the same libraries - unless you either decide I don't get to use an IDE or that my IDE goes into the container. Right?
    - sebastos 7y ago
      Yes, you are exactly right. I mentioned it further down, but my team and I are having preliminary success by integrating with VSCode's "Remote: Docker" extension. They've got it rigged so that all language support can run as remote servers inside the container. They give you a special .json file to control how you'd like to start the image, and you can supply a project-specific dockerfile that takes any additional IDE-specific steps that you want to add in as layers when opening up the project's dev container. It still has some rough edges - for some reason, you can't supply Build-time arguments to the devcontainer Dockerfile I mentioned above, which makes it hard to get permissions to be non-annoying if you're worried your users will have odd UIDs. And there are a couple other random ticky tack things. But it's by far the closest to actually "developing out of a container" as you and I are imagining it, with all of the attendant benefits. When you want a new dev to start working on the project, they basically do this: 1. git clone newproject 2. Open that project in vscode 3. Respond to a pop-up that yes, I do want to open this in a container 4. <wait for the dev-specific layers to be built> 5. Do a normal project build 6. Enjoy full intellisense, syntax highlighting, import resolution, etc. in exactly the environment that's going to run. In addition, by including the Dockerfile that builds the base image (the one that the dev container FROM's) into the source, developers can now change library versions and dependencies as a git commit, and then coerce other developers into automatically updating their environments by pushing a new image with a new tag to your gitlab / w/e, and then updating the tag mentioned in the FROM in your .devcontainer/Dockerfile. It's like how you would expect development to feel in 2020.
      - hcrisp 7y ago
        I just figured out a similar workflow using PyCharm Pro. I set up a Deployment mapping from my Windows laptop to my Linux workstation which deploys code whenever I save. Then the Linux folder is volume-mounted into the Docker container. I set up a Remote Debugging configuration, and then after the Docker container is brought up, it will attach via SSH to the Python instance running in the remote Docker container. (Your code needs only have the pydevd_pycharm library installed.) From there you can breakpoint, watch variables, interact at the console, etc. It was a bit of bear to set up, but now I find it very tractable to run and immensely useful.
- deleted 7y ago
  [deleted]
taleodor 7y ago
Fat jars is the way - works fine with containers; java ee is another story - but should be considered legacy these days.
dclusin 7y ago
I solve this problem by having everything testable. I'm not saying this is a replacement for running your code as a assembled app before committing... But like, you should know that the code is going to work as intended before you incur the cost of running the application. For exploratory programming I used to just enable remote debugging and set break points. This should be going away in favor of the REPL introduced in Java 9 tho... Haven't met lots of developers that have integrated it into their gradle builds yet tho...
- geofft 7y ago
  Are you testing your code inside or outside of Docker? How do you do exploratory programming in languages/environments that don't give you a remote REPL? (For instance, how do you do it in C?)
  - dclusin 7y ago
    The common setups I've worked with are Docker instances for the automated CI tests and then running tests on your dev machine w/o a container. If I had to do exploratory programming without a REPL I'd probably leverage my test environment to write small bits of code, step through them with a debugger, and then use the results to structure a larger program.

gravypod 7y ago

I've done some initial investigation a while ago. You should be able to attach gdb to a process in a different cgroup. To more quickly edit configs just make it so your containers build and start quickly. This is very easy to do if you are aware of the docker layer cache. Hot reload is basically `while (true) build() && restart()`

geofft 7y ago
Oh, you can definitely attach gdb from outside the container to processes in the container. But I need to load debug symbols, libraries, etc. I'd like gdb to be able to see the source code I'm editing. I've done it and it doesn't quite work (I have stories about gdb's strange attempts at container-awareness... it spawns a thread and tries to stick that thread in the container so it can find files). Also, now your gdb setup isn't containerized, and that is genuinely part of your "dev environment" (it's a thing you want to share with other developers), so you lose out on many of the advantages of Dockerizing your dev environments. Note that you can't gdb/strace across containers because you can't see pids in one container from another container. (Similarly, a Kubernetes sidecar with gdb/strace isn't really going to work.) Not all of my software restarts quickly, and it's not because I don't know how to make containers do things - it's because the underlying build system is slow even on the local machine, or because they do a lot of processing at startup, or whatever. (Very few of the things I work on are webapps!)
- amazingman 7y ago
  > Note that you can't gdb/strace across containers because you can't see pids in one container from another container. (Similarly, a Kubernetes sidecar with gdb/strace isn't really going to work.) In Kubernetes, you can configure a Pod to share a single process namespace among its containers. https://kubernetes.io/docs/tasks/configure-pod-container/share-process-namespace/ https://kubernetes.io/docs/tasks/configure-pod-container/sha...
  - geofft 7y ago
    Wow, I totally missed this - good to know!

dclusin 7y ago

JVM has remote debugging as well. Works great with Intellij and Eclipse.