4 ms·
> If you need a hot fix "RIGHT NOW" you might be doing something wrong in the first place. Maybe I phrased this point badly. I haven't found myself to need a
by murkt 6y ago
> If you need a hot fix "RIGHT NOW" you might be doing something wrong in the first place.
Maybe I phrased this point badly. I haven't found myself to need a hot fix RIGHT NOW for a long time. What I had in mind was this: sometimes there is an issue that is very hard to reproduce locally. If a developer doesn't understand it from the get go and have to experiment a bit, it can be frustrating to commit, wait for deploy (even to stage environment), test what happens and repeat.
> Not sure what you mean with lingering containers. Do you launch yours through SSH manually?
Of course the application itself is run automatically, web server, workers, etc.
However, we often need to do some bespoke data analysis, so we often ssh into a server, type `make shell` to launch a REPL and type/paste some stuff into it.
- compsciphd 6y agodocker exec -it <container id> kubectl exec -it pod -c container
- murkt 6y agoNot sure what do you want to say with this command.
- compsciphd 6y agoyou want to be able to "ssh in and make changes to figure out how to fix things". why doesn't that do that for you?
- reportgunner 6y agoSorry, but did you not forget the --rm option to remove the container once it dies ?
- AkshatM 6y agoI interpret it as a suggestion that you allow developers access to the running container in production to debug the issue There are more sane ways to do this than through raw kubectl access, of course: see e.g. telepresence (https://www.telepresence.io/ https://www.telepresence.io/)
- funcDropShadow 6y agoBy sane, you mean yet another piece of infrastructure that has to be installed, documented, packaged into containers, deployed and updated for a problem that did not exist without containers. For a very large organization (e.g. hundreds of devs) this make sense, but for a medium sized company (> billion EUR turnover per year outside of software business) this soon becomes just another piece of overhead.
- wwright 6y agoThis problem certainly exists without containers. - How do you provide easy visibility into running applications? - How do you prevent inspecting an application from affecting the behavior of an application?
- sseagull 6y agoYou put this fairly succinctly. And I agree wholeheartedly. I think of the phrase “adding epicycles” when this kind of stuff starts happening.
- gnfargbl 6y agoI think you might be falling into a cycle that I often find myself falling in to. It goes like this: Oh, there's a bug. But it's obviously caused by _this line here_, so I won't even bother testing it offline, I'll just fix it and run CI and then... oh no, wait, it was more complex than I thought, but I'm in the change-CI-test cycle now so I'll keep doing that. And then, all of a sudden it's taken me three hours or more to fix what was in reality quite a small bug. The solution is to force yourself not to use the CI/container system at all during debugging, and instead to build the binary (or whatever) standalone. That's hard because you invariably aren't tooled up to run the component outside its deployment system so you have to do some extra tricks, but in the long tun it seems to be the way to go.
- murkt 6y agoOver the years I've learned to do this very rarely. However, I'm not the lone wolf in the woods, I also have other developers working for me. I can talk to them however long I want and motivate them to debug locally to understand the problem, but in the end they have to gain some experience with their own pain. I would prefer that their experience gaining would be faster, so it would cost me less money :)
- paloaltokid 6y ago> I would prefer that their experience gaining would be faster, so it would cost me less money :) Maybe your current strategy isn't working?