3 ms·

Because of the insanity of python I run everything in Docker (through compose). No more issues with it not working on some dev's computer because of a missing w

Because of the insanity of python I run everything in Docker (through compose). No more issues with it not working on some dev's computer because of a missing wheel, or that they need to have X and Y c++ toolchain packages installed locally etc. No more trying to fix a broken setup after upgrading python or poetry versions, just "docker compose build" and you're up and running. No spending days getting a freshly cloned project up and running. Then I point Pycharm to use that as my remote interpreter through docker compose. Not watertight, but a hundred times better dev experience.

DangitBobby 4y ago
I've tried this a couple of times but just generally not liked the experience and gone back. When you get a shell into the container so you can use your project's python env, the tools you have installed on your machine (bashrc, rg, aliases) aren't available. I ultimately prefer the hassle of managing environments natively over forfeiting the use of my tools or juggling shells.
LtWorf 4y ago
.deb packages…
- groestl 4y ago
  Do you happen to have a link to a python application that ships it's own dependencies and has state-of-the art debian packaging?
  - vbernat 4y ago
    It depends what you mean by state-of-the-art, but it's possible with dh-virtualenv: https://github.com/vincentbernat/pragmatic-debian-packages/tree/master/puppetboard/debian https://github.com/vincentbernat/pragmatic-debian-packages/t... (not something that can become an official package as it goes against Debian policies)
  - LtWorf 4y ago
    > that ships it's own dependencies You mean a link to a malformed .deb packages? It's not difficult to do but it's also the wrong way to do it.
    - groestl 4y ago
      Honestly, there is a lot more I'd like to do with my life than caring about some debian packaging practices when all I want to do is ship a company's internal application with it's exact dependency versions. But it needs to work, and, e.g. "The virtualenv relocate-able flag has been always experimental, and never really worked;" So in practice we're back to shipping whole distributions using Docker (or, like we did, abandon Python as an option for development but keep .deb packages for deployment).
- u801e 4y ago
  On that note, why are we trying to use language specific packaging tools to build packages rather than just building OS specific packages where dependencies are handled by apt for deb packages or dnf for rpm packages? These are language agnostic and can get the job done.
  - groestl 4y ago
    Because I have dozens of colleagues, each with their own os/distribution, developing an application that needs to lock down dot releases of dependencies to the versions running in production and update them at the same time. There is no way to do that using distribution specific tools without going insane.
    - u801e 4y ago
      >> why are we trying to use language specific packaging tools to build packages rather than just building OS specific packages where dependencies are handled by apt for deb packages or dnf for rpm packages? >> >> These are language agnostic and can get the job done. > Because I have dozens of colleagues, each with their own os/distribution, developing an application that needs to lock down dot releases of dependencies to the versions running in production I assume that the production environment isn't running dozens of os versions and distributions. For development, using a VM or container running the same os version and distro that's used in production and using that OS's package format for packaging the software and installing it in the dev environment (on the container or VM) would work. You're testing to see if the software works in the production environment, not someone's preferred os/distribution. > and update them at the same time I'm not as familiar with apt, but dnf has a version lock feature that would allow you to lock down the dependencies to specific versions. You could update them and test them during development and update the version lock file to pull down the updated version of the dependency when updating production.
      - groestl 4y ago
        For every dependency that's missing from the upstream distribution in the exact version we need, we'd need to package that appropriately. We have nothing to gain here, nobody pays for that.
goodpoint 4y ago
No way. Docker is not designed for security.
- matsemann 4y ago
  Well, it's probably more secure than doing `pip install` and letting every dependency run whatever to install itself on your host...
  - goodpoint 4y ago
    Actually not at all assuming you are using a VM for a production workload. Additionally, even if you only rely on a container - still no: other container engines have a lower attack surface.
    - matsemann 4y ago
      I'm talking about local dev here.
wheelerof4te 4y ago
When the solution to basic problem like this is "use Docker", you realise how deeply flawed modern software development (in Python, at least) is.
- jessekv 4y ago
  Not sure I agree with "deeply flawed" verdict. The flaws of python's packaging system (as well as the GIL and other hangups) emerge from the tradeoffs of inventing a language optimised for reusing and connecting as many disparate system binaries as possible. Its not surprising that such a server scripting language is tightly coupled with the server environment that it runs in. Docker is just one way to ensure a reproducible server environment across machines. What do you think it could have done differently?
  - lmm 4y ago
    What could have been done differently: - Make dependency resolution deterministic by default. Unfortunately the whole ecosystem has to buy into it, but the benefits are huge. - Stop building castles of sand by making more layers of tooling that have to run in Python (and will therefore run in some random poorly managed Python). The language install can't manage the build system install - if anything the build system install should manage the language install. Adding pip to the language distribution was such a backwards decision that it marks the point where I gave up on Python ever fixing their stuff. - Ignore Linux system package managers (apt etc.), they have a fundamentally broken model and will infect your language ecosystem with that breakage if you try to cater to them.
- rtpg 4y ago
  to be fair to Python, almost every ecosystem is in "use Docker" land, even Go binaries will offer docker images!
groestl 4y ago
I secretly believe Docker only exists because of how f'up Python's distribution story is.
- jessekv 4y ago
  I also have often wondered how much Docker owes its success to the shortcomings of python.

DangitBobby 4y ago

I've tried this a couple of times but just generally not liked the experience and gone back. When you get a shell into the container so you can use your project's python env, the tools you have installed on your machine (bashrc, rg, aliases) aren't available. I ultimately prefer the hassle of managing environments natively over forfeiting the use of my tools or juggling shells.

LtWorf 4y ago

.deb packages…

groestl 4y ago
Do you happen to have a link to a python application that ships it's own dependencies and has state-of-the art debian packaging?
- vbernat 4y ago
  It depends what you mean by state-of-the-art, but it's possible with dh-virtualenv: https://github.com/vincentbernat/pragmatic-debian-packages/tree/master/puppetboard/debian https://github.com/vincentbernat/pragmatic-debian-packages/t... (not something that can become an official package as it goes against Debian policies)
- LtWorf 4y ago
  > that ships it's own dependencies You mean a link to a malformed .deb packages? It's not difficult to do but it's also the wrong way to do it.
  - groestl 4y ago
    Honestly, there is a lot more I'd like to do with my life than caring about some debian packaging practices when all I want to do is ship a company's internal application with it's exact dependency versions. But it needs to work, and, e.g. "The virtualenv relocate-able flag has been always experimental, and never really worked;" So in practice we're back to shipping whole distributions using Docker (or, like we did, abandon Python as an option for development but keep .deb packages for deployment).
u801e 4y ago
On that note, why are we trying to use language specific packaging tools to build packages rather than just building OS specific packages where dependencies are handled by apt for deb packages or dnf for rpm packages? These are language agnostic and can get the job done.
- groestl 4y ago
  Because I have dozens of colleagues, each with their own os/distribution, developing an application that needs to lock down dot releases of dependencies to the versions running in production and update them at the same time. There is no way to do that using distribution specific tools without going insane.
  - u801e 4y ago
    >> why are we trying to use language specific packaging tools to build packages rather than just building OS specific packages where dependencies are handled by apt for deb packages or dnf for rpm packages? >> >> These are language agnostic and can get the job done. > Because I have dozens of colleagues, each with their own os/distribution, developing an application that needs to lock down dot releases of dependencies to the versions running in production I assume that the production environment isn't running dozens of os versions and distributions. For development, using a VM or container running the same os version and distro that's used in production and using that OS's package format for packaging the software and installing it in the dev environment (on the container or VM) would work. You're testing to see if the software works in the production environment, not someone's preferred os/distribution. > and update them at the same time I'm not as familiar with apt, but dnf has a version lock feature that would allow you to lock down the dependencies to specific versions. You could update them and test them during development and update the version lock file to pull down the updated version of the dependency when updating production.
    - groestl 4y ago
      For every dependency that's missing from the upstream distribution in the exact version we need, we'd need to package that appropriately. We have nothing to gain here, nobody pays for that.

goodpoint 4y ago

No way. Docker is not designed for security.

matsemann 4y ago
Well, it's probably more secure than doing `pip install` and letting every dependency run whatever to install itself on your host...
- goodpoint 4y ago
  Actually not at all assuming you are using a VM for a production workload. Additionally, even if you only rely on a container - still no: other container engines have a lower attack surface.
  - matsemann 4y ago
    I'm talking about local dev here.

wheelerof4te 4y ago

When the solution to basic problem like this is "use Docker", you realise how deeply flawed modern software development (in Python, at least) is.

jessekv 4y ago
Not sure I agree with "deeply flawed" verdict. The flaws of python's packaging system (as well as the GIL and other hangups) emerge from the tradeoffs of inventing a language optimised for reusing and connecting as many disparate system binaries as possible. Its not surprising that such a server scripting language is tightly coupled with the server environment that it runs in. Docker is just one way to ensure a reproducible server environment across machines. What do you think it could have done differently?
- lmm 4y ago
  What could have been done differently: - Make dependency resolution deterministic by default. Unfortunately the whole ecosystem has to buy into it, but the benefits are huge. - Stop building castles of sand by making more layers of tooling that have to run in Python (and will therefore run in some random poorly managed Python). The language install can't manage the build system install - if anything the build system install should manage the language install. Adding pip to the language distribution was such a backwards decision that it marks the point where I gave up on Python ever fixing their stuff. - Ignore Linux system package managers (apt etc.), they have a fundamentally broken model and will infect your language ecosystem with that breakage if you try to cater to them.
rtpg 4y ago
to be fair to Python, almost every ecosystem is in "use Docker" land, even Go binaries will offer docker images!

groestl 4y ago

I secretly believe Docker only exists because of how f'up Python's distribution story is.

jessekv 4y ago
I also have often wondered how much Docker owes its success to the shortcomings of python.