11 ms·

> Don't write internal cli tools in python A lot of the advice is good but I take an issue with this one. With poetry and docker, packaging Python apps for eas

by torton 5y ago

> Don't write internal cli tools in python A lot of the advice is good but I take an issue with this one. With poetry and docker, packaging Python apps for easy consumption is a non-issue. Same for Ruby. If you can get your team to standardize on poetry, you might not even need containers -- but, honestly, running these tools from CI or automation (anywhere) is so useful that you probably want container versions anyway. Golang is not a good fit for exploratory CLIs that work with complex data structures and are written for one-off, low-CPU consumption purposes -- not for scale-up API services. Just having an interactive shell (or `pry` in Ruby, those two are identical for the purpose) saved me probably weeks of time. Trying to unit test any moderately complex API surface brings me to tears when I compare it to trivial object mocking in something like Ruby. Python (or Ruby) are ideal for this and have excellent frameworks for CLI tools.

p_l 5y ago
Ruby is pretty good in this, but Python I'd heavily argue against. Maybe in another decade when they manage to restabilize what they wrought - it used to be easy. Python gets more and more complex the further away the people running the tool are from fancy recentish distros (Fedora, Ubuntu, Arch) or special constrained environments (Nix, running the CLI in docker). The moment you have to deal with unspecified RHEL version (6 is still reasonably common) or derivative, or Mac or Windows, kiss any expectation of python packaging being nice "bye bye". Unless of course you have a platform team that can handle the packaging and distribution for you, but then it probably falls a bit under "constrained environment".
- zrail 5y ago
  Ruby is also a pain in the butt, fwiw. It's just slightly better because gems are standardized, but you still need something like rbenv to manage Ruby versions, and gods help you if you need openssl.
  - p_l 5y ago
    In my experience the major difference is that rbenv feels more of a convenience in upgrading ruby on truly outdated systems (or when you can't use prepackaged for reasons), and that for majority work minor version differences are at most one line in Gemspec away. I really, really can't say that about Python.
duped 5y ago
> With poetry and docker, packaging Python apps for easy consumption is a non-issue. This is why you shouldn't write CLI tools in Python, you need frigging docker to package them
- frenchyatwork 5y ago
  You most certainly don't, but many people seem to have lost their minds around packaging.
  - disgruntledphd2 5y ago
    Well it depends, right? My most recent Python packaging adventure was managed through Docker, but this was because I also had Java-based dependencies (reAgent RL framework). Given how much easier that was than getting people to use conda/pip (conda is better for DS stuff as it handles C-level dependencies), I completely understand how people just suggest Docker. Like, if you're doing pure Python lots of this may not be necessary, but as soon as you start having multiple C-level dependencies pip breaks (and pip didn't even do dependency resolution till last year(!)).
habitue 5y ago
> With poetry and docker, packaging Python apps for easy consumption is a non-issue. Yeah, apps it's worth doing this for, cli tools it's not.
chousuke 5y ago
I'd state it as "don't use external dependencies carelessly". It applies to more than just Python. Writing and deploying Python tools is easy if all your dependencies come in distro packages.
- p_l 5y ago
  And your code is carefully written to run perfectly fine on both 2.7 and anything between at least 3.4 and 3.10, but might be better to handle as early as 3.0 ... I might have somewhat similar scars to TFA author, I guess...
  - disgruntledphd2 5y ago
    The only big problems with such an approach are string formatting (f-strings are really, really nice) but also dictionary ordering is massively different between 3.5 and 3.7 Presumably this is known to all the people who've been doing Python for a long time, but it bit me in the ass relatively recently.
    - p_l 5y ago
      People who regularly use python probably also have their local environment set up so that it's not a problem to have Python CLI. Problems start when going elsewhere :/
  - chousuke 5y ago
    Well, that depends on what you need to support; even if RHEL7 is the oldest distribution that you need to support, you can still assume you have at least Python 3.6, or python 3.4 for RHEL6 (which is already EOL). There's no obligation to support every potential platform your script might ever need to run on until that need actually arises. You just need to define what platforms you want to support and work within their constraints. Unfortunately for lots of software that platform is essentially "whatever the developer managed to install on their system". Constraining yourself to support a particular platform might mean there will be libraries you will not be able to use even if they might be useful, but that's the tradeoff you make when building software against a stable platform.
9dev 5y ago
I recently had to package a machine learning Jupyter notebook created by a very smart person, albeit a scientist, not a developer. Getting the tool somehow into production, making it reproducible, testable, and maintainable, has proven to be a major headache. Before, I only casually dabbled in the python world, but this was the first time I had to care about packaging, dependencies, CI and the like. Turns out, as TFA said, nobody knows how to package python apps right. For what it’s worth, I wasn’t even able to find some kind of best practice to manage friggin dependencies. There’s like a myriad of package managers, all of them work differently, and nobody seemed to had something like redistributing an app to other people on their mind. Coming from PHP, JavaScript and Go, this was utterly ridiculous to me. Go ahead, tell me I got it all wrong and it’s really easy using tool Xyz, but if a developer with some experience under their belt isn’t able to figure this out in a few days, things are just broken.
- worik 5y ago
  It has taken everybody else thirty years to realise: Python is not very good....
  - burnished 5y ago
    Python is fucking phenomenal. Packaging python is an exercise in frustration until you become a level 3 wizard.
    - nerdponx 5y ago
      It really isn't that hard to package for Python. Certainly a lot easier than packaging for Debian, and not much different from packaging for Ruby or Node. The bad part is the Setuptools documentation, which is slowly improving, but is (and has been for years) so bad that almost nobody can learn from reading it.
      - burnished 5y ago
        I think we might be in agreement because I see that these >>Packaging python is an exercise in frustration >>Setuptools documentation [...] is [...] so bad that almost nobody can learn from reading it and these >> until you become a level 3 wizard >> It really isn't that hard to package for Python. as being fundamentally equivalent statements.
pphysch 5y ago
> Golang is not a good fit for exploratory CLIs that work with complex data structures and are written for one-off, low-CPU consumption purposes "exploratory CLIs that work with complex data structures and are written for one-off ... purposes" Sounds like a maintenance nightmare if you are actually "deploying" Python scripts that fit this definition. Yeah, golang will usually require a bit more boilerplate up front, but it is going to make your workflows infinitely more maintainable & flexible in the long term due to type safety + easy (and docker-less) packaging. IMHO, if it's not a literal one-liner in ($SHELL|curl|awk|jq|*), it should probably be done right the first time in golang/etc or go back to the drawing board.
- maleldil 5y ago
  There's a category of scripts that are complex enough so that shell is too convoluted, but can be a 10-20 lines Python script that would be 50-100 in Go. By the way, OP specifically mentioned "one-off", which you also quoted. Why mention deployment?
  - pphysch 5y ago
    > Why mention deployment? Because the article does.
Thaxll 5y ago
No one wants a CLI in Ruby, and if you look arround there is no popular CLI written in Ruby it's one of the worse language to write CLI in because no one has the Ruby runtime installed on their machine, then you have different architectures, different OS etc ... Go is way way better than Ruby on that topic. As for complex data structure, I don't understand exactly what do you mean, as if a dynamic language would model that easier than a strongly typed language. When you get a single binary for CLI it's hard to use anything else, the pain of using pip for anything python based.
- faizshah 5y ago
  Homebrew is written in ruby, not only is it one of the most popular cli tools but it is also often cited as one of the best designed cli user experiences.
- dragonwriter 5y ago
  > there is no popular CLI written in Ruby Chef?
  - amdelamar 5y ago
    And [Brew](https://brew.sh https://brew.sh)
    - freedomben 5y ago
      Also Metasploit
  - deleted 5y ago
    [deleted]
- pphysch 5y ago
  Vagrant is written in Ruby but is currently getting ported to Golang.
- deleted 5y ago
  [deleted]
boardwaalk 5y ago
The packaging thing (and I don't necessarily think "put it in Docker" is a great answer, especially if you're using Python as glue -- why isolate your glue?) is something you kind of solve once and are done with. You probably already want/need a common development environment between people that is close to what you deploy, and adding a fixed Python into the mix with pyenv or whatever is not a big deal. I don't know, I've generally had a good time with. You have to know how to do it and where the foot guns are (I'm still learning), but it's better than being comparatively gimped in development velocity using Go, never mind Rust or C++.

by torton 5y ago

p_l 5y ago

Ruby is pretty good in this, but Python I'd heavily argue against. Maybe in another decade when they manage to restabilize what they wrought - it used to be easy. Python gets more and more complex the further away the people running the tool are from fancy recentish distros (Fedora, Ubuntu, Arch) or special constrained environments (Nix, running the CLI in docker). The moment you have to deal with unspecified RHEL version (6 is still reasonably common) or derivative, or Mac or Windows, kiss any expectation of python packaging being nice "bye bye". Unless of course you have a platform team that can handle the packaging and distribution for you, but then it probably falls a bit under "constrained environment".

zrail 5y ago
Ruby is also a pain in the butt, fwiw. It's just slightly better because gems are standardized, but you still need something like rbenv to manage Ruby versions, and gods help you if you need openssl.
- p_l 5y ago
  In my experience the major difference is that rbenv feels more of a convenience in upgrading ruby on truly outdated systems (or when you can't use prepackaged for reasons), and that for majority work minor version differences are at most one line in Gemspec away. I really, really can't say that about Python.

duped 5y ago

> With poetry and docker, packaging Python apps for easy consumption is a non-issue. This is why you shouldn't write CLI tools in Python, you need frigging docker to package them

frenchyatwork 5y ago
You most certainly don't, but many people seem to have lost their minds around packaging.
- disgruntledphd2 5y ago
  Well it depends, right? My most recent Python packaging adventure was managed through Docker, but this was because I also had Java-based dependencies (reAgent RL framework). Given how much easier that was than getting people to use conda/pip (conda is better for DS stuff as it handles C-level dependencies), I completely understand how people just suggest Docker. Like, if you're doing pure Python lots of this may not be necessary, but as soon as you start having multiple C-level dependencies pip breaks (and pip didn't even do dependency resolution till last year(!)).

habitue 5y ago

> With poetry and docker, packaging Python apps for easy consumption is a non-issue. Yeah, apps it's worth doing this for, cli tools it's not.

chousuke 5y ago

I'd state it as "don't use external dependencies carelessly". It applies to more than just Python. Writing and deploying Python tools is easy if all your dependencies come in distro packages.

p_l 5y ago
And your code is carefully written to run perfectly fine on both 2.7 and anything between at least 3.4 and 3.10, but might be better to handle as early as 3.0 ... I might have somewhat similar scars to TFA author, I guess...
- disgruntledphd2 5y ago
  The only big problems with such an approach are string formatting (f-strings are really, really nice) but also dictionary ordering is massively different between 3.5 and 3.7 Presumably this is known to all the people who've been doing Python for a long time, but it bit me in the ass relatively recently.
  - p_l 5y ago
    People who regularly use python probably also have their local environment set up so that it's not a problem to have Python CLI. Problems start when going elsewhere :/
- chousuke 5y ago
  Well, that depends on what you need to support; even if RHEL7 is the oldest distribution that you need to support, you can still assume you have at least Python 3.6, or python 3.4 for RHEL6 (which is already EOL). There's no obligation to support every potential platform your script might ever need to run on until that need actually arises. You just need to define what platforms you want to support and work within their constraints. Unfortunately for lots of software that platform is essentially "whatever the developer managed to install on their system". Constraining yourself to support a particular platform might mean there will be libraries you will not be able to use even if they might be useful, but that's the tradeoff you make when building software against a stable platform.

9dev 5y ago

I recently had to package a machine learning Jupyter notebook created by a very smart person, albeit a scientist, not a developer. Getting the tool somehow into production, making it reproducible, testable, and maintainable, has proven to be a major headache. Before, I only casually dabbled in the python world, but this was the first time I had to care about packaging, dependencies, CI and the like. Turns out, as TFA said, nobody knows how to package python apps right. For what it’s worth, I wasn’t even able to find some kind of best practice to manage friggin dependencies. There’s like a myriad of package managers, all of them work differently, and nobody seemed to had something like redistributing an app to other people on their mind. Coming from PHP, JavaScript and Go, this was utterly ridiculous to me. Go ahead, tell me I got it all wrong and it’s really easy using tool Xyz, but if a developer with some experience under their belt isn’t able to figure this out in a few days, things are just broken.

worik 5y ago
It has taken everybody else thirty years to realise: Python is not very good....
- burnished 5y ago
  Python is fucking phenomenal. Packaging python is an exercise in frustration until you become a level 3 wizard.
  - nerdponx 5y ago
    It really isn't that hard to package for Python. Certainly a lot easier than packaging for Debian, and not much different from packaging for Ruby or Node. The bad part is the Setuptools documentation, which is slowly improving, but is (and has been for years) so bad that almost nobody can learn from reading it.
    - burnished 5y ago
      I think we might be in agreement because I see that these >>Packaging python is an exercise in frustration >>Setuptools documentation [...] is [...] so bad that almost nobody can learn from reading it and these >> until you become a level 3 wizard >> It really isn't that hard to package for Python. as being fundamentally equivalent statements.

pphysch 5y ago

> Golang is not a good fit for exploratory CLIs that work with complex data structures and are written for one-off, low-CPU consumption purposes "exploratory CLIs that work with complex data structures and are written for one-off ... purposes" Sounds like a maintenance nightmare if you are actually "deploying" Python scripts that fit this definition. Yeah, golang will usually require a bit more boilerplate up front, but it is going to make your workflows infinitely more maintainable & flexible in the long term due to type safety + easy (and docker-less) packaging. IMHO, if it's not a literal one-liner in ($SHELL|curl|awk|jq|*), it should probably be done right the first time in golang/etc or go back to the drawing board.

maleldil 5y ago
There's a category of scripts that are complex enough so that shell is too convoluted, but can be a 10-20 lines Python script that would be 50-100 in Go. By the way, OP specifically mentioned "one-off", which you also quoted. Why mention deployment?
- pphysch 5y ago
  > Why mention deployment? Because the article does.

Thaxll 5y ago

No one wants a CLI in Ruby, and if you look arround there is no popular CLI written in Ruby it's one of the worse language to write CLI in because no one has the Ruby runtime installed on their machine, then you have different architectures, different OS etc ... Go is way way better than Ruby on that topic. As for complex data structure, I don't understand exactly what do you mean, as if a dynamic language would model that easier than a strongly typed language. When you get a single binary for CLI it's hard to use anything else, the pain of using pip for anything python based.

faizshah 5y ago
Homebrew is written in ruby, not only is it one of the most popular cli tools but it is also often cited as one of the best designed cli user experiences.
dragonwriter 5y ago
> there is no popular CLI written in Ruby Chef?
- amdelamar 5y ago
  And [Brew](https://brew.sh https://brew.sh)
  - freedomben 5y ago
    Also Metasploit
- deleted 5y ago
  [deleted]
pphysch 5y ago
Vagrant is written in Ruby but is currently getting ported to Golang.
deleted 5y ago
[deleted]

boardwaalk 5y ago

The packaging thing (and I don't necessarily think "put it in Docker" is a great answer, especially if you're using Python as glue -- why isolate your glue?) is something you kind of solve once and are done with. You probably already want/need a common development environment between people that is close to what you deploy, and adding a fixed Python into the mix with pyenv or whatever is not a big deal. I don't know, I've generally had a good time with. You have to know how to do it and where the foot guns are (I'm still learning), but it's better than being comparatively gimped in development velocity using Go, never mind Rust or C++.