7 ms·
Exchanging Bash for Python is a folly. Python seems to handle complexity better, but only on the surface. (Basically Python has nice data structures, great str
by pas 9y ago
Exchanging Bash for Python is a folly.
Python seems to handle complexity better, but only on the surface. (Basically Python has nice data structures, great string formatting, and .. that's it.)
Setting up the Python environment requires something, because Python comes in many flavors (2 vs 3, system installed, user installed, /usr/local) and there are a bunch of factors (pip, pyenv, pipenv, PYTHONPATH, current directory with modules, requirement for python-dev headers, and GCC to compile native parts of packages, and then of course maybe extra shared libraries too) that can affect the "script".
Yet Python provides no way to sanity check itself. Mypy is wonderful, but doesn't really help with scripts, where the enemy is raised exceptions, not simple type errors.
And Python lacks efficient, quick and dirty process control. Sure, there are nice packages for handling subprocesses, but that's really a lot more batteries-included in Bash.
Do I like Bash? No, but debugging a bash script is -x, even the most vile many thousand lines of Bash will eventually stop somewhere with a clean-ish error or finish running.
I have no great recommendation for alternative (maybe Ammonite, maybe scripting in Rust), but I'm interested in what people think about this.
- simias 9y agoIt's really not about data structure or anything like that. The big problem with any large shell script is that it's utterly difficult to do proper error handling. Usually the best you can do is check return values to detect an error and 'exit' immediately, hoping that your "trap" works correctly to clean up behind you. >Python lacks efficient, quick and dirty process control. Yeah, quick and dirty, that's kind of the problem really. It's great for 20-line scripts but it turns into a huge mess for larger projects unless you enforce super strict guidelines and everybody has their doctorate in bash. Python, perl and friends make it harder to spawn 3rd party programs and pipe them together for instance, but it also makes it massively easier to handle errors and give decent diagnostics to the user.
- pas 9y ago> trap I don't have more faith in Python's except than in Bash's trap. If you use 3rd party code or a subprocess, then both are pretty weak compared to something kernel enforced. > makes it massively easier to handle errors and give decent diagnostics to the user. I don't really find that. You have to fight for input/output for subprocesses in Python. Sure, logging is easier in Python/perl/PHP/whatever, than in Bash, but error handling is about the same. You have to do everything manually, and there's no compiler, nor type (or other magical) system, that'd warn you if you have missed something.
- chronial 9y ago> You have to do everything manually You are aware that error handling is done via exceptions python?
- pas 9y agoYou are aware that there's no compiler, and that you have to write the try-except-finally? I'm talking about how weak guarantees Python gives compared to Bash about the correctness of the program. It's easy to get None-s, it's too easy to forget to handle file not found or lack of permission.
- xapata 9y agoMaybe you should start raising exceptions instead of returning None. I've almost never wanted to have my program do anything but crash upon file not found or permission error.
- kemitche 9y agoFor most scripts, handling file not found isn't gonna happen anyway. Often the result is just printing to standard out "hey I couldn't find file X" then bailing. An uncaught exception works well enough for this in many cases, such as build scripts. Same goes for permissions issues and a whole slew of similar scenarios. For many use cases, the lack of forced error checking is a feature. Concrete example: build scripts. As long as the cause of the build failure is clear from the uncaught exception, there's no reason to handle it. I don't need a friendly UI to avoid scaring off non technical users, I just need something that handles the happy path well and crashes with a meaningful stack trace when something unexpected happens.
- simias 9y agoLet's be clear, I'm not a huge fan of python or exception-based error handling either but I'll take it any day over... whatever you want to call shell ad-hoc error handling or lack thereof. Trap is not like an exception, it's more like "atexit" or a signal handler. You can't really nest them, you can't really recover from them. So I definitely disagree that error handling is about the same, python's a lot better. >there's no compiler, nor type (or other magical) system, that'd warn you if you have missed something. Well that's the whole static vs. dynamic debate but that's a different story. Use C# or Rust or whatever you feel like if you really can't stand python or scheme. Maybe there's a niche to be filled for statically typed scripting languages, there are no popular ones that I'm aware of. Still, that's besides the point, while I personally prefer static typing like you there's really no contest between dynamic typing and "everything is a string and you can only concatenate them" typing.
- dkrikun 9y agoThat's because Python is not a scripting language. Dynamic is not scripting.
- _sdegutis 9y agoI always add ‘set -ex’ to the top of my bash scripts to exit on any non zero top level return code and to print out all statements being run including expanded variables or inner statements.
- hossbeast 9y agoBased on the rest of your comment (which I agree with), I think you meant "Exchanging Bash for Python is a folly."
- laurent123456 9y agoI had some success with Node.js (though it was in a Node-based project to begin with). I haven't tried piping commands, but to run a bunch of commands and then have all the logic in JavaScript it's quite convenient. Here's an example that builds a React Native app, while updating version numbers and Git tags: https://github.com/laurent22/joplin/blob/master/Tools/release-android.js https://github.com/laurent22/joplin/blob/master/Tools/releas...
- mattnewton 9y agoMy main problem with scripting in Node vs python is the event loop gets in your way when just trying to write a quick and dirty sequence of steps. Sure you can use async await but now you are transpiling your script.. seems easier to just bang it out in python. I love Node but I use it for backend where the event-loop model is useful.
- chrisfinazzo 9y agoI would be remiss if I didn't mention shellcheck, which will also warn you against doing stupid things. Very useful when debugging. https://github.com/koalaman/shellcheck https://github.com/koalaman/shellcheck
- computerfriend 9y agoAlso fantastic for asynchronous linting, if you're into that.
- coldtea 9y ago>Python seems to handle complexity better, but only on the surface. (Basically Python has nice data structures, great string formatting, and .. that's it.) It's a complete scripting language, with tons of features, from a comprehensive standard library with something for most needs, to a packaging system, isolated environments, async io, and more. And lots of expressivity in structuring your program (including optional types). So, not sure what the above "and that's it" means. That it's not Haskell? Yes, it's not. >Setting up the Python environment requires something, because Python comes in many flavors (2 vs 3, system installed, user installed, /usr/local) and there are a bunch of factors (pip, pyenv, pipenv, PYTHONPATH, current directory with modules, requirement for python-dev headers, and GCC to compile native parts of packages, and then of course maybe extra shared libraries too) that can affect the "script". If you're doing dev-ops with such scripts, you already need to handle the environment in general.
- pas 9y ago> not sure what the above "and that's it" means. That it's not Haskell? Yes, it's not. That scripting in Python is pretty bad. It makes thing harder and at the same time you still have to think a lot about what can go wrong, because there's no enforced error handling.
- jamespo 9y agoSome actual examples would be nice; this just sounds like nirvana fallacy.
- dkrikun 9y agoDunno, why do "scripting", it's a rather clean oop language. For scripting use bash or perl maybe.
- tormeh 9y agoYeah, even Java doesn't force you to handle errors. Please, calm down. "Scripting" in a compiled language is not practical. People use the word "script" for a reason - it's because they don't want to deal with a language that forces them to think about everything that could go wrong. For some tasks that's perfectly sensible. For other tasks, there is Rust, Haskell and many others.
- hamandcheese 9y agoIts possible to write 2-3 compatible Python code. Furthermore, for basic "scripting" its reasonable to assume no external dependencies, so no need to worry about packaging, pip, pipenv, etc. The major disadvantage of any "comfortable" language aside from Python is that they aren't installed by default on nearly every distribution of every OS. This is why, for example, Ansible only requires Python 2.6 on managed nodes, allowing literally zero installation on almost any host you want to manage. If you already have a good mechanism for packaging and distributing software to all your machines (which I think is a reasonable expectation for Google) then go ahead and use whatever language you're well equipped for. But know you'll be sacrificing the easy and wide distribution that using Bash or Python brings you.
- pas 9y ago> This is why, for example, Ansible only requires Python 2.6 on managed nodes, allowing literally zero installation on almost any host you want to manage. Yes, Ansible transfers everything. It's ugly and slow :( I pretty much gave up on [vanilla] OpenStack because their idea of DevOps is custom python script generated Ansible plays.
- hamandcheese 9y agoSlow? Maybe. Ugly? I find it rather elegant.
- TheCondor 9y agoYou’ve hit most of it. The desire is a lowest common denominator. Python, Ruby, etc would be great except that in some of the places there arent current versions and beyond the most mundane you end up wanting 3rd party packages, which complicates your packaging or removes that lowest common denominator property. What it begs is for a new tool. A single file, statically linked “super shell” that lets you do structured scripting and has builtins for a bunch of common things (reading in addresses, daemonizing processes, reading the process table, etc. and that tool needs to be build able on nearly everything. Now I like rust and the attention it is getting and the interest in safety and correctness. I can’t think of many places where I’d reach for bash but be happier writing rust, a lot of it is 15minute disposable programming jobs. At a glance, I think I’d trade 5minutes of set -e to find some error I skipped for 45 minutes of trying to rejigger things to properly borrow some shared value.
- pas 9y agoI thought of Rust because of the strict typing. If something TypeScript-like could compile to Bash, that'd be amazing. (Maybe there's something like that already.) But now, to think of it, just using Node + TS (+ yarn) sounds better than Python.
- h1d 9y agoshelljs? https://github.com/shelljs/shelljs https://github.com/shelljs/shelljs
- rkeene2 9y agoTcl meets all of these requirements. I have a Linux distribution whose PID 1 is a Tcl shell and I can setup everything from there without spawning any new processes.
- oblio 9y agoYeah, but Tcl is old and quite crufty. The docs are a mess, everything is a string, a lot of concepts it uses are not very familiar and/or mainstream (upvar?). A modern take on Tcl is what we need, in my opinion. Surely we should be able to build something nicer 30 years after the first release of Tcl.
- Intermernet 9y agoPersonally I've found myself writing lots of simple command line utilities in Go. YMMV but the standard library handles file IO, network stuff, running external commands and a bunch of standard encodings really well. It also has static typing, and awesome concurrency features. It's obviously not perfect, but it has replaced lots of Bash scripts for me.
- taesis 9y agoRe. a recommended alternative to quick and dirty bash scripts, do you think golang would do? I've barely dabbled with the language, but its default packaging (a single, static binary) and build system (`go build x`) seem well suited to rapid setup, deployment and testing, which is presumably what you'd want in this scenario.
- pas 9y agoMany people use go for such things. I simply don't like go, it's too crude (too C), and the packaging system is. Well, it's not Cargo. But probably for deploy scripts, go with a static binary (hosted on, let's say your GitLab instance - using the GitLab HTTP API with a token to wget/curl it) is nigh unbeatable in end-to-end development and deployment time.
- taesis 9y agoAah, makes. I'm not a huge fan of it for similar reasons, but the appeal of a language that's quick, dirty, and relatively acceptable by coworkers is strong.
- sametmax 9y agoThey have a standard setup everywhere, with the same Python version, and the same libs available. So basically, for them, Python requires no fiddling and works as is. Same for the shell. They say "use [[]]" because they know that compat is not a problem: it's the same bash everywhere. Google has millions of servers, billions of lines of code and surely of lot of experience with deployment, managing complexity and handling teams. I'm kinda trusting them on this choice.
- deleted 9y ago[deleted]
- int0x80 9y agoAgree. If you want to do shell work in something closer to the shell and have a real programming language, Perl is the winner.