5 ms·

> It's horrendous and I seriously don't get what's so difficult in just coding these scripts in a programming language that provides single statically linked bi

> It's horrendous and I seriously don't get what's so difficult in just coding these scripts in a programming language that provides single statically linked binaries (like Golang) and just distribute it with your images -- or run them in CI/CD and have init containers and never include them in the images in the first place. It takes longer to develop (for sufficiently small scripts), it's harder to verify, it's harder to debug, it requires a build step and it's a lot more effort to modify it in-place. The process for calling other programs is also extremely streamlined, making it perfect for integration tasks like CI/CD pipelines. Some of these issues don't exist when using an interpreted language like Python instead, but this comes with its own problems. Shell is pretty universal, well-known and is, for sufficiently small problems, the quickest solution.

throwaway482945 2y ago
Python's facilities for calling subprocesses are pretty inconvenient compared to bash IMO. It defaults to binary output instead of UTF-8 so I almost always have to set an option for that. I wind up having to define threads in order to run programs in the background and do anything with their output in real time, which has an awkward syntax. The APIs for checking the exit code vs raising an error are pretty non-obvious and I have to look them up every time. And I always wind up having to write some boilerplate code to strip whitespace from the end of each line and filter out empty lines, like p.stdout.rstrip().split('\n') which can be subtly incorrect depending on what program I'm invoking.
- theamk 2y ago
  It looks like you used some old tutorials? "subprocess.run" appeared in python 3.5, and it's pretty nice - for example you so "check=True" to raise on error exit code, and omit it if you want to check exit code yourself. And to get text output you put "text=True" (or encoding="utf-8" if you are unsure what the system encoding is) As for your boilerplate, it seems "p.stdout.splitlines()" is what you want? it's what you normally want to use to parse process output line-by-line The background process is the hardest part, but for the most common case, you don't need any thread: proc = subprocess.Popen(["slow-app", "arg"], stdout=subprocess.PIPE, text=True) for line in proc.stdout: print("slow-app said:", line.rstrip()) print("slow-app finished, exit code", proc.wait()) sadly if you need to parse multiple streams, threads are often the easiest.
docandrew 2y ago
My team is also in the process of refactoring a number of shell script CI pipelines into Go. With `go run` it might as well be interpreted, from a UX standpoint it’s as easy to call as a shell script but with the benefit of static types and a robust ecosystem of supporting libraries and not having to sed/awk your way through command output.
pdimitar 2y ago
> It takes longer to develop (for sufficiently small scripts) Obviously, but at one point I sat down and tried to estimate how much time I wasted on using `set -euo pipefail` in my scripts and still having to chase after silent failures. I might be biased at this point but it still seemed quite a lot. For a lot of one-off tasks shell scripting is 90% superior (unless you are super comfy with Golang) because it takes literal minutes to write and then iterate on it. Sure. But my threshold for when to reach for a proper program has been getting lower and lower lately because I very quickly arrive at a point where I need typing, better error-handling, retries and a few others. Not everyone's case, surely. It seems my journey was more along the lines of "I was using shell scripting much more than I had to and I am making a partial comeback to proper programs".

throwaway482945 2y ago

Python's facilities for calling subprocesses are pretty inconvenient compared to bash IMO. It defaults to binary output instead of UTF-8 so I almost always have to set an option for that. I wind up having to define threads in order to run programs in the background and do anything with their output in real time, which has an awkward syntax. The APIs for checking the exit code vs raising an error are pretty non-obvious and I have to look them up every time. And I always wind up having to write some boilerplate code to strip whitespace from the end of each line and filter out empty lines, like p.stdout.rstrip().split('\n') which can be subtly incorrect depending on what program I'm invoking.

theamk 2y ago
It looks like you used some old tutorials? "subprocess.run" appeared in python 3.5, and it's pretty nice - for example you so "check=True" to raise on error exit code, and omit it if you want to check exit code yourself. And to get text output you put "text=True" (or encoding="utf-8" if you are unsure what the system encoding is) As for your boilerplate, it seems "p.stdout.splitlines()" is what you want? it's what you normally want to use to parse process output line-by-line The background process is the hardest part, but for the most common case, you don't need any thread: proc = subprocess.Popen(["slow-app", "arg"], stdout=subprocess.PIPE, text=True) for line in proc.stdout: print("slow-app said:", line.rstrip()) print("slow-app finished, exit code", proc.wait()) sadly if you need to parse multiple streams, threads are often the easiest.

docandrew 2y ago

My team is also in the process of refactoring a number of shell script CI pipelines into Go. With `go run` it might as well be interpreted, from a UX standpoint it’s as easy to call as a shell script but with the benefit of static types and a robust ecosystem of supporting libraries and not having to sed/awk your way through command output.

pdimitar 2y ago

> It takes longer to develop (for sufficiently small scripts) Obviously, but at one point I sat down and tried to estimate how much time I wasted on using `set -euo pipefail` in my scripts and still having to chase after silent failures. I might be biased at this point but it still seemed quite a lot. For a lot of one-off tasks shell scripting is 90% superior (unless you are super comfy with Golang) because it takes literal minutes to write and then iterate on it. Sure. But my threshold for when to reach for a proper program has been getting lower and lower lately because I very quickly arrive at a point where I need typing, better error-handling, retries and a few others. Not everyone's case, surely. It seems my journey was more along the lines of "I was using shell scripting much more than I had to and I am making a partial comeback to proper programs".