4 ms·
How do you write "foo | bar" in Python?
by mbrock 10y ago
How do you write "foo | bar" in Python?
- IshKebab 10y agoHow often do you actually need to do that. 99% of `foo | bar` command could easily be `foo > a && bar < a` which is pretty trivial to do in Python.
- mbrock 10y agoWell, how do you write that in Python? I'm curious because these simple things that are delightfully easy in bash often turn out to be surprisingly tedious in other languages. Of course, some things are tedious in bash too. But a basic principle of shell scripting is that you call other programs to do the stuff you don't want to do in shell.
- IshKebab 10y agoLike this: http://stackoverflow.com/a/1996540 http://stackoverflow.com/a/1996540 I agree it is tedious, but to be honest, reading and writing to stdin/out isn't something that would commonly need to be done in a robust system. If the world were perfect you would use library functions. I definitely think there is scope for a language that works well as an interactive shell, and as a general purpose language. They have somewhat conflicting constraints but I'm sure we can do better than Bash. Have you seen how [ is implemented?
- mbrock 10y agoNo stdout or stdin for robust systems? I disagree. Yeah, a nicer shell-like language would be cool. I've been thinking about it for a while. Bash is quirky but it gets a lot of stuff right and once you understand it it can be extremely ergonomic and productive. And not depending on language run times other than shell can be really glorious in some situations, too...
- prodigal_erik 10y agoNow you have to clean up 'a'. And decide whether /tmp or /var/tmp or a dir on some other filesystem has enough space to hold all of 'a' until 'bar' is finished. Is it a security problem that other processes could snoop the contents of 'a' or even tamper with it?
- mpeg 10y agobar(foo) When you're not writing shell, just use the tools the language gives you. For the matter, I think a shell script is cleaner than a python script for devops; but I don't think the composability of unix tools is that much of an advantage compared to the amount of python libraries out there.
- mbrock 10y agoWhen I use shell it's often exactly because I want to construct pipelines of processes and FIFOs and do all the other things that shell does very well and has done well for decades. I'm likely to be using Python programs and other programs in those shell scripts. The beauty of shell is that it makes it so easy to compose programs written in different languages.
- lmm 10y agoShell does things well provided all the intermediate states are naturally expressible as streams of bytes. Otherwise not so much. I think the advantages of using a single language for everything outweigh the disadvantages - see e.g. http://www.teamten.com/lawrence/writings/java-for-everything.html http://www.teamten.com/lawrence/writings/java-for-everything... (though actually my single language is Scala)
- mbrock 10y agoEverything in a computer is a stream of bytes... My shell scripts often use tools like jq and jshon to deal with JSON structures, etc. File hierarchies can also be very pleasant data structures. The kinds of scripts I write would be awkward to have as compiled JVM programs, I think. Shell is just way more ergonomic for me for many tasks.
- lmm 10y ago> Everything in a computer is a stream of bytes Data can be meaningfully separated from control and structure in many cases, and failure to do that is a major (perhaps the major) source of security bugs.
- Spiritus 10y agofrom subprocess import Popen, PIPE p1 = Popen(["foo"], stdout=PIPE) p2 = Popen(["bar"], stdin=p1.stdout, stdout=PIPE) p1.stdout.close() # Allow p1 to receive a SIGPIPE if p2 exits. output, _ = p2.communicate() https://docs.python.org/3.5/library/subprocess.html#replacing-shell-pipeline https://docs.python.org/3.5/library/subprocess.html#replacin...
- mbrock 10y agoThanks. Maybe I'll make a public Gist that's a kind of "foo | bar" cookbook for different languages...
- lmm 10y agoWhat you actually do in Python is use https://github.com/kennethreitz/envoy https://github.com/kennethreitz/envoy
- mbrock 10y agoThat seems to recommend just calling the shell to do pipelines? Makes sense... Or maybe it parses that syntax itself?
- lmm 10y agoI believe it parses it itself.
- catchmrbharath 10y agoLooks like its not maintained.
- deleted 10y ago[deleted]
- alexandre_m 10y agoTry python "sh" https://amoffat.github.io/sh https://amoffat.github.io/sh It's pretty elegant. import sh sh.bar(sh.foo())