8 ms·
Easily handle CLI operation via Python instead of regular Bash programs
- daenz 5y agoI love the motivation for this tool. This looks very useful for when you just want to make something easier in an existing shell script, without fully migrating to Python. If you do want to migrate fully to Python, I encourage people to use sh[0] (disclaimer: author). One concern that I have is that it could be so useful that now your script is filled with `| pz`, and is launching Python processes at every line . Now you have a tangled hybrid of Bash and Python. 0. https://github.com/amoffat/sh https://github.com/amoffat/sh
- yunruse 5y agoDealing with a lot of `| pz` statements could be pretty quickly solved with eg a `pz` interpreter that can be explicitly given multiple sequences of inputs. That said, it would be amazing if `pz` could take in some input arguments and output the Unixy equivalent if it can find one.
- Klasiaster 5y agoOr pz could be a lightweight wrapper that speaks to a python server or starts a new one if there isn't yet. The server could end after X seconds of not getting new requests.
- LeoPanthera 5y agoThat is a really confusing name, since /bin/sh is the default shell in almost every unix.
- dahart 5y agoThis sh is a python lib, not a unix shell, so can’t be easily confused.
- freedomben 5y agoFor people that know Ruby, if you haven't explored Ruby's CLI abilities, you definitely, definitely should[1][2]. When I was building my (free and open source) awk course[3] I fell in love with awk. When I later found out that Ruby has some of the same features, it changed my life: [1]: https://robm.me.uk/2013/11/ruby-enp/ https://robm.me.uk/2013/11/ruby-enp/ [2]: https://benoithamelin.tumblr.com/ruby1line/ https://benoithamelin.tumblr.com/ruby1line/ [3]: https://github.com/FreedomBen/awk-hack-the-planet https://github.com/FreedomBen/awk-hack-the-planet
- js2 5y agoPerl too. The -p, -n, -e switches work identically. -n causes Perl to assume the following loop around your program, which makes it iterate over filename arguments somewhat like sed -n or awk: LINE: while (<>) { ... # your program goes here } -p causes Perl to assume the following loop around your program, which makes it iterate over filename arguments somewhat like sed: LINE: while (<>) { ... # your program goes here } continue { print or die "-p destination: $!\n"; } https://linux.die.net/man/1/perlrun https://linux.die.net/man/1/perlrun
- p_l 5y agoLittle underappreciated part of Ruby these days is that part of it grew out of Perl, and specifically supports similar kind of scripting for daily tasks. Pretty sure both thsoe specific switches and some of the sigils used in syntax came from this Perl legacy.
- freedomben 5y agoYes thanks, I should have added that perl took it (and improved it) from awk, and ruby took it from perl. Most ruby people don't like to remember that ruby has a lot of perl-isms in it, but I think perl was a great language (and probably is, though I liked ruby so much better I never looked back).
- rcthompson 5y agoMy mnemonic for the perl switches is "perl dash lane", i.e. `perl -lane`. And then I remember to replace "n" with "p" if I want to auto-print every line after munging it. I mainly use this as a replacement for sed/awk/grep because PCRE syntax is more familiar to me than whatever flavor(s) of regexes are used by those tools.
- joseph8th 5y agoAnd then there's `xonsh`, "The unholy union of bash and Python."[0] Which I've been using as my daily driver for a few years, and absolutely swear by it. https://xon.sh https://xon.sh
- seanw444 5y agoWhat in the world... that's certainly interesting.
- c0npr 5y agoagreed, but if I am learning a new syntax anyways I will go for ansible.
- ginko 5y agoCare to explain? Xonsh is a python shell. From a cursory look at the very corporate Ansible website and wikipedia it's "an open-source software provisioning, configuration management, and application-deployment tool enabling infrastructure as code" Whatever that means. They seem to do very different things as far as I can tell.
- joseph8th 5y agoNo new syntax, unless you want to write scripts in xonsh, or mix bash and Python in the same command. There's rarely any reason to. 99/100 times I just write pure bash or pure python
- pedrovhb 5y agoI've been daily driving xonsh for a couple of years too. It _sounds_ like a terrible idea for reasons you can't quite put your finger on, but other than a minor annoyance every now and then, it's been great. I especially like iterating over files as Path objects with the simple syntax for f in p`.\*\.jpg`: if f.isdir(): foo = $(echo @(f.absolute()))
- suifbwish 5y ago
- VectorLock 5y agoI've been using a kind of hacky analog of this for JSON for years, rather than messing around with jq, JMESpath, etc. https://gist.github.com/craigcalef/89799df3f2fc7957d84a1b06ad1af3cf https://gist.github.com/craigcalef/89799df3f2fc7957d84a1b06a...
- jamespwilliams 5y agoThis seems very useful! The behaviour of auto-imports https://github.com/CZ-NIC/pz#auto-import https://github.com/CZ-NIC/pz#auto-import seems bit scary: > Caveat: When accessed first time, the auto-import makes the row reprocessed. It may influence your global variables. Use verbose output to see if something has been auto-imported. It sounds like this could cause some very subtle bugs. Maybe a strong and clear warning when a line is being reprocessed would help.
- hauntsaninja 5y agoI wrote a similar tool a while back. My solution for this was to statically analyse the input to discover unused names. The cool part of doing transformations statically is that you're able to see the exact Python code you'd run without running it, e.g. `pyp --explain x[:5]`. https://github.com/hauntsaninja/pyp https://github.com/hauntsaninja/pyp
- visarga 5y agoNice, I made a similar tool for Perl. I chain them up to three rows in terminal, if they get longer I switch to a script.
- uniformlyrandom 5y agoI don't know if I like this approach. Common "regular" Bash programs have a huge advantage - they are common. When I share my 'may accidentally conjure up a monster' one-liners with my team, I am sure they can read it, understand it, and run it. I lose that advantage with the use of a very nice, but obscure tool.
- ogogmad 5y agoBash involves tons and tons of arcania. It also doesn't support features that PL editors have like Intellisense. I sometimes feel like the only people who don't see a problem with the Unix experience are people who've used it day-in day-out for a decade, and don't need to grep man pages more than they'd like to. But by then, it's Stockholm Syndrome? I think this Github project might begin to pull out the chord. I know my Python better than I know my awk/sed/bash/xargs/find/make/bc...
- suifbwish 5y agoBash has one major advantage though: it’s core libraries and and commands almost never change which mean your scripts are most likely going to work forever. Python core library development is filled with asshats who like to deprecate naming conventions in later versions rather than just improving the existing libraries and keeping the method names the same. Python is great for a lot of reasons but if you know bash well, it is far more reliable in the long term.
- dotancohen 5y agoI'm 44 years old. When I read your comment, I think "Great, the Bash script will continue to work the next time I need it (which might be after Elon lands humans on another planet)". But when my 22 year old colleague reads it, he likely thinks "Great, that script is full of incomprehensible syntax from the era of my last diaper change". Thinking long term is a skill that is often realized only after one fails to make a previous long-term investment.
- fivea 5y ago
- rasengan 5y agohttps://github.com/realrasengan/jpipe https://github.com/realrasengan/jpipe A super simple version for JS
- brtknr 5y agoProbably easier to stick to bash rather than learn a new set of commands from scratch. Also bash commands are universally available whereas this needs to be installed first which is not always that straightforward. Also python is slower at some of the examples than compiled executables.
- otikik 5y ago“Universal” is too much of a stretch. You still get tripped by things like “that flag doesn’t exist in OSX sed” or “that option in git was only available in git > 2.8.14” And that’s how you end up with bash scripts wrapped inside Docker images, that I have been seeing for the last couple of years.
- baq 5y agoSuddenly xonsh from a sibling thread doesn’t sound outrageous anymore
- fivea 5y ago> Probably easier to stick to bash rather than the overhead of learning new set of commands that I need to learn from scratch. That's all fine and dandy until your shell script grows beyond a dozen lines of code or so. The moment that happens, shell scripts become barely readable and terribly hard to maintain, unlike alternatives such as python.
- MobiusHorizons 5y agoI have to say I have not found this to be true. Bash is an excellent glue language if what you want to do is fundamentally a shell like activity (eg composing together other more powerful primitives). I think the key (which is often forgotten) is just to remember to use the abstractions provided by the language (also shell-check, which is the most helpful linter I have ever encountered) If you use functions to aid in abstraction, you can easily convert the function to a program in a more capable language as soon as it becomes too complicated. In the end, I usually find that most of the things people replace good old unix tools with are better in one limited area, but miss a large part of what make the originals magic. Usually this is some combination of universal, discoverable, reduced dependencies, composable abstractions.
- fmakunbound 5y agoI will try this, but I still have ptsd from the last time I installed something python based.
- Tijdreiziger 5y agoRelevant xkcd: https://xkcd.com/1987/ https://xkcd.com/1987/
- Naac 5y agoThis looks like an unmaintained version of (the much more complete) pyp: https://github.com/hauntsaninja/pyp https://github.com/hauntsaninja/pyp
- ur-whale 5y agopyp is very cool, especially the --explain flag Didn't know about it, thanks for this. Oh, and small gotcha, if you want to try it, its: pip3 install pypyp rather than the obvious which installs something else entirely.
- kseistrup 5y agoLooks great. It could be even more useful with an `f` variable that holds the ‘words’ obtained from splitting `s` at whitespace.
- hauntsaninja 5y agoI wrote a similar tool a while back that lets you create your own "magic" variables. I use `f` all the time! https://github.com/hauntsaninja/pyp#pyp-lets-you-configure-your-own-magic https://github.com/hauntsaninja/pyp#pyp-lets-you-configure-y...
- kseistrup 5y agoThere it is! I knew I had seen something akin to pz. Thanks for the link!
- aasasd 5y agoAh, yes, the Perlton.
- gpvos 5y agoThat's just perl -pe or -ne.
- nomorecommas 5y agoVery cool, though I probably won't use it.
- est 5y ago> Example: appending '.com' to every line. It was so much easier in CMD.exe mv * *.com
- pixelbeat__ 5y agoLots of similar tools in the vein. Here's my funcpy tool, which has references to similar tools https://www.pixelbeat.org/scripts/funcpy https://www.pixelbeat.org/scripts/funcpy
- McHankHenry 5y agoI just skip the bash etc. alltogether and write scripts entirely in python, and put the most commonly used stuff in a utility module.
- lenkite 5y agoFor people who know LISP/Clojure, Babashka: https://github.com/babashka/babashka https://github.com/babashka/babashka
- j_not_j 5y agoWhat I would like to see (and haven't found yet), is a bash or python shell that automatically does a "more" when a command exceeds some number of lines. So many times I accidently fill up my scroll buffer with an unintended awk or grep mistake.
- throwaway984393 5y agopz versus Perl: $ echo -e "example\nwikipedia" | pz 's += ".com"' $ echo -e "example\nwikipedia" | perl -lpe '$_.=".com"' $ echo "hello world" | pz s[6:] $ echo "hello world" | perl -pe '$_=substr($_,6)' $ echo "hello world" | perl -pe 'substr($_,0,6)=""' $ echo "hello world" | perl -lape '$_=$F[1]' $ tail -f /var/log/syslog | pz -f '{len(s)}: {s}' $ tail -f /var/log/syslog | perl -lpe '$_=length($_).": $_"' $ echo "HELLO" | pz s.lower $ echo "HELLO" | perl -pe '$_=lc' $ echo "hello,5" | pz 's.split(",")[1]' | pz n+7 $ echo "hello,5" | perl -F, -lpe '$_=$F[1]+7' $ pz --findall "(https?://[^\s]+)" < file.log $ perl -lne 'print $1 if m{(https?://\S+)}' foo2.log $ echo -e "1\n2\n3\n4" | pz --end sum $ echo -e "1\n2\n3\n4" | perl -lne 'END{print $t} $t+=$_' $ echo -e "1\n2\n2\n1\n3" | pz --end "sorted(set(lines))" $ echo -e "1\n2\n2\n1\n3" | perl -ne 'END{print sort keys %h} $h{$_}++' $ echo -e "red green\nblue red green" | pz 'C.update(s.split())' --end C.most_common $ echo -e "red green\nblue red green" | perl -lae 'END{print "$_, $h{$_}" for sort keys %h} $h{$_}++ for @F'
- randoglando 5y agoFor people familiar with python, the first one is immediately understandable, and the second one is incomprehensible. Learning Perl may have been a better choice long-term, but this might be a good stop-gap fix for those who know Python already.
- 35mm 5y agoAs someone who considers themselves ‘not a real programmer’ but uses Python, shell scripts, etc to ‘just get things done’, I really love simple utilities like this. Maybe it’s not the ‘right’ way to get things done, but it’s quick and it works.
- tym0 5y agoBabashka[0] is really lovely to do data transformation with if you're more familiar with Clojure. [0] https://github.com/babashka/babashka https://github.com/babashka/babashka