6 ms·
Shshsh is a bridge connects Python and shell
- dhon_ 3y agoWhat are the similarities/differences to plumbum? https://plumbum.readthedocs.io/en/latest/ https://plumbum.readthedocs.io/en/latest/
- red_admiral 3y agoThere are many reasons why one might want to call into the shell from a programming language - but ls, grep and cat are not among them. Python has libraries for this already? I can also almost feel the spirit of Rachel Kroll (rachelbythebay) glaring disapprovingly at this, because the last time someone tried the equivalent of `exec('mkdir -p ' + path)` instead of doing it properly, it led to an SEV: https://rachelbythebay.com/w/2021/12/24/mkdir/ https://rachelbythebay.com/w/2021/12/24/mkdir/ Yes, the page has something like "prepared statements" to avoid obvious injection holes, but it might not be obvious to the everyday user how important that is to get right.
- deleted 3y ago[deleted]
- Spivak 3y agoPython definitely doesn't have a grep. The reason most programs have support for calling grep, ripgrep, or ag is because you can't just write or import a searcher that's as fast as these tools. You can find plenty a library that wrap them but none that compete with them with an independent implementation.
- nijave 3y agoIt seems better/easier if these tools just exposed an API so they could be used as a library directly without sticking exec wrappers in.
- burntsushi 3y agoripgrep does expose a library. It's just vast, complex and exposes an API that is lower level than what the CLI provides. It's a lot easier to just do `rg --json` and be done with it. ripgrep has on the order of 100 flags. I don't have the bandwidth to maintain both a CLI and a stable library interface for all of that. So while it might seem better to you, there are realities that make it difficult to do given constrained time and resources.
- nijave 3y agoTbh I'm surprised no one is already maintaining a decent abstraction (or maybe they're just not discoverable). Afaik openssl is the same way and there's some high-level libraries that are common (although I suppose some of that is built into language stdlibs)
- burntsushi 3y agoopenssl is likely categorically different. When you need SSL, there isn't too much choice. But when you need to search stuff, especially in a specific case, you often don't need the full functionality of something like ripgrep or even its speed. Often, just iterating over lines and doing a regex search for each is Good Enough. Bottom line is that a naive solution can get you a lot of mileage, where as there really is no naive and simplr solution to SSL. On top of that, ripgrep has a `--json` flag with a pretty simple format. Running another process is often not a Big Deal, and thus, there is another off ramp to avoid the complexity of ripgrep's libraries that doesn't really exist for SSL.
- yjftsjthsd-h 3y agoSure, if we could have every program built as a library that then had a front end wrapped around it that would be great. However, in practice that's not particularly common.
- nijave 3y agoI mostly meant around ag/ripgrep since I thought these optimized search tools were pretty popular & widely used.
- CableNinja 3y ago.. wat? Have you not heard of `re` and its functions, search, match, and sub?
- edgyquant 3y agoThat isn’t grep
- CableNinja 3y agoGrep is a regex capable search. re is more powerful than grep. Python supports "grepping" via re.search. Absolutely is grep. If you need a shell tool to search for something in a python script, youre doin it entirely wrong
- burntsushi 3y agoA simple loop over lines using Python's `re` package is unlikely to be as fast as an optimized grep. If your inputs are small, the difference might be negligible. But if they're large, the difference might be enormous. And that's not even accounting for just regex engine throughput on its own. ripgrep for example uses rust/regex. Compare its performance with python/re: https://github.com/BurntSushi/rebar#summary-of-search-time-benchmarks https://github.com/BurntSushi/rebar#summary-of-search-time-b...
- jaimebuelta 3y agoPerhaps you want to interact with more complex command line programs, like kubectl to automate some tasks
- ses1984 3y agoKubectl interacts with the kubernetes api and there is a python client for that.
- ur-whale 3y ago> but ls, grep and cat are not among them. Really? Let me see you write this in native python with the same level of conciseness as the shell can pull off.
- cuu508 3y agofrom pathlib import Path for path in Path().glob("*test*"): print(path.name)
- zinodaur 3y agoi think your python version of `ls` is missing a few options
- cuu508 3y agoThis is a pure-python version of the first example in shshsh's README. The example uses ls without any arguments.
- ur-whale 3y agoSo that's ls Two characters instead of 3 lines Where's grep ? And where's the pipe ?
- d0mine 3y agoglob plays grep's role here. No need for a pipe. If you want to run a shell pipeline that is more challenging to port to pure Python, then you could the shell directly: S.run("a | b", shell=True) You could emulate it in Python using the subprocess module but it would be more verbose. It depends on specific use-case whether it is worth the trouble. Calling the shell pipeline from Python gets you the best of both worlds: a shell one liner is used as a concise domain specific language to run commands , and more complex logic, proper error handling, etc is left to Python.
- carreau 3y agosee also https://xon.sh/ https://xon.sh/
- jaimebuelta 3y agoCheck also the sh module https://sh.readthedocs.io/en/latest/ https://sh.readthedocs.io/en/latest/ It is brilliant to do quick interactions with existing bash commands or “pseudo-bash scripts” in Python
- ur-whale 3y agoVery neat. Can't count the number of times I've needed to go back and forth between shell and python and the built-in Python apis were found sorely lacking.
- geophile 3y agoI wrote a shell, marcel, that pipes Python values instead of strings: https://marceltheshell.org https://marceltheshell.org. It also does the inverse, allowing you to run marcel commands from Python, e.g. https://www.marceltheshell.org/scripting-1 https://www.marceltheshell.org/scripting-1
- fernmyth 3y agoI love it! I’ll have to play around with it today
- jawns 3y agoWhat's up with the overloaded operators? `I >>` and `>= keep` might be terse, but they're hard to reason about unless you're already familiar with the library. This probably isn't the kind of stuff you want to employ code-golf syntax for.
- frou_dh 3y agoI have bookmarked/tried so many Python/Shell mashups over the years. IMHO the following is about the only one that's tasteful in terms of project scope and not going off the deep end: https://github.com/hauntsaninja/pyp https://github.com/hauntsaninja/pyp
- zinodaur 3y agoI was really in to this stuff as well - eventually I just settled for awk, it works pretty well
- geophile 3y agoI'd be curious on your thoughts on my entry: https://marceltheshell.org https://marceltheshell.org. (This website has lots of examples, so you can get a sense of how it works without installing.)
- frou_dh 3y agoPersonally I wouldn't use a full-on replacement interactive shell because I have a stacked .bashrc that I've constructed over a long time, and things I install from my package manager all come with bash_completion support etc, and even scripts written in Python can vend bash completions (https://pypi.org/project/argcomplete/ https://pypi.org/project/argcomplete/). Also shellcheck's static analysis of scripts is very educational and that feeds back into interactive use. I like that you support the scripting use case with vanilla Python syntax. That's something that made Xonsh a non-starter for me (I'm not going to write scripts when syntax highlighting and auto-formatting don't work because it's a superset of valid Python syntax). Evidently my definition of "going off the deep end" here is giving up lots of working tooling and integrations.
- geophile 3y agoThanks. Lately I've been finding it useful to write bash scripts with some marcel, e.g. #!/bin/bash echo ... pushd ... marcel <<EOF ... EOF
- 3y ago
- benatkin 3y agoI've been using ! in ipython a lot. Maybe I should switch to fish though. I haven't figured out how to get the history database in zsh to work the way I want. It is often convenient to save the output of ! and run Python on it, though.
- stainablesteel 3y agoahh this is cool unlike xonsh that can't be ran as a python script and must be ran through its own interpreter, this actually brings the shell into a python script which looks nifty! it makes for an easier time collecting, within the python ecosystem, output from a function call that exists outside the python ecosystem, i would highlight that as the main difference rather than just being a "bridge" to the shell, nice work
- nrclark 3y agoI've always liked Python's "batteries-included" approach, and my take on libraries like this is that they introduce extra dependencies into scripts I write. And often, the juice doesn't feel like it's worth the squeeze. For me, a little bit of syntax sugar (and maybe a line or two of code savings) isn't worth taking on the dependency burden, especially in production code. Each and every dependency is one where I invite risk of requirements collision, or that the upstream goes away, or that the maintainer breaks compatibility, etc. It's more time spent installing dependencies, and more time spent processing lockfiles. Why carry that burden when subprocess.Popen() exists and works fine?
- sneed_chucker 3y agoI agree with you for the most part, I really like writing shebangable dependency-free python scripts. But I 100% understand the motivation for this project. Like you said, subprocess.Popen works well enough, but it's so much less ergonomic than shelling out in Ruby, Perl, or even Awk.
- grrandalf 3y agoGreat points. Agreed. In certain situations though, the readability/maintainability gains from using this library might outweigh the added complexity of a layer of indirection. Situations like this: whether to build a utility solution completely [1] in shell, or invert things and use mix of Python and shell. E.g., wrapping 'git' to create a note-taking CLI, or shelling out to $WEIRD_INTERNAL_TOOL and scraping its logs and stdout/stderr. I think infrastructure like this repo can result in more readable and maintainable code than an all-shell OR all-Python solution in such situations. You'll be less tempted to start doing things completely in shell and then get locked into that lock minimum. However, you MUST be prepared to understand the source code of all dependencies you take [except the language's standard library, or similarly mature external libs]. e.g., I pick bottlepy.org vs Flask for personal projects. With a small/medium library like this, I might import it to get a headstart and will be ok with it ultimately ending up as an incompatible, heavily-modified fork of the upstream project. Which IMO doesn't reflect poorly on the user or the library author. [1] "completely" means some non-trivial processing that makes you use hash tables, arrays, lightweight tokenizing using read, arithmetic evaluation, sed+awk to massage command outputs so Bash's lightweight data structures can handle them, os.path manipulations.
- shhsshs 3y agoI was surprised to see what looked like my username on the front page… not quite!
- d0mine 3y agoIf you need ssh, there is fabric library https://docs.fabfile.org/en/latest/getting-started.html#run-commands-via-connections-and-run https://docs.fabfile.org/en/latest/getting-started.html#run-...
- michalc 3y agoMy own attempt at bridging the Python and, well, maybe not quite shell but more subprocess, boundary: https://github.com/uktrade/iterable-subprocess https://github.com/uktrade/iterable-subprocess.