9 ms·

IPython as a System Shell

141 points by laktak 6y ago

fredley 6y ago
> var = !cmd syntax Is this a standard anywhere outside IPython? I would love to use this sort of thing (python with seamless integration with a system shell) in place of bash scripts. Yes, you can use subprocess but it's a lot less readable.
- tyingq 6y ago
  You might like Perl. The syntax to invoke and interact with processes is much more terse than Python's. And there's quite a lot of history of using it instead of shell scripts. Edit: Things like: $output=`ls`; $output=qx#ls -l#; system('/bin/echo','avoid','subshell','invocation'); open("APIPELINE","ls -l |"); while (<APIPELINE>) { chomp; print "Got line: [$_]\n"; } close(APIPELINE); And access to child exit status in a way that's familiar ($?), etc.
  - dale_glass 6y ago
    That Perl is quite out of date. These days, Perl has 3 argument open, and can use a variable for a filehandle. So: open(my $pipeline, "-|", "ls -l"); This is a good thing, because you can treat a filehandle as a standard variable, and can separate the special stuff from the command itself. This way if you're taking in user input, they don't get to mess with that. You can also do: open(my $pipeline, "-|", "ls", "-l", $directory); Which lets you get away from the mess of correct quoting and just pass every parameter exactly as intended.
- mumblemumble 6y ago
  The bang syntax is specific to ipython. I'm not sure it's really desirable outside of that context. It lacks a lot of process control features you'd want in order to make things robust for general purpose scripting purposes.
- chrisshroba 6y ago
  It's also similar to a vim construct where prefixing a command with ! makes it run it as an external program instead of a vim command.
- fooblat 6y ago
  While I haven't personally used it, another commenter mentioned Xonsh[0] and that is what it appears to be: A very smooth integration of python into a shell that still feels like a unix shell. 0. https://xon.sh/ https://xon.sh/
- notagoodidea 6y ago
  It depends, for other python REPL/Env not really but it is the standard way to call shell command from ed [1] for example. Julia have also a shell mode in its REPL [2]. Personally, I like to use Raku [3] instead of Python or Shell script. It make it easy to drop one-liner, call shell commands [4] and you can super easily make a script a small CLI utilities by writing a MAIN sub [5]. [1] https://www.gnu.org/software/ed/manual/ed_manual.html#Commands https://www.gnu.org/software/ed/manual/ed_manual.html#Comman... [2] https://docs.julialang.org/en/v1/stdlib/REPL/#man-shell-mode https://docs.julialang.org/en/v1/stdlib/REPL/#man-shell-mode [3] https://raku.org/ https://raku.org/ [4] https://docs.raku.org/language/quoting#index-entry-quote_qqx-Shell_quoting_with_interpolation:_qqx https://docs.raku.org/language/quoting#index-entry-quote_qqx... [5] https://docs.raku.org/language/create-cli https://docs.raku.org/language/create-cli
- neolog 6y ago
  That syntax exists in `git alias` too.
- gandvor 6y ago
  You might like invoke: https://www.pyinvoke.org/ https://www.pyinvoke.org/
jjuel 6y ago
Couldn't you just use Xonsh? https://xon.sh/ https://xon.sh/
- musingsole 6y ago
  The question to me is what gaps exist in using IPython directly versus a more dedicated shell like Xonsh? I've inadvertently found myself using the IPython interpreter as my shell more and more. Opening vim in IPython always makes me smile. But I don't try to push it's abilities as a shell too much out of a healthy fear that it'll break unexpectedly in the worst way when I absolutely need it not to.
- salamander014 6y ago
  xonsh is absolutely the right way to go here. There are a lot of 'gotchas' when it comes to building a shell that isn't 'just' a novelty but actually usable. Shells are most successful when you don't need to think in order to use them. Especially since everyone that's used linux in the past 30(?!) years has the same basic foundation for how to use the command line, the best Shell will feel familiar to sh and bash, but better. Math, logic, string and value manipulation, all those things I need to google because I don't remember whether an if statement in bash needs single [ ] or double [[ ]] or '' or "" or ; or spaces or all those weird gotchas. I understand that many people do know how to program in bash effectively. That doesn't mean it's the future. It's like the Perl vs Python thread from the other day. Xonsh is intuitive. Xonsh is well designed. Long live Xonsh.
  - billconan 6y ago
    I'm curious , if I type "ls -la" how does it know if it is "variable ls minus variable la", or "command list with command argument -la"?
    - jonfw 6y ago
      It checks if the variables 'ls' and 'la' exist, and if they do, it'll run it as python. If not, it'll try to parse it as bash. If it can't parse it as bash, it'll try again in python mode. https://xon.sh/tutorial.html#python-mode-vs-subprocess-mode https://xon.sh/tutorial.html#python-mode-vs-subprocess-mode
ducktective 6y ago
- slow startup - no pipe/redirection mechanism - high overhead in version/library control(pyenv, venv)
- spicybright 6y ago
  - no support for one liners because of python syntax - commands too verbose. a line with two lambda expressions would look terrible (All my personal preference, obviously. But I'd rather have any other language than python, even lua)
  - Pxtl 6y ago
    Lua? Really? My first go-to complaint about Lua is exactly that same one - poor support for lambda-style programming for expressive one-liners.
    - spicybright 6y ago
      When I said I'd even take lua over python, that's only if I was forced to pick a high level language for a shell. Lua is really easy to conjure up higher level algos, like javascript can. But I'm not convinced any high level language is a viable replacement for bash. It would have to be more functional than bash whilst still having good support for standard unix tools. One very interesting idea I found was a macro library for racket that turns s-expressions from deeply nested structures into a much flatter representation. Lisp might be a good option with toy with because of flexibility like that. https://www.reddit.com/r/Racket/comments/mands9/fluent_unix_style_pipes_and_lambda_shorthand_to/ https://www.reddit.com/r/Racket/comments/mands9/fluent_unix_...
      - jhgb 6y ago
        So, you want Scsh, then?
  - CraneWorm 6y ago
    > But I'd rather have any other language than python, even lua seriously? lua shell won't even evaluate bare expressions
ruph123 6y ago
I tried this a few years ago when I was younger and more optimistic. But man it was not a great experience. Having said that I think there is still room for modern shells which are easier to script in. For a while I am using fish and thought scripting would be at least as easy and straightforward as using lua. But the syntax is really awkward. For example comparing two variables needs to be put in a math "2 + 2" Urg.
AzzieElbab 6y ago
I've been using amm on and off https://ammonite.io/#Ammonite-Shell https://ammonite.io/#Ammonite-Shell pretty nice if you know scala, still have to use regular shell(s) so I do not forget them
agumonkey 6y ago
can't say i don't like the idea, a certain howardism did similar things in emacs org-mode and live document / os code blend feels very right I can hear the smalltalkers laugh from afar though
canjobear 6y ago
Although I like IPython, I wouldn't want my system shell to be vulnerable to sudden destruction upon updating Python packages.
- stosto88 6y ago
  Yup
stinos 6y ago
Used this a while ago and it's ok-ish, but I've also used Powershell and despite it not being as nice of a language as Python (imo) and also can suck in other ways, what is striking for this particular example is that Powershell doesn't need most of the explanation on this page because the topic is basically 'here is how to do grep/sed on plain text in a list of strings which contains file information and hope you extract the correct field' and Powershell did nail that pretty well by piping objects.
- 1-6 6y ago
  Is there a sh to powershell conversion tool? When I run a local environment in Google Colab, I find myself having to write !commands for MS and Linux filesystems. It'd be convenient to write once.
  - stinos 6y ago
    Not really (for a couple of often-used commands like cd and ls there are aliases in powershell), but I also wouldn't now how that would work. You mean to convert the complete syntax?
fctorial 6y ago
It's too slow (1 second on an nvme machine). Command argument completion doesn't work either.
dsp_person 6y ago
I've enjoyed using pyp[1] with fish to make some nice one-liners For example, custom history command to make early commands at the top, scroll to the bottom, and show the date and command on the same line function hist; history -t -R | pyp '["\t".join(pair) for pair in zip(lines[::2],lines[1::2])]' | less +G; end; funcsave hist [1] https://github.com/hauntsaninja/pyp https://github.com/hauntsaninja/pyp
vinceguidry 6y ago
Anyone interested in a Ruby equivalent may like Pry's shell integrations: https://github.com/pry/pry/wiki/Shell-Integration https://github.com/pry/pry/wiki/Shell-Integration
xiaq 6y ago
If you want a full-fledged programming language in a shell, you might also be interested in Elvish (https://elv.sh https://elv.sh). Start with the quick tour: https://elv.sh/learn/tour.html https://elv.sh/learn/tour.html
nickbauman 6y ago
Babashka has picked up a lot of adherents lately by using a more hybrid approach to this idea. Use clojure to where you want to succinctly and powerfully manipulate data structures (even simple things like (assoc ...) is alien technology in bash) User the shell for everything else.
Rendello 6y ago
TempleOS uses (Holy)C as a system shell (and for everything else). It certainly has its limitations at the command line, but on the flip side it makes writing "shell" scripts a whole lot easier!

141 points by laktak 6y ago

> var = !cmd syntax Is this a standard anywhere outside IPython? I would love to use this sort of thing (python with seamless integration with a system shell) in place of bash scripts. Yes, you can use subprocess but it's a lot less readable.

tyingq 6y ago
You might like Perl. The syntax to invoke and interact with processes is much more terse than Python's. And there's quite a lot of history of using it instead of shell scripts. Edit: Things like: $output=`ls`; $output=qx#ls -l#; system('/bin/echo','avoid','subshell','invocation'); open("APIPELINE","ls -l |"); while (<APIPELINE>) { chomp; print "Got line: [$_]\n"; } close(APIPELINE); And access to child exit status in a way that's familiar ($?), etc.
- dale_glass 6y ago
  That Perl is quite out of date. These days, Perl has 3 argument open, and can use a variable for a filehandle. So: open(my $pipeline, "-|", "ls -l"); This is a good thing, because you can treat a filehandle as a standard variable, and can separate the special stuff from the command itself. This way if you're taking in user input, they don't get to mess with that. You can also do: open(my $pipeline, "-|", "ls", "-l", $directory); Which lets you get away from the mess of correct quoting and just pass every parameter exactly as intended.
mumblemumble 6y ago
The bang syntax is specific to ipython. I'm not sure it's really desirable outside of that context. It lacks a lot of process control features you'd want in order to make things robust for general purpose scripting purposes.
chrisshroba 6y ago
It's also similar to a vim construct where prefixing a command with ! makes it run it as an external program instead of a vim command.
fooblat 6y ago
While I haven't personally used it, another commenter mentioned Xonsh[0] and that is what it appears to be: A very smooth integration of python into a shell that still feels like a unix shell. 0. https://xon.sh/ https://xon.sh/
notagoodidea 6y ago
It depends, for other python REPL/Env not really but it is the standard way to call shell command from ed [1] for example. Julia have also a shell mode in its REPL [2]. Personally, I like to use Raku [3] instead of Python or Shell script. It make it easy to drop one-liner, call shell commands [4] and you can super easily make a script a small CLI utilities by writing a MAIN sub [5]. [1] https://www.gnu.org/software/ed/manual/ed_manual.html#Commands https://www.gnu.org/software/ed/manual/ed_manual.html#Comman... [2] https://docs.julialang.org/en/v1/stdlib/REPL/#man-shell-mode https://docs.julialang.org/en/v1/stdlib/REPL/#man-shell-mode [3] https://raku.org/ https://raku.org/ [4] https://docs.raku.org/language/quoting#index-entry-quote_qqx-Shell_quoting_with_interpolation:_qqx https://docs.raku.org/language/quoting#index-entry-quote_qqx... [5] https://docs.raku.org/language/create-cli https://docs.raku.org/language/create-cli
neolog 6y ago
That syntax exists in `git alias` too.
gandvor 6y ago
You might like invoke: https://www.pyinvoke.org/ https://www.pyinvoke.org/

jjuel 6y ago

Couldn't you just use Xonsh? https://xon.sh/ https://xon.sh/

musingsole 6y ago
The question to me is what gaps exist in using IPython directly versus a more dedicated shell like Xonsh? I've inadvertently found myself using the IPython interpreter as my shell more and more. Opening vim in IPython always makes me smile. But I don't try to push it's abilities as a shell too much out of a healthy fear that it'll break unexpectedly in the worst way when I absolutely need it not to.
salamander014 6y ago
xonsh is absolutely the right way to go here. There are a lot of 'gotchas' when it comes to building a shell that isn't 'just' a novelty but actually usable. Shells are most successful when you don't need to think in order to use them. Especially since everyone that's used linux in the past 30(?!) years has the same basic foundation for how to use the command line, the best Shell will feel familiar to sh and bash, but better. Math, logic, string and value manipulation, all those things I need to google because I don't remember whether an if statement in bash needs single [ ] or double [[ ]] or '' or "" or ; or spaces or all those weird gotchas. I understand that many people do know how to program in bash effectively. That doesn't mean it's the future. It's like the Perl vs Python thread from the other day. Xonsh is intuitive. Xonsh is well designed. Long live Xonsh.
- billconan 6y ago
  I'm curious , if I type "ls -la" how does it know if it is "variable ls minus variable la", or "command list with command argument -la"?
  - jonfw 6y ago
    It checks if the variables 'ls' and 'la' exist, and if they do, it'll run it as python. If not, it'll try to parse it as bash. If it can't parse it as bash, it'll try again in python mode. https://xon.sh/tutorial.html#python-mode-vs-subprocess-mode https://xon.sh/tutorial.html#python-mode-vs-subprocess-mode

ducktective 6y ago

- slow startup - no pipe/redirection mechanism - high overhead in version/library control(pyenv, venv)

spicybright 6y ago
- no support for one liners because of python syntax - commands too verbose. a line with two lambda expressions would look terrible (All my personal preference, obviously. But I'd rather have any other language than python, even lua)
- Pxtl 6y ago
  Lua? Really? My first go-to complaint about Lua is exactly that same one - poor support for lambda-style programming for expressive one-liners.
  - spicybright 6y ago
    When I said I'd even take lua over python, that's only if I was forced to pick a high level language for a shell. Lua is really easy to conjure up higher level algos, like javascript can. But I'm not convinced any high level language is a viable replacement for bash. It would have to be more functional than bash whilst still having good support for standard unix tools. One very interesting idea I found was a macro library for racket that turns s-expressions from deeply nested structures into a much flatter representation. Lisp might be a good option with toy with because of flexibility like that. https://www.reddit.com/r/Racket/comments/mands9/fluent_unix_style_pipes_and_lambda_shorthand_to/ https://www.reddit.com/r/Racket/comments/mands9/fluent_unix_...
    - jhgb 6y ago
      So, you want Scsh, then?
- CraneWorm 6y ago
  > But I'd rather have any other language than python, even lua seriously? lua shell won't even evaluate bare expressions

ruph123 6y ago

I tried this a few years ago when I was younger and more optimistic. But man it was not a great experience. Having said that I think there is still room for modern shells which are easier to script in. For a while I am using fish and thought scripting would be at least as easy and straightforward as using lua. But the syntax is really awkward. For example comparing two variables needs to be put in a math "2 + 2" Urg.

AzzieElbab 6y ago

I've been using amm on and off https://ammonite.io/#Ammonite-Shell https://ammonite.io/#Ammonite-Shell pretty nice if you know scala, still have to use regular shell(s) so I do not forget them

agumonkey 6y ago

can't say i don't like the idea, a certain howardism did similar things in emacs org-mode and live document / os code blend feels very right I can hear the smalltalkers laugh from afar though

canjobear 6y ago

Although I like IPython, I wouldn't want my system shell to be vulnerable to sudden destruction upon updating Python packages.

stosto88 6y ago
Yup

stinos 6y ago

Used this a while ago and it's ok-ish, but I've also used Powershell and despite it not being as nice of a language as Python (imo) and also can suck in other ways, what is striking for this particular example is that Powershell doesn't need most of the explanation on this page because the topic is basically 'here is how to do grep/sed on plain text in a list of strings which contains file information and hope you extract the correct field' and Powershell did nail that pretty well by piping objects.

1-6 6y ago
Is there a sh to powershell conversion tool? When I run a local environment in Google Colab, I find myself having to write !commands for MS and Linux filesystems. It'd be convenient to write once.
- stinos 6y ago
  Not really (for a couple of often-used commands like cd and ls there are aliases in powershell), but I also wouldn't now how that would work. You mean to convert the complete syntax?

fctorial 6y ago

It's too slow (1 second on an nvme machine). Command argument completion doesn't work either.

dsp_person 6y ago

I've enjoyed using pyp[1] with fish to make some nice one-liners For example, custom history command to make early commands at the top, scroll to the bottom, and show the date and command on the same line function hist; history -t -R | pyp '["\t".join(pair) for pair in zip(lines[::2],lines[1::2])]' | less +G; end; funcsave hist [1] https://github.com/hauntsaninja/pyp https://github.com/hauntsaninja/pyp

vinceguidry 6y ago

Anyone interested in a Ruby equivalent may like Pry's shell integrations: https://github.com/pry/pry/wiki/Shell-Integration https://github.com/pry/pry/wiki/Shell-Integration

xiaq 6y ago

If you want a full-fledged programming language in a shell, you might also be interested in Elvish (https://elv.sh https://elv.sh). Start with the quick tour: https://elv.sh/learn/tour.html https://elv.sh/learn/tour.html

nickbauman 6y ago

Babashka has picked up a lot of adherents lately by using a more hybrid approach to this idea. Use clojure to where you want to succinctly and powerfully manipulate data structures (even simple things like (assoc ...) is alien technology in bash) User the shell for everything else.

Rendello 6y ago

TempleOS uses (Holy)C as a system shell (and for everything else). It certainly has its limitations at the command line, but on the flip side it makes writing "shell" scripts a whole lot easier!