4 ms·

There are many reasons why one might want to call into the shell from a programming language - but ls, grep and cat are not among them. Python has libraries for

There are many reasons why one might want to call into the shell from a programming language - but ls, grep and cat are not among them. Python has libraries for this already? I can also almost feel the spirit of Rachel Kroll (rachelbythebay) glaring disapprovingly at this, because the last time someone tried the equivalent of `exec('mkdir -p ' + path)` instead of doing it properly, it led to an SEV: https://rachelbythebay.com/w/2021/12/24/mkdir/ https://rachelbythebay.com/w/2021/12/24/mkdir/ Yes, the page has something like "prepared statements" to avoid obvious injection holes, but it might not be obvious to the everyday user how important that is to get right.

deleted 3y ago
[deleted]
Spivak 3y ago
Python definitely doesn't have a grep. The reason most programs have support for calling grep, ripgrep, or ag is because you can't just write or import a searcher that's as fast as these tools. You can find plenty a library that wrap them but none that compete with them with an independent implementation.
- nijave 3y ago
  It seems better/easier if these tools just exposed an API so they could be used as a library directly without sticking exec wrappers in.
  - burntsushi 3y ago
    ripgrep does expose a library. It's just vast, complex and exposes an API that is lower level than what the CLI provides. It's a lot easier to just do `rg --json` and be done with it. ripgrep has on the order of 100 flags. I don't have the bandwidth to maintain both a CLI and a stable library interface for all of that. So while it might seem better to you, there are realities that make it difficult to do given constrained time and resources.
    - nijave 3y ago
      Tbh I'm surprised no one is already maintaining a decent abstraction (or maybe they're just not discoverable). Afaik openssl is the same way and there's some high-level libraries that are common (although I suppose some of that is built into language stdlibs)
      - burntsushi 3y ago
        openssl is likely categorically different. When you need SSL, there isn't too much choice. But when you need to search stuff, especially in a specific case, you often don't need the full functionality of something like ripgrep or even its speed. Often, just iterating over lines and doing a regex search for each is Good Enough. Bottom line is that a naive solution can get you a lot of mileage, where as there really is no naive and simplr solution to SSL. On top of that, ripgrep has a `--json` flag with a pretty simple format. Running another process is often not a Big Deal, and thus, there is another off ramp to avoid the complexity of ripgrep's libraries that doesn't really exist for SSL.
  - yjftsjthsd-h 3y ago
    Sure, if we could have every program built as a library that then had a front end wrapped around it that would be great. However, in practice that's not particularly common.
    - nijave 3y ago
      I mostly meant around ag/ripgrep since I thought these optimized search tools were pretty popular & widely used.
- CableNinja 3y ago
  .. wat? Have you not heard of `re` and its functions, search, match, and sub?
  - edgyquant 3y ago
    That isn’t grep
    - CableNinja 3y ago
      Grep is a regex capable search. re is more powerful than grep. Python supports "grepping" via re.search. Absolutely is grep. If you need a shell tool to search for something in a python script, youre doin it entirely wrong
      - burntsushi 3y ago
        A simple loop over lines using Python's `re` package is unlikely to be as fast as an optimized grep. If your inputs are small, the difference might be negligible. But if they're large, the difference might be enormous. And that's not even accounting for just regex engine throughput on its own. ripgrep for example uses rust/regex. Compare its performance with python/re: https://github.com/BurntSushi/rebar#summary-of-search-time-benchmarks https://github.com/BurntSushi/rebar#summary-of-search-time-b...
jaimebuelta 3y ago
Perhaps you want to interact with more complex command line programs, like kubectl to automate some tasks
- ses1984 3y ago
  Kubectl interacts with the kubernetes api and there is a python client for that.
ur-whale 3y ago
> but ls, grep and cat are not among them. Really? Let me see you write this in native python with the same level of conciseness as the shell can pull off.
- cuu508 3y ago
  from pathlib import Path for path in Path().glob("*test*"): print(path.name)
  - zinodaur 3y ago
    i think your python version of `ls` is missing a few options
    - cuu508 3y ago
      This is a pure-python version of the first example in shshsh's README. The example uses ls without any arguments.
  - ur-whale 3y ago
    So that's ls Two characters instead of 3 lines Where's grep ? And where's the pipe ?
    - d0mine 3y ago
      glob plays grep's role here. No need for a pipe. If you want to run a shell pipeline that is more challenging to port to pure Python, then you could the shell directly: S.run("a | b", shell=True) You could emulate it in Python using the subprocess module but it would be more verbose. It depends on specific use-case whether it is worth the trouble. Calling the shell pipeline from Python gets you the best of both worlds: a shell one liner is used as a concise domain specific language to run commands , and more complex logic, proper error handling, etc is left to Python.

deleted 3y ago

[deleted]

Spivak 3y ago

Python definitely doesn't have a grep. The reason most programs have support for calling grep, ripgrep, or ag is because you can't just write or import a searcher that's as fast as these tools. You can find plenty a library that wrap them but none that compete with them with an independent implementation.

nijave 3y ago
It seems better/easier if these tools just exposed an API so they could be used as a library directly without sticking exec wrappers in.
- burntsushi 3y ago
  ripgrep does expose a library. It's just vast, complex and exposes an API that is lower level than what the CLI provides. It's a lot easier to just do `rg --json` and be done with it. ripgrep has on the order of 100 flags. I don't have the bandwidth to maintain both a CLI and a stable library interface for all of that. So while it might seem better to you, there are realities that make it difficult to do given constrained time and resources.
  - nijave 3y ago
    Tbh I'm surprised no one is already maintaining a decent abstraction (or maybe they're just not discoverable). Afaik openssl is the same way and there's some high-level libraries that are common (although I suppose some of that is built into language stdlibs)
    - burntsushi 3y ago
      openssl is likely categorically different. When you need SSL, there isn't too much choice. But when you need to search stuff, especially in a specific case, you often don't need the full functionality of something like ripgrep or even its speed. Often, just iterating over lines and doing a regex search for each is Good Enough. Bottom line is that a naive solution can get you a lot of mileage, where as there really is no naive and simplr solution to SSL. On top of that, ripgrep has a `--json` flag with a pretty simple format. Running another process is often not a Big Deal, and thus, there is another off ramp to avoid the complexity of ripgrep's libraries that doesn't really exist for SSL.
- yjftsjthsd-h 3y ago
  Sure, if we could have every program built as a library that then had a front end wrapped around it that would be great. However, in practice that's not particularly common.
  - nijave 3y ago
    I mostly meant around ag/ripgrep since I thought these optimized search tools were pretty popular & widely used.
CableNinja 3y ago
.. wat? Have you not heard of `re` and its functions, search, match, and sub?
- edgyquant 3y ago
  That isn’t grep
  - CableNinja 3y ago
    Grep is a regex capable search. re is more powerful than grep. Python supports "grepping" via re.search. Absolutely is grep. If you need a shell tool to search for something in a python script, youre doin it entirely wrong
    - burntsushi 3y ago
      A simple loop over lines using Python's `re` package is unlikely to be as fast as an optimized grep. If your inputs are small, the difference might be negligible. But if they're large, the difference might be enormous. And that's not even accounting for just regex engine throughput on its own. ripgrep for example uses rust/regex. Compare its performance with python/re: https://github.com/BurntSushi/rebar#summary-of-search-time-benchmarks https://github.com/BurntSushi/rebar#summary-of-search-time-b...

jaimebuelta 3y ago

Perhaps you want to interact with more complex command line programs, like kubectl to automate some tasks

ses1984 3y ago
Kubectl interacts with the kubernetes api and there is a python client for that.

ur-whale 3y ago

> but ls, grep and cat are not among them. Really? Let me see you write this in native python with the same level of conciseness as the shell can pull off.

cuu508 3y ago
from pathlib import Path for path in Path().glob("*test*"): print(path.name)
- zinodaur 3y ago
  i think your python version of `ls` is missing a few options
  - cuu508 3y ago
    This is a pure-python version of the first example in shshsh's README. The example uses ls without any arguments.
- ur-whale 3y ago
  So that's ls Two characters instead of 3 lines Where's grep ? And where's the pipe ?
  - d0mine 3y ago
    glob plays grep's role here. No need for a pipe. If you want to run a shell pipeline that is more challenging to port to pure Python, then you could the shell directly: S.run("a | b", shell=True) You could emulate it in Python using the subprocess module but it would be more verbose. It depends on specific use-case whether it is worth the trouble. Calling the shell pipeline from Python gets you the best of both worlds: a shell one liner is used as a concise domain specific language to run commands , and more complex logic, proper error handling, etc is left to Python.