8 ms·

If you have your shell ready, curl -sS 'https://www.google.com/complete/search?client=hp&hl=en&xhr=t&q=aurora' \ | json_pp | sed -nE '/<\/?b>/{s```

by jxy 10y ago

If you have your shell ready, curl -sS 'https://www.google.com/complete/search?client=hp&hl=en&xhr=t&q=aurora' \ | json_pp | sed -nE '/<\/?b>/{s```g;s`"|,|^ *``g;p}'

devoply 10y ago
I have learned the hard way never to type stuff like this into a shell because I might not like the results. It looks fine, but never again. ;)
- curiousgal 10y ago
  What happened?
  - devoply 10y ago
    This was like 13 years ago, but it in essence rm -rvf / ... but it was a obfuscated one liner that some dick put out there as a joke. Result was having to restore a site and lose a few weeks of posts.
    - 0x0dea 10y ago
      $ echo landline | tr denial -/\ fmr
  - verandaguy 10y ago
    It's generally an easy way to introduce malicious shell scripts into remote machines operated by people who don't know about this. There's a blog dedicated to shaming sites that do this: https://curlpipesh.tumblr.com/#_=_ https://curlpipesh.tumblr.com/#_=_
    - __s 10y ago
      But they aren't piping to sh..
      - cyphar 10y ago
        Yeah, but if there's a code execution vulnerability in any of the tools in the pipeline it's as good as piping to sh.
        deleted 10y ago
        [deleted]
        duaneb 10y ago
        Yup! Using curl at all is downright dangerous. /s
        cyphar 10y ago
        Well, it's all about risk assessment. Curl has had RCE security bugs before[1]. That doesn't mean curl is "downright dangerous" it just means that "it's safe because it doesn't pipe to sh" is also not a correct thing to say. [1]: http://blog.volema.com/curl-rce.html http://blog.volema.com/curl-rce.html
        duaneb 10y ago
        Piping to jq and sed is just fine. I'm not sure where sh comes into this...
        cyphar 10y ago
        Unless there's a bug in curl, jq or sed that leads to an RCE, which is what my point was.
        verandaguy 10y ago
        Incidentally, cURL just had a major security audit. The dev is doing due diligence to avoid RCE vulnerabilities.
      - TeMPOraL 10y ago
        I suppose 'devoply pattern-matched "curl -sS <sth> | <sth else>" to "don't do that". I think that because that's exactly what I did now; it took me a few seconds to realize that - this time - the invocation is mostly harmless.
    - niij 10y ago
      Have you ever downloaded and installed a program through a web browser? Is there a practical difference between that and piping to sh, besides a GUI?
      - OskarS 10y ago
        If you install through a GUI or a package manager, it will often check a cryptographic signature to validate that you downloaded the genuine thing, and you weren't man-in-the-middle'd with some malware. Also, it will check that nothing was corrupted in transit. But generally speaking, you're right. You're taking a risk every time you run executable code from the internet, and if you don't want to live in a paranoid state of fear constantly, you have to balance security and convenience. If you check that it says "https" in the beginning and that you trust the source, then you're probably fine.
jsrn 10y ago
Nice! To run this with macOS, I had to use the GNU version of sed. I installed it with $ brew install gnu-sed And it is then called with 'gsed' instead of 'sed'. As an avid Perl programmer, I had json_pp in my $PATH - for everyone else - it is here: https://metacpan.org/pod/JSON::PP https://metacpan.org/pod/JSON::PP You can install it with cpanm: $ cpanm JSON::PP If you don't have cpanm, you can install it with $ curl -L https://cpanmin.us | perl - App::cpanminus The modified command line from above then becomes: $ curl -sS 'https://www.google.com/complete/search?client=hp&hl=en&xhr=t&q=aurora' \ | json_pp | gsed -nE '/<\/?b>/{s```g;s`"|,|^ *``g;p}' Here is a little Bash function to encapsulate this: $ function c() { curl -sS "https://www.google.com/complete/search?client=hp&hl=en&xhr=t&q=$1" | json_pp | gsed -nE '/<\/?b>/{s```g;s`"|,|^ *``g;p}'; } Which then allows you to use it like this: $ c hacker hacker news hacker typer hackerrank hackerman hackers movie hacker fare hackerone hackers cast hacker pschorr For spaces in your query, use a '+': $ c New+York new york times new york and company new york giants new york post new york daily news new york weather
- eridius 10y ago
  You can use BSD sed by just inserting a semicolon into the sed pattern: sed -nE '/<\/?b>/{s```g;s`"|,|^ *``g;p;}' (Edited to remove previous advice about inserting a newline)
- ekns 10y ago
  Improved version using jq. This properly URL encodes the query parameter and uses a much simpler sed command. Additionally the API returns UTF-8 encoded data when the user agent is specified. Requires curl >= 7.18.0. function c() { url='https://www.google.com/complete/search?client=hp&hl=en&xhr=t' # NB: user-agent must be specified to get back UTF-8 data! curl -H 'user-agent: Mozilla/5.0' -sSG --data-urlencode "q=$*" "$url" | jq -r .[1][][0] | sed 's,</\?b>,,g' } Example: $ c ':)' ':(' ) ( meaning ) ( emoticon ) ( ͡° ͜ʖ ͡°) ) ( emoticon meaning
kevinastone 10y ago
Would think jq[0] is more suitable if you're going to install a cli json parser. [0]: https://stedolan.github.io/jq/ https://stedolan.github.io/jq/
- jsrn 10y ago
  Here is the command with jq instead of json_pp $ curl -sS 'https://www.google.com/complete/search?client=hp&hl=en&xhr=t&q=aurora' | jq . | gsed -nE '/<\/?b>/{s```g;s`"|,|^ *``g;p}' (see my other comment about gsed vs sed)
  - eridius 10y ago
    Here's the above modified to work with BSD sed: curl -sS 'https://www.google.com/complete/search?client=hp&hl=en&xhr=t&q=aurora' | jq . | sed -nE '/<\/?b>/{s```g;s`"|,|^ *``g;p;}' (Edited to remove previous advice about inserting a newline)
  - kevinastone 10y ago
    You don't need sed, you can do it all in jq: curl -sS 'https://www.google.com/complete/search?client=hp&hl=en&xhr=t&q=aurora' | jq -r '.[1] | .[][0] | sub("</?b>"; ""; "gi")'

by jxy 10y ago

If you have your shell ready, curl -sS 'https://www.google.com/complete/search?client=hp&hl=en&xhr=t&q=aurora' \ | json_pp | sed -nE '/<\/?b>/{s```g;s`"|,|^ *``g;p}'

devoply 10y ago

I have learned the hard way never to type stuff like this into a shell because I might not like the results. It looks fine, but never again. ;)

curiousgal 10y ago
What happened?
- devoply 10y ago
  This was like 13 years ago, but it in essence rm -rvf / ... but it was a obfuscated one liner that some dick put out there as a joke. Result was having to restore a site and lose a few weeks of posts.
  - 0x0dea 10y ago
    $ echo landline | tr denial -/\ fmr
- verandaguy 10y ago
  It's generally an easy way to introduce malicious shell scripts into remote machines operated by people who don't know about this. There's a blog dedicated to shaming sites that do this: https://curlpipesh.tumblr.com/#_=_ https://curlpipesh.tumblr.com/#_=_
  - __s 10y ago
    But they aren't piping to sh..
    - cyphar 10y ago
      Yeah, but if there's a code execution vulnerability in any of the tools in the pipeline it's as good as piping to sh.
      - deleted 10y ago
        [deleted]
      - duaneb 10y ago
        Yup! Using curl at all is downright dangerous. /s
        cyphar 10y ago
        Well, it's all about risk assessment. Curl has had RCE security bugs before[1]. That doesn't mean curl is "downright dangerous" it just means that "it's safe because it doesn't pipe to sh" is also not a correct thing to say. [1]: http://blog.volema.com/curl-rce.html http://blog.volema.com/curl-rce.html
        duaneb 10y ago
        Piping to jq and sed is just fine. I'm not sure where sh comes into this...
        cyphar 10y ago
        Unless there's a bug in curl, jq or sed that leads to an RCE, which is what my point was.
        verandaguy 10y ago
        Incidentally, cURL just had a major security audit. The dev is doing due diligence to avoid RCE vulnerabilities.
    - TeMPOraL 10y ago
      I suppose 'devoply pattern-matched "curl -sS <sth> | <sth else>" to "don't do that". I think that because that's exactly what I did now; it took me a few seconds to realize that - this time - the invocation is mostly harmless.
  - niij 10y ago
    Have you ever downloaded and installed a program through a web browser? Is there a practical difference between that and piping to sh, besides a GUI?
    - OskarS 10y ago
      If you install through a GUI or a package manager, it will often check a cryptographic signature to validate that you downloaded the genuine thing, and you weren't man-in-the-middle'd with some malware. Also, it will check that nothing was corrupted in transit. But generally speaking, you're right. You're taking a risk every time you run executable code from the internet, and if you don't want to live in a paranoid state of fear constantly, you have to balance security and convenience. If you check that it says "https" in the beginning and that you trust the source, then you're probably fine.

jsrn 10y ago

Nice! To run this with macOS, I had to use the GNU version of sed. I installed it with $ brew install gnu-sed And it is then called with 'gsed' instead of 'sed'. As an avid Perl programmer, I had json_pp in my $PATH - for everyone else - it is here: https://metacpan.org/pod/JSON::PP https://metacpan.org/pod/JSON::PP You can install it with cpanm: $ cpanm JSON::PP If you don't have cpanm, you can install it with $ curl -L https://cpanmin.us | perl - App::cpanminus The modified command line from above then becomes: $ curl -sS 'https://www.google.com/complete/search?client=hp&hl=en&xhr=t&q=aurora' \ | json_pp | gsed -nE '/<\/?b>/{s```g;s`"|,|^ *``g;p}' Here is a little Bash function to encapsulate this: $ function c() { curl -sS "https://www.google.com/complete/search?client=hp&hl=en&xhr=t&q=$1" | json_pp | gsed -nE '/<\/?b>/{s```g;s`"|,|^ *``g;p}'; } Which then allows you to use it like this: $ c hacker hacker news hacker typer hackerrank hackerman hackers movie hacker fare hackerone hackers cast hacker pschorr For spaces in your query, use a '+': $ c New+York new york times new york and company new york giants new york post new york daily news new york weather

eridius 10y ago
You can use BSD sed by just inserting a semicolon into the sed pattern: sed -nE '/<\/?b>/{s```g;s`"|,|^ *``g;p;}' (Edited to remove previous advice about inserting a newline)
ekns 10y ago
Improved version using jq. This properly URL encodes the query parameter and uses a much simpler sed command. Additionally the API returns UTF-8 encoded data when the user agent is specified. Requires curl >= 7.18.0. function c() { url='https://www.google.com/complete/search?client=hp&hl=en&xhr=t' # NB: user-agent must be specified to get back UTF-8 data! curl -H 'user-agent: Mozilla/5.0' -sSG --data-urlencode "q=$*" "$url" | jq -r .[1][][0] | sed 's,</\?b>,,g' } Example: $ c ':)' ':(' ) ( meaning ) ( emoticon ) ( ͡° ͜ʖ ͡°) ) ( emoticon meaning

kevinastone 10y ago

Would think jq[0] is more suitable if you're going to install a cli json parser. [0]: https://stedolan.github.io/jq/ https://stedolan.github.io/jq/

jsrn 10y ago
Here is the command with jq instead of json_pp $ curl -sS 'https://www.google.com/complete/search?client=hp&hl=en&xhr=t&q=aurora' | jq . | gsed -nE '/<\/?b>/{s```g;s`"|,|^ *``g;p}' (see my other comment about gsed vs sed)
- eridius 10y ago
  Here's the above modified to work with BSD sed: curl -sS 'https://www.google.com/complete/search?client=hp&hl=en&xhr=t&q=aurora' | jq . | sed -nE '/<\/?b>/{s```g;s`"|,|^ *``g;p;}' (Edited to remove previous advice about inserting a newline)
- kevinastone 10y ago
  You don't need sed, you can do it all in jq: curl -sS 'https://www.google.com/complete/search?client=hp&hl=en&xhr=t&q=aurora' | jq -r '.[1] | .[][0] | sub("</?b>"; ""; "gi")'