3 ms·

My humble submission to this genre is this pipeline I wrote once, when I needed an HTTP proxy, but I couldn't use install a real one do to work limitations, and

My humble submission to this genre is this pipeline I wrote once, when I needed an HTTP proxy, but I couldn't use install a real one do to work limitations, and I couldn't use firewall rules because of weird limitations of the Mac OS at the time: echo "" > /tmp/buffer; tail -f /tmp/buffer | netcat httpbin.org 80 | netcat -lp 8080 > /tmp/buffer I call it a circular pipeline.

dundarious 5y ago
I know this is a toy meant for short-term use, but you can use a fifo/named pipe so you don't have a growing file used for data transfer. mkfifo -m0600 /tmp/buffer && netcat httpbin.org 80 < /tmp/buffer | netcat -lp 8080 > /tmp/buffer Might want the file if you want a record of course.
- maxbond 5y ago
  Thanks! TIL! That's a much cleaner way of doing it. Regarding keeping the file, if memory serves the "real" pipeline had some `tee`s in it. I recreated this much more recently to make a GitHub repo of funny code snippets (https://github.com/MaxBondABE/one-thread-crashing/ https://github.com/MaxBondABE/one-thread-crashing/). You reminded me of a different toy I made many years ago. I created FIFOs for files that you might try to read when exploiting an arbitrary file read in a web app, say /etc/www/apache.conf (if that's the right path - it's been a while since I've configured a web server!) and I'd have a program that opened them. This would block until someone opened the other end of the FIFO, at which point I'd raise an alarm (which was just a print statement).
thyrox 5y ago
That's awesome but would you mind explaining this a little bit more to those not very unix savvy? I think I get the first three parts: 1) Create a temp buffer file 2) Read the file constantly to STDOUT? 3) What does netcat httpbin.org 80 | netcat -lp 8080 do? What would be a good use of this circular pipeline? Can you give me some example of what you use this for? Thanks
- dundarious 5y ago
  I'm not OP, but as they said, it's an HTTP proxy. Run it, then point your browser at http://localhost:8080 http://localhost:8080. The message is proxied over to http://httpbin.org http://httpbin.org (on port 80, the normal HTTP port). If you're in an overly restricted environment and have a legit need to proxy to a very limited set of hosts, you can ad-hoc make a proxy mapping from local port to a single specific remote host. Just change the 8080, destination host, and buffer file. And I mean this as encouragement, but you should read the synopsis from the GNU netcat manual. It's short and explains it well enough. netcat httpbin.org 80 is using connect mode, the -l one is using listen mode, and -p is --local-port (the port to listen on in listen mode).
- maxbond 5y ago
  Of course! A.) Some context: - Our goal here is to match up to two network connections, so that anything we receive on one is forwarded to the other. - netcat is a "networking swiss army knife", it's a tool for making network connections - HTTP and bash pipelines both happen to be "line oriented", meaning they operate one line at a time. Exploiting this happy coincedence is what makes this work - httpbin.org is a metanym for any website you need. So: echo "" > /tmp/buffer Initialize our buffer to an empty state tail -f /tmp/buffer Read each line in the buffer, one at a time, and feed them to the next command. Do this forever (-f). netcat httpbin.org 80 Connect to the website httpbin.org. Transmit each line from our buffer to the website. netcat -lp 8080 > /tmp/buffer Accept a connection. Transmit any data we receive from httpbin.org to this client; write any data we receive from the client to the buffer, so that it can be transmitted to httpbin.org. B.) I don't think there is one! I've never needed to do this again in the, gosh, seven intervening years. Normally bash pipelines operate like a bucket brigade, moving data unidirectionally through various processing steps, and usually dumping it into a file at the end. They aren't normally for creating long-lived programs that manage bidirectional flows of data. Which is why writing a web server in bash is a fun challenge.
  - thyrox 5y ago
    Thank you for the detailed explanation. Learned a thing or two too! :)

dundarious 5y ago

I know this is a toy meant for short-term use, but you can use a fifo/named pipe so you don't have a growing file used for data transfer. mkfifo -m0600 /tmp/buffer && netcat httpbin.org 80 < /tmp/buffer | netcat -lp 8080 > /tmp/buffer Might want the file if you want a record of course.

maxbond 5y ago
Thanks! TIL! That's a much cleaner way of doing it. Regarding keeping the file, if memory serves the "real" pipeline had some `tee`s in it. I recreated this much more recently to make a GitHub repo of funny code snippets (https://github.com/MaxBondABE/one-thread-crashing/ https://github.com/MaxBondABE/one-thread-crashing/). You reminded me of a different toy I made many years ago. I created FIFOs for files that you might try to read when exploiting an arbitrary file read in a web app, say /etc/www/apache.conf (if that's the right path - it's been a while since I've configured a web server!) and I'd have a program that opened them. This would block until someone opened the other end of the FIFO, at which point I'd raise an alarm (which was just a print statement).

thyrox 5y ago

That's awesome but would you mind explaining this a little bit more to those not very unix savvy? I think I get the first three parts: 1) Create a temp buffer file 2) Read the file constantly to STDOUT? 3) What does netcat httpbin.org 80 | netcat -lp 8080 do? What would be a good use of this circular pipeline? Can you give me some example of what you use this for? Thanks

dundarious 5y ago
I'm not OP, but as they said, it's an HTTP proxy. Run it, then point your browser at http://localhost:8080 http://localhost:8080. The message is proxied over to http://httpbin.org http://httpbin.org (on port 80, the normal HTTP port). If you're in an overly restricted environment and have a legit need to proxy to a very limited set of hosts, you can ad-hoc make a proxy mapping from local port to a single specific remote host. Just change the 8080, destination host, and buffer file. And I mean this as encouragement, but you should read the synopsis from the GNU netcat manual. It's short and explains it well enough. netcat httpbin.org 80 is using connect mode, the -l one is using listen mode, and -p is --local-port (the port to listen on in listen mode).
maxbond 5y ago
Of course! A.) Some context: - Our goal here is to match up to two network connections, so that anything we receive on one is forwarded to the other. - netcat is a "networking swiss army knife", it's a tool for making network connections - HTTP and bash pipelines both happen to be "line oriented", meaning they operate one line at a time. Exploiting this happy coincedence is what makes this work - httpbin.org is a metanym for any website you need. So: echo "" > /tmp/buffer Initialize our buffer to an empty state tail -f /tmp/buffer Read each line in the buffer, one at a time, and feed them to the next command. Do this forever (-f). netcat httpbin.org 80 Connect to the website httpbin.org. Transmit each line from our buffer to the website. netcat -lp 8080 > /tmp/buffer Accept a connection. Transmit any data we receive from httpbin.org to this client; write any data we receive from the client to the buffer, so that it can be transmitted to httpbin.org. B.) I don't think there is one! I've never needed to do this again in the, gosh, seven intervening years. Normally bash pipelines operate like a bucket brigade, moving data unidirectionally through various processing steps, and usually dumping it into a file at the end. They aren't normally for creating long-lived programs that manage bidirectional flows of data. Which is why writing a web server in bash is a fun challenge.
- thyrox 5y ago
  Thank you for the detailed explanation. Learned a thing or two too! :)