21 ms·
Show HN: A pure bash web server. No netcat, socat, etc.
- spicybright 5y agoDefinitely scary, but I love things like this. Takes a lot of creativity! There was a post a bit ago that had a one liner for a basic web server using net cat: while : ; do cat page.html | nc -l 80; done
- dzove855 5y agoYeah sure. I have see it, but basically this one use bash builtins instead of external commands.
- hericium 5y agoor even while :; do nc -l 80 < page.html; done
- skrtskrt 5y agowhat is the difference in these two approaches? I always use the pipe, it is intuitive to me to think left-to-right, output of this becomes input of next. But I always see stackoverflow answers using the other approach.
- arbitrage 5y agoThe difference is in the first code example, you're launching both cat and nc. In the second block, you're just launching one program that reads input, not two. It's slightly more efficient. I also tend to build large *NIX pipelines using `cat $foo | [...]`, but it's some times regarded as bad form. See: "useless use of cat" for more examples.
- skrtskrt 5y agoAren't you just using another built-in to read the file contents then? Maybe it's lighter than cat, but whatever code path is triggered by the `<` can't be zero-cost
- aidenn0 5y agoIf you want to do left-to-right, you can do that: while :; do <page.html nc -l 80 ; done
- tyingq 5y agoBut, bash can't do listen sockets. "loadable accept builtin" Ahh. Gawk can be a web server also, typically just "as-shipped", without any new extensions. Doesn't scale well though :) https://news.ycombinator.com/item?id=22085459 https://news.ycombinator.com/item?id=22085459
- drvdevd 5y agoThis was very interesting to me - I was expecting using linux socket virtual FS paths to create the socket.
- tyingq 5y agoPretty printed: https://gist.github.com/tyingq/4e568425e2e68e6390f3105e588787c6 https://gist.github.com/tyingq/4e568425e2e68e6390f3105e58878... The /inet/tcp/8080/0/0 is a gawk-ism, versus a Linux thing: https://www.gnu.org/software/gawk/manual/html_node/TCP_002fIP-Networking.html https://www.gnu.org/software/gawk/manual/html_node/TCP_002fI... Unfortunately, gawk doesn't let you control the listen/accept part of it, so it doesn't scale well. Not sure why they did it that way. A separate accept() call would have made it actually usable in a non-toy way.
- genewitch 5y agoProbably designed that way on purpose (or at least post hoc) so that a company wouldn't hit a wall with 10,000 users on the gawkttpd. This is how you get stuff like hiPHoP.
- BeefWellington 5y agoI appreciate the ingenuity here given the other recent posts, but this still requires executing an external application to perform the accept() part. Definitely closest to the mark though, kudos!
- dzove855 5y agoBasically yes, but it's a loadable builtin.
- maxbond 5y agoMy humble submission to this genre is this pipeline I wrote once, when I needed an HTTP proxy, but I couldn't use install a real one do to work limitations, and I couldn't use firewall rules because of weird limitations of the Mac OS at the time: echo "" > /tmp/buffer; tail -f /tmp/buffer | netcat httpbin.org 80 | netcat -lp 8080 > /tmp/buffer I call it a circular pipeline.
- dundarious 5y agoI know this is a toy meant for short-term use, but you can use a fifo/named pipe so you don't have a growing file used for data transfer. mkfifo -m0600 /tmp/buffer && netcat httpbin.org 80 < /tmp/buffer | netcat -lp 8080 > /tmp/buffer Might want the file if you want a record of course.
- maxbond 5y agoThanks! TIL! That's a much cleaner way of doing it. Regarding keeping the file, if memory serves the "real" pipeline had some `tee`s in it. I recreated this much more recently to make a GitHub repo of funny code snippets (https://github.com/MaxBondABE/one-thread-crashing/ https://github.com/MaxBondABE/one-thread-crashing/). You reminded me of a different toy I made many years ago. I created FIFOs for files that you might try to read when exploiting an arbitrary file read in a web app, say /etc/www/apache.conf (if that's the right path - it's been a while since I've configured a web server!) and I'd have a program that opened them. This would block until someone opened the other end of the FIFO, at which point I'd raise an alarm (which was just a print statement).
- thyrox 5y agoThat's awesome but would you mind explaining this a little bit more to those not very unix savvy? I think I get the first three parts: 1) Create a temp buffer file 2) Read the file constantly to STDOUT? 3) What does netcat httpbin.org 80 | netcat -lp 8080 do? What would be a good use of this circular pipeline? Can you give me some example of what you use this for? Thanks
- 5y ago
- encryptluks2 5y agoI used to see value in scripts like this, but often these Bash scripts do quite the opposite. It may be readable for those that only know Bash, but you'll probably be happier if you learn Go or C. There are also a good amount of lightweight C web servers like darkhttpd.
- drran 5y agoIt's written in C: http://git.savannah.gnu.org/cgit/bash.git/tree/examples/loadables/accept.c http://git.savannah.gnu.org/cgit/bash.git/tree/examples/load...
- encryptluks2 5y agoThis is like saying my Python app is written in C because Python is C.
- umvi 5y agoHa, this is more like a puzzle than anything - trying to implement a protocol in a highly restrictive environment. For anyone looking for an easy way to spin up a command line web server, python has one built in, so if you are running linux usually you can do: `python3 -m http.server` From a terminal and it will spin up a web server that serves the current directory. I use this all the time for quick tinkering vs. installing/configuring nginx/node.js/caddy or whatever the cool kids are using these days.
- pcranaway 5y agoI use darkhttpd from time to time, it supports streaming: https://github.com/emikulic/darkhttpd https://github.com/emikulic/darkhttpd
- masklinn 5y agoThe biggest issue with `http.server` is it's a blocking single-threaded server, so it can serve a single client at a time. This makes it really really not great when trying to serve pages (with various attending static files), or to ad-hoc share large resources with other people on the network.
- javajosh 5y agoA single-threaded server is fine if you're only mildly IO bound, which is the case when you're serving files-as-resources. My understanding anyway is that http.server is explicitly not for production use!
- masklinn 5y ago> A single-threaded server is fine if you're only mildly IO bound, which is the case when you're serving files-as-resources. It's fine if you're also evented, but that's not the case here, http.server literally can only interact with one connection at a time, the next connection(s) will be queued up waiting for http.server to finish its thing and accept them. > My understanding anyway is that http.server is explicitly not for production use! Sure, but "I'm at a lan party and have the installer for $game so we don't explode the 'net connection" is not the sort of "production use" that's intended by this statement. Neither is "I want to open this HTML file I wrote which has a bunch of static assets booting up the webapp".
- deleted 5y ago[deleted]
- pdkl95 5y agoRegarding the creation and removal of a tempfile (line 72 & 85): local tmpFile="${TMPDIR:-/tmp/bash-web-server.$$}" # ... rm "$tmpFile" To avoid collisions when the PID is reused, and to clean up0 the tempfile on errors, I recommend using mktemp and trap: local tmpfile="$(mktemp --tmpdir="${TMPDIR:-/tmp}" bash-web-server.XXXXXX)" trap "rm -f \"${tmpfile}\"" RETURN EXIT # ... # Do nothing at the end of the function; trap will # remove the file at RETURN automatically. Otherwise, I like the implementation! It's nice to see good bash techniques like parsing with "IFS='&' read -ra data" and rewriting variables with %%/etc.
- dzove855 5y agoI always use mktemp and trap to remove tmp files. But in this script i would like to avoid external commands. I will even remove the use of tmp files kn thw future.
- goombacloud 5y agoTo spare you the trap you can open the file as FD in your bash process (e.g., exec {my_fd}>"$TMPFILE") and then directly delete it before doing anything else, and use the /proc handle to access it ("/proc/$$/fd/${my_fd}")
- goombacloud 5y agoOne more fun experiment: avoid the tempfile on disk/tmpfs by storing data into the pipe connecting two processes, where the first one can already exit for the pipe to give EOF when reading beyond the buffered data. true | sleep infinity & STDINPID="$!" TMPPIPE="/proc/$STDINPID/fd/0" echo hello > "$TMPPIPE" echo write more > "$TMPPIPE" cat "$TMPPIPE" echo write again > "$TMPPIPE" cat "$TMPPIPE" kill -9 "$STDINPID" # clean up pipe Now I would like to avoid the additional process by using the current shell process instead.
- goombacloud 5y agoStill needs to spawn a temporary process but it can be killed early: true | sleep infinity & STDINPID="$!" exec {mypipe}<"/proc/$STDINPID/fd/0" # hijack FD TMPPIPE="/proc/$$/fd/${mypipe}" disown "$STDINPID" # suppress killed message on stderr kill -9 "$STDINPID" # clean up process now already echo hello > "$TMPPIPE" echo write more > "$TMPPIPE" cat "$TMPPIPE" echo write again > "$TMPPIPE" cat "$TMPPIPE"
- pmarreck 5y agoThe other day I wrote this bash code after getting frustrated with the password generators available to me, just so I had something to use right in the shell: https://gist.github.com/pmarreck/f4dbb02396c4532762a7a64511cd8e52 https://gist.github.com/pmarreck/f4dbb02396c4532762a7a64511c... It actually took an entire afternoon and went through a couple iterations, and I hadn't even written a test for it yet (and it just so happens that I started an MVP shell-script testing library for just such an occasion when I ALSO got frustrated with the bash-accessible testing libraries available to me, also in Bash: https://github.com/pmarreck/tinytestlib https://github.com/pmarreck/tinytestlib) But then I immediately felt bad. I could have written all of this in less than an hour in a language like Elixir, with test coverage (and the testing/assertion lib is built-in, so it would have been free). It is impressive and code-golfish that someone can "write a webserver in pure Bash" but Bash is a crap language to script in at this point, and adding more Bash code to the world just seems wrong now. Bourne shells were invented in the 70's (!), long before many language advancements and understandings, and they thus have MANY warts (many of which we're familiar with). The next time I feel the urge to code anything more than a one-liner in Bash, I'm writing it in elixir script https://thinkingelixir.com/2019-04-running-an-elixir-file-as-a-script/ https://thinkingelixir.com/2019-04-running-an-elixir-file-as... and just sucking up the extra run time (or compiling it into an executable with something like https://github.com/spawnfest/bakeware/ https://github.com/spawnfest/bakeware/ and sucking up the extra megabytes), and then MAYBE writing a wrapper function for my .profile Incidentally, if you DO want a shell that "tastes like" Bourne but is functional and thus sucks a lot less, check out https://github.com/wryun/es-shell https://github.com/wryun/es-shell, or my fork of it https://github.com/pmarreck/es-shell/ https://github.com/pmarreck/es-shell/
- dzove855 5y agoBash is not mean for kind of projects like this. I even develop on other languages. But i like bash. A good bash script, is like reading a novel, because it's challenging to have a clean script.
- pmarreck 5y agoI get it. The challenging part is fun. You can learn all the corner cases, the clever bash-isms, and then show off. But the code you end up with is essentially unmaintainable, especially since you don't have any test coverage, so...
- sillysaurusx 5y agoThis script taught me a few things! Since it's kind of impossible to google, what does this construct mean? while :; do It's clearly a while true loop, but why does an ascii crying face produce true?
- veltas 5y ago: is a sort of 'null' function in bash and bourne shells. It also always returns 0 so it's an alternative to 'true'. https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_16 https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V...
- framecowbird 5y agohttps://stackoverflow.com/a/3224910 https://stackoverflow.com/a/3224910
- StanAngeloff 5y agoSee https://stackoverflow.com/questions/3224878/what-is-the-purpose-of-the-colon-gnu-bash-builtin https://stackoverflow.com/questions/3224878/what-is-the-purp... $ type : : is a shell builtin
- vultour 5y agoWhat's impossible to google about that? "bash colon" gives tons of results
- tyingq 5y agoIncluding the somewhat unexpected "Bottoms Up Bash - Colon Cancer Prevention Project".
- bewuethr 5y agoBash also has the equivalent "true" as a builtin, and Coreutils provides one as well, allowing to write this instead: while true; do
- 5y ago
- mro_name 5y agoI like those unorthodox, against-cargo-cult approaches. Thx!
- dheera 5y agoIf Bash is Turing-complete could we just make a Python to Bash transpiler?
- nottorp 5y agoI wonder if it's faster than the same thing in javascript...
- dzove855 5y agoWell benchmarking it os currently not possible. I will try it in a few days and provide some benchmarks.
- tyingq 5y agoMost of the node.js solutions I'm aware of would support http/1.1 and async so I can't imagine this faring well in a performance comparison.
- nottorp 5y agoThe famous HN lack of humour shows its head :)
- ape4 5y agoSort of related, if you do "man read" you the the huge "bash" man page. It would be nice if man could be improved to take you to the "read" command within that page. Or a new command "manbash read"
- dzove855 5y agoWell just type: help BUILTIN Example : help read
- garaetjjte 5y agoThis prints abridged usage help, though. You can use `info bash read`, but confusingly it might not always contain the same content as manpages.
- ape4 5y agoThanks I didn't know! So many years of searching for "read" (etc) in the huge bash man page and, of course, in the case of "read" finding many false positives. Still it would be nice if `man` could be improved to know about builtin commands - the obvious thing is something like an html anchor in the big bash man page for each built in.
- pxeger1 5y agoprintf '%s\n' "$(<$tmpFile)" Why not just cat $tmpFile printf '\n'
- dzove855 5y agoto avoid external calls
- benbristow 5y agoRight, time to make this into a Docker image and ship it to production
- codeivore 5y agoone day this code will end up on a small/home office router
- hyperupcall 5y agoI had this same idea, but just haven't gotten around to implementing it - so thank you for this reference! This also gives me the perfect excuse for finally getting around to implement automatic dynamic loading of Bash builtins for my Bash package manager This would be very useful as a fallback - in case something like `socat` isn't installed, this can be used to display some sort of dumbed down version of the site
- dzove855 5y agoI follow some projects from you on github. Keep up the work! Falling back to something like socat, would destroy the whole idea behind it. I think the best way is patching accept and shipping it with the project. Like this no need to fallback. And considering using something like socat, will force you to fork each request into a new process, which will be slow as hell. I worked before on a web framework, which was used behind httpd or nginx as cgi, it was fast. But the moment where you get more connections it was slow as hell. The idea of this project is writing some kind of fastCGI (like php-fpm), but for this i would need, to allow multithreading etc..
- yeetaccount4 5y ago…and I like to live dangerously.
- dzove855 5y agoThe project has been patched, now we can handle multiple connections!