11 ms·

Show HN: A Unixy approach to WebSockets

421 points by joewalnes 12y ago

detaro 12y ago
Looks very nice for quick prototype kind of things in any language or to make existing tools quickly available to a small circle of users! for larger-scale usage the overhead probably is to big.
- hippich 12y ago
  Depending on the job. If your job involve long lasting connections, overhead of setting up websocket connection and spawning new process wouldn't matter. But this is clearly will not cut for typical HTTP static stuff. So if your server side keeps connection for a long period of time (think IRC server, or netcat?) - this might be good.
  - phamilton 12y ago
    The overhead isn't the setup. It's the memory footprint of a process. A minimal process (just sleeping) will use ~128KB. One process per connection means a server will need well over 1GB memory per 8000 connected users. A python process just sleeping will be almost 3MB, which means you get 333 connections per GB. Compare this with an efficient green threads implementation. In Erlang you can have 100k sleeping "processes" in just 200MB of memory.
    - hippich 12y ago
      Yes, there is memory overhead, just like there is programmers overhead to create daemon doing what you need, instead of just piping existing tool's stdio into websocket. I guess each tool should be used for tasks it is good at.
- joewalnes 12y ago
  OP here. The overhead is really dependent on your process overhead. I typically use it for Python/Ruby/C apps which are relatively lightweight compared to something JVM based. The runtime profile of WebSockets tends to be different to typical HTTP requests too. With typical HTTP you're often optimizing for 1000s of short requests per second, whereas with WebSockets the requests are much longer lived.
  - jeremiep 12y ago
    The JVM is optimized for long-running processes as it takes time for the JIT to kick in. Other than the memory footprint your code gets quite a performance boost; I'm always surprised how much of a difference it makes before and after. You wouldn't want to boot a JVM process per connection but rather implement the listening socket yourself and dispatch connections within that same process with all of the required services already initialized. A WebSocket server is no different.
NathanOsullivan 12y ago
A project in a similar vein is websockify ( https://github.com/kanaka/websockify https://github.com/kanaka/websockify ) , which makes an existing app with a TCP endpoint available over websockets.
- joewalnes 12y ago
  Nice! I hadn't seen that. Yeah, that's pretty sweet.
- espadrine 12y ago
  What are the differences? Do I miss anything? 1. websocketify allows binary, while websocketd is text-only 2. websocketify does I/O through a socket that the program uses (and that it sniffes through rebind.so), while websocketd relies on stdin / stdout 3. The arguments are different websocketd --port=8080 my-program websocketify 8080 -D -- my-program
  - joewalnes 12y ago
    I haven't used websocketify. I'd be interested to have users try out both and write up their experiences.
  - asergeyev 12y ago
    Also websocketd is multiplatform and does not require python. Neither good or bad, just another difference :) Also, not sure if websocketify can but websocketd also has --dir argument and can supervise and route many sockets to many different programs.
    - cpncrunch 12y ago
      websockify can be run on Node.js as well as python. Both versions seem to be highly reliable (I'm using them on multiple production servers with paying customers for web conferencing). The python version doesn't seem to work on Windows servers, but the Node version works fine.
joewalnes 12y ago
OP here. Something I forgot to mention on the page is how much this simplified life for a server admin. Using "ps" you can see each connection, the memory overhead, CPU usage. It's easy for a sysadmin to kill bad connections without taking down the whole server.
- comboy 12y ago
  I like the project. But it says: > Full duplex messaging And the examples only show single direction. Is my understanding correct that everything received goes as STDIN? And is it also possible to run websocketd as a client?
  - jakejake 12y ago
    In the source code /examples folder most of the languages have a bi-directional example called "greeter"
- itsbits 12y ago
  can i use other web socket clients like socket.io??
  - tracker1 12y ago
    websockets after upgrade are not message based like the same as socket.io (which adds fallback/fall-forward) and sockjs does this providing a stream interface... If you don't need to support a web browser client, you don't need the extra overhead.. and if you do, you're better off just using socket.io or shoe and having your server in node.js ... as an aside, I'd probably just use raw sockets in node if I didn't need browser support.
    - joewalnes 12y ago
      WebSockets are message based.
      - 12y ago
frknbasaran 12y ago
I'll try this. Looks good!
nubb 12y ago
Link is down =(
angersock 12y ago
Oh, this is going to be so fun for doing terrible things.
- asergeyev 12y ago
  Untold rule is not to run "sudo websocketd --devconsole bash" :)
  - cturner 12y ago
    The browser will not implement terminal controls. You could get by though. This sort of scenario is why ed is never obsolete.
    - joewalnes 12y ago
      ed is for the weak. Real developers edit bits using the magnetic forces of oceans.
angersock 12y ago
Oh, this is going to be so fun for doing terrible things.
minimax 12y ago
The thing with WebSockets is that they are message oriented. WebSockets endpoints are presented with a series of distinct messages, whereas stdin/stdout are stream based and you have to build messaging on top, if that's what you want. I guess the idea here is that you are just using '\n' as the message delimiter?
- asergeyev 12y ago
  Pretty much. Good example is Joe's vmstats https://github.com/joewalnes/web-vmstats https://github.com/joewalnes/web-vmstats
elierotenberg 12y ago
Unless you need to handle a very small number of concurrent connections, using 1 process per connection seems to be a huge overhead, although I can think of some use cases. However, I can imagine a similar tool doing multi/demultiplexing, eg the handler process would take input text lines prepended with a connection identifier (eg. "$socketID $message") and output using a similar formatting. Pretty much like websocketd but with multiplexing and unixy/pipe-friendly (eg. you can apply grep/awk/etc before and after). How would this fit compared to websocketd?
- asergeyev 12y ago
  There is closed ticket (github.com/joewalnes/websocket) where we were battling this out, you're always welcome to join with good ideas, that would be appreciated.
- ayani 12y ago
  Indeed, there is no free lunch. At is stands, this is only really workable for low traffic (so it doesn't eat memory) where connections do not come and go frequently (so it doesn't eat process management CPU). Once you start doing multiplexing for the sake of making this more reasonable in terms of resource usage, the simplicity benefits kind of fall away as you move closer to a full concurrent web framework. I guess it really depends what you're tuning for, what your use case is, and how much hardware budget you have to throw at the problem.
  - platz 12y ago
    About wow many connections for an average machine with 8GB ram would be deemed 'OK' with this tool?
    - jakejake 12y ago
      I was just playing around with this tonight using PHP and each process was about 5Mb of RAM. I'd imagine if you wrote your server code in, say C instead, then the memory footprint would be much smaller. There's also a limit to the number of processes allowed as well. On my OSX laptop with 16Gb RAM for example the default limit is 709 (kinda strange number???). The command ulimit -a will tell you the value of "max user processes" for your machine.
lordlarm 12y ago
For simple one-directional communication (Server to client(s)) as shown in the example, it may for many people be simpler to use EventSource [1] I've used this for UI's where the server continuously sends/pushes updates to the clients. Really handy, and multiple implementations and libraries available in most languages. Of interest is perhaps also the spec [2] [1]: https://developer.mozilla.org/en-US/docs/Web/API/EventSource https://developer.mozilla.org/en-US/docs/Web/API/EventSource [2]: https://html.spec.whatwg.org/multipage/comms.html#the-eventsource-interface https://html.spec.whatwg.org/multipage/comms.html#the-events...
- asergeyev 12y ago
  Not supported in any of IE versions... https://status.modern.ie/serversenteventseventsource?term=EventSource https://status.modern.ie/serversenteventseventsource?term=Ev... And WebSockets is working since IE10 https://status.modern.ie/websocket?term=WebSocket https://status.modern.ie/websocket?term=WebSocket
  - stephenr 12y ago
    Due to it's nature, EventSource can be quite easily implemented using a polyfill for legacy browsers. It's also MUCH easier to setup/run in a multi-layered stack (e.g. Pound/HAProxy > Varnish > App Server). As per usual, WebSockets is what the "cool kids" use, even when it's often much less appropriate and much less flexible.
    - tokenizerrr 12y ago
      So what would you use to implement the polyfill for IE10? Is there something more appropriate than websockets? If not, are we not back to square one where you may as well use websockets from the start?
      - stephenr 12y ago
        There are multiple polyfills for EventSource - mostly all it needs is a working XHR implementation. The key thing is that a proxy/load balancer doesn't need special cases to support EventSource like it needs (and often doesn't have) with websockets.
keyle 12y ago
Oh my! I have been waiting for this, trying to write my own websocket server implementation in multiple esoteric languages! Fantastic!
riobard 12y ago
I guess the next logical step is to build a virtual filesystem such that each connection is represented by a file. Then you can further decouple the connection handling from the actual application. Applications can start later to talk to a connection established before.
- nine_k 12y ago
  That would be less 'unixy' and more 'plan-niney' :) I like the idea of having a FS interface, e.g. using named pipes made available by the daemon.
- dangayle 12y ago
  There was SpockFS here on HN a while back: https://github.com/unbit/spockfs https://github.com/unbit/spockfs
- espadrine 12y ago
  That's the idea behind https://thefiletree.com/ https://thefiletree.com/. The decoupling works well for applications tied to a textual format (see eg. https://thefiletree.com/heap/diagram.sequence https://thefiletree.com/heap/diagram.sequence). It means collaborative SVG editing, for instance, is not a challenge. On the other hand, binary data (bitmap images, sound, video) would need to be treated specially, I think, unless we had a common structure format and protocol that would map to each of those. (The code is here: https://github.com/garden/ https://github.com/garden/)
Udo 12y ago
This is a good idea. Myself, I noticed that almost all my websockets projects could easily share one single code base, and finally I just made a Websockets boilerplate repo that I can pull from for any given project. This is what node.js really excels at, and it's an execution model fundamentally different from websocketd: being a message broker between the client and the request-based web server. Some people don't like the separation between web server and websockets server, but when you think about it they don't belong quite on the same level of abstraction. Plus, it's usually orders of magnitude easier to reason about single requests than to reason about a complex persistent application server's state.
jeremiep 12y ago
> Write programs to handle text streams, because that is a universal interface. I always hated that part of UNIX. It would be so much better if programs could handle data-structure streams instead. Having text streams causes every single program to implement ad-hoc parsers and serializers. I now use the command-line only for trivial one-time actions. For everything else I'll use a scripting language and forget about reading and writing text streams altogether.
- chriswarbo 12y ago
  The main problem is that there are many kinds of incompatible "data-structure streams". Using a scripting language doesn't solve the problem, it just standardises on one particular ad-hoc parser/serialiser combination (say, Python's pickle, for example). That's fine for personal use, or in a single project, but doesn't scale like "dumb" text does.
  - nine_k 12y ago
    If there was a non-ideal but reasonable standard serialization format, like maybe IFF [1] or BSON [2], it would still simplify things, and common tools like `cat` might support decoding them. It would be easy to wrap an arbitrary other packed format inside a binary string, or an arbitrary text format inside a text string. IFF is quite successful in certain areas, BTW. [1]: http://en.wikipedia.org/wiki/Interchange_File_Format http://en.wikipedia.org/wiki/Interchange_File_Format [2]: http://en.wikipedia.org/wiki/BSON http://en.wikipedia.org/wiki/BSON
    - tracker1 12y ago
      You could always pass a message per line (so to speak) and use msgpack or json... if you want compression you could use json+gz Of course, you could also do something similar with a 0mq adapter as well for a request/response server, which is available pretty much everywhere you might need it (language wise)... It's an interesting generic interface... however as I mostly use node for newer development, it's so trivial to use sockets or websockets directly for this, I'd just assume go that route... or for that matter, something like sockjs/shoe.
deleted 12y ago
[deleted]
dkarapetyan 12y ago
This is cool. I can see this being used for prototyping and validating an idea before going full steam ahead with a more scalable framework.
jkarneges 12y ago
Elegant approach. Reminds me of inetd.
- colanderman 12y ago
  I'm surprised this isn't just implemented as an inetd handler, given the "do one thing and do it well" mantra.
  - fit2rule 12y ago
    Its the 21st Century. People don't look backwards any more. inetd is passé..
    - colanderman 12y ago
      I'm assuming you're being sarcastic. This tool wants to "do things the Unix way"... which was invented in the 1970s.
spiralpolitik 12y ago
I assume you are treating `stdin` and `stdout` as just a stream of bytes rather than trying to do anything clever a la Python 3 ?
- joewalnes 12y ago
  Yep. The only magic thing is a line break (e.g. \n) is used to terminate each message.
cypher543 12y ago
> "Write programs that do one thing and do it well." So why does this WebSocket daemon also serve static files and CGI applications?
- spoiler 12y ago
  Because you need to implement a tiny HTTP server for WebSockets anyway (if you want't them to work from browsers). So, adding the ability to serve static files on top of that is trivial. Not sure why they added CGI, though.
  - joewalnes 12y ago
    That indeed was a dilemma. Although it focusses on doing one thing well, there were also a few core things that are useful to support a websocket based app - namely a place to serve the actual web content from (static) and the ability to use websocketd using vanilla http instead of websockets (cgi).
nichochar 12y ago
Holy shit that's cool
ianbicking 12y ago
I don't understand how I'd use this. WebSockets are generally for server-initiated events, but this doesn't make it very easy to initiate events on the server. Usually it will be some pubsub situation, so I'd want a server process to be able to emit a message to one or many connections - but instead the connection is tied to one process for its life. I'd like to see an example like a simple chat room.
- andrewstuart2 12y ago
  Actually, I think you're looking for Server-Sent Events [1] for something of a pub/sub nature. WebSockets, on the other hand, are intended for realtime bidirectional communication over a single TCP connection. Pub/Sub is much more one-directional. A single subscribe event initiates a stream of all published events. Server-Sent Events are more suited for this because the server will never have to bother checking the TCP connection for incoming data. [1] http://www.w3.org/TR/2011/WD-eventsource-20110208/ http://www.w3.org/TR/2011/WD-eventsource-20110208/
- tracker1 12y ago
  The solution in question could well be used for pub/sub servers or for request/response patterns... it's really pretty cool. I'm needing to expose a very small utility that uses C# for the micro service... I'm considering this for that reason... aside from that, I think the solution in question is pretty cool.
- joewalnes 12y ago
  I've built chatrooms with websocketd. To do this, you a shared pub/sub store. Redis is great, so is Postgres (with its pub/sub add-ons). Storing chat data in a store is more durable than keeping it in memory on a single machine.
runewell 12y ago
This is perfect for admin tools. I'd rather parse stdout strings than code a custom API.
hurin 12y ago
Could anyone give an example of what this is used for?
- runewell 12y ago
  It appears to me that it is useful for eliminating the "middle man" scripts/code often written for web application and server interaction. If I want to create a web application that displays the status of my server (CPU, memory, I/O, etc) I can use this software to call vmstat directly instead of writing an API (PHP, Node.js, RoR, etc) that executes the same command or uses a library that requires even more code to get the exact same data.
  - hurin 12y ago
    I'm sure this this is probably because I don't have do this for work - but from that description it's hard for me to tell what the advantages are. A script to feed vmstat (or other command data) with a python socket would be ~50 lines or so.
- joewalnes 12y ago
  It allows you to write a WebSocket end point, in any programming language, without having to deal with building socket servers, dealing with WebSocket protocol handshake/parsing/formatting/etc, worrying about threading etc. Just write a script that reads and writes to stdin/out.
curiously 12y ago
this is fucking cool. I got it instantly just glancing at the code example. I immediately understood how it worked. That is powerful stuff. My only worry of course is, how would you scale this up? What's really going beneath the hood. I'm really excited and trying to think of something so I can use it as an excuse to use this. The only other suggestion I would make is maybe change the name to something more catchy and brandable. Websocketd...okay like systemd...but I don't know, something as good as this deserves a brandable name like Jupiter, or some Greek goddess or clever hacky name.
- joewalnes 12y ago
  Thanks. And I suck at naming stuff.
aabajian 12y ago
This looks great, without knowing much I was able to get: ./websocketd --staticdir=. --port=8123 ps to give me a simple output of processes. What I'd really LOVE is to get this to work: ./websocketd --staticdir=. --port=8123 htop It'd be great if I could see the output of htop on the web...from anywhere. I guess htop is setting up a different video mode or something that isn't compatible?
- olalonde 12y ago
  I think you would need to feed what you are getting to a Javascript terminal emulator. (e.g. https://github.com/chjj/tty.js/ https://github.com/chjj/tty.js/)
romanpoet 12y ago
Why is this better than netcat? https://en.wikipedia.org/wiki/Netcat https://en.wikipedia.org/wiki/Netcat
- colanderman 12y ago
  Because it handles the WebSocket protocol for you.
Lerc 12y ago
I used this a while ago on my own insane project. I had a Webdav server and websocketd providing a Web interface to a Linux box. Even at it's young age (bug reports #5 and #7 were mine), it allowed me to progress a lot further in my project before I needed to write a server designed specifically for the task at hand. In the end I wrote userserv https://github.com/Lerc/userserv https://github.com/Lerc/userserv for the one must have feature I needed. I needed logins and responses delivered from a process with the UID of the CookieToken. So, thanks Joe. Notanos got further because of websocketd and while I'm not currently using it, There's a high chance of me doing so in future projects.
- joewalnes 12y ago
  Great to hear! Thanks :)
tomerbd 12y ago
beautiful website may I ask with which template / framework it was written? (is it in github by any chance?)
- joewalnes 12y ago
  Didn't use any templates or frameworks (apart from font-awesome for icons and prism for syntax highlighting the code examples). Just some good old fashioned hand written HTML + CSS. https://github.com/joewalnes/websocketd/tree/gh-pages https://github.com/joewalnes/websocketd/tree/gh-pages
vram22 12y ago
I had come across websocketd a while ago, and to try it out, wrote a simple Python program and a web page. The two posts about it are here: 1. Use WebSockets and Python for web-based system monitoring: http://jugad2.blogspot.in/2014/01/use-websockets-and-python-for-web-based.html http://jugad2.blogspot.in/2014/01/use-websockets-and-python-... 2. websocketd and Python for system monitoring - the JavaScript WebSocket client: http://jugad2.blogspot.in/2014/01/websocketd-and-python-for-system.html http://jugad2.blogspot.in/2014/01/websocketd-and-python-for-... Note: As it says in one of the posts, you have to: set PYTHONUNBUFFERED=true for the Python program to work as a websocket server, though it works fine without that if only run directly at the command line (without websocketd). Thanks to Joe for pointing this out. websocketd is a nice utility.
MatthewWilkes 12y ago
It's a petty comment, I know, but: for count in range(0, 10): print count + 1 sleep(0.5) makes me feel sad.
- asergeyev 12y ago
  Please ignore complexity of supplied examples... It's really not for that kind of use. I have lots of scripts that do something like this (and most of them are not in bash :)...): while read ARG; do if validate $ARG; then run_something $ARG fi done Most of my "validate" pieces are bash functions and most "run_something" return lengthy CSV datablocks. Then, there is the javascript that uses user mouse (and other signals) to generate arguments to send to websocketd and draw pretty visuals based on data that arrives back.
kalmi10 12y ago
How are message boundaries handled? WebSockets are message-based. UNIX streams are not.
- joewalnes 12y ago
  websocketd treats each line as a message. i.e. boundaries are marked with \n
- joewalnes 12y ago
  websocketd treats each line as a message. i.e. boundaries are marked with \n
kragen 12y ago
> Each inbound WebSocket connection runs your program in a dedicated process. Connections are isolated by process. That sounds bad; it is like “CGI, twenty years later”, as they say. In 2000 at KnowNow, we were able to support over ten thousand concurrent Comet connections using a hacked-up version of thttpd, on a 1GHz CPU with 1GiB of RAM. I’ll be surprised if you can support ten thousand Comet connections using WebSockets and websocketd even on a modern machine, say, with a quad-core 3GHz CPU and 32GiB of RAM. Why would you want ten thousand concurrent connections? Well, normal non-Comet HTTP is pretty amazingly lightweight on the server side, due to REST. Taking an extreme example, this HN discussion page takes 5 requests to load, which takes about a second, but much of that is network latency — a total of maybe ½s of time on the server side. But it contains 7000 words to read, which takes about 2048 seconds. So a single process or thread on the server can handle about 4096 concurrent HN readers. So a relatively normal machine can handle hundreds of thousands of concurrent users without breaking a sweat. On the other hand, Linux has gotten a lot better since 2000 at managing large numbers of runnable processes and doing things like fork and exit. httpdito (http://canonical.org/~kragen/sw/dev3/server.s http://canonical.org/~kragen/sw/dev3/server.s) can handle tens of thousands of hits on a single machine nowadays, even though each hit forks a new child process (which then exits). http://canonical.org/~kragen/sw/dev3/httpdito-readme http://canonical.org/~kragen/sw/dev3/httpdito-readme has more performance notes. On the gripping hand, httpdito’s virtual memory size is up to 16kiB, so Linux may be able to handle httpdito processes better than regular processes.
- asergeyev 12y ago
  Difference with cgi is that they are live and die, with each user request. websocketd programs are more like "one per user session"... Makes sense when your user sessions are lengthy.
  - kragen 12y ago
    Yes — that’s exactly the problem. Handling ten thousand concurrent users with CGI is easy, even on 2000-era hardware. Ten thousand concurrent users might be four requests per second. But ten thousand concurrent users using websocketd means you have ten thousand processes. And if you’re doing some kind of pub/sub thing, every once in a while, all of those processes will go into the run queue at once, because there’s a message for all of them. Have you ever administered a server with a load average over 1000? Still, the O(N) scheduler work in current Linux might make that kind of thing survivable.

421 points by joewalnes 12y ago

Looks very nice for quick prototype kind of things in any language or to make existing tools quickly available to a small circle of users! for larger-scale usage the overhead probably is to big.

hippich 12y ago
Depending on the job. If your job involve long lasting connections, overhead of setting up websocket connection and spawning new process wouldn't matter. But this is clearly will not cut for typical HTTP static stuff. So if your server side keeps connection for a long period of time (think IRC server, or netcat?) - this might be good.
- phamilton 12y ago
  The overhead isn't the setup. It's the memory footprint of a process. A minimal process (just sleeping) will use ~128KB. One process per connection means a server will need well over 1GB memory per 8000 connected users. A python process just sleeping will be almost 3MB, which means you get 333 connections per GB. Compare this with an efficient green threads implementation. In Erlang you can have 100k sleeping "processes" in just 200MB of memory.
  - hippich 12y ago
    Yes, there is memory overhead, just like there is programmers overhead to create daemon doing what you need, instead of just piping existing tool's stdio into websocket. I guess each tool should be used for tasks it is good at.
joewalnes 12y ago
OP here. The overhead is really dependent on your process overhead. I typically use it for Python/Ruby/C apps which are relatively lightweight compared to something JVM based. The runtime profile of WebSockets tends to be different to typical HTTP requests too. With typical HTTP you're often optimizing for 1000s of short requests per second, whereas with WebSockets the requests are much longer lived.
- jeremiep 12y ago
  The JVM is optimized for long-running processes as it takes time for the JIT to kick in. Other than the memory footprint your code gets quite a performance boost; I'm always surprised how much of a difference it makes before and after. You wouldn't want to boot a JVM process per connection but rather implement the listening socket yourself and dispatch connections within that same process with all of the required services already initialized. A WebSocket server is no different.

NathanOsullivan 12y ago

A project in a similar vein is websockify ( https://github.com/kanaka/websockify https://github.com/kanaka/websockify ) , which makes an existing app with a TCP endpoint available over websockets.

joewalnes 12y ago
Nice! I hadn't seen that. Yeah, that's pretty sweet.
espadrine 12y ago
What are the differences? Do I miss anything? 1. websocketify allows binary, while websocketd is text-only 2. websocketify does I/O through a socket that the program uses (and that it sniffes through rebind.so), while websocketd relies on stdin / stdout 3. The arguments are different websocketd --port=8080 my-program websocketify 8080 -D -- my-program
- joewalnes 12y ago
  I haven't used websocketify. I'd be interested to have users try out both and write up their experiences.
- asergeyev 12y ago
  Also websocketd is multiplatform and does not require python. Neither good or bad, just another difference :) Also, not sure if websocketify can but websocketd also has --dir argument and can supervise and route many sockets to many different programs.
  - cpncrunch 12y ago
    websockify can be run on Node.js as well as python. Both versions seem to be highly reliable (I'm using them on multiple production servers with paying customers for web conferencing). The python version doesn't seem to work on Windows servers, but the Node version works fine.

joewalnes 12y ago

OP here. Something I forgot to mention on the page is how much this simplified life for a server admin. Using "ps" you can see each connection, the memory overhead, CPU usage. It's easy for a sysadmin to kill bad connections without taking down the whole server.

comboy 12y ago
I like the project. But it says: > Full duplex messaging And the examples only show single direction. Is my understanding correct that everything received goes as STDIN? And is it also possible to run websocketd as a client?
- jakejake 12y ago
  In the source code /examples folder most of the languages have a bi-directional example called "greeter"
itsbits 12y ago
can i use other web socket clients like socket.io??
- tracker1 12y ago
  websockets after upgrade are not message based like the same as socket.io (which adds fallback/fall-forward) and sockjs does this providing a stream interface... If you don't need to support a web browser client, you don't need the extra overhead.. and if you do, you're better off just using socket.io or shoe and having your server in node.js ... as an aside, I'd probably just use raw sockets in node if I didn't need browser support.
  - joewalnes 12y ago
    WebSockets are message based.
    - 12y ago

frknbasaran 12y ago

I'll try this. Looks good!

nubb 12y ago

Link is down =(

angersock 12y ago

Oh, this is going to be so fun for doing terrible things.

asergeyev 12y ago
Untold rule is not to run "sudo websocketd --devconsole bash" :)
- cturner 12y ago
  The browser will not implement terminal controls. You could get by though. This sort of scenario is why ed is never obsolete.
  - joewalnes 12y ago
    ed is for the weak. Real developers edit bits using the magnetic forces of oceans.

angersock 12y ago

Oh, this is going to be so fun for doing terrible things.

minimax 12y ago

The thing with WebSockets is that they are message oriented. WebSockets endpoints are presented with a series of distinct messages, whereas stdin/stdout are stream based and you have to build messaging on top, if that's what you want. I guess the idea here is that you are just using '\n' as the message delimiter?

asergeyev 12y ago
Pretty much. Good example is Joe's vmstats https://github.com/joewalnes/web-vmstats https://github.com/joewalnes/web-vmstats

elierotenberg 12y ago

Unless you need to handle a very small number of concurrent connections, using 1 process per connection seems to be a huge overhead, although I can think of some use cases. However, I can imagine a similar tool doing multi/demultiplexing, eg the handler process would take input text lines prepended with a connection identifier (eg. "$socketID $message") and output using a similar formatting. Pretty much like websocketd but with multiplexing and unixy/pipe-friendly (eg. you can apply grep/awk/etc before and after). How would this fit compared to websocketd?

asergeyev 12y ago
There is closed ticket (github.com/joewalnes/websocket) where we were battling this out, you're always welcome to join with good ideas, that would be appreciated.
ayani 12y ago
Indeed, there is no free lunch. At is stands, this is only really workable for low traffic (so it doesn't eat memory) where connections do not come and go frequently (so it doesn't eat process management CPU). Once you start doing multiplexing for the sake of making this more reasonable in terms of resource usage, the simplicity benefits kind of fall away as you move closer to a full concurrent web framework. I guess it really depends what you're tuning for, what your use case is, and how much hardware budget you have to throw at the problem.
- platz 12y ago
  About wow many connections for an average machine with 8GB ram would be deemed 'OK' with this tool?
  - jakejake 12y ago
    I was just playing around with this tonight using PHP and each process was about 5Mb of RAM. I'd imagine if you wrote your server code in, say C instead, then the memory footprint would be much smaller. There's also a limit to the number of processes allowed as well. On my OSX laptop with 16Gb RAM for example the default limit is 709 (kinda strange number???). The command ulimit -a will tell you the value of "max user processes" for your machine.

lordlarm 12y ago

For simple one-directional communication (Server to client(s)) as shown in the example, it may for many people be simpler to use EventSource [1] I've used this for UI's where the server continuously sends/pushes updates to the clients. Really handy, and multiple implementations and libraries available in most languages. Of interest is perhaps also the spec [2] [1]: https://developer.mozilla.org/en-US/docs/Web/API/EventSource https://developer.mozilla.org/en-US/docs/Web/API/EventSource [2]: https://html.spec.whatwg.org/multipage/comms.html#the-eventsource-interface https://html.spec.whatwg.org/multipage/comms.html#the-events...

asergeyev 12y ago
Not supported in any of IE versions... https://status.modern.ie/serversenteventseventsource?term=EventSource https://status.modern.ie/serversenteventseventsource?term=Ev... And WebSockets is working since IE10 https://status.modern.ie/websocket?term=WebSocket https://status.modern.ie/websocket?term=WebSocket
- stephenr 12y ago
  Due to it's nature, EventSource can be quite easily implemented using a polyfill for legacy browsers. It's also MUCH easier to setup/run in a multi-layered stack (e.g. Pound/HAProxy > Varnish > App Server). As per usual, WebSockets is what the "cool kids" use, even when it's often much less appropriate and much less flexible.
  - tokenizerrr 12y ago
    So what would you use to implement the polyfill for IE10? Is there something more appropriate than websockets? If not, are we not back to square one where you may as well use websockets from the start?
    - stephenr 12y ago
      There are multiple polyfills for EventSource - mostly all it needs is a working XHR implementation. The key thing is that a proxy/load balancer doesn't need special cases to support EventSource like it needs (and often doesn't have) with websockets.

keyle 12y ago

Oh my! I have been waiting for this, trying to write my own websocket server implementation in multiple esoteric languages! Fantastic!

riobard 12y ago

I guess the next logical step is to build a virtual filesystem such that each connection is represented by a file. Then you can further decouple the connection handling from the actual application. Applications can start later to talk to a connection established before.

nine_k 12y ago
That would be less 'unixy' and more 'plan-niney' :) I like the idea of having a FS interface, e.g. using named pipes made available by the daemon.
dangayle 12y ago
There was SpockFS here on HN a while back: https://github.com/unbit/spockfs https://github.com/unbit/spockfs
espadrine 12y ago
That's the idea behind https://thefiletree.com/ https://thefiletree.com/. The decoupling works well for applications tied to a textual format (see eg. https://thefiletree.com/heap/diagram.sequence https://thefiletree.com/heap/diagram.sequence). It means collaborative SVG editing, for instance, is not a challenge. On the other hand, binary data (bitmap images, sound, video) would need to be treated specially, I think, unless we had a common structure format and protocol that would map to each of those. (The code is here: https://github.com/garden/ https://github.com/garden/)

Udo 12y ago

This is a good idea. Myself, I noticed that almost all my websockets projects could easily share one single code base, and finally I just made a Websockets boilerplate repo that I can pull from for any given project. This is what node.js really excels at, and it's an execution model fundamentally different from websocketd: being a message broker between the client and the request-based web server. Some people don't like the separation between web server and websockets server, but when you think about it they don't belong quite on the same level of abstraction. Plus, it's usually orders of magnitude easier to reason about single requests than to reason about a complex persistent application server's state.

jeremiep 12y ago

> Write programs to handle text streams, because that is a universal interface. I always hated that part of UNIX. It would be so much better if programs could handle data-structure streams instead. Having text streams causes every single program to implement ad-hoc parsers and serializers. I now use the command-line only for trivial one-time actions. For everything else I'll use a scripting language and forget about reading and writing text streams altogether.

chriswarbo 12y ago
The main problem is that there are many kinds of incompatible "data-structure streams". Using a scripting language doesn't solve the problem, it just standardises on one particular ad-hoc parser/serialiser combination (say, Python's pickle, for example). That's fine for personal use, or in a single project, but doesn't scale like "dumb" text does.
- nine_k 12y ago
  If there was a non-ideal but reasonable standard serialization format, like maybe IFF [1] or BSON [2], it would still simplify things, and common tools like `cat` might support decoding them. It would be easy to wrap an arbitrary other packed format inside a binary string, or an arbitrary text format inside a text string. IFF is quite successful in certain areas, BTW. [1]: http://en.wikipedia.org/wiki/Interchange_File_Format http://en.wikipedia.org/wiki/Interchange_File_Format [2]: http://en.wikipedia.org/wiki/BSON http://en.wikipedia.org/wiki/BSON
  - tracker1 12y ago
    You could always pass a message per line (so to speak) and use msgpack or json... if you want compression you could use json+gz Of course, you could also do something similar with a 0mq adapter as well for a request/response server, which is available pretty much everywhere you might need it (language wise)... It's an interesting generic interface... however as I mostly use node for newer development, it's so trivial to use sockets or websockets directly for this, I'd just assume go that route... or for that matter, something like sockjs/shoe.

deleted 12y ago

[deleted]

dkarapetyan 12y ago

This is cool. I can see this being used for prototyping and validating an idea before going full steam ahead with a more scalable framework.

jkarneges 12y ago

Elegant approach. Reminds me of inetd.

colanderman 12y ago
I'm surprised this isn't just implemented as an inetd handler, given the "do one thing and do it well" mantra.
- fit2rule 12y ago
  Its the 21st Century. People don't look backwards any more. inetd is passé..
  - colanderman 12y ago
    I'm assuming you're being sarcastic. This tool wants to "do things the Unix way"... which was invented in the 1970s.

spiralpolitik 12y ago

I assume you are treating `stdin` and `stdout` as just a stream of bytes rather than trying to do anything clever a la Python 3 ?

joewalnes 12y ago
Yep. The only magic thing is a line break (e.g. \n) is used to terminate each message.

cypher543 12y ago

> "Write programs that do one thing and do it well." So why does this WebSocket daemon also serve static files and CGI applications?

spoiler 12y ago
Because you need to implement a tiny HTTP server for WebSockets anyway (if you want't them to work from browsers). So, adding the ability to serve static files on top of that is trivial. Not sure why they added CGI, though.
- joewalnes 12y ago
  That indeed was a dilemma. Although it focusses on doing one thing well, there were also a few core things that are useful to support a websocket based app - namely a place to serve the actual web content from (static) and the ability to use websocketd using vanilla http instead of websockets (cgi).

nichochar 12y ago

Holy shit that's cool

ianbicking 12y ago

I don't understand how I'd use this. WebSockets are generally for server-initiated events, but this doesn't make it very easy to initiate events on the server. Usually it will be some pubsub situation, so I'd want a server process to be able to emit a message to one or many connections - but instead the connection is tied to one process for its life. I'd like to see an example like a simple chat room.

andrewstuart2 12y ago
Actually, I think you're looking for Server-Sent Events [1] for something of a pub/sub nature. WebSockets, on the other hand, are intended for realtime bidirectional communication over a single TCP connection. Pub/Sub is much more one-directional. A single subscribe event initiates a stream of all published events. Server-Sent Events are more suited for this because the server will never have to bother checking the TCP connection for incoming data. [1] http://www.w3.org/TR/2011/WD-eventsource-20110208/ http://www.w3.org/TR/2011/WD-eventsource-20110208/
tracker1 12y ago
The solution in question could well be used for pub/sub servers or for request/response patterns... it's really pretty cool. I'm needing to expose a very small utility that uses C# for the micro service... I'm considering this for that reason... aside from that, I think the solution in question is pretty cool.
joewalnes 12y ago
I've built chatrooms with websocketd. To do this, you a shared pub/sub store. Redis is great, so is Postgres (with its pub/sub add-ons). Storing chat data in a store is more durable than keeping it in memory on a single machine.

runewell 12y ago

This is perfect for admin tools. I'd rather parse stdout strings than code a custom API.

hurin 12y ago

Could anyone give an example of what this is used for?

runewell 12y ago
It appears to me that it is useful for eliminating the "middle man" scripts/code often written for web application and server interaction. If I want to create a web application that displays the status of my server (CPU, memory, I/O, etc) I can use this software to call vmstat directly instead of writing an API (PHP, Node.js, RoR, etc) that executes the same command or uses a library that requires even more code to get the exact same data.
- hurin 12y ago
  I'm sure this this is probably because I don't have do this for work - but from that description it's hard for me to tell what the advantages are. A script to feed vmstat (or other command data) with a python socket would be ~50 lines or so.
joewalnes 12y ago
It allows you to write a WebSocket end point, in any programming language, without having to deal with building socket servers, dealing with WebSocket protocol handshake/parsing/formatting/etc, worrying about threading etc. Just write a script that reads and writes to stdin/out.

curiously 12y ago

this is fucking cool. I got it instantly just glancing at the code example. I immediately understood how it worked. That is powerful stuff. My only worry of course is, how would you scale this up? What's really going beneath the hood. I'm really excited and trying to think of something so I can use it as an excuse to use this. The only other suggestion I would make is maybe change the name to something more catchy and brandable. Websocketd...okay like systemd...but I don't know, something as good as this deserves a brandable name like Jupiter, or some Greek goddess or clever hacky name.

joewalnes 12y ago
Thanks. And I suck at naming stuff.

aabajian 12y ago

This looks great, without knowing much I was able to get: ./websocketd --staticdir=. --port=8123 ps to give me a simple output of processes. What I'd really LOVE is to get this to work: ./websocketd --staticdir=. --port=8123 htop It'd be great if I could see the output of htop on the web...from anywhere. I guess htop is setting up a different video mode or something that isn't compatible?

olalonde 12y ago
I think you would need to feed what you are getting to a Javascript terminal emulator. (e.g. https://github.com/chjj/tty.js/ https://github.com/chjj/tty.js/)

romanpoet 12y ago

Why is this better than netcat? https://en.wikipedia.org/wiki/Netcat https://en.wikipedia.org/wiki/Netcat

colanderman 12y ago
Because it handles the WebSocket protocol for you.

Lerc 12y ago

I used this a while ago on my own insane project. I had a Webdav server and websocketd providing a Web interface to a Linux box. Even at it's young age (bug reports #5 and #7 were mine), it allowed me to progress a lot further in my project before I needed to write a server designed specifically for the task at hand. In the end I wrote userserv https://github.com/Lerc/userserv https://github.com/Lerc/userserv for the one must have feature I needed. I needed logins and responses delivered from a process with the UID of the CookieToken. So, thanks Joe. Notanos got further because of websocketd and while I'm not currently using it, There's a high chance of me doing so in future projects.

joewalnes 12y ago
Great to hear! Thanks :)

tomerbd 12y ago

beautiful website may I ask with which template / framework it was written? (is it in github by any chance?)

joewalnes 12y ago
Didn't use any templates or frameworks (apart from font-awesome for icons and prism for syntax highlighting the code examples). Just some good old fashioned hand written HTML + CSS. https://github.com/joewalnes/websocketd/tree/gh-pages https://github.com/joewalnes/websocketd/tree/gh-pages

vram22 12y ago

I had come across websocketd a while ago, and to try it out, wrote a simple Python program and a web page. The two posts about it are here: 1. Use WebSockets and Python for web-based system monitoring: http://jugad2.blogspot.in/2014/01/use-websockets-and-python-for-web-based.html http://jugad2.blogspot.in/2014/01/use-websockets-and-python-... 2. websocketd and Python for system monitoring - the JavaScript WebSocket client: http://jugad2.blogspot.in/2014/01/websocketd-and-python-for-system.html http://jugad2.blogspot.in/2014/01/websocketd-and-python-for-... Note: As it says in one of the posts, you have to: set PYTHONUNBUFFERED=true for the Python program to work as a websocket server, though it works fine without that if only run directly at the command line (without websocketd). Thanks to Joe for pointing this out. websocketd is a nice utility.

MatthewWilkes 12y ago

It's a petty comment, I know, but: for count in range(0, 10): print count + 1 sleep(0.5) makes me feel sad.

asergeyev 12y ago
Please ignore complexity of supplied examples... It's really not for that kind of use. I have lots of scripts that do something like this (and most of them are not in bash :)...): while read ARG; do if validate $ARG; then run_something $ARG fi done Most of my "validate" pieces are bash functions and most "run_something" return lengthy CSV datablocks. Then, there is the javascript that uses user mouse (and other signals) to generate arguments to send to websocketd and draw pretty visuals based on data that arrives back.

kalmi10 12y ago

How are message boundaries handled? WebSockets are message-based. UNIX streams are not.

joewalnes 12y ago
websocketd treats each line as a message. i.e. boundaries are marked with \n
joewalnes 12y ago
websocketd treats each line as a message. i.e. boundaries are marked with \n

kragen 12y ago

> Each inbound WebSocket connection runs your program in a dedicated process. Connections are isolated by process. That sounds bad; it is like “CGI, twenty years later”, as they say. In 2000 at KnowNow, we were able to support over ten thousand concurrent Comet connections using a hacked-up version of thttpd, on a 1GHz CPU with 1GiB of RAM. I’ll be surprised if you can support ten thousand Comet connections using WebSockets and websocketd even on a modern machine, say, with a quad-core 3GHz CPU and 32GiB of RAM. Why would you want ten thousand concurrent connections? Well, normal non-Comet HTTP is pretty amazingly lightweight on the server side, due to REST. Taking an extreme example, this HN discussion page takes 5 requests to load, which takes about a second, but much of that is network latency — a total of maybe ½s of time on the server side. But it contains 7000 words to read, which takes about 2048 seconds. So a single process or thread on the server can handle about 4096 concurrent HN readers. So a relatively normal machine can handle hundreds of thousands of concurrent users without breaking a sweat. On the other hand, Linux has gotten a lot better since 2000 at managing large numbers of runnable processes and doing things like fork and exit. httpdito (http://canonical.org/~kragen/sw/dev3/server.s http://canonical.org/~kragen/sw/dev3/server.s) can handle tens of thousands of hits on a single machine nowadays, even though each hit forks a new child process (which then exits). http://canonical.org/~kragen/sw/dev3/httpdito-readme http://canonical.org/~kragen/sw/dev3/httpdito-readme has more performance notes. On the gripping hand, httpdito’s virtual memory size is up to 16kiB, so Linux may be able to handle httpdito processes better than regular processes.

asergeyev 12y ago
Difference with cgi is that they are live and die, with each user request. websocketd programs are more like "one per user session"... Makes sense when your user sessions are lengthy.
- kragen 12y ago
  Yes — that’s exactly the problem. Handling ten thousand concurrent users with CGI is easy, even on 2000-era hardware. Ten thousand concurrent users might be four requests per second. But ten thousand concurrent users using websocketd means you have ten thousand processes. And if you’re doing some kind of pub/sub thing, every once in a while, all of those processes will go into the run queue at once, because there’s a message for all of them. Have you ever administered a server with a load average over 1000? Still, the O(N) scheduler work in current Linux might make that kind of thing survivable.