6 ms·

Websockets 101

289 points by the_mitsuhiko 14y ago

romaniv 14y ago
Why do standards like WebSockets don't include anything that can be used in pure HTML? This would be a great tool for building efficient dynamic apps, if you could "submit" forms without going to a new page, and then get incremental page updates as a response.
- j2labs 14y ago
  Because it's a sockets layer. You could very easily send HTML over it if you want.
- dbaupp 14y ago
  What is so bad about a little Javascript? And, how does one get sufficiently customisable incremental updates without having to standardise (and implement) a huge complex new language to control them?
  - romaniv 14y ago
    What is so bad about a little Javascript? What's so bad about having a declarative way of doing something that a lot of people will find useful? And, how does one get sufficiently customisable incremental updates without having to standardise (and implement) a huge complex new language to control them? By using element IDs. Every element in the incoming frame that has and ID that matches and ID of an existing element replaces it in the document. People have been doing this for years. It's called AHAH (Asychronous HTML and HTTP). Doing this without any custom code and with every update being as cheap as sending WebSockets frame would be absolutely awesome. It could be used to create very sophisticated and dynamic applications with very little complexity either on the client or on the server side.
- xionon 14y ago
  I'm not sure I see the use-case there as being more useful than JS based WebSockets. Writing a general JS library to do what you suggest would be pretty straightforward, and the server-side work wouldn't be any different (serving partial HTML based on request type). If your users are worried about security / performance enough to turn off JavaScript, they probably wouldn't want "HTMLSockets" enabled, either. If your users are browsing with a browser so old it doesn't have a reasonable JavaScript engine, then it wouldn't support "HTMLSockets," either. Basically, there's nothing you could do with "HTMLSockets" that you can't do with WebSockets, and with only a small layer to make it so. However, there's a whole host of utility that you get from WebSockets that you could never get from HTMLSockets, or would require a huge amount of hacking in order to get it to work. And all this is beside the "pureness" argument, where one might say that HTML is for data, CSS is for design, JavaScript is for behavior; WebSockets are behavior, not data.
  - romaniv 14y ago
    This is like asking why we should bother having any (new?) HTML or CSS standards at all, when you could write a custom rendering engine in JavaScript for every website. There is tremendous value in having a solid, usable foundation for web development that doesn't rely on an imperative language to work. To start with, declarative features are semantic and allow for semantic upgrades on wide-scale basis. Do you like your Firefox/Chrome spell-checking in text areas? Do you like them being resizable? Well, this wouldn't be possible if all textareas were some JavaScript hack. (Hey, with only small layer you could actually fake a textarea. Does that seem like a good idea in retrospect, though?) Also, it would be great to be able to develop simple dynamic web applications (or prototypes) without doing something complex. Progressive enhancement is buried far too early. If your users are worried about security / performance enough to turn off JavaScript, they probably wouldn't want "HTMLSockets" enabled, either. I frequently browse with JavaScript blocked by default, and I can tell you right here that this assumption is invalid. There is a world of difference between JavaScript library and a standard, declarative technology. The latter is more likely to be faster, have less bugs, no side-effects and be secure.
- TazeTSchnitzel 14y ago
  This exists in Firefox. HTTP Server Push. https://en.wikipedia.org/wiki/Push_technology#HTTP_server_push https://en.wikipedia.org/wiki/Push_technology#HTTP_server_pu...
btmorex 14y ago
If anyone finds themselves needing to write a Websockets implementation, there's an awesome protocol test suite at http://autobahn.ws/testsuite http://autobahn.ws/testsuite I wrote a C++ implementation for a side project and with the aforementioned test suite I actually found it pretty easy to get to 100% compliance. There's some ugliness in the protocol because of proxies, but it's definitely not the worst protocol in the world. The only big missing feature is compression and there's a proposal for that (you could certainly do application level compression, but I'd rather avoid writing compression code in JS).
zafriedman 14y ago
An even quicker introduction to the final specification of WebSockets: http://socket.io http://socket.io. P.S. I'm aware that a) this doesn't help anyone not using Node.js on their server (it's not even part of my production stack at work (yet!), even though I'm bringing it to light here) and b) it's more than just WebSockets, for instance it will gracefully degrade on legacy browsers. I just can't miss an opportunity to sing its praises because it has so many benefits over the simple implementation.
- chrisduesing 14y ago
  It is not actually limited to Node. https://github.com/learnboost/socket.io/wiki https://github.com/learnboost/socket.io/wiki Though some of these libs do tend to lag in version support.
peterwwillis 14y ago
Encryption is not a panacea. As more websites use HTTPS, more corporate networks are installing content filters that passively filter HTTPS content. Yes. I just said corporate networks are spying on HTTPS connections. It's really simple to implement. Get some kind of web proxy that has support (Websense is pretty popular, amazingly) and generate a root cert on the box. Then use a group policy on your AD server to distribute the root cert to all the client systems. Now Websense can decode the HTTPS traffic by issuing its own fake certs to clients and handle the "real" HTTPS handshake on the frontend proxy side. Result? Your websockets are still going to get fucked with by proxies. Can we please stop building fake protocols on top of real protocols now?
- the_mitsuhiko 14y ago
  This won't work with our games. We verify that the certificate was created by our CA and reject anything else. Someone would have to patch the game to bypass that. It also won't work on mobile phones that people use themselves.
  - peterwwillis 14y ago
    How? Is this some JavaScript magic i'm unaware of? Because the actual connection from the proxy to your site uses your site's valid certificates; only the client to the proxy uses a "self-signed" one. Are you using client certificates?
    - the_mitsuhiko 14y ago
      We're more in the area of things that actually are not websites than websites. The websocket stuff I added as an experiment and if people want to use our stuff to make HTML5 applications without a proxy server in between. If it's indeed between browser and website someone could MITM it of course if that person can also give you a fake security certificate into your browser. But on the large scale that will never happen. Maybe in corporate networks but I doubt that this will become widespread.
      - 14y ago
jschrf 14y ago
Excellent article, thank you for sharing. As someone who has implemented a WS server and client and would like to be able to host them in Amazon's cloud, the notes about ELB are worrysome.
- gabrtv 14y ago
  Websockets through ELB works fine if you use the TCP listener. You lose out on some HTTP-specific features like Cookie-based session stickiness, but it works great otherwise. We've been using Websockets/ELB in production for almost a year now.
tinco 14y ago
Very nice readable article. Having dabbled with implementation a bit I agree with most of his points. The author complains about websockets being frame-based instead of stream-based, because he argues if he would want a frame-based system he could easily model it on top of it. I think frames are awesome. It is easier to build a stream on top of frames than it is to build frames on top of a stream in my opinion (you don't need an extra protocol for starters). Besides that almost anything you would like to do can be expressed in frames, saving most of us a lot of headaches. I would like to add that websockets are not the end-all answer for web-game programmers. It makes some stuff easier, and reduces some overhead, but it's still on TCP, meaning realtime multiplayer games are still as good as impossible to implement. Sadly all the issues with proxies go for UDP and then some, so I don't think we'll see any ubiquitous browser UDP protocol anytime soon :(
- the_mitsuhiko 14y ago
  > I think frames are awesome. It is easier to build a stream on top of frames than it is to build frames on top of a stream in my opinion (you don't need an extra protocol for starters). Besides that almost anything you would like to do can be expressed in frames, saving most of us a lot of headaches. Maybe I'm just old fashioned but I really don't like the idea of frames forced onto me. I much rather get my data in small pieces and can already start processing data as it comes in than having something buffer up everything and then giving it me as a large chunk. Streaming json for instance over websockets is a lot harder than TCP directly.
  - ishbits 14y ago
    IMO the built in framing is one of the things I really like about websockets. I'm sure it will become my transport of choice for non performance critical messaging even when a browser is not one of the peers.
- wmf 14y ago
  WebRTC is already there if you want UDP.
  - 14y ago
TazeTSchnitzel 14y ago
(Shameless plug...) I have a list of websocket libraries and a small echo server demo here: http://ajf.me/websocket/#libs http://ajf.me/websocket/#libs
- kibwen 14y ago
  That first link there to your Github repo is broken: https://github.com/TazeTSchnitzel/WebSocket.us https://github.com/TazeTSchnitzel/WebSocket.us
  - TazeTSchnitzel 14y ago
    Oops. I moved it to my main website's repo: https://github.com/TazeTSchnitzel/ajf.me/tree/master/htdocs/websocket https://github.com/TazeTSchnitzel/ajf.me/tree/master/htdocs/...
bowmessage 14y ago
Loving the article! Just thought I'd point out: "needles complexity"
rizwan 14y ago
For folks thinking about using Socket.IO for websockets, it works very well, but remember that most open-source implementations ONLY implement WebSockets and NONE of the other fall-backs that are in the Socket.IO spec. So without some work, these libraries are often useless for mobile development, for instance.
- digisth 14y ago
  I'm using SockJS (client and server: https://github.com/sockjs/sockjs-client https://github.com/sockjs/sockjs-client) for project I'm working on, which does implement a variety of fallback methods (iframe tricks, JSON polling, etc.) and it's been working pretty well (some work better than others.) For those who to opt for these libs, I recommend the following for protocols/fallbacks: protocols_whitelist: ['websocket', 'xhr-streaming', 'iframe-eventsource', 'iframe-htmlfile', 'xhr-polling', 'iframe-xhr-polling'] This gets you a working system in pretty much every modern browser supporting WS, and /usually/ works pretty well in IE8/9. Just try to avoid page refreshes in those two browsers (very spotty results if you don't - I'd recommend putting up a message for your users) and implement the workarounds for finicky browsers (things like catching ESC in Firefox.) I've been pretty happy with it so far.
kaiserama 14y ago
Pop-up Video Excerpt: The author (Armin Ronacher) also authored Flask (Python web micro framework) and its template engine Jinja2. I've used Flask for a number of my most recent projects and really have enjoyed working with it. Back to your scheduled broadcast!
chadrs 14y ago
Man, I would have loved to have this about a year ago. Although then I would have missed out on some of the hilarious drama on the websocket mailing list (e.g. http://www.ietf.org/mail-archive/web/hybi/current/msg08668.html http://www.ietf.org/mail-archive/web/hybi/current/msg08668.h...)
Evbn 14y ago
> Messages sent from the client to the server are obfuscated with a basic transmission mask of 4 bytes which is sent within each websocket package. What is this? Some pretend security feature?
- pjscott 14y ago
  The purpose of the masking is to prevent malicious applications from being able to send arbitrary bytes over the wire, for fear that they may be able to screw with buggy proxies.
jorangreef 14y ago
Browser vendors must start to move away from top-down innovation, where they hoard APIs such as TCP and UDP and release sub-standard specs in their stead. Instead they must expose only the bare minimum OS low-level APIs (TCP, UDP, POSIX), keeping the surface area as small and powerful and direct as possible, and then let open source grow around this. Innovation needs to be decentralized, bottom-up. Not designed by committee. For that to happen, we need to stop conflating "web apps" (trusted, installed, annointed with machine power) with "web pages" (accessed by single link click). At present, Web Apps are suffering and crippled by being lumped under the same security policy as web pages. But Web Apps need to have access to raw machine resources in the same way that Native Apps have access. Those that don't seem to care for any of this insistence, tend also to be naive as to the massive differences between TCP and WebSockets, and IP and TCP and the whole stack in general. The WebSocket spec is a good example of people doing things in the most indirect way possible, with a maximum of red tape, as opposed to people doing things in the most direct way possible, with a minimum of red tape. The Web as we have it in these respects is very much Animal Farm and 1984. There appears to be little thought leadership from the major stakeholders in this regard. People like Tim Berners-Lee are asking for change (http://lists.w3.org/Archives/Public/public-webapps/2012JanMar/0464.html http://lists.w3.org/Archives/Public/public-webapps/2012JanMa...), but the new incumbents don't seem to want to see.
- milkshakes 14y ago
  I don't understand the point you're trying to make. Are you against standardization? What specific actions would you take to "minimize red tape"? You do know that TCP, UDP, and POSIX were all developed by committees right? How exactly is the Web in any way like Animal Farm or 1984?
  - jorangreef 14y ago
    Standardisation for the web targets too much surface area. Browser vendors need to agree on a smaller set of more powerful, more dangerous, more direct APIs so as to put the responsibility for innovation back in the hands of developers. Developers are being wedged further and further away from bare metal, with no right to return, even if they should so wish - this process needs to be reversed.

289 points by the_mitsuhiko 14y ago

Why do standards like WebSockets don't include anything that can be used in pure HTML? This would be a great tool for building efficient dynamic apps, if you could "submit" forms without going to a new page, and then get incremental page updates as a response.

j2labs 14y ago
Because it's a sockets layer. You could very easily send HTML over it if you want.
dbaupp 14y ago
What is so bad about a little Javascript? And, how does one get sufficiently customisable incremental updates without having to standardise (and implement) a huge complex new language to control them?
- romaniv 14y ago
  What is so bad about a little Javascript? What's so bad about having a declarative way of doing something that a lot of people will find useful? And, how does one get sufficiently customisable incremental updates without having to standardise (and implement) a huge complex new language to control them? By using element IDs. Every element in the incoming frame that has and ID that matches and ID of an existing element replaces it in the document. People have been doing this for years. It's called AHAH (Asychronous HTML and HTTP). Doing this without any custom code and with every update being as cheap as sending WebSockets frame would be absolutely awesome. It could be used to create very sophisticated and dynamic applications with very little complexity either on the client or on the server side.
xionon 14y ago
I'm not sure I see the use-case there as being more useful than JS based WebSockets. Writing a general JS library to do what you suggest would be pretty straightforward, and the server-side work wouldn't be any different (serving partial HTML based on request type). If your users are worried about security / performance enough to turn off JavaScript, they probably wouldn't want "HTMLSockets" enabled, either. If your users are browsing with a browser so old it doesn't have a reasonable JavaScript engine, then it wouldn't support "HTMLSockets," either. Basically, there's nothing you could do with "HTMLSockets" that you can't do with WebSockets, and with only a small layer to make it so. However, there's a whole host of utility that you get from WebSockets that you could never get from HTMLSockets, or would require a huge amount of hacking in order to get it to work. And all this is beside the "pureness" argument, where one might say that HTML is for data, CSS is for design, JavaScript is for behavior; WebSockets are behavior, not data.
- romaniv 14y ago
  This is like asking why we should bother having any (new?) HTML or CSS standards at all, when you could write a custom rendering engine in JavaScript for every website. There is tremendous value in having a solid, usable foundation for web development that doesn't rely on an imperative language to work. To start with, declarative features are semantic and allow for semantic upgrades on wide-scale basis. Do you like your Firefox/Chrome spell-checking in text areas? Do you like them being resizable? Well, this wouldn't be possible if all textareas were some JavaScript hack. (Hey, with only small layer you could actually fake a textarea. Does that seem like a good idea in retrospect, though?) Also, it would be great to be able to develop simple dynamic web applications (or prototypes) without doing something complex. Progressive enhancement is buried far too early. If your users are worried about security / performance enough to turn off JavaScript, they probably wouldn't want "HTMLSockets" enabled, either. I frequently browse with JavaScript blocked by default, and I can tell you right here that this assumption is invalid. There is a world of difference between JavaScript library and a standard, declarative technology. The latter is more likely to be faster, have less bugs, no side-effects and be secure.
TazeTSchnitzel 14y ago
This exists in Firefox. HTTP Server Push. https://en.wikipedia.org/wiki/Push_technology#HTTP_server_push https://en.wikipedia.org/wiki/Push_technology#HTTP_server_pu...

btmorex 14y ago

If anyone finds themselves needing to write a Websockets implementation, there's an awesome protocol test suite at http://autobahn.ws/testsuite http://autobahn.ws/testsuite I wrote a C++ implementation for a side project and with the aforementioned test suite I actually found it pretty easy to get to 100% compliance. There's some ugliness in the protocol because of proxies, but it's definitely not the worst protocol in the world. The only big missing feature is compression and there's a proposal for that (you could certainly do application level compression, but I'd rather avoid writing compression code in JS).

zafriedman 14y ago

An even quicker introduction to the final specification of WebSockets: http://socket.io http://socket.io. P.S. I'm aware that a) this doesn't help anyone not using Node.js on their server (it's not even part of my production stack at work (yet!), even though I'm bringing it to light here) and b) it's more than just WebSockets, for instance it will gracefully degrade on legacy browsers. I just can't miss an opportunity to sing its praises because it has so many benefits over the simple implementation.

chrisduesing 14y ago
It is not actually limited to Node. https://github.com/learnboost/socket.io/wiki https://github.com/learnboost/socket.io/wiki Though some of these libs do tend to lag in version support.

peterwwillis 14y ago

Encryption is not a panacea. As more websites use HTTPS, more corporate networks are installing content filters that passively filter HTTPS content. Yes. I just said corporate networks are spying on HTTPS connections. It's really simple to implement. Get some kind of web proxy that has support (Websense is pretty popular, amazingly) and generate a root cert on the box. Then use a group policy on your AD server to distribute the root cert to all the client systems. Now Websense can decode the HTTPS traffic by issuing its own fake certs to clients and handle the "real" HTTPS handshake on the frontend proxy side. Result? Your websockets are still going to get fucked with by proxies. Can we please stop building fake protocols on top of real protocols now?

the_mitsuhiko 14y ago
This won't work with our games. We verify that the certificate was created by our CA and reject anything else. Someone would have to patch the game to bypass that. It also won't work on mobile phones that people use themselves.
- peterwwillis 14y ago
  How? Is this some JavaScript magic i'm unaware of? Because the actual connection from the proxy to your site uses your site's valid certificates; only the client to the proxy uses a "self-signed" one. Are you using client certificates?
  - the_mitsuhiko 14y ago
    We're more in the area of things that actually are not websites than websites. The websocket stuff I added as an experiment and if people want to use our stuff to make HTML5 applications without a proxy server in between. If it's indeed between browser and website someone could MITM it of course if that person can also give you a fake security certificate into your browser. But on the large scale that will never happen. Maybe in corporate networks but I doubt that this will become widespread.
    - 14y ago

jschrf 14y ago

Excellent article, thank you for sharing. As someone who has implemented a WS server and client and would like to be able to host them in Amazon's cloud, the notes about ELB are worrysome.

gabrtv 14y ago
Websockets through ELB works fine if you use the TCP listener. You lose out on some HTTP-specific features like Cookie-based session stickiness, but it works great otherwise. We've been using Websockets/ELB in production for almost a year now.

tinco 14y ago

Very nice readable article. Having dabbled with implementation a bit I agree with most of his points. The author complains about websockets being frame-based instead of stream-based, because he argues if he would want a frame-based system he could easily model it on top of it. I think frames are awesome. It is easier to build a stream on top of frames than it is to build frames on top of a stream in my opinion (you don't need an extra protocol for starters). Besides that almost anything you would like to do can be expressed in frames, saving most of us a lot of headaches. I would like to add that websockets are not the end-all answer for web-game programmers. It makes some stuff easier, and reduces some overhead, but it's still on TCP, meaning realtime multiplayer games are still as good as impossible to implement. Sadly all the issues with proxies go for UDP and then some, so I don't think we'll see any ubiquitous browser UDP protocol anytime soon :(

the_mitsuhiko 14y ago
> I think frames are awesome. It is easier to build a stream on top of frames than it is to build frames on top of a stream in my opinion (you don't need an extra protocol for starters). Besides that almost anything you would like to do can be expressed in frames, saving most of us a lot of headaches. Maybe I'm just old fashioned but I really don't like the idea of frames forced onto me. I much rather get my data in small pieces and can already start processing data as it comes in than having something buffer up everything and then giving it me as a large chunk. Streaming json for instance over websockets is a lot harder than TCP directly.
- ishbits 14y ago
  IMO the built in framing is one of the things I really like about websockets. I'm sure it will become my transport of choice for non performance critical messaging even when a browser is not one of the peers.
wmf 14y ago
WebRTC is already there if you want UDP.
- 14y ago

TazeTSchnitzel 14y ago

(Shameless plug...) I have a list of websocket libraries and a small echo server demo here: http://ajf.me/websocket/#libs http://ajf.me/websocket/#libs

kibwen 14y ago
That first link there to your Github repo is broken: https://github.com/TazeTSchnitzel/WebSocket.us https://github.com/TazeTSchnitzel/WebSocket.us
- TazeTSchnitzel 14y ago
  Oops. I moved it to my main website's repo: https://github.com/TazeTSchnitzel/ajf.me/tree/master/htdocs/websocket https://github.com/TazeTSchnitzel/ajf.me/tree/master/htdocs/...

bowmessage 14y ago

Loving the article! Just thought I'd point out: "needles complexity"

rizwan 14y ago

For folks thinking about using Socket.IO for websockets, it works very well, but remember that most open-source implementations ONLY implement WebSockets and NONE of the other fall-backs that are in the Socket.IO spec. So without some work, these libraries are often useless for mobile development, for instance.

digisth 14y ago
I'm using SockJS (client and server: https://github.com/sockjs/sockjs-client https://github.com/sockjs/sockjs-client) for project I'm working on, which does implement a variety of fallback methods (iframe tricks, JSON polling, etc.) and it's been working pretty well (some work better than others.) For those who to opt for these libs, I recommend the following for protocols/fallbacks: protocols_whitelist: ['websocket', 'xhr-streaming', 'iframe-eventsource', 'iframe-htmlfile', 'xhr-polling', 'iframe-xhr-polling'] This gets you a working system in pretty much every modern browser supporting WS, and /usually/ works pretty well in IE8/9. Just try to avoid page refreshes in those two browsers (very spotty results if you don't - I'd recommend putting up a message for your users) and implement the workarounds for finicky browsers (things like catching ESC in Firefox.) I've been pretty happy with it so far.

kaiserama 14y ago

Pop-up Video Excerpt: The author (Armin Ronacher) also authored Flask (Python web micro framework) and its template engine Jinja2. I've used Flask for a number of my most recent projects and really have enjoyed working with it. Back to your scheduled broadcast!

chadrs 14y ago

Man, I would have loved to have this about a year ago. Although then I would have missed out on some of the hilarious drama on the websocket mailing list (e.g. http://www.ietf.org/mail-archive/web/hybi/current/msg08668.html http://www.ietf.org/mail-archive/web/hybi/current/msg08668.h...)

Evbn 14y ago

> Messages sent from the client to the server are obfuscated with a basic transmission mask of 4 bytes which is sent within each websocket package. What is this? Some pretend security feature?

pjscott 14y ago
The purpose of the masking is to prevent malicious applications from being able to send arbitrary bytes over the wire, for fear that they may be able to screw with buggy proxies.

jorangreef 14y ago

Browser vendors must start to move away from top-down innovation, where they hoard APIs such as TCP and UDP and release sub-standard specs in their stead. Instead they must expose only the bare minimum OS low-level APIs (TCP, UDP, POSIX), keeping the surface area as small and powerful and direct as possible, and then let open source grow around this. Innovation needs to be decentralized, bottom-up. Not designed by committee. For that to happen, we need to stop conflating "web apps" (trusted, installed, annointed with machine power) with "web pages" (accessed by single link click). At present, Web Apps are suffering and crippled by being lumped under the same security policy as web pages. But Web Apps need to have access to raw machine resources in the same way that Native Apps have access. Those that don't seem to care for any of this insistence, tend also to be naive as to the massive differences between TCP and WebSockets, and IP and TCP and the whole stack in general. The WebSocket spec is a good example of people doing things in the most indirect way possible, with a maximum of red tape, as opposed to people doing things in the most direct way possible, with a minimum of red tape. The Web as we have it in these respects is very much Animal Farm and 1984. There appears to be little thought leadership from the major stakeholders in this regard. People like Tim Berners-Lee are asking for change (http://lists.w3.org/Archives/Public/public-webapps/2012JanMar/0464.html http://lists.w3.org/Archives/Public/public-webapps/2012JanMa...), but the new incumbents don't seem to want to see.

milkshakes 14y ago
I don't understand the point you're trying to make. Are you against standardization? What specific actions would you take to "minimize red tape"? You do know that TCP, UDP, and POSIX were all developed by committees right? How exactly is the Web in any way like Animal Farm or 1984?
- jorangreef 14y ago
  Standardisation for the web targets too much surface area. Browser vendors need to agree on a smaller set of more powerful, more dangerous, more direct APIs so as to put the responsibility for innovation back in the hands of developers. Developers are being wedged further and further away from bare metal, with no right to return, even if they should so wish - this process needs to be reversed.