6 ms·

I have thoughts, and I agree but also disagree a bit. We are in the dark-ages with respect to streaming sockets, and I am the architect of one of the worlds la

I have thoughts, and I agree but also disagree a bit. We are in the dark-ages with respect to streaming sockets, and I am the architect of one of the worlds largest WebSocket and streaming services. First off, unseating the operational benefits provided by request response is a huge challenge. It's possible, and I've done it. One of these days, my co-authors and I will present an OSDI paper with the broad strokes. It's very easy to list a lot of the challenges with scaling WebSocket, but each of them can be solved. I believe we will have a nicer world with QUIC --> Http/3 and the up and coming WebTransport discussion. SADLY, that will result in a decade long "caniuse" issue, so we should expect to see WebTransport polyfills to arrive... Second, HTML over WebSockets is nice, but what is better is reactive data binding over the socket. Let the browser maintain a DOM tree that templates over a JSON object, then reactive-ly update and it's an amazing experience. You have minimal data transfer, and minimal rendering updates. There is a great deal of power in having your UI be a stateless function over a giant JSON object. Finally, the big challenge is dealing with the server state in a meaningful way. The exceptionally nice property of using request response for everything is the shared-nothing stateless nature of the web server. It's exceptionally forgiving in both operational and developer issues. It's very easy to fuck up the implicit state tied to the connection, and this requires discipline to get right. It's very easy to ship unreliable software that then has issues based on the connection. Now, I'm a HUGE believer in that we need to use streams for everything, but it requires a mind shift. The key thing to tackle is dealing with the server side state, and this is where I have my shitty idea of a programming language for board games. In essence, I am building a DIY database such that people connect directly to the database and get that sweet giant JSON object along with a stream of updates. The DIY part is related to the fact that I invented a programming language to transact on the JSON object such that you get all the benefits of durability with the mental model of working within a single machine. http://www.adama-lang.org/ http://www.adama-lang.org/

rraihansaputra 6y ago
> Second, HTML over WebSockets is nice, but what is better is reactive data binding over the socket. Let the browser maintain a DOM tree that templates over a JSON object, then reactive-ly update and it's an amazing experience. You have minimal data transfer, and minimal rendering updates. There is a great deal of power in having your UI be a stateless function over a giant JSON object. I think a lot of the "best-practices" for SPA is trying to do this. But as streaming the data transfer is not yet common, the data is abstracted as a giant JSON object to be a parameter of the UI as a stateless function (React+Redux/MobX/etc, Vue+VueX). When enforced correctly on the project level, what's complicated _is_ syncing the data to the server. I think you'll like Supabase's subscription approach to reactively bind the JSON for the client. I think a separation between the info-web (document-content heavy) and the app-web (needing stateful data sync) will be more clearly defined in the next few years. Nobody likes trying to sync data between the client and server over HTTP calls; we just have to wait for streaming to be the norm for the app-web (and native apps, ofcourse).
methyl 6y ago
> Let the browser maintain a DOM tree that templates over a JSON object, then reactive-ly update and it's an amazing experience Totally agreed. I even pulled off an Elixir library as a POC to show this concept, here's an example project: https://github.com/surferseo/live_data/tree/master/examples/phoenix_livedata_todomvc https://github.com/surferseo/live_data/tree/master/examples/... (most relevant part of API is here: https://github.com/surferseo/live_data/blob/master/examples/phoenix_livedata_todomvc/assets/js/containers/TodoList.js#L22 https://github.com/surferseo/live_data/blob/master/examples/... and here: https://github.com/surferseo/live_data/blob/master/examples/phoenix_livedata_todomvc/lib/phoenix_livedata_todomvc_web/live_data/todo_state.ex https://github.com/surferseo/live_data/blob/master/examples/...)
hfjtktkf 6y ago
I am doing something somewhat similar - I have a Postgres database and I send all the user tables to the client on connection and then stream all the updates. I use Vue and Vuex-ORM to make the client tables reactive, and put a view on that. When something changes in the database, the client UI will also update. It is indeed a very powerful and reliable way of ensuring synced state.
- mxxx 6y ago
  Sounds like you’re describing a system similar to Meteor.
- qmmmur 6y ago
  I do the same for a collaborative music app. User gets the state of a session on load as a JSON object then just received the diffs. It's pretty efficient.
RangerScience 6y ago
Got a question for you - I'm making a chat bot thing (https://www.brian.bot/ https://www.brian.bot/) and have added socket-based web chat... but I don't even know how to phrase the question of "how do I horizontally scale the socket connections". Webchat <-> Server <-> Slack. Once I have more than one server dyno (heroku), then I can't ensure that the Slack event hits the dyno with the corresponding socket connection. Thoughts / tips?
- mathgladiator 6y ago
  I'll be completely honest... you're mostly screwed. I say this with full love in my heart. Your best bet is to use Amazon and then leverage their ELB offering with sticky routing. HOWEVER, now you have a new problem as capacity dies, gets removed, cycled, added, etc. Sticky routing was designed with the assumption that it was an efficiency play as opposed to a deterministic play. What you will see happen is that your internal state will be split brain, and people joining the chat will not be globally consistent. Some people will be on host X while some will be on host Y. So, you have two choices. You could use a database to provide the global state, at which point now you have a whole bunch of problems. You can start with polling the database, and the key advantage of the websocket is moving the polling from client to server, so that will be a win for users. However, it will cost you. Second, you could introduce a message broker, but then you have the problem of how to initialize the state. Ultimately, everyone ends up with a hybrid database for storage and message broker for real-time. In your model, the database becomes like Slack, and your Server becomes a proxy. This could be super neat, but what is the value-add of the Server? The interesting thing to observe is that the solution is how to turn your front-end app into a database like thing. If the load balancer could route traffic like a database traffic controller thing, then you can put all your state in the front-end and move crazy fast. However, you will move so fast that you will fuck up your state. State is VERY HARD and exceptionally unforgiving to manage yourself. There is great wisdom in using a database with versioning and all that. It’s a great abstraction having lasted so long. I’m working on the state bits and trying to figure out the people side of discipline with my silly language, and I have some novel contributions to provide over the coming years. There are some linked pdfs on http://www.adama-lang.org/blog/some-thinky-thoughts-2021 http://www.adama-lang.org/blog/some-thinky-thoughts-2021 which may provide some more insight.
anderspitman 6y ago
> HTML over WebSockets is nice, but what is better is reactive data binding over the socket. Let the browser maintain a DOM tree that templates over a JSON object, then reactive-ly update and it's an amazing experience. You have minimal data transfer, and minimal rendering updates. There is a great deal of power in having your UI be a stateless function over a giant JSON object. Interesting that you're exploring this idea in the context of board games. I implemented[0] a similar idea while making a browser-based magic the gathering interface. Basically send commands to update the server state (a big JSON object), and the server does state diffs per client and sends lz compressed JSON patches. Worked quite well, but I haven't played with it for a couple years. [0]: https://github.com/anderspitman/pojo_flow https://github.com/anderspitman/pojo_flow
brundolf 6y ago
> The key thing to tackle is dealing with the server side state The core problem of web UIs has always been managing state. People like statically-rendered pages because most state gets kept in a) the URL, and b) transient HTML elements, and both are highly visible and easy to deal with in robust ways. When you're forced to have server-side state it's exceptionally painful, but people usually just avoid it altogether which in most cases pushes things towards the virtue of having less state anyway. For complex UIs with lots of little interactions the above story doesn't really work, so we've invented state management systems for the client. Managing complex state still sucks, but it sucks less with these tools. Sounds like this approach is just moving that state store back to the server, with a greased pipeline so it isn't so horribly painful to add new pieces of state even though both sides are involved. But you'll still face all the same problems you face with a front-end store: transientness, cache-clearing, even modeling is hard when you're dealing with a big blob of mutable state. And there will be the additional challenge of associating this stuff with a user (which you get for free on the front-end, even with modern state stores). > Let the browser maintain a DOM tree that templates over a JSON object, then reactive-ly update and it's an amazing experience. You have minimal data transfer, and minimal rendering updates. There is a great deal of power in having your UI be a stateless function over a giant JSON object. > this is where I have my shitty idea of a programming language for board games It's funny, I started building a web-based board game to play with family during quarantine, and I landed on this exact approach totally by coincidence :) I'm using polling requests instead of websockets because I'm lazy/it doesn't need to scale, but it's the exact same idea where the server just publishes a giant JSON object of state and the clients are pure functions of it (and send messages to the server to mutate it). I wonder if there's some aspect of board games that pushes one towards that line of thinking.
- jamesjyu 6y ago
  > I'm using polling requests instead of websockets because I'm lazy/it doesn't need to scale, but it's the exact same idea where the server just publishes a giant JSON object of state and the clients are pure functions of it (and send messages to the server to mutate it). I like this approach as well, but I do worry that users today expect optimistic updates on the client, and the injection of a X00ms delay would make the app feel sluggish.
radicalbyte 6y ago
This sounds like what Microsoft were trying to do 15 years ago with webforms: the browser was treated like a dumb terminal with the UI rendered on the server and pushed to the client. The UX back then was horrible, however technology has come a long way since.
djedr 6y ago
Interesting ideas, somewhat in line with what I've been thinking about. Perhaps you or someone in this topic might be interested in my little project[1] fit for innovative ideas like this. It is not very far from being available for experimental use in place of JSON. A future direction is experimenting with replacing HTML[2]. [1] https://www.tree-annotation.org/ https://www.tree-annotation.org/ [2] https://imgur.com/a/ER5qwtZ https://imgur.com/a/ER5qwtZ
jlokier 6y ago
The architecture you're describing is awesome. It sounds like the one I was working with ~18 years ago. Except it used HTTP (long polling or comet) instead of HTTP/3 for the transport. And more partial rendering to DOM-diffs on the server side because it was faster than doing it all on the client, but the client still applied diffs. The issues with full DOM state synchronisation, handling network errors and recovery, transport of templates and data, minimisation of transport subject to maximising responsiveness, some amount of anticipatory prefetching, were much the same then as now. A lovely thing about all this stateful streaming cleverness is you end up building a mostly stateless, functional-reactive model on top of it again, that ends up extremely robust. It's just faster, and it can react immediately to server side data changes as well as client side. And that model can be declarative, it doesn't particularly need to be in JavaScript despite running on the client.
- bullen 6y ago
  Yes, HTTP/1.1 is awesome but you need to use comet-stream, comet is the same thing as long-polling and it's terrible and was only required because IE broke the XHR standard until IE8!
  - jlokier 6y ago
    IE did XHR before everyone else, and many years before there was a standard. It invented XHR; it was shipped in IE4. IE's behaviour defined the standard everyone else copied. Then other browsers copied it, adding their own quirks. The XHR standard wasn't created until about the same time as IE8 was released. So it's not possible for IE before IE8 to break the XHR standard, as there wasn't an XHR standard before IE8. Long-polling is still used as a fallback today. The modern alternative is WebSockets, but they do not work reliably in every network environment, either at the client or server side. Last time I looked, Facebook and Gmail were using long-polling to good effect. Despite the conceptual ugliness, long-polling when implemented properly works quite well and is extremely reliable. In practice, "quick and dirty" long-polling is not always implemented well, but it's not hard, just takes a little thinking about unreliable networks. XHR has nothing to do with long-polling particularly, and it doesn't remove the need for long-polling either. Long-polling is usually done with XHR these days, but it can be done easily without XHR, using invisible iframes or dynamic script elements. The latter is called JSONP and you've probably heard of it. SSE can also be used. All of these things have nothing to do with HTTP/1.1. They work perfectly well over HTTP/1.0, 1.1, HTTP/2 and HTTP/3.
jayd16 6y ago
>In essence, I am building a DIY database such that people connect directly to the database and get that sweet giant JSON object along with a stream of updates Isn't that Firebase DB?
- mathgladiator 6y ago
  It could be, but I own it and the roadmap. The key thing that I can do is await on players decision, so I can block the entire process for multiple people to do things.
- hn_throwaway_99 6y ago
  Yeah, that was my reaction to a lot of the discussion on this article. Firebase Realtime DB has been one of the most amazing pieces of software I've used. I don't have to worry about the connection details at all, I just have this data store that feels like a local DB except it's "magically" synced with the server, and any other users of the DB "magically" get updates when I change it.
  - thu2111 6y ago
    How do you implement ACLs and other privacy-related transformations in that model?
    - hn_throwaway_99 6y ago
      You write Firebase Security Rules [1] that restrict access to paths within your database, it's quite awesome. Those rules can be used to implement ACLs and object ownership, and also role-based access according to claims that exist on the JWT of the authenticated user. [1] https://firebase.google.com/docs/rules https://firebase.google.com/docs/rules
- JamesSwift 6y ago
  And Realm, and couch/pouch
toomim 6y ago
> Second, HTML over WebSockets is nice, but what is better is reactive data binding over the socket. This is the approach we are building in https://braid.org https://braid.org. We are extending HTTP from a state transfer protocol into a state synchronization protocol, so that when you do a GET request, you can also be promised to receive all updated versions of that resource. This extends very nicely with software that binds the resource directly into a DOM element.
- dgellow 6y ago
  Your website doesn't seem to be reachable.
  - toomim 6y ago
    Sorry, there was an outage this morning for a few hours, but it's back now.
dynamite-ready 6y ago
I have a personal project based on this approach (JSON over Websockets): https://quixical.com/ https://quixical.com/ It appears to work well. Security was tough work though (at least I thought it was). And scaling up, no matter what tech you have access to, can be costly. Also, determining how to handle disconnections is quite an interesting UI problem. Auto reconnect might be an option, but then you need to think about a retry policy. And it can be harder than you think... Mobile phones drop connections with surprising frequency, so it's worth thinking about very early on.
- mathgladiator 6y ago
  Security is exceptionally interesting, and it is why my language handles privacy as a first class citizen. The scaling up is a challenge due to the lack of common investments. It can be made absolutely cheap, and it is way cheaper than polling. People that dismiss websockets have a point that it throws away decades of wisdom, but the winnings and potential are there. The challenge is curating the right ideas and educating everyone... is hard.
rafale 6y ago
You are over complicating things. UI state should live in the client. Having the server maintain it is not only more complex but will also introduce a latency. Let's say I click a button to delete an item in the list. The button state goes to pressed, list loses focus, button takes focus, button unpressed, delete animation starts on deleted item, translation animations on the other items to slide into its place, new items devirtualized if the list is too long,... Now that's 1 user input. Imagine multiple consecutive input like mouse scroll. Or drag-and-drop. Only state shared between multiple clients should go to the server.
- jiriknesl 6y ago
  "UI state should live in the client." this is wrong and I think you have missed the point of the article. In server-rendered app, there's nothing like UI state. There's a request. This request and the database is enough to build response. There is no UI state. Your app can be (and often is) absolutely stateless and restult depends only on request and a state of the storage. While when you have SPA, API & server, you lose the application state and it's not possible to store the whole app state in the client for many reasons (security, scalability, synchronisation...). So you have to store subset of it. And you have to build a machinery that will do all this synchronization so your states stay in sync. It's double the effort to get 5% improvement.
  - pier25 6y ago
    > In server-rendered app, there's nothing like UI state. Of course there is. For example, any state related to interactivity like forms, client-side validation, widgets, filters, etc.
    - jiriknesl 6y ago
      All these things can be set up declaratively on the render time and managed through Ajax/socket. Local state is voluntary here.
      - pier25 6y ago
aruggirello 6y ago
Not the author, but since nobody did, I feel compelled to mention htmx: https://htmx.org/ https://htmx.org/
bullen 6y ago
What advantage is this providing except job security for developers that have to port working HTTP/1.1 systems to HTTP/2&3 and WebSockets? Have you used HTTP/1.1 comet-stream? I recommend you look at this sites source before you commit further: http://fuse.rupy.se http://fuse.rupy.se
rkangel 6y ago
> It's very easy to fuck up the implicit state tied to the connection, and this requires discipline to get right. It's very easy to ship unreliable software that then has issues based on the connection. I agree with this and it's why I believe that Phoenix LiveView is the best solution to this problem. The programming model and tooling that you get with the BEAM (along with things Phoenix provides like PubSub) are the best there is to manage this complexity.
john-shaffer 6y ago
> In essence, I am building a DIY database such that people connect directly to the database and get that sweet giant JSON object along with a stream of updates. biff implements this as a framework, backed by the Crux bi-temporal DB. It natively uses EDN rather than JSON, but emitting JSON should be trivial if that's needed. https://findka.com/biff/#introduction https://findka.com/biff/#introduction https://opencrux.com/main/index.html https://opencrux.com/main/index.html
m33k44 6y ago
> There is a great deal of power in having your UI be a stateless function over a giant JSON object. Something like this: http://jasonette.com http://jasonette.com
- mathgladiator 6y ago
  Kind of, except the JSON is pure data rather than a rendering tree. That's a very interesting project...

rraihansaputra 6y ago

> Second, HTML over WebSockets is nice, but what is better is reactive data binding over the socket. Let the browser maintain a DOM tree that templates over a JSON object, then reactive-ly update and it's an amazing experience. You have minimal data transfer, and minimal rendering updates. There is a great deal of power in having your UI be a stateless function over a giant JSON object. I think a lot of the "best-practices" for SPA is trying to do this. But as streaming the data transfer is not yet common, the data is abstracted as a giant JSON object to be a parameter of the UI as a stateless function (React+Redux/MobX/etc, Vue+VueX). When enforced correctly on the project level, what's complicated _is_ syncing the data to the server. I think you'll like Supabase's subscription approach to reactively bind the JSON for the client. I think a separation between the info-web (document-content heavy) and the app-web (needing stateful data sync) will be more clearly defined in the next few years. Nobody likes trying to sync data between the client and server over HTTP calls; we just have to wait for streaming to be the norm for the app-web (and native apps, ofcourse).

methyl 6y ago

> Let the browser maintain a DOM tree that templates over a JSON object, then reactive-ly update and it's an amazing experience Totally agreed. I even pulled off an Elixir library as a POC to show this concept, here's an example project: https://github.com/surferseo/live_data/tree/master/examples/phoenix_livedata_todomvc https://github.com/surferseo/live_data/tree/master/examples/... (most relevant part of API is here: https://github.com/surferseo/live_data/blob/master/examples/phoenix_livedata_todomvc/assets/js/containers/TodoList.js#L22 https://github.com/surferseo/live_data/blob/master/examples/... and here: https://github.com/surferseo/live_data/blob/master/examples/phoenix_livedata_todomvc/lib/phoenix_livedata_todomvc_web/live_data/todo_state.ex https://github.com/surferseo/live_data/blob/master/examples/...)

hfjtktkf 6y ago

I am doing something somewhat similar - I have a Postgres database and I send all the user tables to the client on connection and then stream all the updates. I use Vue and Vuex-ORM to make the client tables reactive, and put a view on that. When something changes in the database, the client UI will also update. It is indeed a very powerful and reliable way of ensuring synced state.

mxxx 6y ago
Sounds like you’re describing a system similar to Meteor.
qmmmur 6y ago
I do the same for a collaborative music app. User gets the state of a session on load as a JSON object then just received the diffs. It's pretty efficient.

RangerScience 6y ago

Got a question for you - I'm making a chat bot thing (https://www.brian.bot/ https://www.brian.bot/) and have added socket-based web chat... but I don't even know how to phrase the question of "how do I horizontally scale the socket connections". Webchat <-> Server <-> Slack. Once I have more than one server dyno (heroku), then I can't ensure that the Slack event hits the dyno with the corresponding socket connection. Thoughts / tips?

mathgladiator 6y ago
I'll be completely honest... you're mostly screwed. I say this with full love in my heart. Your best bet is to use Amazon and then leverage their ELB offering with sticky routing. HOWEVER, now you have a new problem as capacity dies, gets removed, cycled, added, etc. Sticky routing was designed with the assumption that it was an efficiency play as opposed to a deterministic play. What you will see happen is that your internal state will be split brain, and people joining the chat will not be globally consistent. Some people will be on host X while some will be on host Y. So, you have two choices. You could use a database to provide the global state, at which point now you have a whole bunch of problems. You can start with polling the database, and the key advantage of the websocket is moving the polling from client to server, so that will be a win for users. However, it will cost you. Second, you could introduce a message broker, but then you have the problem of how to initialize the state. Ultimately, everyone ends up with a hybrid database for storage and message broker for real-time. In your model, the database becomes like Slack, and your Server becomes a proxy. This could be super neat, but what is the value-add of the Server? The interesting thing to observe is that the solution is how to turn your front-end app into a database like thing. If the load balancer could route traffic like a database traffic controller thing, then you can put all your state in the front-end and move crazy fast. However, you will move so fast that you will fuck up your state. State is VERY HARD and exceptionally unforgiving to manage yourself. There is great wisdom in using a database with versioning and all that. It’s a great abstraction having lasted so long. I’m working on the state bits and trying to figure out the people side of discipline with my silly language, and I have some novel contributions to provide over the coming years. There are some linked pdfs on http://www.adama-lang.org/blog/some-thinky-thoughts-2021 http://www.adama-lang.org/blog/some-thinky-thoughts-2021 which may provide some more insight.

anderspitman 6y ago

> HTML over WebSockets is nice, but what is better is reactive data binding over the socket. Let the browser maintain a DOM tree that templates over a JSON object, then reactive-ly update and it's an amazing experience. You have minimal data transfer, and minimal rendering updates. There is a great deal of power in having your UI be a stateless function over a giant JSON object. Interesting that you're exploring this idea in the context of board games. I implemented[0] a similar idea while making a browser-based magic the gathering interface. Basically send commands to update the server state (a big JSON object), and the server does state diffs per client and sends lz compressed JSON patches. Worked quite well, but I haven't played with it for a couple years. [0]: https://github.com/anderspitman/pojo_flow https://github.com/anderspitman/pojo_flow

brundolf 6y ago

> The key thing to tackle is dealing with the server side state The core problem of web UIs has always been managing state. People like statically-rendered pages because most state gets kept in a) the URL, and b) transient HTML elements, and both are highly visible and easy to deal with in robust ways. When you're forced to have server-side state it's exceptionally painful, but people usually just avoid it altogether which in most cases pushes things towards the virtue of having less state anyway. For complex UIs with lots of little interactions the above story doesn't really work, so we've invented state management systems for the client. Managing complex state still sucks, but it sucks less with these tools. Sounds like this approach is just moving that state store back to the server, with a greased pipeline so it isn't so horribly painful to add new pieces of state even though both sides are involved. But you'll still face all the same problems you face with a front-end store: transientness, cache-clearing, even modeling is hard when you're dealing with a big blob of mutable state. And there will be the additional challenge of associating this stuff with a user (which you get for free on the front-end, even with modern state stores). > Let the browser maintain a DOM tree that templates over a JSON object, then reactive-ly update and it's an amazing experience. You have minimal data transfer, and minimal rendering updates. There is a great deal of power in having your UI be a stateless function over a giant JSON object. > this is where I have my shitty idea of a programming language for board games It's funny, I started building a web-based board game to play with family during quarantine, and I landed on this exact approach totally by coincidence :) I'm using polling requests instead of websockets because I'm lazy/it doesn't need to scale, but it's the exact same idea where the server just publishes a giant JSON object of state and the clients are pure functions of it (and send messages to the server to mutate it). I wonder if there's some aspect of board games that pushes one towards that line of thinking.

jamesjyu 6y ago
> I'm using polling requests instead of websockets because I'm lazy/it doesn't need to scale, but it's the exact same idea where the server just publishes a giant JSON object of state and the clients are pure functions of it (and send messages to the server to mutate it). I like this approach as well, but I do worry that users today expect optimistic updates on the client, and the injection of a X00ms delay would make the app feel sluggish.

radicalbyte 6y ago

This sounds like what Microsoft were trying to do 15 years ago with webforms: the browser was treated like a dumb terminal with the UI rendered on the server and pushed to the client. The UX back then was horrible, however technology has come a long way since.

djedr 6y ago

Interesting ideas, somewhat in line with what I've been thinking about. Perhaps you or someone in this topic might be interested in my little project[1] fit for innovative ideas like this. It is not very far from being available for experimental use in place of JSON. A future direction is experimenting with replacing HTML[2]. [1] https://www.tree-annotation.org/ https://www.tree-annotation.org/ [2] https://imgur.com/a/ER5qwtZ https://imgur.com/a/ER5qwtZ

jlokier 6y ago

The architecture you're describing is awesome. It sounds like the one I was working with ~18 years ago. Except it used HTTP (long polling or comet) instead of HTTP/3 for the transport. And more partial rendering to DOM-diffs on the server side because it was faster than doing it all on the client, but the client still applied diffs. The issues with full DOM state synchronisation, handling network errors and recovery, transport of templates and data, minimisation of transport subject to maximising responsiveness, some amount of anticipatory prefetching, were much the same then as now. A lovely thing about all this stateful streaming cleverness is you end up building a mostly stateless, functional-reactive model on top of it again, that ends up extremely robust. It's just faster, and it can react immediately to server side data changes as well as client side. And that model can be declarative, it doesn't particularly need to be in JavaScript despite running on the client.

bullen 6y ago
Yes, HTTP/1.1 is awesome but you need to use comet-stream, comet is the same thing as long-polling and it's terrible and was only required because IE broke the XHR standard until IE8!
- jlokier 6y ago
  IE did XHR before everyone else, and many years before there was a standard. It invented XHR; it was shipped in IE4. IE's behaviour defined the standard everyone else copied. Then other browsers copied it, adding their own quirks. The XHR standard wasn't created until about the same time as IE8 was released. So it's not possible for IE before IE8 to break the XHR standard, as there wasn't an XHR standard before IE8. Long-polling is still used as a fallback today. The modern alternative is WebSockets, but they do not work reliably in every network environment, either at the client or server side. Last time I looked, Facebook and Gmail were using long-polling to good effect. Despite the conceptual ugliness, long-polling when implemented properly works quite well and is extremely reliable. In practice, "quick and dirty" long-polling is not always implemented well, but it's not hard, just takes a little thinking about unreliable networks. XHR has nothing to do with long-polling particularly, and it doesn't remove the need for long-polling either. Long-polling is usually done with XHR these days, but it can be done easily without XHR, using invisible iframes or dynamic script elements. The latter is called JSONP and you've probably heard of it. SSE can also be used. All of these things have nothing to do with HTTP/1.1. They work perfectly well over HTTP/1.0, 1.1, HTTP/2 and HTTP/3.

jayd16 6y ago

>In essence, I am building a DIY database such that people connect directly to the database and get that sweet giant JSON object along with a stream of updates Isn't that Firebase DB?

mathgladiator 6y ago
It could be, but I own it and the roadmap. The key thing that I can do is await on players decision, so I can block the entire process for multiple people to do things.
hn_throwaway_99 6y ago
Yeah, that was my reaction to a lot of the discussion on this article. Firebase Realtime DB has been one of the most amazing pieces of software I've used. I don't have to worry about the connection details at all, I just have this data store that feels like a local DB except it's "magically" synced with the server, and any other users of the DB "magically" get updates when I change it.
- thu2111 6y ago
  How do you implement ACLs and other privacy-related transformations in that model?
  - hn_throwaway_99 6y ago
    You write Firebase Security Rules [1] that restrict access to paths within your database, it's quite awesome. Those rules can be used to implement ACLs and object ownership, and also role-based access according to claims that exist on the JWT of the authenticated user. [1] https://firebase.google.com/docs/rules https://firebase.google.com/docs/rules
JamesSwift 6y ago
And Realm, and couch/pouch

toomim 6y ago

> Second, HTML over WebSockets is nice, but what is better is reactive data binding over the socket. This is the approach we are building in https://braid.org https://braid.org. We are extending HTTP from a state transfer protocol into a state synchronization protocol, so that when you do a GET request, you can also be promised to receive all updated versions of that resource. This extends very nicely with software that binds the resource directly into a DOM element.

dgellow 6y ago
Your website doesn't seem to be reachable.
- toomim 6y ago
  Sorry, there was an outage this morning for a few hours, but it's back now.

dynamite-ready 6y ago

I have a personal project based on this approach (JSON over Websockets): https://quixical.com/ https://quixical.com/ It appears to work well. Security was tough work though (at least I thought it was). And scaling up, no matter what tech you have access to, can be costly. Also, determining how to handle disconnections is quite an interesting UI problem. Auto reconnect might be an option, but then you need to think about a retry policy. And it can be harder than you think... Mobile phones drop connections with surprising frequency, so it's worth thinking about very early on.

mathgladiator 6y ago
Security is exceptionally interesting, and it is why my language handles privacy as a first class citizen. The scaling up is a challenge due to the lack of common investments. It can be made absolutely cheap, and it is way cheaper than polling. People that dismiss websockets have a point that it throws away decades of wisdom, but the winnings and potential are there. The challenge is curating the right ideas and educating everyone... is hard.

rafale 6y ago

You are over complicating things. UI state should live in the client. Having the server maintain it is not only more complex but will also introduce a latency. Let's say I click a button to delete an item in the list. The button state goes to pressed, list loses focus, button takes focus, button unpressed, delete animation starts on deleted item, translation animations on the other items to slide into its place, new items devirtualized if the list is too long,... Now that's 1 user input. Imagine multiple consecutive input like mouse scroll. Or drag-and-drop. Only state shared between multiple clients should go to the server.

jiriknesl 6y ago
"UI state should live in the client." this is wrong and I think you have missed the point of the article. In server-rendered app, there's nothing like UI state. There's a request. This request and the database is enough to build response. There is no UI state. Your app can be (and often is) absolutely stateless and restult depends only on request and a state of the storage. While when you have SPA, API & server, you lose the application state and it's not possible to store the whole app state in the client for many reasons (security, scalability, synchronisation...). So you have to store subset of it. And you have to build a machinery that will do all this synchronization so your states stay in sync. It's double the effort to get 5% improvement.
- pier25 6y ago
  > In server-rendered app, there's nothing like UI state. Of course there is. For example, any state related to interactivity like forms, client-side validation, widgets, filters, etc.
  - jiriknesl 6y ago
    All these things can be set up declaratively on the render time and managed through Ajax/socket. Local state is voluntary here.
    - pier25 6y ago

aruggirello 6y ago

Not the author, but since nobody did, I feel compelled to mention htmx: https://htmx.org/ https://htmx.org/

bullen 6y ago

What advantage is this providing except job security for developers that have to port working HTTP/1.1 systems to HTTP/2&3 and WebSockets? Have you used HTTP/1.1 comet-stream? I recommend you look at this sites source before you commit further: http://fuse.rupy.se http://fuse.rupy.se

rkangel 6y ago

> It's very easy to fuck up the implicit state tied to the connection, and this requires discipline to get right. It's very easy to ship unreliable software that then has issues based on the connection. I agree with this and it's why I believe that Phoenix LiveView is the best solution to this problem. The programming model and tooling that you get with the BEAM (along with things Phoenix provides like PubSub) are the best there is to manage this complexity.

john-shaffer 6y ago

> In essence, I am building a DIY database such that people connect directly to the database and get that sweet giant JSON object along with a stream of updates. biff implements this as a framework, backed by the Crux bi-temporal DB. It natively uses EDN rather than JSON, but emitting JSON should be trivial if that's needed. https://findka.com/biff/#introduction https://findka.com/biff/#introduction https://opencrux.com/main/index.html https://opencrux.com/main/index.html

m33k44 6y ago

> There is a great deal of power in having your UI be a stateless function over a giant JSON object. Something like this: http://jasonette.com http://jasonette.com

mathgladiator 6y ago
Kind of, except the JSON is pure data rather than a rendering tree. That's a very interesting project...