9 ms·

Data fetching on the web still sucks

155 points by bcherny 6y ago

gajus 6y ago
Relay modern does all of the things that this article sets as a criteria for good data fetching.
- ch4s3 6y ago
  It doesn't have a great story around error handling IMHO.
  - orta 6y ago
    I've found Relay to be pretty solid with error handling, though I think the ability in how to handle GraphQL errors does hurt Relay here, but it does for all GraphQL clients. Re: parent, I also was surprised to see Relay dismissed on the list, it solves every problem they bring up in the list, perhaps maybe not the real-time aspects
    - ch4s3 6y ago
      I should have clarified that Relay flavored GraphQL from a backend perspective is lacking in this respect. Yeah, I think relay is pretty good, but I'm not sure the broader ecosystem is really there yet. Outside of node, the backend libraries aren't really fully baked yet, and the standard seems to have some holes. But I'm somewhat optimistic.
- bcherny 6y ago
  I'd agree Relay is relatively far along, compared to other frameworks. I might lump its support into a few groups: - Full (eg. request batching) - Full, but painful (eg. colocation) - Partial (eg. mutation queuing) - Non-existent (eg. durability guarantees) (I've been using Relay Modern daily for a few years.)
robertoandred 6y ago
SWR or react-query?
- dsego 6y ago
  I've used swr, but they seem very similar. Very easy to use for server synced state imho. Try both and pick the one that feels nicer to you.
- dinkleberg 6y ago
  I started my current project using SWR and then switched to react-query after dealing with some annoyances with SWR. Overall react-query is impressive for data fetching, but I’ve run into issues with mutations and cache invalidation.
  - AgentME 6y ago
    I've been disappointed at how SWR and react-query seem to only support the bare minimum for mutations. As far as I can tell, they don't offer any kind of protection from refreshes overwriting any local mutations that haven't been saved to the server yet. They don't do any handling around updating the data on the server, so you need to handle issues yourself like retrying, deciding when to send to the server, and making sure you never have overlapping update requests racing each other. I've had trouble finding any library in the React or general JS ecosystem that handles this stuff.
  - shuding 6y ago
    Hey! SWR author here. Mind sharing the annoyances and how I can make it better for you?
    - andrewrothman 6y ago
      Thanks for your contribution and outreach! SWR has been really helpful for us at work.
patates 6y ago
What are examples of data-fetching libs/standards outside the web, that check all (or a relevant subset) of those criteria?
- dhritzkiv 6y ago
  I too am curious. All of the platform-specific frameworks which do networking+datastore, that I've used, check even fewer boxes. Especially when it comes to reactive UI and optimistic responses.
joshxyz 6y ago
I'm currently experimenting with React and WebSockets and they seem to be a perfect fit. No need to write wrappers for Fetch, network errors and reconnects can be handled on high-level, handlers for each message type can be mounted and unmounted on useEffect hooks, all back-end jobs can notify the user in realtime, all session-based client-side data can be updated in realtime (in single or multiple open tabs). I'm also using uWebSockets.js[0] which is great in terms of API design, stability, and performance. Their benchmarks[1] are just convincing. Highly recommend people using ExpressJS / Koa / whatever to try it. [0] https://github.com/uNetworking/uWebSockets.js/ https://github.com/uNetworking/uWebSockets.js/ [1] https://github.com/uNetworking/uWebSockets/tree/master/benchmarks https://github.com/uNetworking/uWebSockets/tree/master/bench...
- azangru 6y ago
  > and they seem to be a perfect fit Needs a fairly capable server though, I suppose. One that can handle lots and lots of open websocket connections.
  - IggleSniggle 6y ago
    Guess it really depends on how many concurrent users you expect to have... Sounds like 30 GiB RAM for 1 million concurrent users is not unreasonable: https://stackoverflow.com/a/17453704 https://stackoverflow.com/a/17453704
  - joshxyz 6y ago
    Yes that's some genuine concern. Though on B2B and B2C SaaS where you got paying users, I think each of them deserve a websocket, and it could* alleviate some headaches for developers. * Anecdotal of course, at least for us.
  - e_y_ 6y ago
    Also your load balancer stack needs to support it, which is becoming less of a problem these days but there's still some stragglers. You could also end up with an imbalance where some servers have a lot of connections and others have few.
- 6y ago
lexicality 6y ago
> You shouldn’t have to ship kilobytes of metadata describing your data schema to clients in order to fetch data. I feel like this requirement makes every other requirement impossible unless you use some kind of compiled language feature and even then you're not going to be able to have type safety without some kind of metadata. Also, I don't think it's possible to make any one library that solves all "data fetching" scenarios. How could you reasonably live data (eg stock ticker), enormous data sets (eg looking through individual analytics events) slow but largely static lookups (eg searching a library catalog) and fast but uncachable (eg a SPA forum) into the same API without making it horrendous?
- lexicality 6y ago
  That said, we do need some better standards. I look after a bunch of API integrations at work and we have everything from SOAP to GraphQL to downloading a magically named .xml.gzip file in a FTP directory with at least 10 other homemade REST implementation between and every single one of them is basically returning the same info but in wildly incompatible ways. At least the SOAP one crashes when they change the API without telling us :/
tyingq 6y ago
I didn't see "show download progress" as one of the needs. I remember seeing complaints about the fetch api not exposing an easy way to do that. Issues like this one seem to remain: https://github.com/AnthumChris/fetch-progress-indicators/issues/13 https://github.com/AnthumChris/fetch-progress-indicators/iss...
- pseudosavant 6y ago
  This is possible with the Fetch API actually. You can get the chunked up contents of the request as they arrive. This link has a good example: https://javascript.info/fetch-progress https://javascript.info/fetch-progress
  - tyingq 6y ago
    The issue is with "Content-Encoding: gzip". In that case, "Content-Length" isn't enough. You can make it work if you toss in an extra header server-side.
rektide 6y ago
> Realtime updates. Sometimes, you want data to be pushed, not pulled (eg. for things like notifications). But the APIs for pushing and pulling data are often different, both on the client and on the server. Two and a half months since Google announced the removal of HTTP Push[1]. There'd been a number of threads open (some 5 years old now) asking the Fetch spec to please include some way to allow the page to be notified of Push[2], which would have made HTTP Push super interesting & useful. After 5 years, we never got anywhere, never got any help from standards body/google in trying to get Push useful. The topic came up a couple times when Abort was being worked on, and it seemed like maybe we'd see some progress, but those finer points got drowned among the noise of the Abort work. And then Google announced we developers weren't using Push enough, & that they were going to drop it. Without ever having tried to make it useful in the first place. HTTP/2 seemed to have such promise, was such elation finally getting released in 2015. Seeing so much promise go unfulfilled & then get cancelled has been quite dark. Unlike so so many of the alternatives that we are up to for data fetching, this path was both realtime & still http/resource centric. It still felt like core web architecture, in a way few others do. The unfulfilment of this destiny, this loss, has been jarring, & continues to leave us with these open quandries of what new entirely different alternatives do we invent, over WebSockets, over SSE, over WebTransport, &c. HTTP (almost) was here. [1] https://news.ycombinator.com/item?id=25064855 https://news.ycombinator.com/item?id=25064855 [2] https://github.com/whatwg/fetch/issues/51 https://github.com/whatwg/fetch/issues/51 Edit: there was a deleted comment saying that this kind of use case is not what Push was for, is not there to send unprompted data. Technically, Push can only be done in reply to a http request, so there is a factual limitation. Web Push Protocol gets around this by holding a GET open, to allow "unprompted" data. I think a sizable amount of engineers at the time were indeed thinking of Push as an optimization, a way to get pages to load faster. But this contingent of folks who saw HTTP/2 Push enabling pushing data capabilities, who were hungry for it, who counted on it's maturation: I think they had reasonable expectations, and that we're in a very weird place on the web, with HTTP technically having the ability to push semi-unprompted, but browsers never delivering that capability that's now inbuilt in HTTP. Using Push is still possible for so called unprompted updates even! At least it was till Push got removed. The trick is that the server needed some way to tell the browser which resources to send: so the server might be pushing resources, then using SSE to notify the page of those updates, for ex.
joverholt 6y ago
Ember.js seems to check off a lot of those boxes, if you are willing to use a complete solution and not do the 'pick and choose' method of react and friends. While it has lost popularity in the last few years, it has been moving forward technically. It has become leaner and meaner. Really learning the "Ember Way" to do things can reduce some of the friction the author mentions. Ember Data can allow you to talk to multiple api backends (differing schemas) while presenting the same model to your UI. If you use the default JsonApi[0] based backend, you get a lot of things for free: powerful filtering, side loading data (relationships), consistent error handling. Sometimes it can be chatty, but that's a spot where HTTP2 can help. Use ember-concurrency add-on and you have a nice way to manage your requests, things like de-bouncing, handling loading spinners, etc. [0]https://jsonapi.org/format/ https://jsonapi.org/format/
- Nextgrid 6y ago
  I'm saddened to see that almost a decade after Ember's release, the front-end world still insists rolling their own (terrible) reimplementation of it. I would've expected that a decade later we would've settled on an Ember-like framework for the front-end and moved beyond constructing URLs and parsing JSON responses manually but apparently not.
  - whalesalad 6y ago
    The rest of the world moved on from Ember because Ember is a pain in the ass to use.
    - enspandi 6y ago
      It has improved a lot though - checkout ember octane
    - Nextgrid 6y ago
      As someone who's not a front-end developer, but who works near them and sees the (React) code they produce and their general productivity (or lack thereof), what's so bad about it? To me as a backend developer just reading Ember's docs, it seems like Ember provides most of the conveniences I take for granted in a backend web framework such as Django, Rails or Laravel. On the other hand, the front-end code I see from the people I work with seems to have no standard for structure (every project has its own), reinvents the wheel all the time (using Axios and building the URLs manually with string concatenation for example), etc. Most of the stuff they do (and redo) from scratch seems like something that would be handled by Ember to begin with. So what's so bad? I feel like (as an outsider - feel free to prove me wrong) Ember is fine for most purposes, and edge-cases where React or alternative approaches do provide a benefit can be used ad-hoc without having to use it for your entire application.
philplckthun 6y ago
Not to toot our own horn, but while this mentions GraphQL with Relay / Apollo as fetching clients, with urql and its normalised cache Graphcache we started approaching more of these problems. Solving the mentioned problems in the article for best practices around fragments is on our near future roadmap, but there are some other points here that we've worked on that especially Apollo did not (yet) Request batching is in my humble opinion not quite needed with GraphQL and especially with HTTP/2 and edge caching via persisted queries, however we have stronger guarantees around commutative application of responses from the server. We also have optimistic updates and a lot of intuitive safe guards around how these and other updates are applied to all normalised data. They're applied in a pre-determined order and optimistic updates are applied in such a way that the optimistic/temporary data can never be mixed with "permanent" data in the cache. It also prevents races by queueing up queries that would otherwise overwrite optimistic data accidentally and defers them up until all optimistic updates are completed, which will all settle in a single batch, rather than one by one. I find this article really interesting since it seems to summarise a lot of the efforts that we've also identified as "weaknesses" in normalised caching and GraphQL data fetching, and common problems that come up during development with data fetching clients that aren't aware of these issues. Together with React and their (still experimental / upcoming) Suspense API it's actually rather easy to build consistent loading experiences as well. The same goes for Vue 3's Suspense boundaries too. Edit: Also this all being said, in most cases Relay actually also does a great job on most of the criticism that the author lays out here, so if the only complaint that a reader here picks up are the DX around fragments and nothing else applies this once again shows how solid Relay can be as well.
benlivengood 6y ago
Safe client-side SQL sounds like what the author wants. * Consistency when fetching data: define a UI that matches a particular SQL view. * Request batching: It's probably not hard to send deltas on a large view with server-side support for sessions. * Avoiding network waterfalls: basically the definition of a SQL view, with some session management to de-duplicate traffic when updating the view. A lot of clients support batching queries. * UX when fetching data: this seems entirely like an oversite of some UI packages, but when all the other points line up there are very distinct points in application flow when the UI will know a query and potential update is in progress. * Colocation: the definition of a view is necessarily local. * File size: SQL is pretty lightweight on the schema details in results * Type safety: SQL doesn't support dependent types, but e.g. postgres is pretty type-y * Versioning: no explicit support in SQL but for changes to the UI a new view can be created. For changes to the data model a compatible change to the view can be committed in the same transaction as the data model change. * Realtime updates: the glaring hole in the glory of SQL. You'll have to do some user-defined triggers server-side to support this. * Consistency when updating data: the relational model with constraints enforces this * Optimistic updates: if the theoretical differential view update described above exists then this is pretty trivial. Compare local and remote deltas when refreshing the view and indicate to the user the expected state that wasn't committed. * Request queueing: this one seems to be more of a data model problem. If the data model approximates a state machine and queries are on transitions then yes, they need to be serialized. But why not do complex updates all at once with a multi-statement update transaction that commits or fails atomically? * UX when updating data: presumably a UI tied to a single view is easier to indicate busy-ness for. If an update transaction is in flight, display a spinner. When it fails or commits display an OK or error. * Durability when updating data: this could almost certainly be improved for most sql clients. However, idempotence is definitely covered by conditional update transactions. * Type safety: user defined types if necessary but constraints are where a lot of the type-safety happens in SQL. * Versioning: backward-compatibility by updating mutable views in the same transaction as data model changes. Forward-compatibility by rendering directly from the view that's returned, not expecting any particular view.
kabes 6y ago
Meteor actually seems to come pretty close of you're ok with mongodb
heipei 6y ago
This post feels like a uninformed and undifferentiated rant against "things are too complex". Let's start with the first paragraph: What does the JavaScript fetch API have to do with data management? How can you compare the fetch() API with Swagger (an API documentation format) with Protobuf (a serialisation format)? That doesn't even make sense. Second paragraph: "The UI should automatically update everywhere the data is used". Again, what does this have to do with any of the above? That is state management, yeah, and you can build proper state management with any HTTP library and any message serialisation format. Request batching: How would that happen "automatically"? By waiting to fire requests and then batching them? UX when fetching data: What does that have to do with any of the above? You still have to decide how your UI displays the fact that a piece of data is loading. What do you expect there to be in place? Best thing I could imagine is to have a global operations indicators a la Adobe Lightroom which tells you how many HTTP requests are in flight. I could go on, but the last paragraphs maybe highlights the lack of understanding the author had: "UI Frameworks (at this point, React has won)". If React had "won" then why would we be having this discussion. React hasn't "won" because it solves one piece of the puzzle: Rendering. For every little other thing you have to incorporate another library or figure out your own solution: Routing, State Management, CRUD / HTTP API, etc. If anything, Ember.js would most closely fit the bill of incorporating most of the things the author seems to care about yet can't articulate clearly.
- batoure 6y ago
  What this person said! ^ almost all of the mentioned issues have solutions out their but the author is jumping around the stack with no sense of true purpose. How does graphql impact your SPA updating views, it doesn’t..
  - chrisco255 6y ago
    Of course it does, because GraphQL allows you to roll up queries for separate entities and get the data back for those in a single HTTP request. And that capability is something Relay and Apollo take advantage of. In this way a component can reference only the data it needs and the query rollup and caching happens automatically.
gitpusher 6y ago
The author does a good job of summarizing the relevant aspects of fetching data in a modern web-app. However, by NO means should all of this be handled by one "data-fetching library" - as he suggests.
ficklepickle 6y ago
This is a solved problem with Nuxt/Vue and the fetch hook. Client side, server side, no problem. One of the many reasons I love nuxt/Vue.
whalesalad 6y ago
This post is hard to digest - it is a rant on 10 different things. All are valid hurdles but I don't see how we can just tie it all up in a bag and call it "data fetching". Architectures are what help solve these sorts of problems - not tools or libraries. Placing the blame on tools means the real issue has not been identified.
SimeVidas 6y ago
I’m curious if the author thinks that people can actually read the ultra-low-contrast text in the side column.
iskin 6y ago
Completely agree! I believe that op-log sync with CRDT-based atomic operations are a better solution. Logux is one of the example of this approach: https://logux.io/ https://logux.io/
super_hhh 6y ago
Vue > React
- viviansolide 6y ago
  Suspense feature is great.
anderspitman 6y ago
If your UI is complicated enough to worry about most things on this list, maybe simplify your UI?
- beders 6y ago
  Valid point, if your app is a todo list manager. There's a level of complexity in the domain logic where all these things mentioned by the author become pain points. That complexity can't be reduced by "simplify your UI" statements.
  - anderspitman 6y ago
    Is that actually true, or is it what we've been told to believe by people selling frameworks?
irrational 6y ago
> UI frameworks (at this point, React has won) Isn't this like someone in 2011 saying "UI frameworks (at this point, jQuery has won)"? I think things like Svelte and other up and coming UI frameworks are still being developed because we all recognize that React is not the ultimate UI Framework. Perhaps we will never get there, but surely we can do better than where we are at.
- deergomoo 6y ago
  This is highly subjective of course, but I think Vue has already surpassed React from a technical and ease-of-use standpoint. Where React still leads handsomely is tooling (the official VS Code Vue plug-in is buggy as all hell), breadth of ecosystem, and employability.
- pseudosavant 6y ago
  ^ React is just the new jQuery. We have people who basically don't really know HTML, CSS, or JavaScript but make React apps day-in day-out. Ten years ago the exact same thing was happening with jQuery. I still cringe when I see questions on StackOverflow where they ask "how can I XYZ with jQuery" where XYZ is something that has absolutely nothing to do with the DOM.
pier25 6y ago
Nobody has solved it because there is nothing to solve. That's like saying that "we to optimize SQL for UX". No, no we don't. Fetching data has nothing to do with managing state, rendering UI, or even UX. Each layer has different problems it needs to address. And btw you can totally do GraphQL requests with fetch().
asiando 6y ago
This applies to everything in the browser honestly. Why can’t I bind a variable to the DOM natively? I want the variable X to match the value of <input> and vice versa without having to set up a bunch of listeners and hope they don’t go in a loop.
- uryga 6y ago
  sounds like you want something like Vue baked into the browser. but i'd prefer React baked into the browser! we can't have both, so it's probably best we have neither :) seriously though, why not just write a simple wrapper around event listeners + maybe some proxy magic and get the semantics you prefer? or find a library that does that, i'm sure there's a bunch out there
swiley 6y ago
You can serialize pretty complex graphs of data and UI information in a new format called "HTML." The software capable of consuming this format is able to merge fragments of it into something called a DOM. If you're doing something more complex than this in your app then quit whining about browsers not being convenient for you to abuse. The users don't want your crap any more than browser implementers do.
vim-guru 6y ago
I think Fulcro has mechanisms to tick all your boxes and then some. Welcome to the future
slifin 6y ago
Server side react components has the best data fetching story I've ever seen They demoed an application that has a react component powered directly by a SQL query, when the client next fetched anything from that page, changes in terms of data coming out of the SQL query were automatically propagated to the client No data fetch API just coding like you would in a server side app with React handling the IO I hope someone starts implementing something like Drupal on top for automatic deadlock prevention, declarative data driven interfaces for forms, schema etc

155 points by bcherny 6y ago

gajus 6y ago

Relay modern does all of the things that this article sets as a criteria for good data fetching.

ch4s3 6y ago
It doesn't have a great story around error handling IMHO.
- orta 6y ago
  I've found Relay to be pretty solid with error handling, though I think the ability in how to handle GraphQL errors does hurt Relay here, but it does for all GraphQL clients. Re: parent, I also was surprised to see Relay dismissed on the list, it solves every problem they bring up in the list, perhaps maybe not the real-time aspects
  - ch4s3 6y ago
    I should have clarified that Relay flavored GraphQL from a backend perspective is lacking in this respect. Yeah, I think relay is pretty good, but I'm not sure the broader ecosystem is really there yet. Outside of node, the backend libraries aren't really fully baked yet, and the standard seems to have some holes. But I'm somewhat optimistic.
bcherny 6y ago
I'd agree Relay is relatively far along, compared to other frameworks. I might lump its support into a few groups: - Full (eg. request batching) - Full, but painful (eg. colocation) - Partial (eg. mutation queuing) - Non-existent (eg. durability guarantees) (I've been using Relay Modern daily for a few years.)

robertoandred 6y ago

SWR or react-query?

dsego 6y ago
I've used swr, but they seem very similar. Very easy to use for server synced state imho. Try both and pick the one that feels nicer to you.
dinkleberg 6y ago
I started my current project using SWR and then switched to react-query after dealing with some annoyances with SWR. Overall react-query is impressive for data fetching, but I’ve run into issues with mutations and cache invalidation.
- AgentME 6y ago
  I've been disappointed at how SWR and react-query seem to only support the bare minimum for mutations. As far as I can tell, they don't offer any kind of protection from refreshes overwriting any local mutations that haven't been saved to the server yet. They don't do any handling around updating the data on the server, so you need to handle issues yourself like retrying, deciding when to send to the server, and making sure you never have overlapping update requests racing each other. I've had trouble finding any library in the React or general JS ecosystem that handles this stuff.
- shuding 6y ago
  Hey! SWR author here. Mind sharing the annoyances and how I can make it better for you?
  - andrewrothman 6y ago
    Thanks for your contribution and outreach! SWR has been really helpful for us at work.

patates 6y ago

What are examples of data-fetching libs/standards outside the web, that check all (or a relevant subset) of those criteria?

dhritzkiv 6y ago
I too am curious. All of the platform-specific frameworks which do networking+datastore, that I've used, check even fewer boxes. Especially when it comes to reactive UI and optimistic responses.

joshxyz 6y ago

I'm currently experimenting with React and WebSockets and they seem to be a perfect fit. No need to write wrappers for Fetch, network errors and reconnects can be handled on high-level, handlers for each message type can be mounted and unmounted on useEffect hooks, all back-end jobs can notify the user in realtime, all session-based client-side data can be updated in realtime (in single or multiple open tabs). I'm also using uWebSockets.js[0] which is great in terms of API design, stability, and performance. Their benchmarks[1] are just convincing. Highly recommend people using ExpressJS / Koa / whatever to try it. [0] https://github.com/uNetworking/uWebSockets.js/ https://github.com/uNetworking/uWebSockets.js/ [1] https://github.com/uNetworking/uWebSockets/tree/master/benchmarks https://github.com/uNetworking/uWebSockets/tree/master/bench...

azangru 6y ago
> and they seem to be a perfect fit Needs a fairly capable server though, I suppose. One that can handle lots and lots of open websocket connections.
- IggleSniggle 6y ago
  Guess it really depends on how many concurrent users you expect to have... Sounds like 30 GiB RAM for 1 million concurrent users is not unreasonable: https://stackoverflow.com/a/17453704 https://stackoverflow.com/a/17453704
- joshxyz 6y ago
  Yes that's some genuine concern. Though on B2B and B2C SaaS where you got paying users, I think each of them deserve a websocket, and it could* alleviate some headaches for developers. * Anecdotal of course, at least for us.
- e_y_ 6y ago
  Also your load balancer stack needs to support it, which is becoming less of a problem these days but there's still some stragglers. You could also end up with an imbalance where some servers have a lot of connections and others have few.
6y ago

lexicality 6y ago

> You shouldn’t have to ship kilobytes of metadata describing your data schema to clients in order to fetch data. I feel like this requirement makes every other requirement impossible unless you use some kind of compiled language feature and even then you're not going to be able to have type safety without some kind of metadata. Also, I don't think it's possible to make any one library that solves all "data fetching" scenarios. How could you reasonably live data (eg stock ticker), enormous data sets (eg looking through individual analytics events) slow but largely static lookups (eg searching a library catalog) and fast but uncachable (eg a SPA forum) into the same API without making it horrendous?

lexicality 6y ago
That said, we do need some better standards. I look after a bunch of API integrations at work and we have everything from SOAP to GraphQL to downloading a magically named .xml.gzip file in a FTP directory with at least 10 other homemade REST implementation between and every single one of them is basically returning the same info but in wildly incompatible ways. At least the SOAP one crashes when they change the API without telling us :/

tyingq 6y ago

I didn't see "show download progress" as one of the needs. I remember seeing complaints about the fetch api not exposing an easy way to do that. Issues like this one seem to remain: https://github.com/AnthumChris/fetch-progress-indicators/issues/13 https://github.com/AnthumChris/fetch-progress-indicators/iss...

pseudosavant 6y ago
This is possible with the Fetch API actually. You can get the chunked up contents of the request as they arrive. This link has a good example: https://javascript.info/fetch-progress https://javascript.info/fetch-progress
- tyingq 6y ago
  The issue is with "Content-Encoding: gzip". In that case, "Content-Length" isn't enough. You can make it work if you toss in an extra header server-side.

rektide 6y ago

> Realtime updates. Sometimes, you want data to be pushed, not pulled (eg. for things like notifications). But the APIs for pushing and pulling data are often different, both on the client and on the server. Two and a half months since Google announced the removal of HTTP Push[1]. There'd been a number of threads open (some 5 years old now) asking the Fetch spec to please include some way to allow the page to be notified of Push[2], which would have made HTTP Push super interesting & useful. After 5 years, we never got anywhere, never got any help from standards body/google in trying to get Push useful. The topic came up a couple times when Abort was being worked on, and it seemed like maybe we'd see some progress, but those finer points got drowned among the noise of the Abort work. And then Google announced we developers weren't using Push enough, & that they were going to drop it. Without ever having tried to make it useful in the first place. HTTP/2 seemed to have such promise, was such elation finally getting released in 2015. Seeing so much promise go unfulfilled & then get cancelled has been quite dark. Unlike so so many of the alternatives that we are up to for data fetching, this path was both realtime & still http/resource centric. It still felt like core web architecture, in a way few others do. The unfulfilment of this destiny, this loss, has been jarring, & continues to leave us with these open quandries of what new entirely different alternatives do we invent, over WebSockets, over SSE, over WebTransport, &c. HTTP (almost) was here. [1] https://news.ycombinator.com/item?id=25064855 https://news.ycombinator.com/item?id=25064855 [2] https://github.com/whatwg/fetch/issues/51 https://github.com/whatwg/fetch/issues/51 Edit: there was a deleted comment saying that this kind of use case is not what Push was for, is not there to send unprompted data. Technically, Push can only be done in reply to a http request, so there is a factual limitation. Web Push Protocol gets around this by holding a GET open, to allow "unprompted" data. I think a sizable amount of engineers at the time were indeed thinking of Push as an optimization, a way to get pages to load faster. But this contingent of folks who saw HTTP/2 Push enabling pushing data capabilities, who were hungry for it, who counted on it's maturation: I think they had reasonable expectations, and that we're in a very weird place on the web, with HTTP technically having the ability to push semi-unprompted, but browsers never delivering that capability that's now inbuilt in HTTP. Using Push is still possible for so called unprompted updates even! At least it was till Push got removed. The trick is that the server needed some way to tell the browser which resources to send: so the server might be pushing resources, then using SSE to notify the page of those updates, for ex.

joverholt 6y ago

Ember.js seems to check off a lot of those boxes, if you are willing to use a complete solution and not do the 'pick and choose' method of react and friends. While it has lost popularity in the last few years, it has been moving forward technically. It has become leaner and meaner. Really learning the "Ember Way" to do things can reduce some of the friction the author mentions. Ember Data can allow you to talk to multiple api backends (differing schemas) while presenting the same model to your UI. If you use the default JsonApi[0] based backend, you get a lot of things for free: powerful filtering, side loading data (relationships), consistent error handling. Sometimes it can be chatty, but that's a spot where HTTP2 can help. Use ember-concurrency add-on and you have a nice way to manage your requests, things like de-bouncing, handling loading spinners, etc. [0]https://jsonapi.org/format/ https://jsonapi.org/format/

Nextgrid 6y ago
I'm saddened to see that almost a decade after Ember's release, the front-end world still insists rolling their own (terrible) reimplementation of it. I would've expected that a decade later we would've settled on an Ember-like framework for the front-end and moved beyond constructing URLs and parsing JSON responses manually but apparently not.
- whalesalad 6y ago
  The rest of the world moved on from Ember because Ember is a pain in the ass to use.
  - enspandi 6y ago
    It has improved a lot though - checkout ember octane
  - Nextgrid 6y ago
    As someone who's not a front-end developer, but who works near them and sees the (React) code they produce and their general productivity (or lack thereof), what's so bad about it? To me as a backend developer just reading Ember's docs, it seems like Ember provides most of the conveniences I take for granted in a backend web framework such as Django, Rails or Laravel. On the other hand, the front-end code I see from the people I work with seems to have no standard for structure (every project has its own), reinvents the wheel all the time (using Axios and building the URLs manually with string concatenation for example), etc. Most of the stuff they do (and redo) from scratch seems like something that would be handled by Ember to begin with. So what's so bad? I feel like (as an outsider - feel free to prove me wrong) Ember is fine for most purposes, and edge-cases where React or alternative approaches do provide a benefit can be used ad-hoc without having to use it for your entire application.

philplckthun 6y ago

Not to toot our own horn, but while this mentions GraphQL with Relay / Apollo as fetching clients, with urql and its normalised cache Graphcache we started approaching more of these problems. Solving the mentioned problems in the article for best practices around fragments is on our near future roadmap, but there are some other points here that we've worked on that especially Apollo did not (yet) Request batching is in my humble opinion not quite needed with GraphQL and especially with HTTP/2 and edge caching via persisted queries, however we have stronger guarantees around commutative application of responses from the server. We also have optimistic updates and a lot of intuitive safe guards around how these and other updates are applied to all normalised data. They're applied in a pre-determined order and optimistic updates are applied in such a way that the optimistic/temporary data can never be mixed with "permanent" data in the cache. It also prevents races by queueing up queries that would otherwise overwrite optimistic data accidentally and defers them up until all optimistic updates are completed, which will all settle in a single batch, rather than one by one. I find this article really interesting since it seems to summarise a lot of the efforts that we've also identified as "weaknesses" in normalised caching and GraphQL data fetching, and common problems that come up during development with data fetching clients that aren't aware of these issues. Together with React and their (still experimental / upcoming) Suspense API it's actually rather easy to build consistent loading experiences as well. The same goes for Vue 3's Suspense boundaries too. Edit: Also this all being said, in most cases Relay actually also does a great job on most of the criticism that the author lays out here, so if the only complaint that a reader here picks up are the DX around fragments and nothing else applies this once again shows how solid Relay can be as well.

benlivengood 6y ago

Safe client-side SQL sounds like what the author wants. * Consistency when fetching data: define a UI that matches a particular SQL view. * Request batching: It's probably not hard to send deltas on a large view with server-side support for sessions. * Avoiding network waterfalls: basically the definition of a SQL view, with some session management to de-duplicate traffic when updating the view. A lot of clients support batching queries. * UX when fetching data: this seems entirely like an oversite of some UI packages, but when all the other points line up there are very distinct points in application flow when the UI will know a query and potential update is in progress. * Colocation: the definition of a view is necessarily local. * File size: SQL is pretty lightweight on the schema details in results * Type safety: SQL doesn't support dependent types, but e.g. postgres is pretty type-y * Versioning: no explicit support in SQL but for changes to the UI a new view can be created. For changes to the data model a compatible change to the view can be committed in the same transaction as the data model change. * Realtime updates: the glaring hole in the glory of SQL. You'll have to do some user-defined triggers server-side to support this. * Consistency when updating data: the relational model with constraints enforces this * Optimistic updates: if the theoretical differential view update described above exists then this is pretty trivial. Compare local and remote deltas when refreshing the view and indicate to the user the expected state that wasn't committed. * Request queueing: this one seems to be more of a data model problem. If the data model approximates a state machine and queries are on transitions then yes, they need to be serialized. But why not do complex updates all at once with a multi-statement update transaction that commits or fails atomically? * UX when updating data: presumably a UI tied to a single view is easier to indicate busy-ness for. If an update transaction is in flight, display a spinner. When it fails or commits display an OK or error. * Durability when updating data: this could almost certainly be improved for most sql clients. However, idempotence is definitely covered by conditional update transactions. * Type safety: user defined types if necessary but constraints are where a lot of the type-safety happens in SQL. * Versioning: backward-compatibility by updating mutable views in the same transaction as data model changes. Forward-compatibility by rendering directly from the view that's returned, not expecting any particular view.

kabes 6y ago

Meteor actually seems to come pretty close of you're ok with mongodb

heipei 6y ago

This post feels like a uninformed and undifferentiated rant against "things are too complex". Let's start with the first paragraph: What does the JavaScript fetch API have to do with data management? How can you compare the fetch() API with Swagger (an API documentation format) with Protobuf (a serialisation format)? That doesn't even make sense. Second paragraph: "The UI should automatically update everywhere the data is used". Again, what does this have to do with any of the above? That is state management, yeah, and you can build proper state management with any HTTP library and any message serialisation format. Request batching: How would that happen "automatically"? By waiting to fire requests and then batching them? UX when fetching data: What does that have to do with any of the above? You still have to decide how your UI displays the fact that a piece of data is loading. What do you expect there to be in place? Best thing I could imagine is to have a global operations indicators a la Adobe Lightroom which tells you how many HTTP requests are in flight. I could go on, but the last paragraphs maybe highlights the lack of understanding the author had: "UI Frameworks (at this point, React has won)". If React had "won" then why would we be having this discussion. React hasn't "won" because it solves one piece of the puzzle: Rendering. For every little other thing you have to incorporate another library or figure out your own solution: Routing, State Management, CRUD / HTTP API, etc. If anything, Ember.js would most closely fit the bill of incorporating most of the things the author seems to care about yet can't articulate clearly.

batoure 6y ago
What this person said! ^ almost all of the mentioned issues have solutions out their but the author is jumping around the stack with no sense of true purpose. How does graphql impact your SPA updating views, it doesn’t..
- chrisco255 6y ago
  Of course it does, because GraphQL allows you to roll up queries for separate entities and get the data back for those in a single HTTP request. And that capability is something Relay and Apollo take advantage of. In this way a component can reference only the data it needs and the query rollup and caching happens automatically.

gitpusher 6y ago

The author does a good job of summarizing the relevant aspects of fetching data in a modern web-app. However, by NO means should all of this be handled by one "data-fetching library" - as he suggests.

ficklepickle 6y ago

This is a solved problem with Nuxt/Vue and the fetch hook. Client side, server side, no problem. One of the many reasons I love nuxt/Vue.

whalesalad 6y ago

This post is hard to digest - it is a rant on 10 different things. All are valid hurdles but I don't see how we can just tie it all up in a bag and call it "data fetching". Architectures are what help solve these sorts of problems - not tools or libraries. Placing the blame on tools means the real issue has not been identified.

SimeVidas 6y ago

I’m curious if the author thinks that people can actually read the ultra-low-contrast text in the side column.

iskin 6y ago

Completely agree! I believe that op-log sync with CRDT-based atomic operations are a better solution. Logux is one of the example of this approach: https://logux.io/ https://logux.io/

super_hhh 6y ago

Vue > React

viviansolide 6y ago
Suspense feature is great.

anderspitman 6y ago

If your UI is complicated enough to worry about most things on this list, maybe simplify your UI?

beders 6y ago
Valid point, if your app is a todo list manager. There's a level of complexity in the domain logic where all these things mentioned by the author become pain points. That complexity can't be reduced by "simplify your UI" statements.
- anderspitman 6y ago
  Is that actually true, or is it what we've been told to believe by people selling frameworks?

irrational 6y ago

> UI frameworks (at this point, React has won) Isn't this like someone in 2011 saying "UI frameworks (at this point, jQuery has won)"? I think things like Svelte and other up and coming UI frameworks are still being developed because we all recognize that React is not the ultimate UI Framework. Perhaps we will never get there, but surely we can do better than where we are at.

deergomoo 6y ago
This is highly subjective of course, but I think Vue has already surpassed React from a technical and ease-of-use standpoint. Where React still leads handsomely is tooling (the official VS Code Vue plug-in is buggy as all hell), breadth of ecosystem, and employability.
pseudosavant 6y ago
^ React is just the new jQuery. We have people who basically don't really know HTML, CSS, or JavaScript but make React apps day-in day-out. Ten years ago the exact same thing was happening with jQuery. I still cringe when I see questions on StackOverflow where they ask "how can I XYZ with jQuery" where XYZ is something that has absolutely nothing to do with the DOM.

pier25 6y ago

Nobody has solved it because there is nothing to solve. That's like saying that "we to optimize SQL for UX". No, no we don't. Fetching data has nothing to do with managing state, rendering UI, or even UX. Each layer has different problems it needs to address. And btw you can totally do GraphQL requests with fetch().

asiando 6y ago

This applies to everything in the browser honestly. Why can’t I bind a variable to the DOM natively? I want the variable X to match the value of <input> and vice versa without having to set up a bunch of listeners and hope they don’t go in a loop.

uryga 6y ago
sounds like you want something like Vue baked into the browser. but i'd prefer React baked into the browser! we can't have both, so it's probably best we have neither :) seriously though, why not just write a simple wrapper around event listeners + maybe some proxy magic and get the semantics you prefer? or find a library that does that, i'm sure there's a bunch out there

swiley 6y ago

You can serialize pretty complex graphs of data and UI information in a new format called "HTML." The software capable of consuming this format is able to merge fragments of it into something called a DOM. If you're doing something more complex than this in your app then quit whining about browsers not being convenient for you to abuse. The users don't want your crap any more than browser implementers do.

vim-guru 6y ago

I think Fulcro has mechanisms to tick all your boxes and then some. Welcome to the future

slifin 6y ago

Server side react components has the best data fetching story I've ever seen They demoed an application that has a react component powered directly by a SQL query, when the client next fetched anything from that page, changes in terms of data coming out of the SQL query were automatically propagated to the client No data fetch API just coding like you would in a server side app with React handling the IO I hope someone starts implementing something like Drupal on top for automatic deadlock prevention, declarative data driven interfaces for forms, schema etc