4 ms·

Structured Page Fragments by YouTube

31 points by nikolay 11y ago

bshimmin 11y ago
I know it's practically a HackerNews trope by now to point this out, but this is a really poor name choice, given the abbreviation that they seem keen to use. (I'm referring to https://en.wikipedia.org/wiki/Sender_Policy_Framework https://en.wikipedia.org/wiki/Sender_Policy_Framework in case it's not obvious.)
jordanlev 11y ago
This looks great. I'm curious how the server knows which fragment of the page to return... or is there an assumption with this that only 1 section of your site changes from page to page?
- aaronem 11y ago
  I haven't used it myself, but judging from (a brief review of) the documentation, the server dispatches on the request URI and responds with content updates for an arbitrary number of fragments, each identified by DOM ID.
  - jordanlev 11y ago
    Ah, gotcha. So you are potentially returning more content then will be rendered, but still a faster situation then a full page reload. Thanks!
    - aaronem 11y ago
      Well, no; when I say "arbitrary number of fragments", I mean there isn't a set limit established by the client code, not that the server should promiscuously return data the client may or may not actually need. A correctly designed application should never make a state transition that requires the server to guess about what the client's going to need to render.
      - jordanlev 11y ago
        Understood, which brings me back to the original question of wondering how exactly the server knows which fragments the client wants.
        aaronem 11y ago
        Again, that's entirely a matter of application design. The general answer is that the server uses the request URI and HTTP headers to determine which fragments are needed, but there are as many specific answers as there are applications following this paradigm. (Said paradigm, incidentally, is just that of the single-page application; this is only YouTube's specific implementation of it. It sounds like you might get more use out of a general treatment of how SPA's work, than of a deeper dive into the details of the specific implementation under discussion; it might be worth your while to take a look at http://singlepageappbook.com http://singlepageappbook.com, which, while I haven't actually read it in any close detail, looks to be a pretty good overview and high-level discussion of the topic.)
        jordanlev 11y ago
        I appreciate your responses, but I think we got off track somehow. I'm quite familiar with the general architecture of SPA's... my question is specifically about how this library implements this feature. I didn't see anything in the docs about how to use URI or HTTP headers (or anything else) to specify which fragments you want -- instead it just says "?spf=navigate will be appended to the url". So I was wondering how this library specifically thinks more fine-grained details about what to return should be communicated to the server.
        aaronem 11y ago
        As far as I can tell, that's pretty much all you've got. Given the heavy concentration this library seems to place on converting static navigation to dynamic, my surmise is that, instead of responding to a navigation link with an entire HTML document, it just responds with the content that document would contain in the response object's "body" member, with content fragments organized by the ID of the element they're meant to populate. View changes seem to be supported by sending the appropriate template fragments in the "head", "title", and optionally "foot" members of the response JSON object. More complex state transitions such as form submissions appear to be provided for in the second argument of the spf.navigate method, which can take an object including a POST method specification and the corresponding data. It looks like that's about all the library provides in terms of request flexibility; per the code, setting custom request headers on navigation requests is definitely unavailable, but it doesn't do anything to override the default behavior by which XHR passes on e.g. Authorization headers already set by client code, so (in conjunction with e.g. OAuth) you'd be able to identify and authenticate the user making the request. I guess the idea is that, for the purpose the library's meant to answer, what it provides in the URI is just about all the information the server should need on what content to include in the response.
jolt 11y ago
I love YouTube. I use it daily. I have some grief with the interface though. It seems like it doesn't want to be a webpage. So much of it feels custom-made as opposed to browser based. Navigation, scrolling, and image loading breaks for me all the time regardless of browser. So for me it feels risky to adopt their interface implementation. I realize that YouTube is a crazy hard problem to tackle, and that is probably why stuff like this is necessary.
pjc50 11y ago
This sounds surprisingly like the old HTML "frames"..
- jordanlev 11y ago
  I suppose they are attempting to achieve the same goal, but this does not suffer from the drawbacks that frames do (most notably the weird "which page am I actually on"? confusion and potential for frames-within-frames-within-frames-etc).
swang 11y ago
This is similar to Facebook's "BigPipe"[0] right? 0: https://www.facebook.com/notes/facebook-engineering/bigpipe-pipelining-web-pages-for-high-performance/389414033919 https://www.facebook.com/notes/facebook-engineering/bigpipe-...
carsongross 11y ago
Interesting, but a lot heavier weight than the structural library that I built: http://intercoolerjs.org http://intercoolerjs.org. From the docs: In dynamic navigation, only fragments are sent, using JSON as transport. Why not HTML as a transport?
- kodablah 11y ago
  I don't think the added JSON.parse (which is very fast these days) is a "lot" heavier weight. Also, although I haven't looked, I would assume a more rigidly structured format like JSON for multiple fragments would be easier to programmatically reference different ones instead of parsing HTML or making some other assumptions about how the disparate fragments are delimited.
pwenzel 11y ago
Sounds like a fresh approach to PJAX (https://github.com/defunkt/jquery-pjax https://github.com/defunkt/jquery-pjax)
pdkl95 11y ago
Is this in any way different than the common rails feature? <div id="bar"></div> <% link_to "Load #{@widget}", @widget, remote: true, id: 'foo' %> $("#foo").on("ajax:success", function (e, data, status, xhr) { $("#bar").html(xhr.responseText); }); I thought rendering partials and dynamically replacing them has been a standard feature of most frameworks. Rails even renders a working link so the page works without the javascript.
triskweline 11y ago
We built something similar with Up.js [0]. Rather than wrapping some responses in JSON, Up.js doesn't require any server-side changes. Fragments are selected client-side via CSS selectors. 0: http://upjs.io/ http://upjs.io/

31 points by nikolay 11y ago

I know it's practically a HackerNews trope by now to point this out, but this is a really poor name choice, given the abbreviation that they seem keen to use. (I'm referring to https://en.wikipedia.org/wiki/Sender_Policy_Framework https://en.wikipedia.org/wiki/Sender_Policy_Framework in case it's not obvious.)

jordanlev 11y ago

This looks great. I'm curious how the server knows which fragment of the page to return... or is there an assumption with this that only 1 section of your site changes from page to page?

aaronem 11y ago
I haven't used it myself, but judging from (a brief review of) the documentation, the server dispatches on the request URI and responds with content updates for an arbitrary number of fragments, each identified by DOM ID.
- jordanlev 11y ago
  Ah, gotcha. So you are potentially returning more content then will be rendered, but still a faster situation then a full page reload. Thanks!
  - aaronem 11y ago
    Well, no; when I say "arbitrary number of fragments", I mean there isn't a set limit established by the client code, not that the server should promiscuously return data the client may or may not actually need. A correctly designed application should never make a state transition that requires the server to guess about what the client's going to need to render.
    - jordanlev 11y ago
      Understood, which brings me back to the original question of wondering how exactly the server knows which fragments the client wants.
      - aaronem 11y ago
        Again, that's entirely a matter of application design. The general answer is that the server uses the request URI and HTTP headers to determine which fragments are needed, but there are as many specific answers as there are applications following this paradigm. (Said paradigm, incidentally, is just that of the single-page application; this is only YouTube's specific implementation of it. It sounds like you might get more use out of a general treatment of how SPA's work, than of a deeper dive into the details of the specific implementation under discussion; it might be worth your while to take a look at http://singlepageappbook.com http://singlepageappbook.com, which, while I haven't actually read it in any close detail, looks to be a pretty good overview and high-level discussion of the topic.)
        jordanlev 11y ago
        I appreciate your responses, but I think we got off track somehow. I'm quite familiar with the general architecture of SPA's... my question is specifically about how this library implements this feature. I didn't see anything in the docs about how to use URI or HTTP headers (or anything else) to specify which fragments you want -- instead it just says "?spf=navigate will be appended to the url". So I was wondering how this library specifically thinks more fine-grained details about what to return should be communicated to the server.
        aaronem 11y ago
        As far as I can tell, that's pretty much all you've got. Given the heavy concentration this library seems to place on converting static navigation to dynamic, my surmise is that, instead of responding to a navigation link with an entire HTML document, it just responds with the content that document would contain in the response object's "body" member, with content fragments organized by the ID of the element they're meant to populate. View changes seem to be supported by sending the appropriate template fragments in the "head", "title", and optionally "foot" members of the response JSON object. More complex state transitions such as form submissions appear to be provided for in the second argument of the spf.navigate method, which can take an object including a POST method specification and the corresponding data. It looks like that's about all the library provides in terms of request flexibility; per the code, setting custom request headers on navigation requests is definitely unavailable, but it doesn't do anything to override the default behavior by which XHR passes on e.g. Authorization headers already set by client code, so (in conjunction with e.g. OAuth) you'd be able to identify and authenticate the user making the request. I guess the idea is that, for the purpose the library's meant to answer, what it provides in the URI is just about all the information the server should need on what content to include in the response.

jolt 11y ago

I love YouTube. I use it daily. I have some grief with the interface though. It seems like it doesn't want to be a webpage. So much of it feels custom-made as opposed to browser based. Navigation, scrolling, and image loading breaks for me all the time regardless of browser. So for me it feels risky to adopt their interface implementation. I realize that YouTube is a crazy hard problem to tackle, and that is probably why stuff like this is necessary.

pjc50 11y ago

This sounds surprisingly like the old HTML "frames"..

jordanlev 11y ago
I suppose they are attempting to achieve the same goal, but this does not suffer from the drawbacks that frames do (most notably the weird "which page am I actually on"? confusion and potential for frames-within-frames-within-frames-etc).

swang 11y ago

This is similar to Facebook's "BigPipe"[0] right? 0: https://www.facebook.com/notes/facebook-engineering/bigpipe-pipelining-web-pages-for-high-performance/389414033919 https://www.facebook.com/notes/facebook-engineering/bigpipe-...

carsongross 11y ago

Interesting, but a lot heavier weight than the structural library that I built: http://intercoolerjs.org http://intercoolerjs.org. From the docs: In dynamic navigation, only fragments are sent, using JSON as transport. Why not HTML as a transport?

kodablah 11y ago
I don't think the added JSON.parse (which is very fast these days) is a "lot" heavier weight. Also, although I haven't looked, I would assume a more rigidly structured format like JSON for multiple fragments would be easier to programmatically reference different ones instead of parsing HTML or making some other assumptions about how the disparate fragments are delimited.

pwenzel 11y ago

Sounds like a fresh approach to PJAX (https://github.com/defunkt/jquery-pjax https://github.com/defunkt/jquery-pjax)

pdkl95 11y ago

Is this in any way different than the common rails feature? <div id="bar"></div> <% link_to "Load #{@widget}", @widget, remote: true, id: 'foo' %> $("#foo").on("ajax:success", function (e, data, status, xhr) { $("#bar").html(xhr.responseText); }); I thought rendering partials and dynamically replacing them has been a standard feature of most frameworks. Rails even renders a working link so the page works without the javascript.

triskweline 11y ago

We built something similar with Up.js [0]. Rather than wrapping some responses in JSON, Up.js doesn't require any server-side changes. Fragments are selected client-side via CSS selectors. 0: http://upjs.io/ http://upjs.io/