3 ms·

The site is open source. Why not code a solution and send it along to PG?

by ericb 18y ago

jcl 18y ago
As I understand it, the problem is that the web server stores the user session state in continuations, and old continuations need to be discarded regularly to keep the server from filling up. The error appears when the user tries to do something that requires a continuation that has already been discarded. The easiest solution, then, is to simply increase the length of time that the continuations are held -- hence increasing the number of continuations held at once. If this were as simple as changing a number somewhere, I assume PG would have already done it. Hence it is likely that the server is already retaining as many continuations as it can at any one time, expiring them as it goes. So the obvious solutions are: Get a heftier web server; rewrite mzscheme's continuations to take less resources; or rewrite HN to not use continuations. The first cannot be done with source code alone, the second has yet to be accomplished by Scheme's brightest minds, and the third is unlikely to appeal to PG. (Edit: Not that I want to discourage anyone from improving Scheme, arc, or HN... just pointing out that the solution is harder than a simple bug fix.)
- jganetsk 18y ago
  Disclaimer: I haven't looked at the source yet. My particular example was for browsing the thread list. Let's s/continuation/browse context/. You can generate exactly one browse continuation a minute, and hand that one out to everyone who visits the page during that minute. It contains the effective state for that continuation. So it's a little out of date... big deal. Let's say a browse context is 1 kilobyte (which can fit a list of several hundred IDs and scores). With 1 gigabyte of RAM, you can store 1.9 years worth of browse contexts.
  - palish 18y ago
    "Browse contexts" would not be able to be stored client-side, so the server cannot hand one out to everyone who visits a page. The server hands out an ID that references a browse context that exists on the server. Each browse context must be unique if you're logged in (otherwise you would see News.YC from someone else's perspective). When a user is logged in, one browse context is generated per link-that-requires-a-browse-context. Each "reply" link references a new browse context each time a page is loaded, for example. So if there are 50 comments on a page, then there are 50 "reply" links, each of which references a unique browse context. (If you mouseover each reply link in Firefox, you can see the ID of its browse context in the status bar. If you refresh the page, each reply link then references a different, unique, server-side browse context.) So it's easy to see why this could get a little expensive. It's essentially a time-consuming optimization problem to get rid of the "unknown or expired link" error, at this point. And it is not hard to work around.
    - jcl 18y ago
      Hmm... I notice, however, that the "reply" link takes you to a fairly similar page as the "link" link in the comment's header: both contain a reply box. The "link" link, however, doesn't contain a fnid in the URL -- just a REST-like comment id. I wonder if these fnid-less URLs avoid generating continuations? If so, perhaps we could drop the "reply" link entirely and just label the "link" link as "link/reply"? If I understand your explanation correctly, that would both eliminate the error for initial replies and dramatically reduce the rate that continuations expire when using other site features...
      - palish 18y ago
        Yes, you're correct. That would both eliminate the error for initial replies and dramatically reduce the rate that continuations expire when using other site features. Unfortunately, that would completely remove the ability to "reply". :) When a logged-in user loads a Hacker News page, the server must generate a unique set of continuations for that user (one per each link that requires a continuation.) In other words, if everyone shared the same set of continuations, everyone would see the exact same Hacker News. The username in the upper-right corner would be the same, everyone's comments would be tagged with the same username, etc. Does that make sense? So the "reply" link requires a continuation because you're adding a comment that's tagged with your username. But the "link" link does not require one because you're simply viewing a comment.
        jcl 18y ago
        Did you click on the "link" link? It has a box and reply button just like the reply page. So removing the "reply" link wouldn't remove the ability to reply. I assume the inputs for the reply function are adequately captured by the URL (comment ID) and cookies (logged-in user) and thus don't require a continuation.
        palish 18y ago
        Oh snaps son. I got burned. You're right, I didn't click on the "link" link and assumed that it was only for viewing the comment. I sincerely apologize.
- soundsop 18y ago
  Could you write old continuations to disk? That way older continuations would take longer to resurrect, because they would come from disk, but at least they would still be around. You could probably store them for a few days before expiring them from the disk cache. I guess this could totally thrash performance by causing a lot of disk I/O, but I have no sense of how likely this is. If you had some statistics about the rate of continuation-cache misses and how often cache evictions occur, you could probably get a good idea whether it's feasible.
  - jcl 18y ago
    The impression I get from the following newsgroup thread is that mzscheme doesn't support serializing continuations to disk... at least not easily: http://www.cs.brown.edu/pipermail/plt-scheme/2007-February/016343.html http://www.cs.brown.edu/pipermail/plt-scheme/2007-February/0... Perhaps it is possible under other Schemes, though?

The site is open source. Why not code a solution and send it along to PG?

by ericb 18y ago

The site is open source. Why not code a solution and send it along to PG?

jcl 18y ago

As I understand it, the problem is that the web server stores the user session state in continuations, and old continuations need to be discarded regularly to keep the server from filling up. The error appears when the user tries to do something that requires a continuation that has already been discarded. The easiest solution, then, is to simply increase the length of time that the continuations are held -- hence increasing the number of continuations held at once. If this were as simple as changing a number somewhere, I assume PG would have already done it. Hence it is likely that the server is already retaining as many continuations as it can at any one time, expiring them as it goes. So the obvious solutions are: Get a heftier web server; rewrite mzscheme's continuations to take less resources; or rewrite HN to not use continuations. The first cannot be done with source code alone, the second has yet to be accomplished by Scheme's brightest minds, and the third is unlikely to appeal to PG. (Edit: Not that I want to discourage anyone from improving Scheme, arc, or HN... just pointing out that the solution is harder than a simple bug fix.)

jganetsk 18y ago
Disclaimer: I haven't looked at the source yet. My particular example was for browsing the thread list. Let's s/continuation/browse context/. You can generate exactly one browse continuation a minute, and hand that one out to everyone who visits the page during that minute. It contains the effective state for that continuation. So it's a little out of date... big deal. Let's say a browse context is 1 kilobyte (which can fit a list of several hundred IDs and scores). With 1 gigabyte of RAM, you can store 1.9 years worth of browse contexts.
- palish 18y ago
  "Browse contexts" would not be able to be stored client-side, so the server cannot hand one out to everyone who visits a page. The server hands out an ID that references a browse context that exists on the server. Each browse context must be unique if you're logged in (otherwise you would see News.YC from someone else's perspective). When a user is logged in, one browse context is generated per link-that-requires-a-browse-context. Each "reply" link references a new browse context each time a page is loaded, for example. So if there are 50 comments on a page, then there are 50 "reply" links, each of which references a unique browse context. (If you mouseover each reply link in Firefox, you can see the ID of its browse context in the status bar. If you refresh the page, each reply link then references a different, unique, server-side browse context.) So it's easy to see why this could get a little expensive. It's essentially a time-consuming optimization problem to get rid of the "unknown or expired link" error, at this point. And it is not hard to work around.
  - jcl 18y ago
    Hmm... I notice, however, that the "reply" link takes you to a fairly similar page as the "link" link in the comment's header: both contain a reply box. The "link" link, however, doesn't contain a fnid in the URL -- just a REST-like comment id. I wonder if these fnid-less URLs avoid generating continuations? If so, perhaps we could drop the "reply" link entirely and just label the "link" link as "link/reply"? If I understand your explanation correctly, that would both eliminate the error for initial replies and dramatically reduce the rate that continuations expire when using other site features...
    - palish 18y ago
      Yes, you're correct. That would both eliminate the error for initial replies and dramatically reduce the rate that continuations expire when using other site features. Unfortunately, that would completely remove the ability to "reply". :) When a logged-in user loads a Hacker News page, the server must generate a unique set of continuations for that user (one per each link that requires a continuation.) In other words, if everyone shared the same set of continuations, everyone would see the exact same Hacker News. The username in the upper-right corner would be the same, everyone's comments would be tagged with the same username, etc. Does that make sense? So the "reply" link requires a continuation because you're adding a comment that's tagged with your username. But the "link" link does not require one because you're simply viewing a comment.
      - jcl 18y ago
        Did you click on the "link" link? It has a box and reply button just like the reply page. So removing the "reply" link wouldn't remove the ability to reply. I assume the inputs for the reply function are adequately captured by the URL (comment ID) and cookies (logged-in user) and thus don't require a continuation.
        palish 18y ago
        Oh snaps son. I got burned. You're right, I didn't click on the "link" link and assumed that it was only for viewing the comment. I sincerely apologize.
soundsop 18y ago
Could you write old continuations to disk? That way older continuations would take longer to resurrect, because they would come from disk, but at least they would still be around. You could probably store them for a few days before expiring them from the disk cache. I guess this could totally thrash performance by causing a lot of disk I/O, but I have no sense of how likely this is. If you had some statistics about the rate of continuation-cache misses and how often cache evictions occur, you could probably get a good idea whether it's feasible.
- jcl 18y ago
  The impression I get from the following newsgroup thread is that mzscheme doesn't support serializing continuations to disk... at least not easily: http://www.cs.brown.edu/pipermail/plt-scheme/2007-February/016343.html http://www.cs.brown.edu/pipermail/plt-scheme/2007-February/0... Perhaps it is possible under other Schemes, though?