8 ms·

The HN server uses a table of closures to implement those links (the id code for the closure is the bit after fnid= in the url). When the HN server starts runn

by mooism2 15y ago

The HN server uses a table of closures to implement those links (the id code for the closure is the bit after fnid= in the url). When the HN server starts running out of memory, it drops entries from this table. When your browser asks for an entry that is no longer in this table, you get the "Unknown or expired link" error. This is a crazy design, but unless someone would like to patch the source code and get PG to accept it, we're stuck with it.

gnaritas 15y ago
It's not crazy at all, it greatly simplifies development to use callbacks for actions rather than manually encoding the necessary state into the URL. Techniques such as this are what enable a single developer to be so productive by automating boring and time consuming stuff.
- srdev 15y ago
  Except it doesn't appear to work robustly, which makes it poor design. "Automating boring and time consuming stuff," is all well and good if it actually produces a functional system, but that concern is secondary to robustness.
  - blahedo 15y ago
    It'd work robustly enough if the links didn't expire, and if we believe other posts on this page, the links are expiring due to memory limits on the system. (The other possibility is a timeout, I guess, which is easily fixed.) If it's running out of memory to store the closures it would run out of memory to store the interaction state. In other words, there's a problem here, but it's not the programming model that pg chose.
    - srdev 15y ago
      Except the links do expire, so its not robust. I expect that when I visit a web page, I can let it sit for an extended period of time before moving on to the next page and have it work. HN doesn't work. Furthermore, the technique of holding important state authoritatively in memory like this is not a good web-development practice for various reasons. Doubly so if its state data which can be round-tripped. Links should not break when the web server or cache (I'm not sure which one it is) runs low on memory. So yes, there is a problem with the programming model that pg chose.
      - gnaritas 15y ago
        If he hadn't used that technique, there would be no hacker news for you to use at all. You're entirely missing the point that this is a technique to make hobby programming more fun, it's not about being robust or best practice, it's about making programming simpler so pg finds it worth his time to build this site in the first place.
        srdev 15y ago
        blahedo's point was that the technique was not fundamentally a problem from a robustness point of view, and I disagree with that point. It is a problem, and I was pointing that out. Your point seems to be that since Hacker News is a "hobby project," that we may forgive sacrificing a bit of robustness to make the programming exercise more pleasant. That point was not clear to me from your original posting. Rather, the point seemed to be that the technique was good because it was clever and fun, and I disagreed with that sentiment. PG seems to be saying elsewhere that it was used as a rapid prototyping technique. That seems to be a fair justification of the technique, in my estimation.
    - dasil003 15y ago
      > If it's running out of memory to store the closures it would run out of memory to store the interaction state. Not necessarily. The way I would approach this is to keep the link cache in memory, but have the links contain the minimal necessary state to reconstruct the link from disk-based storage in the case where the cache is gone. That gives the same excellent median performance without any breakage.
      - gnaritas 15y ago
        > The way I would approach this is to keep the link cache in memory, but have the links contain the minimal necessary state to reconstruct the link from disk-based storage in the case where the cache is gone. It's not a cache, and you clearly don't understand the issue. These are callbacks to closures, embedding the state in the URL is what they attempt to avoid because doing that is tedious.
        dasil003 15y ago
        Sorry, bad choice of words. I was just shooting from the hip here in response to the parent that suggests you can't both keep something in RAM for the majority of cases and still make it robust. The fact that embedding state in the URL is tedious is neither here nor there, but in any case it's a pretty garbage excuse. You know what's more tedious than writing code to pass a few integers around in links? Thousands of people losing carefully written paragraphs of enlightened prose on a regular basis. In fact the more carefully considered, the more likely the text is to be lost. If it took 24 hours or even 12 hours for links to expire then maybe you could justify the approach, but it seems to be well under an hour on average before a given closure is purged. This site seems hardly so complex as to be gaining much from a pure continuation approach, and if you can't ease this problem in a lisp then are all of us building services for non-hackers doomed to life of bitter tedium?
        gnaritas 15y ago
        You seem to not be aware of the architecture of this site, it's all run out of ram, no database; just simple lazy load on demand files on a single server running in a single process. Because of this, memory is tight, that's why those closures are purged. It simply means pg hasn't had the time to convert more of the prototyped code into production stateless code that always works. But that's his prerogative, this site is a hobby for him, he doesn't need to make any excuses about anything. If you don't like his site, go somewhere else.
  - gnaritas 15y ago
    > but that concern is secondary to robustness. To you, but that's a value judgement, it obviously was the other way around for pg. Had he not taken those shortcuts, there would be no hacker news at all; be thankful he automating that boring stuff and bothered to build the site.
- wvenable 15y ago
  Simplifies development? This is not a complicated piece of software and the techniques to build it are well known. You wouldn't need any more state than the id number you need for the callback anyway. Building broken software is always much easier than building robust correct software so this is hardly a good argument.
  - gnaritas 15y ago
    > You wouldn't need any more state than the id number you need for the callback anyway. I don't think you grasp the issue... the link is expired because the callback no longer exists to link to.
    - wvenable 15y ago
      No, I understand the issue. But you're presupposing a specific implementation here. If you were just designing this in, for example, PHP then you'd just need one piece of state: the page # (for more...) or the parent comment id (for the comment) and so on. The real issue is that there's a whole bunch of saved state on the server for operations that could be (and should be) completely stateless.
      - gnaritas 15y ago
        No, the issue is time, specifically, pg's time; using callbacks takes less programmer time than manually building every URL statelessly. Yes, it could be done another way, but it wouldn't exist at all if he'd had to do that for every link because it'd have taken too much of his time.
        wvenable 15y ago
        What is it about Arc or the architecture of HN that makes callbacks so easy and stateless URLs so hard? I've written a forum in under 3 hours the "traditional" way in PHP. I find the stateless concept a lot simpler in general. None of these links should require any server state at all.
- debaserab2 15y ago
  It's not productive to have a site that randomly locks out visitors. No matter how clever the code design is, this is a product flaw.
  - gnaritas 15y ago
    No, it's a design choice, he favors ease of programming more than user experience. You might not agree with that choice, but it's not a flaw, he did it on purpose and knew the consequences.
    - DougBTX 15y ago
      A better name might be technical debt. It is a flaw from the perspective of someone seeing the error message. But for the developer it is a way of saving development time, which can be paid off later to fix it.
      - gnaritas 15y ago
        Yes, this is a better way to say it.
    - wnight 15y ago
      It's not a feature so it's a flaw. You're right that pg did it this way in the beginning to save time but years have gone by and the site is now more central to his business - especially as a tech demo. This back and forth argument presupposes that there isn't a better fix than the naive 'use old-style code' solution. Anyways, the discussion is worth having. Only by pointing out problems do you fix them.
- mooism2 15y ago
  You're justifying code that doesn't work on the grounds that it was quick to write. (Facetious comment: if the website doesn't have to work, I can write the whole thing in under a minute. Someone wrote a HN clone on these lines a few weeks back, but I don't know how long it took them.) Given that the HN code was written by an increasingly busy man in his spare time, his use of an unreliable but quick-to-write implementation technique may be an acceptable trade-off. But it doesn't make the design any less crazy.
  - gnaritas 15y ago
    > You're justifying code that doesn't work on the grounds that it was quick to write. No, the code works fine for an acceptable period of time, it doesn't just not work.

by mooism2 15y ago

gnaritas 15y ago

It's not crazy at all, it greatly simplifies development to use callbacks for actions rather than manually encoding the necessary state into the URL. Techniques such as this are what enable a single developer to be so productive by automating boring and time consuming stuff.

srdev 15y ago
Except it doesn't appear to work robustly, which makes it poor design. "Automating boring and time consuming stuff," is all well and good if it actually produces a functional system, but that concern is secondary to robustness.
- blahedo 15y ago
  It'd work robustly enough if the links didn't expire, and if we believe other posts on this page, the links are expiring due to memory limits on the system. (The other possibility is a timeout, I guess, which is easily fixed.) If it's running out of memory to store the closures it would run out of memory to store the interaction state. In other words, there's a problem here, but it's not the programming model that pg chose.
  - srdev 15y ago
    Except the links do expire, so its not robust. I expect that when I visit a web page, I can let it sit for an extended period of time before moving on to the next page and have it work. HN doesn't work. Furthermore, the technique of holding important state authoritatively in memory like this is not a good web-development practice for various reasons. Doubly so if its state data which can be round-tripped. Links should not break when the web server or cache (I'm not sure which one it is) runs low on memory. So yes, there is a problem with the programming model that pg chose.
    - gnaritas 15y ago
      If he hadn't used that technique, there would be no hacker news for you to use at all. You're entirely missing the point that this is a technique to make hobby programming more fun, it's not about being robust or best practice, it's about making programming simpler so pg finds it worth his time to build this site in the first place.
      - srdev 15y ago
        blahedo's point was that the technique was not fundamentally a problem from a robustness point of view, and I disagree with that point. It is a problem, and I was pointing that out. Your point seems to be that since Hacker News is a "hobby project," that we may forgive sacrificing a bit of robustness to make the programming exercise more pleasant. That point was not clear to me from your original posting. Rather, the point seemed to be that the technique was good because it was clever and fun, and I disagreed with that sentiment. PG seems to be saying elsewhere that it was used as a rapid prototyping technique. That seems to be a fair justification of the technique, in my estimation.
  - dasil003 15y ago
    > If it's running out of memory to store the closures it would run out of memory to store the interaction state. Not necessarily. The way I would approach this is to keep the link cache in memory, but have the links contain the minimal necessary state to reconstruct the link from disk-based storage in the case where the cache is gone. That gives the same excellent median performance without any breakage.
    - gnaritas 15y ago
      > The way I would approach this is to keep the link cache in memory, but have the links contain the minimal necessary state to reconstruct the link from disk-based storage in the case where the cache is gone. It's not a cache, and you clearly don't understand the issue. These are callbacks to closures, embedding the state in the URL is what they attempt to avoid because doing that is tedious.
      - dasil003 15y ago
        Sorry, bad choice of words. I was just shooting from the hip here in response to the parent that suggests you can't both keep something in RAM for the majority of cases and still make it robust. The fact that embedding state in the URL is tedious is neither here nor there, but in any case it's a pretty garbage excuse. You know what's more tedious than writing code to pass a few integers around in links? Thousands of people losing carefully written paragraphs of enlightened prose on a regular basis. In fact the more carefully considered, the more likely the text is to be lost. If it took 24 hours or even 12 hours for links to expire then maybe you could justify the approach, but it seems to be well under an hour on average before a given closure is purged. This site seems hardly so complex as to be gaining much from a pure continuation approach, and if you can't ease this problem in a lisp then are all of us building services for non-hackers doomed to life of bitter tedium?
        gnaritas 15y ago
        You seem to not be aware of the architecture of this site, it's all run out of ram, no database; just simple lazy load on demand files on a single server running in a single process. Because of this, memory is tight, that's why those closures are purged. It simply means pg hasn't had the time to convert more of the prototyped code into production stateless code that always works. But that's his prerogative, this site is a hobby for him, he doesn't need to make any excuses about anything. If you don't like his site, go somewhere else.
- gnaritas 15y ago
  > but that concern is secondary to robustness. To you, but that's a value judgement, it obviously was the other way around for pg. Had he not taken those shortcuts, there would be no hacker news at all; be thankful he automating that boring stuff and bothered to build the site.
wvenable 15y ago
Simplifies development? This is not a complicated piece of software and the techniques to build it are well known. You wouldn't need any more state than the id number you need for the callback anyway. Building broken software is always much easier than building robust correct software so this is hardly a good argument.
- gnaritas 15y ago
  > You wouldn't need any more state than the id number you need for the callback anyway. I don't think you grasp the issue... the link is expired because the callback no longer exists to link to.
  - wvenable 15y ago
    No, I understand the issue. But you're presupposing a specific implementation here. If you were just designing this in, for example, PHP then you'd just need one piece of state: the page # (for more...) or the parent comment id (for the comment) and so on. The real issue is that there's a whole bunch of saved state on the server for operations that could be (and should be) completely stateless.
    - gnaritas 15y ago
      No, the issue is time, specifically, pg's time; using callbacks takes less programmer time than manually building every URL statelessly. Yes, it could be done another way, but it wouldn't exist at all if he'd had to do that for every link because it'd have taken too much of his time.
      - wvenable 15y ago
        What is it about Arc or the architecture of HN that makes callbacks so easy and stateless URLs so hard? I've written a forum in under 3 hours the "traditional" way in PHP. I find the stateless concept a lot simpler in general. None of these links should require any server state at all.
debaserab2 15y ago
It's not productive to have a site that randomly locks out visitors. No matter how clever the code design is, this is a product flaw.
- gnaritas 15y ago
  No, it's a design choice, he favors ease of programming more than user experience. You might not agree with that choice, but it's not a flaw, he did it on purpose and knew the consequences.
  - DougBTX 15y ago
    A better name might be technical debt. It is a flaw from the perspective of someone seeing the error message. But for the developer it is a way of saving development time, which can be paid off later to fix it.
    - gnaritas 15y ago
      Yes, this is a better way to say it.
  - wnight 15y ago
    It's not a feature so it's a flaw. You're right that pg did it this way in the beginning to save time but years have gone by and the site is now more central to his business - especially as a tech demo. This back and forth argument presupposes that there isn't a better fix than the naive 'use old-style code' solution. Anyways, the discussion is worth having. Only by pointing out problems do you fix them.
mooism2 15y ago
You're justifying code that doesn't work on the grounds that it was quick to write. (Facetious comment: if the website doesn't have to work, I can write the whole thing in under a minute. Someone wrote a HN clone on these lines a few weeks back, but I don't know how long it took them.) Given that the HN code was written by an increasingly busy man in his spare time, his use of an unreliable but quick-to-write implementation technique may be an acceptable trade-off. But it doesn't make the design any less crazy.
- gnaritas 15y ago
  > You're justifying code that doesn't work on the grounds that it was quick to write. No, the code works fine for an acceptable period of time, it doesn't just not work.