4 ms·

There's a reason things are set up as they are: * Although HTTP connections are normally stateless, HN has a login system for account holders, which means ther

by lutusp 13y ago

There's a reason things are set up as they are: * Although HTTP connections are normally stateless, HN has a login system for account holders, which means there's a state, a connection, associated with each logged-in user. * Because of the state information associated with logins, A Web server has a limited number of login connections it can handle. * To serve the maximum number of visitors, a server needs to recycle connections that are no longer in use. * There's no reliable heartbeat method between browser and server that a server can use to determine whether a given connection is still in use. Also, heartbeats would represent an extra traffic load on the server. And now that tabs have become popular, there's no way to be sure that a heartbeat response isn't from an unused, forgotten tab among dozens. * Therefore, after a connection has not seen packet requests for a set time interval, the server expires the connection to make it available to other visitors. * The shorter the timeout, the more connections the server can handle. > I would suggest a sensible default of 86400 seconds. Let's say the server can handle 2048 simultaneous login sessions (I don't know the actual number). According to your suggestion, after the first 2048 visitors log in, the server would refuse to accept any new logins for 24 hours. This isn't realistic. Here's how I handle the problem -- if I decide a reply is going to take more than a few seconds, I transfer my reply to a text editor and let the connection expire. When my long-winded reply is ready, I refresh the original page, paste my reply in, and click "Add Comment". I hope this helps.

krapp 13y ago
Although - other sites can manage user state and validating posts just fine without this 'feature.' Even assuming it's all but impossible to fix given the implementation in Arc, surely it would be possible to automatically reassociate the post with whatever the current function id is? It seems to be as much a user experience problem as anything.
- jayrox 13y ago
  i always figured it was because they built each page and cached it to html, then that cache was set to expire after 5 minutes. then once the cache expired you couldn't go to the next page or what ever because the cached file was deleted. IMO the reasons presented for this "Feature" should be changed to "lazy programming". cause that's all it is.
  - krapp 13y ago
    Even then it would make more sense if the location that new comments were POSTed to was not necessarily one which could expire. Would it be impossible for the API (or whatever passes for an API) to present a static endpoint for receiving comments?
    - jayrox 13y ago
      its not impossible, just appears to be coded by someone who over did it with caching in the beginning before the site needed it.
exelib 13y ago
Then it's broken by design. There is no sense to do so, but a great way to anger visitors and destroy UX.
nwh 13y ago
I don't think this is what is going on here, it makes no sense for it to operate like this. To demonstrate this is not the case, I just momentarily unplugged my NIC and then submitted this comment. The connection was definitely dropped and re-associated, but my comment passed perfectly fine. This is more likely a very short cache time on CSRF tokens rather than anything to do with associated sockets.
- phamilton 13y ago
  Http is stateless, so there are no "known sockets". Every request opens a new socket. Unplugging and plugging back in is a no-op in this context.
- brudgers 13y ago
  The architectural model your comment and the initial post assumes to underlay the behavior is incorrect. The reason for 'Unknown or Expired Link' is that HN runs on a webserver that uses continuations to maintain state. The continuations are stored in a table and based on load individual table entries get flushed to make room for new entries as needed. The message is actually always the result of an unknown link because the hash is no longer in the table. Your experiment worked because the relevant continuation [state reference] was available after disconnect e.g. the 'route' so to speak was still valid. Now obviously one might wonder why HN works this way. The reason is that the HN codebase came about as a way of field testing Arc [Graham's 100 year language]. Until rather recently, HN ran on a single server. It still uses relatively few lines of code.
  - acdha 13y ago
    > Now obviously one might wonder why HN works this way. The reason is that the HN codebase came about as a way of field testing Arc [Graham's 100 year language]. This is really the key point: continuations are the wrong way to write this kind of application. It's significantly more work and there's no benefit from doing so but it allowed someone to test a pet idea and, having done so, fixing it would mean admitting that doing something because it's cool isn't good engineering.
Quai 13y ago
This is not the case. Websites, including hacker news, does not handle sessions like that.
kijin 13y ago
CSRF tokens != login sessions != persistent HTTP connections. There is no need to keep an HTTP connection open in order for the server to log people in or recognize when a valid token has been submitted with a form.
grey-area 13y ago
Because of the state information associated with logins, A Web server has a limited number of login connections it can handle. You're making excuses for a particular setup - this is not something which holds true of all web servers. It's quite possible to store the state necessary to allow 10000 simultaneous logins if you use cookies and a db to store your state, rather than relying on in-memory state and fragile continuation links like this (created a few minutes ago): https://news.ycombinator.com/x?fnid=l5qgIWgvnRbHE7owLEaPMw https://news.ycombinator.com/x?fnid=l5qgIWgvnRbHE7owLEaPMw or tying replies to stored state on the server which expires within minutes. Many large scale sites manage it, I suppose because they don't store data like this in memory or urls. I'm not sure about this idea of storing state in urls, it often backfires, and urls should identify resources and actions, not store state. There is no good reason the server cannot receive a POST request from an authenticated user against a node id, and perform the requested action (post my reply please), no matter how many intervening logins/actions it has handled. Looking at the form in question (reply forms) reveals why it does break: <input type="hidden" name="fnid" value="UswPVmT3a8lZnqPBtcx359"> The server stores a continuation in the comment reply form, which (like the more link above) expires very quickly. The reason it expires is presumably because the server keeps a list of these fnids in memory, and as you say the list is finite, but it doesn't have to do that. There are other ways to create a CSRF (if that's what they use this for) which won't expire so quickly, and this problem is not one which many sites experience. It is annoying just how quickly reply links, more links etc expire on this site, and if the site used a different method to sign reply forms or construct other links they could get rid of this issue. Keeping data in memory like this will become more and more of a problem as the site scales - eventually things would break in seconds with enough load - that's clearly not acceptable.
zackbrown 13y ago
A login system does introduce a notion of state between the server and client, but the use of the word 'connection' to describe that is probably a misnomer. I'm unaware of any basic HTTP servers that use a persistent socket connection (which seems to be the sort of connection you're describing, given the resource constraints you suggest) with clients in order to manage authentication. Most often, the server just needs to set a flag associating an auth token with a given user (for whom it has a persistent representation, e.g. in a database.) Lookups against this token => user hash/map/table should be very cheap, especially if it's kept in memory. Further, even for tens of thousands of concurrent users, that's not a lot of memory overhead (1KB/authed user would be a generous upper bound, and that would still only be 10MB of RAM per 10k concurrent users.) Could be that HN is running on a 1990s-era desktop with 256MB of memory in someone's office and that this is an actual resource constraint issue--but my hunch is that this problem lies on the software side, not the hardware/resource side.
newobj 13y ago
I'm not sure what's more disturbing; this reply, or the fact that only 2 out of 5 replies to this reply attempt to refute it.
ordinary 13y ago
This is largely correct. Each reply form includes a hidden input field with the name 'fnid' in their form. Many other pages include a parameter with the same name. This is the unique identifier of a closure that's associated with the page you're on. These closures are deleted after X time, at which point requests from that page can no longer be handled correctly. Yes, obviously it would be great if X were increased. But doing so would require an increase in server resources or a significant change in the way HN's backend works; neither of these options is free. That is the reason why things are this way. This is an old debate. See, for example: https://github.com/HackerNews/HN/issues/11 https://github.com/HackerNews/HN/issues/11
- grey-area 13y ago
  fnids are evil, they just need to drop them and respond properly to posts without them. What more state needs to be recorded which is not already encoded in a POST reply request {user:n,post:7651593,text:'xxx'}? The solution should be pretty straightforward: Reply forms -> Remove fnid, authenticate on post, add CSRF Flag links -> Remove fnid, authenticate on post, add CSRF They already check authentication on these actions anyway as you'll see if you try upvote links from another user without login, all that would be required would be to use CSRF instead of these closures to handle posts like this - the closures are not a workable solution on a large site, and I'd contend are dangerous on a small site - they throw away a lot of the advantages of HTTP for no appreciable gains. Things like the more link are particularly pointless, they could just use something like: https://news.ycombinator.com/x?fnid=xxx -> https://news.ycombinator.com/news/2 This would change over time like the existing home page, but there's no reason for a simple GET request like this to expire.
  - quasque 13y ago
    The fnid is a CSRF token, just with an disadvantageous implementation that leads to a far too short expiry time. I don't know much about the internals of Arc, but maybe it would be possible to serialise the continuation data, combine it with a reasonable expiry time, apply some form of authenticated encryption to this, and supply that as the fnid? And reverse the process - with appropriate integrity checks - when the fnid is submitted. Then you have your state distributed at the client-side instead of all being kept on the server, so it can scale more effectively. EDIT: Nevermind, just realised that this very suggestion is addressed here, and it's more difficult than I anticipated https://github.com/HackerNews/HN/issues/11#issuecomment-32152432 https://github.com/HackerNews/HN/issues/11#issuecomment-3215...
acdha 13y ago
> * Because of the state information associated with logins, A Web server has a limited number of login connections it can handle. a) this is technically incorrect – you can trivially handle this using signed cookies to store the logged in user b) even using traditional database or filesystem based approaches, those limits would be aggressive even if the site was hosted on an iPhone – you're talking tens of bytes per session. > To serve the maximum number of visitors, a server needs to recycle connections that are no longer in use. Also technically wrong: if you aren't doing anything, you aren't causing any load whatsoever on the server. This would be different for something like a chat site using WebSockets but for a vanilla app like HN this is not an issue. Also, again, even if you were using a persistent connection the absolute technical limits are much, much higher than the traffic HN sees - hundreds of thousands of simultaneous users. > There's no reliable heartbeat method between browser and server that a server can use to determine whether a given connection is still in use. Again, technically wrong: plain HTTP sites don't maintain persistent connections but in the case of something like a WebSocket connection you'd simply use TCP keepalives, which are a mature standard feature and very low overhead. You appear to have confused the concept of a session with something which is actually happening at the network level. Try reading something about the HTTP request cycle (e.g. http://www.slideshare.net/cczona/full-stack-full-circle-what-the-heck-happens-in-an-http-requestresponse-cycle http://www.slideshare.net/cczona/full-stack-full-circle-what...) for basic background and after that move on to some discussion about the work done to make servers which can handle many simultaneous connections - the C10K problem was the original work tuning operating systems to handle tens of thousands of simultaneous connections and was large a non-issue at the network / basic HTTP leel by the mid-2000s: http://bulk.fefe.de/scalability/ http://bulk.fefe.de/scalability/ By now a single server can be assumed to handle hundreds of thousands of simultaneous connections at the network level – assuming, of course, that you have capacity to actual do whatever work those connections are used for. Idle users typing text into a form wouldn't be a problem even if it did use a persistent connection.

by lutusp 13y ago

krapp 13y ago

Although - other sites can manage user state and validating posts just fine without this 'feature.' Even assuming it's all but impossible to fix given the implementation in Arc, surely it would be possible to automatically reassociate the post with whatever the current function id is? It seems to be as much a user experience problem as anything.

jayrox 13y ago
i always figured it was because they built each page and cached it to html, then that cache was set to expire after 5 minutes. then once the cache expired you couldn't go to the next page or what ever because the cached file was deleted. IMO the reasons presented for this "Feature" should be changed to "lazy programming". cause that's all it is.
- krapp 13y ago
  Even then it would make more sense if the location that new comments were POSTed to was not necessarily one which could expire. Would it be impossible for the API (or whatever passes for an API) to present a static endpoint for receiving comments?
  - jayrox 13y ago
    its not impossible, just appears to be coded by someone who over did it with caching in the beginning before the site needed it.

exelib 13y ago

Then it's broken by design. There is no sense to do so, but a great way to anger visitors and destroy UX.

nwh 13y ago

I don't think this is what is going on here, it makes no sense for it to operate like this. To demonstrate this is not the case, I just momentarily unplugged my NIC and then submitted this comment. The connection was definitely dropped and re-associated, but my comment passed perfectly fine. This is more likely a very short cache time on CSRF tokens rather than anything to do with associated sockets.

phamilton 13y ago
Http is stateless, so there are no "known sockets". Every request opens a new socket. Unplugging and plugging back in is a no-op in this context.
brudgers 13y ago
The architectural model your comment and the initial post assumes to underlay the behavior is incorrect. The reason for 'Unknown or Expired Link' is that HN runs on a webserver that uses continuations to maintain state. The continuations are stored in a table and based on load individual table entries get flushed to make room for new entries as needed. The message is actually always the result of an unknown link because the hash is no longer in the table. Your experiment worked because the relevant continuation [state reference] was available after disconnect e.g. the 'route' so to speak was still valid. Now obviously one might wonder why HN works this way. The reason is that the HN codebase came about as a way of field testing Arc [Graham's 100 year language]. Until rather recently, HN ran on a single server. It still uses relatively few lines of code.
- acdha 13y ago
  > Now obviously one might wonder why HN works this way. The reason is that the HN codebase came about as a way of field testing Arc [Graham's 100 year language]. This is really the key point: continuations are the wrong way to write this kind of application. It's significantly more work and there's no benefit from doing so but it allowed someone to test a pet idea and, having done so, fixing it would mean admitting that doing something because it's cool isn't good engineering.

Quai 13y ago

This is not the case. Websites, including hacker news, does not handle sessions like that.

kijin 13y ago

CSRF tokens != login sessions != persistent HTTP connections. There is no need to keep an HTTP connection open in order for the server to log people in or recognize when a valid token has been submitted with a form.

grey-area 13y ago

Because of the state information associated with logins, A Web server has a limited number of login connections it can handle. You're making excuses for a particular setup - this is not something which holds true of all web servers. It's quite possible to store the state necessary to allow 10000 simultaneous logins if you use cookies and a db to store your state, rather than relying on in-memory state and fragile continuation links like this (created a few minutes ago): https://news.ycombinator.com/x?fnid=l5qgIWgvnRbHE7owLEaPMw https://news.ycombinator.com/x?fnid=l5qgIWgvnRbHE7owLEaPMw or tying replies to stored state on the server which expires within minutes. Many large scale sites manage it, I suppose because they don't store data like this in memory or urls. I'm not sure about this idea of storing state in urls, it often backfires, and urls should identify resources and actions, not store state. There is no good reason the server cannot receive a POST request from an authenticated user against a node id, and perform the requested action (post my reply please), no matter how many intervening logins/actions it has handled. Looking at the form in question (reply forms) reveals why it does break: <input type="hidden" name="fnid" value="UswPVmT3a8lZnqPBtcx359"> The server stores a continuation in the comment reply form, which (like the more link above) expires very quickly. The reason it expires is presumably because the server keeps a list of these fnids in memory, and as you say the list is finite, but it doesn't have to do that. There are other ways to create a CSRF (if that's what they use this for) which won't expire so quickly, and this problem is not one which many sites experience. It is annoying just how quickly reply links, more links etc expire on this site, and if the site used a different method to sign reply forms or construct other links they could get rid of this issue. Keeping data in memory like this will become more and more of a problem as the site scales - eventually things would break in seconds with enough load - that's clearly not acceptable.

zackbrown 13y ago

A login system does introduce a notion of state between the server and client, but the use of the word 'connection' to describe that is probably a misnomer. I'm unaware of any basic HTTP servers that use a persistent socket connection (which seems to be the sort of connection you're describing, given the resource constraints you suggest) with clients in order to manage authentication. Most often, the server just needs to set a flag associating an auth token with a given user (for whom it has a persistent representation, e.g. in a database.) Lookups against this token => user hash/map/table should be very cheap, especially if it's kept in memory. Further, even for tens of thousands of concurrent users, that's not a lot of memory overhead (1KB/authed user would be a generous upper bound, and that would still only be 10MB of RAM per 10k concurrent users.) Could be that HN is running on a 1990s-era desktop with 256MB of memory in someone's office and that this is an actual resource constraint issue--but my hunch is that this problem lies on the software side, not the hardware/resource side.

newobj 13y ago

I'm not sure what's more disturbing; this reply, or the fact that only 2 out of 5 replies to this reply attempt to refute it.

ordinary 13y ago

This is largely correct. Each reply form includes a hidden input field with the name 'fnid' in their form. Many other pages include a parameter with the same name. This is the unique identifier of a closure that's associated with the page you're on. These closures are deleted after X time, at which point requests from that page can no longer be handled correctly. Yes, obviously it would be great if X were increased. But doing so would require an increase in server resources or a significant change in the way HN's backend works; neither of these options is free. That is the reason why things are this way. This is an old debate. See, for example: https://github.com/HackerNews/HN/issues/11 https://github.com/HackerNews/HN/issues/11

grey-area 13y ago
fnids are evil, they just need to drop them and respond properly to posts without them. What more state needs to be recorded which is not already encoded in a POST reply request {user:n,post:7651593,text:'xxx'}? The solution should be pretty straightforward: Reply forms -> Remove fnid, authenticate on post, add CSRF Flag links -> Remove fnid, authenticate on post, add CSRF They already check authentication on these actions anyway as you'll see if you try upvote links from another user without login, all that would be required would be to use CSRF instead of these closures to handle posts like this - the closures are not a workable solution on a large site, and I'd contend are dangerous on a small site - they throw away a lot of the advantages of HTTP for no appreciable gains. Things like the more link are particularly pointless, they could just use something like: https://news.ycombinator.com/x?fnid=xxx -> https://news.ycombinator.com/news/2 This would change over time like the existing home page, but there's no reason for a simple GET request like this to expire.
- quasque 13y ago
  The fnid is a CSRF token, just with an disadvantageous implementation that leads to a far too short expiry time. I don't know much about the internals of Arc, but maybe it would be possible to serialise the continuation data, combine it with a reasonable expiry time, apply some form of authenticated encryption to this, and supply that as the fnid? And reverse the process - with appropriate integrity checks - when the fnid is submitted. Then you have your state distributed at the client-side instead of all being kept on the server, so it can scale more effectively. EDIT: Nevermind, just realised that this very suggestion is addressed here, and it's more difficult than I anticipated https://github.com/HackerNews/HN/issues/11#issuecomment-32152432 https://github.com/HackerNews/HN/issues/11#issuecomment-3215...

acdha 13y ago

> * Because of the state information associated with logins, A Web server has a limited number of login connections it can handle. a) this is technically incorrect – you can trivially handle this using signed cookies to store the logged in user b) even using traditional database or filesystem based approaches, those limits would be aggressive even if the site was hosted on an iPhone – you're talking tens of bytes per session. > To serve the maximum number of visitors, a server needs to recycle connections that are no longer in use. Also technically wrong: if you aren't doing anything, you aren't causing any load whatsoever on the server. This would be different for something like a chat site using WebSockets but for a vanilla app like HN this is not an issue. Also, again, even if you were using a persistent connection the absolute technical limits are much, much higher than the traffic HN sees - hundreds of thousands of simultaneous users. > There's no reliable heartbeat method between browser and server that a server can use to determine whether a given connection is still in use. Again, technically wrong: plain HTTP sites don't maintain persistent connections but in the case of something like a WebSocket connection you'd simply use TCP keepalives, which are a mature standard feature and very low overhead. You appear to have confused the concept of a session with something which is actually happening at the network level. Try reading something about the HTTP request cycle (e.g. http://www.slideshare.net/cczona/full-stack-full-circle-what-the-heck-happens-in-an-http-requestresponse-cycle http://www.slideshare.net/cczona/full-stack-full-circle-what...) for basic background and after that move on to some discussion about the work done to make servers which can handle many simultaneous connections - the C10K problem was the original work tuning operating systems to handle tens of thousands of simultaneous connections and was large a non-issue at the network / basic HTTP leel by the mid-2000s: http://bulk.fefe.de/scalability/ http://bulk.fefe.de/scalability/ By now a single server can be assumed to handle hundreds of thousands of simultaneous connections at the network level – assuming, of course, that you have capacity to actual do whatever work those connections are used for. Idle users typing text into a form wouldn't be a problem even if it did use a persistent connection.