3 ms·

[1] It's not easy to determine how many root-level comments should be shown in the first page. You need to count sub-comments as well, and second level comments

[1] It's not easy to determine how many root-level comments should be shown in the first page. You need to count sub-comments as well, and second level comments should have loaded as "expanded", it's also not easy to decide how many second level comments should be shown as expanded. The amount of them could be 10 and 100 or 1000. So if you just say like "10" comments, the letters in just one comment could be worth of 100 comments. So you also need a "show more" button for each comment exceeds some character limit. If you do a viewport optimization, you need to calculate the rendered size of a comment section and request only comments visible for the browser viewport. That's why it's difficult. * Elon Mush threads shows only one thread now. https://news.ycombinator.com/item?id=31025061 https://news.ycombinator.com/item?id=31025061 Edit: Looks like it is no longer an issue. It shows latest comments.

stefan_ 5y ago
I don't understand. We're still talking about trivial numbers. This is a tree with ~800 nodes or comments. There is no "dynamic viewport" optimization, HN is static HTML (and we love it for it!).
- fatih-erikli 5y ago
  I was just thinking, what could be the problem in Elon Musk's thread. It shows just one comment. I didn't say there is a dynamic viewport optimization. The issue is not in DOM structure. I'm creator of "arguman.org", maybe you've heard about it (it's down now). I had the same issues there as well. I'm also happy with HN's plain html interface. Also, how would you render a page with 800 nodes with user content, it is still a question. I would like to hear and learn. Btw, yes, Bulletin board style forums is a strait forward solution, there's always some ways to do that, but there's also ways to do it like in Twitter's way. That's the point I was talking about.
  - krapp 5y ago
    >Also, how would you render a page with 800 nodes with user content, it is still a question. I would like to hear and learn. It would be trivial with any site using a relational database. The two basic models used are nested sets[0], which optimize rendering over insertion and deletion, versus the much simpler relational model (rows with id and parent id, sometimes root id), which is much easier to insert and delete from, but requires recursive queries to render an entire tree from any branch other than the root. The vast majority of threaded forums use the former model, many larger sites use the latter, and rendering large threads isn't an issue either way unless maybe you're at Reddit scale, which most forums including HN aren't. Hacker News doesn't use a relational database, it uses flat text files (limiting it to file I/O speed for data not cached in RAM), and it also runs under a single thread with limited RAM. It likely wasn't designed to operate efficiently at the scale that it currently does, and the increasing size and velocity of threads is pushing it to its limit. Edit: looks like the stone passed. [0]https://en.wikipedia.org/wiki/Nested_set_model https://en.wikipedia.org/wiki/Nested_set_model
    - fatih-erikli 5y ago
      Thanks for your comments. Optimizing the render part in insertion and deletion is also difficult. Some nodes might have user-level views, so some of them won't be visible to some other users. So you can't just label the nodes like this is "left" or "right" of that record. My experience with a similar problem: Instead of putting the rendering part to the insertion and deletion (nested set model), I was using a simple nginx file based cache. This is a trade off and I chose the application layer to be "consistent" and "reliable", over "being fast". Keeping the data integrity is difficult when you have a nested set model and it's also expensive to doing that calculation during the insertion. Sometimes you need a message broker to handle such an expensive query, because it's not safe to do it within a request-response lifecycle, even if you do have a transactional database, there's always a barrier of "request time out".
mkl 5y ago
It seems simple enough to just generate the conversation as one big page, then break it up. As you traverse the tree, save a list of indices of top-level comments and store the (recursive) sum of children sizes, then split appropriately. This might be slow with millions of nodes, but I don't understand how that process alone would be slow with hundreds. What would be slow (and may be happening), is generating that dynamically for every page refresh on a busy story by every visitor (i.e. no caching).
- fatih-erikli 5y ago
  Generating one big page is not the scalable way of doing that, I think, It will eventually reach to a point that it's no longer a web-page but a downloadable document with couple of megabytes. I'm interested to brain-storm about how to model and implement "Breaking-up" part of your comment. For the backend part, I think Splay-tree kind of a data structure will help to serve it faster.
  - mkl 5y ago
    I misspoke a little. I wouldn't generate the whole thing as a blob of text and then split that. I would traverse the tree depth-first making HTML blobs for new comments that that hasn't been done for already, generate a linear list of comment items that each know the size of their own subtree (set that when you traverse back up to the node), and make a list of the indices of the top-level comments as I go. Then you can go through the top-level comments, adding them (and their subtrees - everything up to the next top-level index in the list) one by one into your webpage and adding up the total size so far until you reach some threshold of "too big", at which point you finish off that page and start the next one. I think caching would be important to make it fast, and you could cache the HTML of each comment or subtree. Actually, if I was making my own forum and not copying HN I would probably have the comment tree structure represented in the DOM tree, as that seems better for navigation and accessibility. All of this is totally theoretical of course! I haven't actually done it, so may be wrong.
    - fatih-erikli 5y ago

stefan_ 5y ago

I don't understand. We're still talking about trivial numbers. This is a tree with ~800 nodes or comments. There is no "dynamic viewport" optimization, HN is static HTML (and we love it for it!).

fatih-erikli 5y ago
I was just thinking, what could be the problem in Elon Musk's thread. It shows just one comment. I didn't say there is a dynamic viewport optimization. The issue is not in DOM structure. I'm creator of "arguman.org", maybe you've heard about it (it's down now). I had the same issues there as well. I'm also happy with HN's plain html interface. Also, how would you render a page with 800 nodes with user content, it is still a question. I would like to hear and learn. Btw, yes, Bulletin board style forums is a strait forward solution, there's always some ways to do that, but there's also ways to do it like in Twitter's way. That's the point I was talking about.
- krapp 5y ago
  >Also, how would you render a page with 800 nodes with user content, it is still a question. I would like to hear and learn. It would be trivial with any site using a relational database. The two basic models used are nested sets[0], which optimize rendering over insertion and deletion, versus the much simpler relational model (rows with id and parent id, sometimes root id), which is much easier to insert and delete from, but requires recursive queries to render an entire tree from any branch other than the root. The vast majority of threaded forums use the former model, many larger sites use the latter, and rendering large threads isn't an issue either way unless maybe you're at Reddit scale, which most forums including HN aren't. Hacker News doesn't use a relational database, it uses flat text files (limiting it to file I/O speed for data not cached in RAM), and it also runs under a single thread with limited RAM. It likely wasn't designed to operate efficiently at the scale that it currently does, and the increasing size and velocity of threads is pushing it to its limit. Edit: looks like the stone passed. [0]https://en.wikipedia.org/wiki/Nested_set_model https://en.wikipedia.org/wiki/Nested_set_model
  - fatih-erikli 5y ago
    Thanks for your comments. Optimizing the render part in insertion and deletion is also difficult. Some nodes might have user-level views, so some of them won't be visible to some other users. So you can't just label the nodes like this is "left" or "right" of that record. My experience with a similar problem: Instead of putting the rendering part to the insertion and deletion (nested set model), I was using a simple nginx file based cache. This is a trade off and I chose the application layer to be "consistent" and "reliable", over "being fast". Keeping the data integrity is difficult when you have a nested set model and it's also expensive to doing that calculation during the insertion. Sometimes you need a message broker to handle such an expensive query, because it's not safe to do it within a request-response lifecycle, even if you do have a transactional database, there's always a barrier of "request time out".

mkl 5y ago

It seems simple enough to just generate the conversation as one big page, then break it up. As you traverse the tree, save a list of indices of top-level comments and store the (recursive) sum of children sizes, then split appropriately. This might be slow with millions of nodes, but I don't understand how that process alone would be slow with hundreds. What would be slow (and may be happening), is generating that dynamically for every page refresh on a busy story by every visitor (i.e. no caching).

fatih-erikli 5y ago
Generating one big page is not the scalable way of doing that, I think, It will eventually reach to a point that it's no longer a web-page but a downloadable document with couple of megabytes. I'm interested to brain-storm about how to model and implement "Breaking-up" part of your comment. For the backend part, I think Splay-tree kind of a data structure will help to serve it faster.
- mkl 5y ago
  I misspoke a little. I wouldn't generate the whole thing as a blob of text and then split that. I would traverse the tree depth-first making HTML blobs for new comments that that hasn't been done for already, generate a linear list of comment items that each know the size of their own subtree (set that when you traverse back up to the node), and make a list of the indices of the top-level comments as I go. Then you can go through the top-level comments, adding them (and their subtrees - everything up to the next top-level index in the list) one by one into your webpage and adding up the total size so far until you reach some threshold of "too big", at which point you finish off that page and start the next one. I think caching would be important to make it fast, and you could cache the HTML of each comment or subtree. Actually, if I was making my own forum and not copying HN I would probably have the comment tree structure represented in the DOM tree, as that seems better for navigation and accessibility. All of this is totally theoretical of course! I haven't actually done it, so may be wrong.
  - fatih-erikli 5y ago