5 ms·

Scaling a tree structured view is difficult as hell. Elon musk thread shows just one single comment, and "view 557 more comments" button. I guess it's because o

Scaling a tree structured view is difficult as hell. Elon musk thread shows just one single comment, and "view 557 more comments" button. I guess it's because of the number of comments written for the first comment. I would expect to see other threads in the news detail page, not only one, but probably it's some sort of optimization to handle enormous amount of comments. I guess the twitter's way of handling huge threads is much more scalable and I think twitter's acquisition should had such a point.

mkl 5y ago
> Scaling a tree structured view is difficult has hell. Really? Why is that? It doesn't seem like that should be the case here. HN threads are only a few hundred comments, and the web page structure is very linear and formulaic (the threads do not form a tree in the DOM). I.e. it's a depth-first traversal visiting children by ranking, each comment producing a row in a table. For just a few hundred comments I would expect that to be very fast (but I don't know what data structure it's all stored in).
- fatih-erikli 5y ago
  [1] It's not easy to determine how many root-level comments should be shown in the first page. You need to count sub-comments as well, and second level comments should have loaded as "expanded", it's also not easy to decide how many second level comments should be shown as expanded. The amount of them could be 10 and 100 or 1000. So if you just say like "10" comments, the letters in just one comment could be worth of 100 comments. So you also need a "show more" button for each comment exceeds some character limit. If you do a viewport optimization, you need to calculate the rendered size of a comment section and request only comments visible for the browser viewport. That's why it's difficult. * Elon Mush threads shows only one thread now. https://news.ycombinator.com/item?id=31025061 https://news.ycombinator.com/item?id=31025061 Edit: Looks like it is no longer an issue. It shows latest comments.
  - stefan_ 5y ago
    I don't understand. We're still talking about trivial numbers. This is a tree with ~800 nodes or comments. There is no "dynamic viewport" optimization, HN is static HTML (and we love it for it!).
    - fatih-erikli 5y ago
      I was just thinking, what could be the problem in Elon Musk's thread. It shows just one comment. I didn't say there is a dynamic viewport optimization. The issue is not in DOM structure. I'm creator of "arguman.org", maybe you've heard about it (it's down now). I had the same issues there as well. I'm also happy with HN's plain html interface. Also, how would you render a page with 800 nodes with user content, it is still a question. I would like to hear and learn. Btw, yes, Bulletin board style forums is a strait forward solution, there's always some ways to do that, but there's also ways to do it like in Twitter's way. That's the point I was talking about.
      - krapp 5y ago
        >Also, how would you render a page with 800 nodes with user content, it is still a question. I would like to hear and learn. It would be trivial with any site using a relational database. The two basic models used are nested sets[0], which optimize rendering over insertion and deletion, versus the much simpler relational model (rows with id and parent id, sometimes root id), which is much easier to insert and delete from, but requires recursive queries to render an entire tree from any branch other than the root. The vast majority of threaded forums use the former model, many larger sites use the latter, and rendering large threads isn't an issue either way unless maybe you're at Reddit scale, which most forums including HN aren't. Hacker News doesn't use a relational database, it uses flat text files (limiting it to file I/O speed for data not cached in RAM), and it also runs under a single thread with limited RAM. It likely wasn't designed to operate efficiently at the scale that it currently does, and the increasing size and velocity of threads is pushing it to its limit. Edit: looks like the stone passed. [0]https://en.wikipedia.org/wiki/Nested_set_model https://en.wikipedia.org/wiki/Nested_set_model
jedberg 5y ago
> Scaling a tree structured view is difficult as hell. It's not that hard with so few comments, it's just a matter of UX philosophy. At reddit we solved this with the "load more comments" AJAX request and "continue this thread". "Continue" only happens when a tree gets too deep, and then is just sets the current comment as the top of the tree and recalculates. "load more" happens after the comment limit was reached, but since it's AJAX it doesn't really take away from the viewing experience (and I think in new reddit it just auto loads when you scroll to it). Here's the reddit code if you want to see it: https://github.com/reddit-archive/reddit/blob/master/r2/r2/lib/comment_tree.py https://github.com/reddit-archive/reddit/blob/master/r2/r2/l... The main issue for HN is that it's static so when a node gets a lot of children, it breaks a lot of their assumptions and makes for very large blobs of HTML to display the tree. Honestly it's probably a worthwhile tradeoff -- for the most part the threads here don't get so deep that it's a problem, and in exchange we get a super snappy website most of the time.
- fatih-erikli 5y ago
  Thank you! I will look at that.

mkl 5y ago

> Scaling a tree structured view is difficult has hell. Really? Why is that? It doesn't seem like that should be the case here. HN threads are only a few hundred comments, and the web page structure is very linear and formulaic (the threads do not form a tree in the DOM). I.e. it's a depth-first traversal visiting children by ranking, each comment producing a row in a table. For just a few hundred comments I would expect that to be very fast (but I don't know what data structure it's all stored in).

fatih-erikli 5y ago
[1] It's not easy to determine how many root-level comments should be shown in the first page. You need to count sub-comments as well, and second level comments should have loaded as "expanded", it's also not easy to decide how many second level comments should be shown as expanded. The amount of them could be 10 and 100 or 1000. So if you just say like "10" comments, the letters in just one comment could be worth of 100 comments. So you also need a "show more" button for each comment exceeds some character limit. If you do a viewport optimization, you need to calculate the rendered size of a comment section and request only comments visible for the browser viewport. That's why it's difficult. * Elon Mush threads shows only one thread now. https://news.ycombinator.com/item?id=31025061 https://news.ycombinator.com/item?id=31025061 Edit: Looks like it is no longer an issue. It shows latest comments.
- stefan_ 5y ago
  I don't understand. We're still talking about trivial numbers. This is a tree with ~800 nodes or comments. There is no "dynamic viewport" optimization, HN is static HTML (and we love it for it!).
  - fatih-erikli 5y ago
    I was just thinking, what could be the problem in Elon Musk's thread. It shows just one comment. I didn't say there is a dynamic viewport optimization. The issue is not in DOM structure. I'm creator of "arguman.org", maybe you've heard about it (it's down now). I had the same issues there as well. I'm also happy with HN's plain html interface. Also, how would you render a page with 800 nodes with user content, it is still a question. I would like to hear and learn. Btw, yes, Bulletin board style forums is a strait forward solution, there's always some ways to do that, but there's also ways to do it like in Twitter's way. That's the point I was talking about.
    - krapp 5y ago
      >Also, how would you render a page with 800 nodes with user content, it is still a question. I would like to hear and learn. It would be trivial with any site using a relational database. The two basic models used are nested sets[0], which optimize rendering over insertion and deletion, versus the much simpler relational model (rows with id and parent id, sometimes root id), which is much easier to insert and delete from, but requires recursive queries to render an entire tree from any branch other than the root. The vast majority of threaded forums use the former model, many larger sites use the latter, and rendering large threads isn't an issue either way unless maybe you're at Reddit scale, which most forums including HN aren't. Hacker News doesn't use a relational database, it uses flat text files (limiting it to file I/O speed for data not cached in RAM), and it also runs under a single thread with limited RAM. It likely wasn't designed to operate efficiently at the scale that it currently does, and the increasing size and velocity of threads is pushing it to its limit. Edit: looks like the stone passed. [0]https://en.wikipedia.org/wiki/Nested_set_model https://en.wikipedia.org/wiki/Nested_set_model

jedberg 5y ago

> Scaling a tree structured view is difficult as hell. It's not that hard with so few comments, it's just a matter of UX philosophy. At reddit we solved this with the "load more comments" AJAX request and "continue this thread". "Continue" only happens when a tree gets too deep, and then is just sets the current comment as the top of the tree and recalculates. "load more" happens after the comment limit was reached, but since it's AJAX it doesn't really take away from the viewing experience (and I think in new reddit it just auto loads when you scroll to it). Here's the reddit code if you want to see it: https://github.com/reddit-archive/reddit/blob/master/r2/r2/lib/comment_tree.py https://github.com/reddit-archive/reddit/blob/master/r2/r2/l... The main issue for HN is that it's static so when a node gets a lot of children, it breaks a lot of their assumptions and makes for very large blobs of HTML to display the tree. Honestly it's probably a worthwhile tradeoff -- for the most part the threads here don't get so deep that it's a problem, and in exchange we get a super snappy website most of the time.

fatih-erikli 5y ago
Thank you! I will look at that.