5 ms·

Antirez tried, with lamernews: https://github.com/antirez/lamernews/blob/master/app.rb https://github.com/antirez/lamernews/blob/master/app.rb Diff that vs HN

Antirez tried, with lamernews: https://github.com/antirez/lamernews/blob/master/app.rb https://github.com/antirez/lamernews/blob/master/app.rb Diff that vs HN circa 2008: https://github.com/evanrmurphy/SweetScript/blob/master/arc3.1/news.arc https://github.com/evanrmurphy/SweetScript/blob/master/arc3.... Lamernews seems like just another HN clone, but back when it was launched it was a serious attempt at being a competitor. They probably came closest to pulling away a significant chunk of HN. I started to use it. Then I ran into missing features. Little quality of life stuff. I forget what they were, but the whole site felt noticeably harder to use. So I stopped using it. It's so tempting to think "HN is so simple. Just throw Rails at the problem and you can wipe the floor with them." But Lamernews was 2,964 lines of Ruby, and only managed to contain about 60% of the features. news.arc is 2,616 lines. And I don't care what anybody says: if you sat down and forced yourself to read that code for a week straight, you'd find it is actually quite readable. It's terse because it's for getting shit done.

tptacek 9y ago
I'm not saying you'll be as successful as HN. Lobste.rs, which is run carefully by thoughtful people using software which is probably better than HN's, is much less successful. But that's the point.
- sillysaurus3 9y ago
  You said you'd be two generations of tech ahead of HN. I know it was tongue in cheek, and I'm not calling you out. I'm nerding out about how much functionality pg managed to pack into so few lines of code. And it all worked! No data loss, no massive security problem, and the last time HN was down for >24h was because pg manually edited the database and accidentally created a circular comment chain. One way to look at the situation is that pg's original vision for arc failed. Arc was supposed to be what Clojure turned out to be. But even though HN alone is the one significant Arc project, what a project! The world was made materially better thanks to it. And I don't think they could've pulled off HN in some other language. There was too much work for one person to do to be distracted by the thousands of details a modern webdev needs to stay on top of. I think it's possible to do the same thing for modern webdev. Arc's goal was to be a reliable foundation for building webapps. It comes with a user+admin system out of the box, for example. Every Arc app does. I've been playing around with hooking up AngularJS to it, because unlike React or Angular, AngularJS can operate purely in HTML. You normally never want to do that -- usually you want to set up controllers and such. But when you control the language, you end up being able to give it the Babel treatment: the output looks complex, but you get all the features of a modern frontend site. Initial experiments were quite promising. And from there, it's easy to just emit JSX instead. React? Angular? Sure, whatever. It's treated as compiler output. (Indirect sourcemaps are funny... JSX compiles to minified JS, whose sourcemap points back to the JSX. But if Arc spits out JSX, you'll want a sourcemap from JSX -> Arc as well. It's sourcemaps all the way down.)
  - tptacek 9y ago
    You can call me out any time you'd like. I think an actual database backend for this site is only a couple years old. Or do we actually have one yet? I know the API has one, but HN doesn't run the API. It's possible this thing is still relying on flat files.
    - sillysaurus3 9y ago
      Flat files FTW. Proof that a simple solution can scale. And not just in a "Why the fuck would you subject yourself to this torture?" kind of way, but in a "This actually simplifies everything" way. It's why Arc was so fucking fast: remember how impactful memcached was? Arc had that from day one. Here's how it works. Everything is loaded into memory at startup. You can think of it like a bigass JSON tree. Anytime you change the tree, write the whole thing out to disk. Since all the data is loaded from file "foo", and you write out to "foo.tmp" followed by `mv foo.tmp foo`, there is no risk of data loss, ever. So what about sharding? Well, yeah, that's a problem. But it's not a problem that HN had to deal with. You can see how effective it was to throw a single 64GB RAM monster at the problem rather than a cluster of servers. The moment they hooked up Cloudflare to HN, they got all the same benefits a big cluster would provide anyway. People can be prejudiced against this simplicity all they want, but it's hard to argue with results. You don't even need to know SQL to start using it. "But migrations --" well, Arc makes it super easy to define a "template", which is just a set of fields, and each field is whatever you want them to be. https://github.com/evanrmurphy/SweetScript/blob/6b4ed26acc9c2df6fde028f57b19ecb4661943fb/arc3.1/news.arc#L50-L68 https://github.com/evanrmurphy/SweetScript/blob/6b4ed26acc9c... (deftem item id nil type nil by nil ip nil time (seconds) url nil title nil text nil votes nil ; elts each (time ip user type score) score 0 sockvotes 0 flags nil dead nil deleted nil parts nil parent nil kids nil keys nil) So yes, theoretically if you wanted to rename "kids" to "children", you're gonna have to process all the old data. But 1. then don't do that, and 2. even still, it's not hard to do that anyway. It's sort of similar to mongoose record definitions. But look, HN is proof positive that this system can work and can scale to a top-1000 website. 99% of webapps won't even come close to those performance requirements. And when you do approach them, you can find ways to make it scale. My main focus has been updating Arc to run on JS, and to add keyword arguments. That way you'll be able to use the whole JS ecosystem too.
- angersock 9y ago
  Depends on what you consider successful. If the metric is raw numbers and impressions, HN is definitely the winner. If the metric is signal-to-noise for technical content, Lobsters is probably ahead. If the metric is user count, HN wins. If the metric is moderation transparency, Lobsters wins. Just depends what you're looking for.
  - tptacek 9y ago
    I spent some time on Lobsters and found the signal/noise content for technical stuff only marginally better than that of HN, and many of the same pathologies were evident, just at a lower volume (because the site sees overall a much lower volume of comments). Moreover, while the variance on technical badness on Lobsters is marginally lower, so is the variance on technical excellence. That, too, is a straightforward consequence of lower volume: the original authors of important software packages are more likely to comment on HN than on Lobsters. I like Lobsters a lot, and I like a lot about what they do differently than HN, but we shouldn't pretend that it's something it isn't.
  - deleted 9y ago
    [deleted]

tptacek 9y ago

I'm not saying you'll be as successful as HN. Lobste.rs, which is run carefully by thoughtful people using software which is probably better than HN's, is much less successful. But that's the point.

sillysaurus3 9y ago
You said you'd be two generations of tech ahead of HN. I know it was tongue in cheek, and I'm not calling you out. I'm nerding out about how much functionality pg managed to pack into so few lines of code. And it all worked! No data loss, no massive security problem, and the last time HN was down for >24h was because pg manually edited the database and accidentally created a circular comment chain. One way to look at the situation is that pg's original vision for arc failed. Arc was supposed to be what Clojure turned out to be. But even though HN alone is the one significant Arc project, what a project! The world was made materially better thanks to it. And I don't think they could've pulled off HN in some other language. There was too much work for one person to do to be distracted by the thousands of details a modern webdev needs to stay on top of. I think it's possible to do the same thing for modern webdev. Arc's goal was to be a reliable foundation for building webapps. It comes with a user+admin system out of the box, for example. Every Arc app does. I've been playing around with hooking up AngularJS to it, because unlike React or Angular, AngularJS can operate purely in HTML. You normally never want to do that -- usually you want to set up controllers and such. But when you control the language, you end up being able to give it the Babel treatment: the output looks complex, but you get all the features of a modern frontend site. Initial experiments were quite promising. And from there, it's easy to just emit JSX instead. React? Angular? Sure, whatever. It's treated as compiler output. (Indirect sourcemaps are funny... JSX compiles to minified JS, whose sourcemap points back to the JSX. But if Arc spits out JSX, you'll want a sourcemap from JSX -> Arc as well. It's sourcemaps all the way down.)
- tptacek 9y ago
  You can call me out any time you'd like. I think an actual database backend for this site is only a couple years old. Or do we actually have one yet? I know the API has one, but HN doesn't run the API. It's possible this thing is still relying on flat files.
  - sillysaurus3 9y ago
    Flat files FTW. Proof that a simple solution can scale. And not just in a "Why the fuck would you subject yourself to this torture?" kind of way, but in a "This actually simplifies everything" way. It's why Arc was so fucking fast: remember how impactful memcached was? Arc had that from day one. Here's how it works. Everything is loaded into memory at startup. You can think of it like a bigass JSON tree. Anytime you change the tree, write the whole thing out to disk. Since all the data is loaded from file "foo", and you write out to "foo.tmp" followed by `mv foo.tmp foo`, there is no risk of data loss, ever. So what about sharding? Well, yeah, that's a problem. But it's not a problem that HN had to deal with. You can see how effective it was to throw a single 64GB RAM monster at the problem rather than a cluster of servers. The moment they hooked up Cloudflare to HN, they got all the same benefits a big cluster would provide anyway. People can be prejudiced against this simplicity all they want, but it's hard to argue with results. You don't even need to know SQL to start using it. "But migrations --" well, Arc makes it super easy to define a "template", which is just a set of fields, and each field is whatever you want them to be. https://github.com/evanrmurphy/SweetScript/blob/6b4ed26acc9c2df6fde028f57b19ecb4661943fb/arc3.1/news.arc#L50-L68 https://github.com/evanrmurphy/SweetScript/blob/6b4ed26acc9c... (deftem item id nil type nil by nil ip nil time (seconds) url nil title nil text nil votes nil ; elts each (time ip user type score) score 0 sockvotes 0 flags nil dead nil deleted nil parts nil parent nil kids nil keys nil) So yes, theoretically if you wanted to rename "kids" to "children", you're gonna have to process all the old data. But 1. then don't do that, and 2. even still, it's not hard to do that anyway. It's sort of similar to mongoose record definitions. But look, HN is proof positive that this system can work and can scale to a top-1000 website. 99% of webapps won't even come close to those performance requirements. And when you do approach them, you can find ways to make it scale. My main focus has been updating Arc to run on JS, and to add keyword arguments. That way you'll be able to use the whole JS ecosystem too.
angersock 9y ago
Depends on what you consider successful. If the metric is raw numbers and impressions, HN is definitely the winner. If the metric is signal-to-noise for technical content, Lobsters is probably ahead. If the metric is user count, HN wins. If the metric is moderation transparency, Lobsters wins. Just depends what you're looking for.
- tptacek 9y ago
  I spent some time on Lobsters and found the signal/noise content for technical stuff only marginally better than that of HN, and many of the same pathologies were evident, just at a lower volume (because the site sees overall a much lower volume of comments). Moreover, while the variance on technical badness on Lobsters is marginally lower, so is the variance on technical excellence. That, too, is a straightforward consequence of lower volume: the original authors of important software packages are more likely to comment on HN than on Lobsters. I like Lobsters a lot, and I like a lot about what they do differently than HN, but we shouldn't pretend that it's something it isn't.
- deleted 9y ago
  [deleted]