4 ms·

You said you'd be two generations of tech ahead of HN. I know it was tongue in cheek, and I'm not calling you out. I'm nerding out about how much functionality

You said you'd be two generations of tech ahead of HN. I know it was tongue in cheek, and I'm not calling you out. I'm nerding out about how much functionality pg managed to pack into so few lines of code. And it all worked! No data loss, no massive security problem, and the last time HN was down for >24h was because pg manually edited the database and accidentally created a circular comment chain. One way to look at the situation is that pg's original vision for arc failed. Arc was supposed to be what Clojure turned out to be. But even though HN alone is the one significant Arc project, what a project! The world was made materially better thanks to it. And I don't think they could've pulled off HN in some other language. There was too much work for one person to do to be distracted by the thousands of details a modern webdev needs to stay on top of. I think it's possible to do the same thing for modern webdev. Arc's goal was to be a reliable foundation for building webapps. It comes with a user+admin system out of the box, for example. Every Arc app does. I've been playing around with hooking up AngularJS to it, because unlike React or Angular, AngularJS can operate purely in HTML. You normally never want to do that -- usually you want to set up controllers and such. But when you control the language, you end up being able to give it the Babel treatment: the output looks complex, but you get all the features of a modern frontend site. Initial experiments were quite promising. And from there, it's easy to just emit JSX instead. React? Angular? Sure, whatever. It's treated as compiler output. (Indirect sourcemaps are funny... JSX compiles to minified JS, whose sourcemap points back to the JSX. But if Arc spits out JSX, you'll want a sourcemap from JSX -> Arc as well. It's sourcemaps all the way down.)

tptacek 9y ago
You can call me out any time you'd like. I think an actual database backend for this site is only a couple years old. Or do we actually have one yet? I know the API has one, but HN doesn't run the API. It's possible this thing is still relying on flat files.
- sillysaurus3 9y ago
  Flat files FTW. Proof that a simple solution can scale. And not just in a "Why the fuck would you subject yourself to this torture?" kind of way, but in a "This actually simplifies everything" way. It's why Arc was so fucking fast: remember how impactful memcached was? Arc had that from day one. Here's how it works. Everything is loaded into memory at startup. You can think of it like a bigass JSON tree. Anytime you change the tree, write the whole thing out to disk. Since all the data is loaded from file "foo", and you write out to "foo.tmp" followed by `mv foo.tmp foo`, there is no risk of data loss, ever. So what about sharding? Well, yeah, that's a problem. But it's not a problem that HN had to deal with. You can see how effective it was to throw a single 64GB RAM monster at the problem rather than a cluster of servers. The moment they hooked up Cloudflare to HN, they got all the same benefits a big cluster would provide anyway. People can be prejudiced against this simplicity all they want, but it's hard to argue with results. You don't even need to know SQL to start using it. "But migrations --" well, Arc makes it super easy to define a "template", which is just a set of fields, and each field is whatever you want them to be. https://github.com/evanrmurphy/SweetScript/blob/6b4ed26acc9c2df6fde028f57b19ecb4661943fb/arc3.1/news.arc#L50-L68 https://github.com/evanrmurphy/SweetScript/blob/6b4ed26acc9c... (deftem item id nil type nil by nil ip nil time (seconds) url nil title nil text nil votes nil ; elts each (time ip user type score) score 0 sockvotes 0 flags nil dead nil deleted nil parts nil parent nil kids nil keys nil) So yes, theoretically if you wanted to rename "kids" to "children", you're gonna have to process all the old data. But 1. then don't do that, and 2. even still, it's not hard to do that anyway. It's sort of similar to mongoose record definitions. But look, HN is proof positive that this system can work and can scale to a top-1000 website. 99% of webapps won't even come close to those performance requirements. And when you do approach them, you can find ways to make it scale. My main focus has been updating Arc to run on JS, and to add keyword arguments. That way you'll be able to use the whole JS ecosystem too.
  - tptacek 9y ago
    For what it's worth, the published Arc/HN code is rapidly approaching a full decade out of date.
    - sillysaurus3 9y ago
      When you get the design right, there's no need to change it. :) Jokes aside, I refuse to let Arc die. There were genuinely good ideas embedded in it, amidst the rough edges. People just never forced themselves to use it. It's like if Vim was invented but had no documentation, so no one ever realized it's powerful if you embrace it. But yeah, I imagine my version will have a different name and a shiny website. Being a decade out of date does matter, especially for hiring purposes.
      - tptacek 9y ago
        I think I'm just saying that I'm not sure you can conclude they don't have a real database backend now from the news.arc code.
        sillysaurus3 9y ago
        Ah, yeah. (It's odd that Arc went closed-source but kept improving. It's maybe one of the most important custom languages in the industry, relative to HN's impact on the world.)
        krapp 9y ago
        Did Arc go closed-source or does Hacker News run a closed-source fork? As far as I can tell, the arc forums site is still up and running, and a lot of people seem to be focused on Anarki[0]. [0]https://github.com/arclanguage/anarki https://github.com/arclanguage/anarki
        tptacek 9y ago
        Hacker News has been on a closed-source "fork" (it's really the upstream) since the day news.arc was published.
- sillysaurus3 9y ago
  Direct answer: no, probably no database. HN forwards info to Firebase which provides an API. But assuming Arc is still running on Racket, there aren't a lot of options for database bindings (https://docs.racket-lang.org/continue/ https://docs.racket-lang.org/continue/) and flat files work well enough. EDIT: Actually, you may be right. They already forward data to Firebase, which means they could just store the data in Firebase directly. Firebase is pretty great, so it's possible they did this. One bit of supporting evidence: there seems to be no startup time. Loading GBs of data from disk takes a little while, so restarting HN would be painful. We'd notice it as ~30 seconds of downtime. Especially now that HN has grown so large. I haven't noticed this in a couple years (but I'm not omnipresent). So it could go either way, and there are a lot of advantages to the Firebase approach. I guess I take back everything I said. But only because a lightweight alternative exists, not because flat files are inherently bad.
  - soegaard 9y ago
    Lots of database options! See https://docs.racket-lang.org/db/ https://docs.racket-lang.org/db/
krapp 9y ago
>I think it's possible to do the same thing for modern webdev. Arc's goal was to be a reliable foundation for building webapps. It comes with a user+admin system out of the box, for example. Every Arc app does. Forums written in other languages do as well - and any decent framework can just include user management as a dependency. To me it doesn't seem like an unqualified good idea for Arc to bake all of that into the language or the default app.
- sillysaurus3 9y ago
  It's pretty handy. A few times over the years, I've needed to throw together a standard "manage this data" type of app. I just extend the standard blog.arc app that ships with Arc: https://github.com/evanrmurphy/SweetScript/blob/master/arc3.1/blog.arc https://github.com/evanrmurphy/SweetScript/blob/master/arc3.... Bam, you get CRUD for free, you get special superpowers if you're logged in as an admin, and you can put in access controls so that other people can log in as other types of users and see a subset of the data. I dunno, it's pretty great. You even get a REPL at /repl in case you need to hop on and run some code on production. (If something seems amiss, rather than bother with grepping logfiles, you'd head to /repl and start asking questions about the current state of the app. This tends to be a lot faster.)

tptacek 9y ago

You can call me out any time you'd like. I think an actual database backend for this site is only a couple years old. Or do we actually have one yet? I know the API has one, but HN doesn't run the API. It's possible this thing is still relying on flat files.

sillysaurus3 9y ago
Flat files FTW. Proof that a simple solution can scale. And not just in a "Why the fuck would you subject yourself to this torture?" kind of way, but in a "This actually simplifies everything" way. It's why Arc was so fucking fast: remember how impactful memcached was? Arc had that from day one. Here's how it works. Everything is loaded into memory at startup. You can think of it like a bigass JSON tree. Anytime you change the tree, write the whole thing out to disk. Since all the data is loaded from file "foo", and you write out to "foo.tmp" followed by `mv foo.tmp foo`, there is no risk of data loss, ever. So what about sharding? Well, yeah, that's a problem. But it's not a problem that HN had to deal with. You can see how effective it was to throw a single 64GB RAM monster at the problem rather than a cluster of servers. The moment they hooked up Cloudflare to HN, they got all the same benefits a big cluster would provide anyway. People can be prejudiced against this simplicity all they want, but it's hard to argue with results. You don't even need to know SQL to start using it. "But migrations --" well, Arc makes it super easy to define a "template", which is just a set of fields, and each field is whatever you want them to be. https://github.com/evanrmurphy/SweetScript/blob/6b4ed26acc9c2df6fde028f57b19ecb4661943fb/arc3.1/news.arc#L50-L68 https://github.com/evanrmurphy/SweetScript/blob/6b4ed26acc9c... (deftem item id nil type nil by nil ip nil time (seconds) url nil title nil text nil votes nil ; elts each (time ip user type score) score 0 sockvotes 0 flags nil dead nil deleted nil parts nil parent nil kids nil keys nil) So yes, theoretically if you wanted to rename "kids" to "children", you're gonna have to process all the old data. But 1. then don't do that, and 2. even still, it's not hard to do that anyway. It's sort of similar to mongoose record definitions. But look, HN is proof positive that this system can work and can scale to a top-1000 website. 99% of webapps won't even come close to those performance requirements. And when you do approach them, you can find ways to make it scale. My main focus has been updating Arc to run on JS, and to add keyword arguments. That way you'll be able to use the whole JS ecosystem too.
- tptacek 9y ago
  For what it's worth, the published Arc/HN code is rapidly approaching a full decade out of date.
  - sillysaurus3 9y ago
    When you get the design right, there's no need to change it. :) Jokes aside, I refuse to let Arc die. There were genuinely good ideas embedded in it, amidst the rough edges. People just never forced themselves to use it. It's like if Vim was invented but had no documentation, so no one ever realized it's powerful if you embrace it. But yeah, I imagine my version will have a different name and a shiny website. Being a decade out of date does matter, especially for hiring purposes.
    - tptacek 9y ago
      I think I'm just saying that I'm not sure you can conclude they don't have a real database backend now from the news.arc code.
      - sillysaurus3 9y ago
        Ah, yeah. (It's odd that Arc went closed-source but kept improving. It's maybe one of the most important custom languages in the industry, relative to HN's impact on the world.)
        krapp 9y ago
        Did Arc go closed-source or does Hacker News run a closed-source fork? As far as I can tell, the arc forums site is still up and running, and a lot of people seem to be focused on Anarki[0]. [0]https://github.com/arclanguage/anarki https://github.com/arclanguage/anarki
        tptacek 9y ago
        Hacker News has been on a closed-source "fork" (it's really the upstream) since the day news.arc was published.
sillysaurus3 9y ago
Direct answer: no, probably no database. HN forwards info to Firebase which provides an API. But assuming Arc is still running on Racket, there aren't a lot of options for database bindings (https://docs.racket-lang.org/continue/ https://docs.racket-lang.org/continue/) and flat files work well enough. EDIT: Actually, you may be right. They already forward data to Firebase, which means they could just store the data in Firebase directly. Firebase is pretty great, so it's possible they did this. One bit of supporting evidence: there seems to be no startup time. Loading GBs of data from disk takes a little while, so restarting HN would be painful. We'd notice it as ~30 seconds of downtime. Especially now that HN has grown so large. I haven't noticed this in a couple years (but I'm not omnipresent). So it could go either way, and there are a lot of advantages to the Firebase approach. I guess I take back everything I said. But only because a lightweight alternative exists, not because flat files are inherently bad.
- soegaard 9y ago
  Lots of database options! See https://docs.racket-lang.org/db/ https://docs.racket-lang.org/db/

krapp 9y ago

>I think it's possible to do the same thing for modern webdev. Arc's goal was to be a reliable foundation for building webapps. It comes with a user+admin system out of the box, for example. Every Arc app does. Forums written in other languages do as well - and any decent framework can just include user management as a dependency. To me it doesn't seem like an unqualified good idea for Arc to bake all of that into the language or the default app.

sillysaurus3 9y ago
It's pretty handy. A few times over the years, I've needed to throw together a standard "manage this data" type of app. I just extend the standard blog.arc app that ships with Arc: https://github.com/evanrmurphy/SweetScript/blob/master/arc3.1/blog.arc https://github.com/evanrmurphy/SweetScript/blob/master/arc3.... Bam, you get CRUD for free, you get special superpowers if you're logged in as an admin, and you can put in access controls so that other people can log in as other types of users and see a subset of the data. I dunno, it's pretty great. You even get a REPL at /repl in case you need to hop on and run some code on production. (If something seems amiss, rather than bother with grepping logfiles, you'd head to /repl and start asking questions about the current state of the app. This tends to be a lot faster.)