4 ms·

Older HN users may recall when busy discussions had comments split across several pages. This is because the Arc [1] language that HN runs on was originally hos

Older HN users may recall when busy discussions had comments split across several pages. This is because the Arc [1] language that HN runs on was originally hosted on top of Racket [2] and the implementation was too slow to handle giant discussions at HN scale. Around September 2024 Dang et al finished porting Arc to SBCL, and performance increased so much that even the largest discussions no longer need splitting. The server is unresponsive/restarting a lot less frequently since these changes, too, despite continued growth in traffic and comments: https://news.ycombinator.com/item?id=41679215 https://news.ycombinator.com/item?id=41679215 [1] https://paulgraham.com/arc.html https://paulgraham.com/arc.html [2] https://racket-lang.org/ https://racket-lang.org/

rayiner 8mo ago
I love this comment: "Btw, we rolled this out over 3 weeks ago and I think you're the first person to ask about it on HN. There was one earlier question by email. I think that qualifies as a splash-free dive." I had no idea and I'm an HN addict!
- philipkglass 8mo ago
  I was the person who emailed him about it earlier. 2024-09-05, me: On another topic, I just noticed that the 700+ comments on https://news.ycombinator.com/item?id=41445413 https://news.ycombinator.com/item?id=41445413 all render on a single page. Hurray! Is the pagination approach obsolete now? I know that you've commented several times about wanting to optimize the code so pagination wasn't necessary. I don't know if that's finished or if pagination will have to go on the next time there's a big breaking story. Dan G: Yes: the performance improvements I've been working on for years finally got deployed, so pagination is turned off for now. (In case you're curious, the change is that Arc now runs over SBCL instead of Racket.) ... Btw you're the only person I know of who's noticed this and pointed it out so far! I have very mixed feelings about how much I know about this site.
- dang 8mo ago
  The "splash-free dive" metaphor came from the greatly missed sctb and was the springboard for much fun conversation at HNHQ. https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=by%3Adang%20splash%20free&sort=byDate&type=comment https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...
  - ashton314 8mo ago
    I’m curious: was Arc running Racket BC or CS? I understand it got a big performance boost after switching to Chez Scheme.
    - dang 8mo ago
      It was running BC. I had high hopes for switching to CS because I'd heard the same thing you had, but when I tried it, HN slowed to a crawl. This stuff is so unpredictable.
      - gus_massa 8mo ago
        [Warning, I'm only 90% sure about the details.] Arc uses mutable cons, but Racket has immutable cons. So it's a problem. In Racket BC mutable and immutable cons use the same struct at the C level, so both are quite fast and almost interchangeable, if you cross your fingers that the optimization passes don't notice the mess and get annoyed (somewhat like UB in C). In Racket CS immutable cons are implemented as cons in Chez Scheme, but mutable cons are implemented as records in Chez Scheme, so they are not interchangeable at all. Arc used a magic unsafe trick to mutate immutable(at the Racket level) cons that are actually mutable(at the Chez Scheme level) cons. The trick is slow because the Racket to Chez Scheme "transpiler" doesn't understand it and does not generate nice fast code. One solution is to rewrite Arc to use mutable cons in Racket, but they are slow too because they are records in Chez Scheme that have less magic than cons in Chez Scheme. So my guess it that it will be a lot of work and little speed gain. [Also, ¿kogir? asked a long time ago in the email list about how to use more memory in Racket BC, or how to use it better or something like that. I think he made a small patch for HN because it has some unusual requirements. Anyway, I'm not sure if it was still in use.] --- The takeaway is that mutable cons are slow in Racket.
deleted 8mo ago
[deleted]
e12e 8mo ago
I suppose this version of Arc for sbcl is different from what hn runs on?: https://github.com/pauek/arc-sbcl https://github.com/pauek/arc-sbcl And there's no version of Anarki that runs on sbcl?: https://arclanguage.github.io/ https://arclanguage.github.io/
- dang 8mo ago
  It's different, yes. The HN implementation is called clarc. PG suggested we spell it "clerk" as a joke on the British pronunciation of the latter, but I chickened out. I talked to one of the Anarki devs (or at least someone who uses it) about possibly open-sourcing a version of clarc which would run the original open-sourced version of HN, but it's a bit hard because the implementation would require some careful surgery to factor out the relevant parts.
  - e12e 8mo ago
    There's hn specific parts to the clarc implementation of Arc? (As opposed to the hn version of the "news" application)?
    - dang 8mo ago
      Yes because we just add the things we need, at whatever layer it makes the most sense to add them. This type of application stack that includes the language you're writing in and even, when convenient, its implementation, is really satisfying to work with. There is much less need for workarounds, arbitrary choices, and various indirections (e.g. what used to be called dependency injection, for example). The plumbing is simple and it allows us to keep the codebase much smaller than it would otherwise be. I spend basically zero time bitching about having to deal with software dependencies, making me realize how much of my former life as a programnmer was taken up with that. I think of this as sort of the unikernel form of application dev and of course it's a fine fit for a Lisp, since "write the language you want to write your program in as you write your program in it" is the natural style there. The tradeoff is that there's a lot of vertical coupling between the layers. If you want to factor out one layer for general consumption, e.g. to open source the language implementation so other people can build cool things with it, there's a fair bit of work to do. Also, since the language implementation exists to run a specific application, we don't bother supporting what we don't need for HN. That too comes back to bite you when you want to open source! HN has had 15+ years of work since the original news application was open-sourced; that's a lot of things-we-added at-some-point. Most are at the application level but some ended up in Arc and some in the Arc implementation when it was convenient to put them there. This is especially handy when you have limited time to work on the code.
rurban 8mo ago
Also racket since then was ported to use the fastest known scheme, chez, from the quite simple and slow mzscheme.

rayiner 8mo ago

I love this comment: "Btw, we rolled this out over 3 weeks ago and I think you're the first person to ask about it on HN. There was one earlier question by email. I think that qualifies as a splash-free dive." I had no idea and I'm an HN addict!

philipkglass 8mo ago
I was the person who emailed him about it earlier. 2024-09-05, me: On another topic, I just noticed that the 700+ comments on https://news.ycombinator.com/item?id=41445413 https://news.ycombinator.com/item?id=41445413 all render on a single page. Hurray! Is the pagination approach obsolete now? I know that you've commented several times about wanting to optimize the code so pagination wasn't necessary. I don't know if that's finished or if pagination will have to go on the next time there's a big breaking story. Dan G: Yes: the performance improvements I've been working on for years finally got deployed, so pagination is turned off for now. (In case you're curious, the change is that Arc now runs over SBCL instead of Racket.) ... Btw you're the only person I know of who's noticed this and pointed it out so far! I have very mixed feelings about how much I know about this site.
dang 8mo ago
The "splash-free dive" metaphor came from the greatly missed sctb and was the springboard for much fun conversation at HNHQ. https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=by%3Adang%20splash%20free&sort=byDate&type=comment https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...
- ashton314 8mo ago
  I’m curious: was Arc running Racket BC or CS? I understand it got a big performance boost after switching to Chez Scheme.
  - dang 8mo ago
    It was running BC. I had high hopes for switching to CS because I'd heard the same thing you had, but when I tried it, HN slowed to a crawl. This stuff is so unpredictable.
    - gus_massa 8mo ago
      [Warning, I'm only 90% sure about the details.] Arc uses mutable cons, but Racket has immutable cons. So it's a problem. In Racket BC mutable and immutable cons use the same struct at the C level, so both are quite fast and almost interchangeable, if you cross your fingers that the optimization passes don't notice the mess and get annoyed (somewhat like UB in C). In Racket CS immutable cons are implemented as cons in Chez Scheme, but mutable cons are implemented as records in Chez Scheme, so they are not interchangeable at all. Arc used a magic unsafe trick to mutate immutable(at the Racket level) cons that are actually mutable(at the Chez Scheme level) cons. The trick is slow because the Racket to Chez Scheme "transpiler" doesn't understand it and does not generate nice fast code. One solution is to rewrite Arc to use mutable cons in Racket, but they are slow too because they are records in Chez Scheme that have less magic than cons in Chez Scheme. So my guess it that it will be a lot of work and little speed gain. [Also, ¿kogir? asked a long time ago in the email list about how to use more memory in Racket BC, or how to use it better or something like that. I think he made a small patch for HN because it has some unusual requirements. Anyway, I'm not sure if it was still in use.] --- The takeaway is that mutable cons are slow in Racket.

deleted 8mo ago

[deleted]

e12e 8mo ago

I suppose this version of Arc for sbcl is different from what hn runs on?: https://github.com/pauek/arc-sbcl https://github.com/pauek/arc-sbcl And there's no version of Anarki that runs on sbcl?: https://arclanguage.github.io/ https://arclanguage.github.io/

dang 8mo ago
It's different, yes. The HN implementation is called clarc. PG suggested we spell it "clerk" as a joke on the British pronunciation of the latter, but I chickened out. I talked to one of the Anarki devs (or at least someone who uses it) about possibly open-sourcing a version of clarc which would run the original open-sourced version of HN, but it's a bit hard because the implementation would require some careful surgery to factor out the relevant parts.
- e12e 8mo ago
  There's hn specific parts to the clarc implementation of Arc? (As opposed to the hn version of the "news" application)?
  - dang 8mo ago
    Yes because we just add the things we need, at whatever layer it makes the most sense to add them. This type of application stack that includes the language you're writing in and even, when convenient, its implementation, is really satisfying to work with. There is much less need for workarounds, arbitrary choices, and various indirections (e.g. what used to be called dependency injection, for example). The plumbing is simple and it allows us to keep the codebase much smaller than it would otherwise be. I spend basically zero time bitching about having to deal with software dependencies, making me realize how much of my former life as a programnmer was taken up with that. I think of this as sort of the unikernel form of application dev and of course it's a fine fit for a Lisp, since "write the language you want to write your program in as you write your program in it" is the natural style there. The tradeoff is that there's a lot of vertical coupling between the layers. If you want to factor out one layer for general consumption, e.g. to open source the language implementation so other people can build cool things with it, there's a fair bit of work to do. Also, since the language implementation exists to run a specific application, we don't bother supporting what we don't need for HN. That too comes back to bite you when you want to open source! HN has had 15+ years of work since the original news application was open-sourced; that's a lot of things-we-added at-some-point. Most are at the application level but some ended up in Arc and some in the Arc implementation when it was convenient to put them there. This is especially handy when you have limited time to work on the code.

rurban 8mo ago

Also racket since then was ported to use the fastest known scheme, chez, from the quite simple and slow mzscheme.