4 ms·

We have the fastest server we could conveniently get about a year ago. But I just asked Rtm if we could get something better. Unfortunately processors don't s

by pg 16y ago

We have the fastest server we could conveniently get about a year ago. But I just asked Rtm if we could get something better. Unfortunately processors don't seem to get faster at the rate they used to. Now they just have more cores, and I believe MzScheme can only use one core. I think the solution is to make the software faster. There's probably some small hack we could make somewhere in the guts of the system that would make it 10x faster. I'm thinking of having a make-HN-faster contest, but first I have to release a more current version of the code.

rick_2047 16y ago
but first I have to release a more current version of the code. Please do that, some people are dying to see some of the stuff that goes under the hood. Of course things like hell banning may not be released, but some of the ranking and "ask" page stuff would be awesome to look into.
- projectileboy 16y ago
  http://ycombinator.com/arc/arc3.1.tar http://ycombinator.com/arc/arc3.1.tar
  - rick_2047 16y ago
    If you didn't know, that does not have some updates that pg makes on the fly. I have been managing hackerstreet.in, which uses that source, and I can tell it is outdated. Please actually see links before you are linking.
    - projectileboy 16y ago
      It's the most recent code that's publicly available. I'm aware of the fact that there has not been an Arc update recently.
btilly 16y ago
I believe MzScheme can only use one core. How annoying. http://osdir.com/ml/lisp.scheme.plt/2003-01/msg00216.html http://osdir.com/ml/lisp.scheme.plt/2003-01/msg00216.html suggests that you're right, but MzScheme _used_ to support multiple cores. How difficult would it be to divide HN up into multiple cooperating processes that each get their own core? With a little caching, it should be possible to get a lot more headroom.
jacquesm 16y ago
Would it be possible to layer arc on top of clojure or is that too far distant from each other?
- alnayyir 16y ago
  That's a bit much to resort to when AFAIK caching hasn't been pursued aggressively as it could be.
  - jacquesm 16y ago
    Typically if you start the work when you need it you're actually too late, if this is a direction that is viable the work could get underway long before there was an immediate need and it would probably be done that much better for that reason. Again, I have no idea if this is even feasible, the differences between the various dialects of lisp seem to me (as an outsider) to be considerable.
    - alnayyir 16y ago
      Ignoring myriad other differences, Clojure's syntactic improvements on Lisp/Scheme alone would create far more work than would be worthwhile. It's simply unnecessary and excessively painful in the face of much better alternatives. Like cutting the bloody limb off when you haven't even tried a penicillin derivative yet, a common "first step".
    - pmjordan 16y ago
      To my knowledge, Arc, like Scheme, uses mutable conses as its main data structure; Clojure doesn't use conses in the sense that the cdr-equivalent element of a list cell can't be just anything. Clojure's data structures are also immutable. You'd have to do a lot to re-engineer the basic data structures just for that; I doubt Arc would be the same language at the end of it. I'm sure Mz/Arc can be made to run workers in other threads or processes in some way, even if it's not trivial. I also get the impression the caching is implemented in Arc code rather than as a HTTP proxy. The latter could run across multiple cores or even servers. Finally, it may be possible to decouple the user-specific data from the generated pages themselves and cache that information on the HTTP level. I admittedly haven't given it a lot of thought, and I also haven't read through the public HN source yet.
gus_massa 16y ago
Since version 4.2.5 MzScheme can use many cores with future and touch (http://docs.racket-lang.org/guide/performance.html#(part._effective-futures) http://docs.racket-lang.org/guide/performance.html#(part._ef... ) But they apparently they are tricky to use. For example the manual says that to run the computations in parallel, you should no use operations that mix exact ans inexact numbers. It is possible to do this by hand writing the scheme code, but I think that it is more difficult to make the arc->scheme compiler do this automatically.
awwx 16y ago
I agree, given some representative sample data and a goal ("make rendering this page go fast"), finding speed improvements likely wouldn't be very hard.

by pg 16y ago

rick_2047 16y ago

but first I have to release a more current version of the code. Please do that, some people are dying to see some of the stuff that goes under the hood. Of course things like hell banning may not be released, but some of the ranking and "ask" page stuff would be awesome to look into.

projectileboy 16y ago
http://ycombinator.com/arc/arc3.1.tar http://ycombinator.com/arc/arc3.1.tar
- rick_2047 16y ago
  If you didn't know, that does not have some updates that pg makes on the fly. I have been managing hackerstreet.in, which uses that source, and I can tell it is outdated. Please actually see links before you are linking.
  - projectileboy 16y ago
    It's the most recent code that's publicly available. I'm aware of the fact that there has not been an Arc update recently.

btilly 16y ago

I believe MzScheme can only use one core. How annoying. http://osdir.com/ml/lisp.scheme.plt/2003-01/msg00216.html http://osdir.com/ml/lisp.scheme.plt/2003-01/msg00216.html suggests that you're right, but MzScheme _used_ to support multiple cores. How difficult would it be to divide HN up into multiple cooperating processes that each get their own core? With a little caching, it should be possible to get a lot more headroom.

jacquesm 16y ago

Would it be possible to layer arc on top of clojure or is that too far distant from each other?

alnayyir 16y ago
That's a bit much to resort to when AFAIK caching hasn't been pursued aggressively as it could be.
- jacquesm 16y ago
  Typically if you start the work when you need it you're actually too late, if this is a direction that is viable the work could get underway long before there was an immediate need and it would probably be done that much better for that reason. Again, I have no idea if this is even feasible, the differences between the various dialects of lisp seem to me (as an outsider) to be considerable.
  - alnayyir 16y ago
    Ignoring myriad other differences, Clojure's syntactic improvements on Lisp/Scheme alone would create far more work than would be worthwhile. It's simply unnecessary and excessively painful in the face of much better alternatives. Like cutting the bloody limb off when you haven't even tried a penicillin derivative yet, a common "first step".
  - pmjordan 16y ago
    To my knowledge, Arc, like Scheme, uses mutable conses as its main data structure; Clojure doesn't use conses in the sense that the cdr-equivalent element of a list cell can't be just anything. Clojure's data structures are also immutable. You'd have to do a lot to re-engineer the basic data structures just for that; I doubt Arc would be the same language at the end of it. I'm sure Mz/Arc can be made to run workers in other threads or processes in some way, even if it's not trivial. I also get the impression the caching is implemented in Arc code rather than as a HTTP proxy. The latter could run across multiple cores or even servers. Finally, it may be possible to decouple the user-specific data from the generated pages themselves and cache that information on the HTTP level. I admittedly haven't given it a lot of thought, and I also haven't read through the public HN source yet.

gus_massa 16y ago

Since version 4.2.5 MzScheme can use many cores with future and touch (http://docs.racket-lang.org/guide/performance.html#(part._effective-futures) http://docs.racket-lang.org/guide/performance.html#(part._ef... ) But they apparently they are tricky to use. For example the manual says that to run the computations in parallel, you should no use operations that mix exact ans inexact numbers. It is possible to do this by hand writing the scheme code, but I think that it is more difficult to make the arc->scheme compiler do this automatically.

awwx 16y ago

I agree, given some representative sample data and a goal ("make rendering this page go fast"), finding speed improvements likely wouldn't be very hard.