4 ms·

Ask HN: how hard is it to maintain your own clone of HN ?

There's a site I want to start - and I think the code behind news.yc (http://github.com/nex3/arc if I'm right) would be a nice fit for that. I'm not an arc-ist at all, hence my question: is there anyone running its own copy, without much knowledge of Arc ? Has it been hard to tweak, maintain, support ? (I definitely appreciate the learning opportunity it would represent - but I really want to be able to sustain the site, first) Thoughts?

thibaut_barrere 17y ago
I also welcome comments on how hard it was to customize the look or behaviour, if you did so.
- adrianwaj 17y ago
  Two different talented developers have told me with regards to tweaking news.arc: "I think the reason you see a lot of posts on arc-the-language and very few on news-the-application is that arc is small and simple, while news, and arc's whole web application DSL, is very alien and difficult to learn :). " and "My preferred approach would be to do the bulk of the work in PLT Scheme, and then add the necessary interfaces into ac.scm."
jacquesm 17y ago
I've toyed around with it for a couple of days about two minor releases ago, it was quite stable but under load that might be a different situation. My lisp knowledge is around -270 degrees so I had a real hard time understanding the code but bit by bit it started to clear up and once you get past that it looked very compact. The only person that ran a real production clone of HN with substantial traffic that I'm aware of was nickb, together with prakash and miles under 'newmogul.com', since then this has been taken over by markenomics.com, but there doesn't seem to be a whole lot of activity there. As for stability under load, from what I can tell by observing HN the server goes down several times an hour and has a script wrapped around it to keep it up or something like that. Every time it goes down it re-reads the state from a bunch of files and then seems to work quite well until the next crash. The crashes - as far as I can see - rarely lead to data loss though there have been some rare instances of comments getting mixed up in the wrong threads. I hope this helps.
- thibaut_barrere 17y ago
  Thanks for the insight, it's really helpful. I think I'll probably keep this as a learning exercise, then. Thanks!
- andrew1 17y ago
  What causes the crashes? It sounds like a rather bizarre system!
  - jacquesm 17y ago
    I think it has to do in part with requesting pages that cause large datasets to be accessed, in particular requests for the posting history of say edw519, nickb or other top posters seem to have a (very) bad effect on the server. Some of this has been fixed but it still happens plenty of times. Maybe PG can shed some light on the exact cause of this, I'm just making guesses based on my interaction with the service.
    - andrew1 17y ago
      Yes, purely out of curiousity I'd be interested in what's happening behind the scenes. Is memory being leaked? Is garbage collection (assuming it's used) not good enough? Is it a deliberate design choice for HN?
      - jacquesm 17y ago
        Try to retrieve the posting histories of the top 10 of the leaderboard, see if they all come up for you without weirdness happening. I can't get it to work, some will work, some will give blank pages and sometimes the server seems to restart. Would be nice to see what someone else gets.
        streety 17y ago
        I retrieved the comment histories of the top 10 users without a single glitch. I don't follow any particular users so don't view the comments pages all that often so can only comment on this occasion but no problems encountered this time.
iamelgringo 17y ago
If you're interested, I'd be willing to give you a clone of the http://Newsley.com http://Newsley.com codebase. It's a bit bare bones right now, and since I'm the sole developer, I haven't worked on the documentation much. But, it's Python/Django, which might be a little more familiar to you if you're a Ruby/Rails guy. I've thought about open sourcing the code, but some parts are a little ugly right now, so I haven't done it. But if you're interested, ping me and we can talk. Another thought is Slinkset: http://slinkset.com/ http://slinkset.com/ It looks like they still have some active sites, and it looks like they've added features since I last looked a few months back. Anyone know if they're still up?
- thibaut_barrere 17y ago
  Hey - thanks for the links. Look interesting. If you planned to open-source it, go for it, but I won't use it if it's just between you and me. I'll either use an open-source implementation (good support + community) or create it myself (just what I need). Thanks though!
  - iamelgringo 17y ago
    No worries. Any time.
adrianwaj 17y ago
I wrote up a detailed spec for an enhanced version of news.arc, I literally saved the HN item, home and user pages and made changes, which I elaborated upon in the rest of the document. I'd be interested to share it in some sort of collaboration.

thibaut_barrere 17y ago

I also welcome comments on how hard it was to customize the look or behaviour, if you did so.

adrianwaj 17y ago
Two different talented developers have told me with regards to tweaking news.arc: "I think the reason you see a lot of posts on arc-the-language and very few on news-the-application is that arc is small and simple, while news, and arc's whole web application DSL, is very alien and difficult to learn :). " and "My preferred approach would be to do the bulk of the work in PLT Scheme, and then add the necessary interfaces into ac.scm."

jacquesm 17y ago

I've toyed around with it for a couple of days about two minor releases ago, it was quite stable but under load that might be a different situation. My lisp knowledge is around -270 degrees so I had a real hard time understanding the code but bit by bit it started to clear up and once you get past that it looked very compact. The only person that ran a real production clone of HN with substantial traffic that I'm aware of was nickb, together with prakash and miles under 'newmogul.com', since then this has been taken over by markenomics.com, but there doesn't seem to be a whole lot of activity there. As for stability under load, from what I can tell by observing HN the server goes down several times an hour and has a script wrapped around it to keep it up or something like that. Every time it goes down it re-reads the state from a bunch of files and then seems to work quite well until the next crash. The crashes - as far as I can see - rarely lead to data loss though there have been some rare instances of comments getting mixed up in the wrong threads. I hope this helps.

thibaut_barrere 17y ago
Thanks for the insight, it's really helpful. I think I'll probably keep this as a learning exercise, then. Thanks!
andrew1 17y ago
What causes the crashes? It sounds like a rather bizarre system!
- jacquesm 17y ago
  I think it has to do in part with requesting pages that cause large datasets to be accessed, in particular requests for the posting history of say edw519, nickb or other top posters seem to have a (very) bad effect on the server. Some of this has been fixed but it still happens plenty of times. Maybe PG can shed some light on the exact cause of this, I'm just making guesses based on my interaction with the service.
  - andrew1 17y ago
    Yes, purely out of curiousity I'd be interested in what's happening behind the scenes. Is memory being leaked? Is garbage collection (assuming it's used) not good enough? Is it a deliberate design choice for HN?
    - jacquesm 17y ago
      Try to retrieve the posting histories of the top 10 of the leaderboard, see if they all come up for you without weirdness happening. I can't get it to work, some will work, some will give blank pages and sometimes the server seems to restart. Would be nice to see what someone else gets.
      - streety 17y ago
        I retrieved the comment histories of the top 10 users without a single glitch. I don't follow any particular users so don't view the comments pages all that often so can only comment on this occasion but no problems encountered this time.

iamelgringo 17y ago

If you're interested, I'd be willing to give you a clone of the http://Newsley.com http://Newsley.com codebase. It's a bit bare bones right now, and since I'm the sole developer, I haven't worked on the documentation much. But, it's Python/Django, which might be a little more familiar to you if you're a Ruby/Rails guy. I've thought about open sourcing the code, but some parts are a little ugly right now, so I haven't done it. But if you're interested, ping me and we can talk. Another thought is Slinkset: http://slinkset.com/ http://slinkset.com/ It looks like they still have some active sites, and it looks like they've added features since I last looked a few months back. Anyone know if they're still up?

thibaut_barrere 17y ago
Hey - thanks for the links. Look interesting. If you planned to open-source it, go for it, but I won't use it if it's just between you and me. I'll either use an open-source implementation (good support + community) or create it myself (just what I need). Thanks though!
- iamelgringo 17y ago
  No worries. Any time.

adrianwaj 17y ago

I wrote up a detailed spec for an enhanced version of news.arc, I literally saved the HN item, home and user pages and made changes, which I elaborated upon in the rest of the document. I'd be interested to share it in some sort of collaboration.