5 ms·

What caused that outage

74 points by pg 18y ago

HN was down today for around 2 hours. Sorry about that. The News server currently crashes a couple times a day when it runs out of memory. All the comments and stories no longer fit in the 2 GB we can get on a 32 bit machine. We'd been planning to upgrade to a new 64 bit server. In the meantime it was arguably a rather slow form of GC. Unfortunately the process somehow got wedged in the middle of segfaulting. We're not sure why and will probably never know. But that meant the process that usually notices when News is wedged and restarts it was unable to kill it. Fortunately rebooting the machine solved the problem. But now we'll presumably be switching to that new, bigger server sooner rather than later. As far as I can tell it was a coincidence that this happened today. It doesn't seem to have been caused either by the increased traffic, or the excessive number of posts about Erlang.

deleted 18y ago
[deleted]
mixmax 18y ago
That's what happens when you make something in a new and unproven language. And I think that it deserves tremendous respect that you have done so. You could have hacked HN in python (or lisp?) in a weekend with the knowledge that it would scale, yet you decided to go with Arc. Very bold. And a great way to battletest a new language. It's great to see people eat their own dogfood.
- dylanz 18y ago
  Wow, had no idea this was written in Arc. +1 to what mixmax said. It's the only way to keep pushing the envelope.
  - mixmax 18y ago
    Boldness drives innovation
- Jebdm 18y ago
  If I'm not mistaken, "to battletest a new language" was the purpose of building this.
  - rw 18y ago
    Keep in mind that mzscheme is battle tested. Arc is "just" a bunch of macros on top of PLT Scheme; what gives HN its [under]performing character is that foundational JIT compiler and 3m GC.
    - pg 18y ago
      Arc compiles into MzScheme, but it's not implemented as macros. You can see that from the source.
    - elibarzilay 18y ago
      In general, 3m is performing very well, much better than the conservative collector plt used to use by default (the boehm gc). As for the jit: many of the benefit of the jit are irrelevant in Arc since it doesn't use mzscheme modules; this is in addition to using an old mzscheme version, when the jit was rather new -- many many improvements were done since then.
- jrockway 18y ago
  I doubt Arc is the cause of any problems. I think this data model would be problematic in any language. You can't just let your processes die when you are low on memory, you need to evict unused pages from the cache.
  - lacker 18y ago
    Well, in a language with existing web frameworks it would be convenient to use one of them and store your data with SQL. In Arc you would have to write your own SQL wrapper. I suspect if pg was not using Arc he would not be keeping everything in memory. That said, the lisp world could use a good web framework, so perhaps this will inspire some lispers to create one.
    - jrockway 18y ago
      I suspect if pg was not using Arc he would not be keeping everything in memory. I don't want to speak for pg, but I highly doubt this. He chose the architecture he did for good reason. Looking things up in memory is much faster than looking them up in the database. The only issue is that the site is too big to fit entirely into memory now, so he needs to write something that cleanly manages the memory space. (BerkeleyDB calls this a pager; data is stored in pages that can be in memory or disk.) With that in place, the site should perform fine.
    - pg 18y ago
      I would be keeping everything in memory (using lazy loading) regardless. It makes life simpler.
      - lacker 18y ago
        You wouldn't use SQL? Something like django or rails is pretty handy if you are already using those languages, as annoying as it can be.
        jrockway 18y ago
        SQL databases are a very poor fit for sites like news.yc. Things like threaded comments, for example, are hard to model efficiently. If you use an object database (or roll your own in-memory thing), though, it is very easy to model. Relational databases are certainly useful in some situations, but there are usually better options for websites. You should consider learning about other options before telling everyone to use Rails. Rails is not the only way to do things, and it's rarely the best way.
acegopher 18y ago
The server goes down, and when it comes back up the front page is filled with stories about Erlang. Coincidence?
- Derrek 18y ago
  Sign of the Second Coming?
jm4 18y ago
Bummer. Maybe you should consider rewriting in Erlang... :)
mblakele 18y ago
Why was the site using a 32-bit environment to begin with? Opterons have been cheap for years now, and all new Xeons are capable of 64-bit operation.
- MrRage 18y ago
  What chip in put out in the last 2 years or so couldn't run a 64 bit system? I'm pretty sure most of the AMD chips could... Edit: By chip I mean one for a desktop CPU, e.g. Atom don't count.
  - lutorm 18y ago
    Atom 330 is also 64-bit.
- axod 18y ago
  IDK if it's relevant, but 64bit can seriously suck also. If you use a lot of pointers, you just doubled the memory you need for each pointer. In my experience, 64bit isn't a good way to save memory.
natrius2 18y ago
Speaking of bugs, pages don't load for me when I'm logged into my real account (without the 2). The front page cuts off after "1.</td><td><center>" and comments pages cut off right after the first sub-table opens. The leaders and submit pages load fine. I thought it was just a sign that I should get back to work, but maybe my account got put into some sort of inconsistent state?
timtrueman 18y ago
I thought 32-bit addressing gave 4GB of addresses…is there some sort of flag that's taking one bit? Not trying to be a smartass, just curious about the discrepancy.
- spydez 18y ago
  It does give 4 GB of addressable space, but many things are memory mapped by the OS, and so on a 32-bit system you end up with anywhere from 2.8 to 3.5 GB of addressable memory.
  - timtrueman 18y ago
    So by 2GB he meant usable, not addressable? I understand that addressable RAM != 4GB due to graphics card memory, BIOS, etc. but 2GB is way less than you could get on a 32-bit server. I just wanted to understand if there's something I'm missing.
    - gnaritas 18y ago
      2 gig is all you're going to get for a single process anyway, that's some kind of limit if I recall correctly. This site is a single process.
    - deleted 18y ago
      [deleted]
- protomyth 18y ago
  The OS sometimes reserves quite a chunk. For example, Windows XP really only has 2.25GB available. Putting more RAM in is pretty useless unless you switch to 64-bit.
  - vinutheraj 18y ago
    Can't this be changed, you know, by hacking the registry or something ?!
ajkirwin 18y ago
You aren't running this in the standard language/caching/db setup, then? You're storing EVERYTHING in memory? Isn't this kind of.. well, stupid, frankly.
- mishmash 18y ago
  While the comment above may not have the best manners, it does bring up an interesting point. I certainly wouldn't mind hearing more about HN's setup.
  - paulgb 18y ago
    If you're really curious and have the time, you can download the source: http://arclanguage.org/install http://arclanguage.org/install
    - catch404 18y ago
      It'd be interesting to read a write up on the concepts used (such as the in memory database) and the server restart mechanism. It'd be hard to tell from the source for non Lisp programmers.
      - lacker 18y ago
        An in-memory database is pretty simple - it's just a bunch of hash tables. A server restart mechanism is also pretty simple. Write a cron job that activates every few seconds to ps aux and grep for the server name, and if it's not there start the server. That won't catch server hangs rather than deaths, but that's exactly the problem that happened here.
        catch404 18y ago
        Thanks, I understood it uses hash tables - I've done the same myself, but never on such a large scale. This post sums it up: http://news.ycombinator.com/item?id=513259 http://news.ycombinator.com/item?id=513259
jjb 18y ago
Check out god -- it can restart mongrels when they reach a certain memory size: http://god.rubyforge.org/ http://god.rubyforge.org/ (if that's the issue)
adnymarc 18y ago
Judging by the number of Erlang stories on the front page (19 out of 25 currently) I would say that's your problem right there...(wholly in jest, half in earnest)
BonsaiKitt3n 18y ago
add +10 mhz to the heap
callahad 18y ago
Thank God. I was worried you had pushed the big red noprocrast.
ryan-allen 18y ago
pg, I'd love to read a post about how you've dealt with writing HN to work in one process, in 2GB of RAM, it sounds quite novel! Did you have to make manual indexes? How do you arrange the files on the file system? How do you handle voting and concurrency/file locking?
DanielBMarkham 18y ago
Well there's only one good thing for the ultimate in scalability -- arc in the cloud. Which I believe should be called "lightning", right?
lacker 18y ago
Yeah, it's not optimal to let your processes die from running out of memory because maybe some other non-server process actually requested the last bit of memory, and then who knows what sort of state your machine is in. How about making your restarter job notice when the machine is very close to being out of memory and preemptively kill the server then?
lsc 18y ago
hey, so I'd be happy to donate 16GB worth of Xen instances to the project,just to say I did, if you need mirrors. my boxes are 32GB ram/8 core, and the CPU is proportional, so if you want 4x4GB instances over separate servers, I can do that. (well, you will have to wait a bit for me to put up the 4th server.)
elibarzilay 18y ago
Something that could be done for now, is to write a piece of mzscheme code that "marshalls" the data in (utf-8-encoded) byte-strings. Assuming that most of the 2gb is made of strings, and that these strings are mostly ascii, this should reduce the consumption by close to a factor of 4. (I can imagine an interface that is transparent at the arc level, where are strings are just passed to the backend and retrieved from it, and the backend converts them to and from byte strings. Later on it could change to use a FS or a DB or whatever.)

74 points by pg 18y ago

deleted 18y ago

[deleted]

mixmax 18y ago

That's what happens when you make something in a new and unproven language. And I think that it deserves tremendous respect that you have done so. You could have hacked HN in python (or lisp?) in a weekend with the knowledge that it would scale, yet you decided to go with Arc. Very bold. And a great way to battletest a new language. It's great to see people eat their own dogfood.

dylanz 18y ago
Wow, had no idea this was written in Arc. +1 to what mixmax said. It's the only way to keep pushing the envelope.
- mixmax 18y ago
  Boldness drives innovation
Jebdm 18y ago
If I'm not mistaken, "to battletest a new language" was the purpose of building this.
- rw 18y ago
  Keep in mind that mzscheme is battle tested. Arc is "just" a bunch of macros on top of PLT Scheme; what gives HN its [under]performing character is that foundational JIT compiler and 3m GC.
  - pg 18y ago
    Arc compiles into MzScheme, but it's not implemented as macros. You can see that from the source.
  - elibarzilay 18y ago
    In general, 3m is performing very well, much better than the conservative collector plt used to use by default (the boehm gc). As for the jit: many of the benefit of the jit are irrelevant in Arc since it doesn't use mzscheme modules; this is in addition to using an old mzscheme version, when the jit was rather new -- many many improvements were done since then.
jrockway 18y ago
I doubt Arc is the cause of any problems. I think this data model would be problematic in any language. You can't just let your processes die when you are low on memory, you need to evict unused pages from the cache.
- lacker 18y ago
  Well, in a language with existing web frameworks it would be convenient to use one of them and store your data with SQL. In Arc you would have to write your own SQL wrapper. I suspect if pg was not using Arc he would not be keeping everything in memory. That said, the lisp world could use a good web framework, so perhaps this will inspire some lispers to create one.
  - jrockway 18y ago
    I suspect if pg was not using Arc he would not be keeping everything in memory. I don't want to speak for pg, but I highly doubt this. He chose the architecture he did for good reason. Looking things up in memory is much faster than looking them up in the database. The only issue is that the site is too big to fit entirely into memory now, so he needs to write something that cleanly manages the memory space. (BerkeleyDB calls this a pager; data is stored in pages that can be in memory or disk.) With that in place, the site should perform fine.
  - pg 18y ago
    I would be keeping everything in memory (using lazy loading) regardless. It makes life simpler.
    - lacker 18y ago
      You wouldn't use SQL? Something like django or rails is pretty handy if you are already using those languages, as annoying as it can be.
      - jrockway 18y ago
        SQL databases are a very poor fit for sites like news.yc. Things like threaded comments, for example, are hard to model efficiently. If you use an object database (or roll your own in-memory thing), though, it is very easy to model. Relational databases are certainly useful in some situations, but there are usually better options for websites. You should consider learning about other options before telling everyone to use Rails. Rails is not the only way to do things, and it's rarely the best way.

acegopher 18y ago

The server goes down, and when it comes back up the front page is filled with stories about Erlang. Coincidence?

Derrek 18y ago
Sign of the Second Coming?

jm4 18y ago

Bummer. Maybe you should consider rewriting in Erlang... :)

mblakele 18y ago

Why was the site using a 32-bit environment to begin with? Opterons have been cheap for years now, and all new Xeons are capable of 64-bit operation.

MrRage 18y ago
What chip in put out in the last 2 years or so couldn't run a 64 bit system? I'm pretty sure most of the AMD chips could... Edit: By chip I mean one for a desktop CPU, e.g. Atom don't count.
- lutorm 18y ago
  Atom 330 is also 64-bit.
axod 18y ago
IDK if it's relevant, but 64bit can seriously suck also. If you use a lot of pointers, you just doubled the memory you need for each pointer. In my experience, 64bit isn't a good way to save memory.

natrius2 18y ago

Speaking of bugs, pages don't load for me when I'm logged into my real account (without the 2). The front page cuts off after "1.</td><td><center>" and comments pages cut off right after the first sub-table opens. The leaders and submit pages load fine. I thought it was just a sign that I should get back to work, but maybe my account got put into some sort of inconsistent state?

timtrueman 18y ago

I thought 32-bit addressing gave 4GB of addresses…is there some sort of flag that's taking one bit? Not trying to be a smartass, just curious about the discrepancy.

spydez 18y ago
It does give 4 GB of addressable space, but many things are memory mapped by the OS, and so on a 32-bit system you end up with anywhere from 2.8 to 3.5 GB of addressable memory.
- timtrueman 18y ago
  So by 2GB he meant usable, not addressable? I understand that addressable RAM != 4GB due to graphics card memory, BIOS, etc. but 2GB is way less than you could get on a 32-bit server. I just wanted to understand if there's something I'm missing.
  - gnaritas 18y ago
    2 gig is all you're going to get for a single process anyway, that's some kind of limit if I recall correctly. This site is a single process.
  - deleted 18y ago
    [deleted]
protomyth 18y ago
The OS sometimes reserves quite a chunk. For example, Windows XP really only has 2.25GB available. Putting more RAM in is pretty useless unless you switch to 64-bit.
- vinutheraj 18y ago
  Can't this be changed, you know, by hacking the registry or something ?!

ajkirwin 18y ago

You aren't running this in the standard language/caching/db setup, then? You're storing EVERYTHING in memory? Isn't this kind of.. well, stupid, frankly.

mishmash 18y ago
While the comment above may not have the best manners, it does bring up an interesting point. I certainly wouldn't mind hearing more about HN's setup.
- paulgb 18y ago
  If you're really curious and have the time, you can download the source: http://arclanguage.org/install http://arclanguage.org/install
  - catch404 18y ago
    It'd be interesting to read a write up on the concepts used (such as the in memory database) and the server restart mechanism. It'd be hard to tell from the source for non Lisp programmers.
    - lacker 18y ago
      An in-memory database is pretty simple - it's just a bunch of hash tables. A server restart mechanism is also pretty simple. Write a cron job that activates every few seconds to ps aux and grep for the server name, and if it's not there start the server. That won't catch server hangs rather than deaths, but that's exactly the problem that happened here.
      - catch404 18y ago
        Thanks, I understood it uses hash tables - I've done the same myself, but never on such a large scale. This post sums it up: http://news.ycombinator.com/item?id=513259 http://news.ycombinator.com/item?id=513259

jjb 18y ago

Check out god -- it can restart mongrels when they reach a certain memory size: http://god.rubyforge.org/ http://god.rubyforge.org/ (if that's the issue)

adnymarc 18y ago

Judging by the number of Erlang stories on the front page (19 out of 25 currently) I would say that's your problem right there...(wholly in jest, half in earnest)

BonsaiKitt3n 18y ago

add +10 mhz to the heap

callahad 18y ago

Thank God. I was worried you had pushed the big red noprocrast.

ryan-allen 18y ago

pg, I'd love to read a post about how you've dealt with writing HN to work in one process, in 2GB of RAM, it sounds quite novel! Did you have to make manual indexes? How do you arrange the files on the file system? How do you handle voting and concurrency/file locking?

DanielBMarkham 18y ago

Well there's only one good thing for the ultimate in scalability -- arc in the cloud. Which I believe should be called "lightning", right?

lacker 18y ago

Yeah, it's not optimal to let your processes die from running out of memory because maybe some other non-server process actually requested the last bit of memory, and then who knows what sort of state your machine is in. How about making your restarter job notice when the machine is very close to being out of memory and preemptively kill the server then?

lsc 18y ago

hey, so I'd be happy to donate 16GB worth of Xen instances to the project,just to say I did, if you need mirrors. my boxes are 32GB ram/8 core, and the CPU is proportional, so if you want 4x4GB instances over separate servers, I can do that. (well, you will have to wait a bit for me to put up the 4th server.)

elibarzilay 18y ago

Something that could be done for now, is to write a piece of mzscheme code that "marshalls" the data in (utf-8-encoded) byte-strings. Assuming that most of the 2gb is made of strings, and that these strings are mostly ascii, this should reduce the consumption by close to a factor of 4. (I can imagine an interface that is transparent at the arc level, where are strings are just passed to the backend and retrieved from it, and the backend converts them to and from byte strings. Later on it could change to use a FS or a DB or whatever.)