4 ms·
It's true that this is elementary in and of itself, but looking at things with a bit of a wider lens shows the complexity. We're a small engineering team (10 pe
by harryh 16y ago
It's true that this is elementary in and of itself, but looking at things with a bit of a wider lens shows the complexity. We're a small engineering team (10 people) working on a product that is growing extremely fast both in terms of usage and feature set. Meanwhile we're also pretty much constantly re-architecting things to keep up with growth and also doing the immense work of growing the company up from 3 people to 33 and beyond (this has turned out to be WAY HARDER than I would have guessed going in).
Further, there are lots and lots of different things that we need to be monitoring at any different time to make sure that everything is going ok and we aren't about to run into a wall. Automated tools can help a lot with this, but these tools still need to be properly set up and maintained.
I'm not saying we didn't screw up. We had 17 hours of downtime over two days. We screwed up bad, and we feel horrible about it, and are doing a lot to make sure that we don't screw up the same way again.
But it's not because we're morons that never thought about the fact that we should be monitoring memory usage. We just got overwhelmed with the complexity of all that we're doing at once.
-harryh, foursquare eng lead
- chewbranca 16y agoI agree completely about the complexity of monitoring solutions for small teams with fluctuating applications. However, something as simple as htop running on an extra monitor would have alerted you of this issue long before downtime resulted.
- cullenking 16y agoAwww come on, it's easy to say from an outside perspective. Regardless, the problem was handled well in the end, and we all get the benefit of understanding these limitations better. I think us tech people got the good side (information) out of this ordeal :)
- chewbranca 16y agoI work on a team of 2 where I'm responsible for a handful of servers. I chimed in because I'm in a similar position, I've been looking for a monitoring solution for a while now. Things like nagios and zenoss are over kill, but lack of time has prevented me from finding an ideal solution. That said, I keep htop open and running at all times, and its saved my ass on more than one occasion. I say htop because of the color coding it provides, if things start going red it attracts my attention.
- sjs 16y agoNagios is pretty nice. It's dead easy to write custom monitors and clients are everywhere, there's even a Firefox extension. It requires a bit of learning to get going with but it's not so bad and the pay-off is big. That said I'm looking at monit too. I hear it's quite nice and has less of a learning curve.
- rhizome 16y agoNagios/Cacti/etc. are indeed overkill for a few servers, but Munin is dead-simple and has notifications: http://munin-monitoring.org/wiki/HowToContact http://munin-monitoring.org/wiki/HowToContact
- mootothemax 16y agoSecond vote for Munin, is simple, quick to install and set up and covers all the basics.
- dedward 16y agoStop thinking about it and set it up. There isn't anything to think about. Nagios (I like using it through groundwork) are certainly not overkill... it's thinking like that that is what gets you in trouble in the first place. Get that tool in place NOW while you are small, so it's there when you get bigger... so you can hire people to watch it while you move on to other things. You will always have a lack of time, and always be busy. Spend a weekend putting up Nagios, setting up alerts over SMS or XMPP or Twitter or whatever you want, and then move on. This is fundamental to systems management. Dont' over think it - just get nagios/cacti (again, try groundwork) up and running and graph EVERYTHING you can... yuo'll be very, very glad you did.
- jranck 16y agoI didn't realize you only had 10 guys behind the scenes, that's pretty impressive in and of itself. In all actuality this is a good problem to have and one I wouldn't mind having. It means you're growing and fast.
- nkallen 16y agoSuggestion: In additions to hiring scalability experts (many will claim to be experts but most aren't), talk to your investors to find outside technical advisors who have worked on and are still working on similar problems. Find advisors from Google, Facebook, and other places who have dealt with these issues. In my experience, you have as much to learn from people who have made mistakes as from people who have succeeded. Also, in my judgement, you guys are a little too risk-tolerant for your significance (first major Scala 2.8 upgrade, largest MongoDB deploy...).
- foobarbazetc 16y agoThe fail here isn't just that you should have set up a shard 12 hours before or whatever. The other fail is that it seems (from the groups post) that you were relying on your data set only fitting in RAM but the underlying I/O system wasn't quite up to scratch in terms of being able to page in/out that additional 1GB of data. It's not like your 66GB is hot all the time (or even big by any sort of measure), so that makes no sense to me, and that part of things needs to be further explained by 10gen and/or 4sq.