3 ms·

Exactly. How the hell does a fast growing well funded 24 employee startup NOT have load monitoring on their database servers! Pay the 10c the hour for a micro E

by hartror 16y ago

Exactly. How the hell does a fast growing well funded 24 employee startup NOT have load monitoring on their database servers! Pay the 10c the hour for a micro EC2 instance and run Zabbix or one of the half dozen other awesome monitoring packages out there.

binspace 16y ago
Other things get prioritized.
- hartror 16y ago
  Like not having a massive downtime that gives their users a reason to try out some massive social network's offering that has already started eating their lunch? Oh wait no they didn't prioritize that. My point is, and perhaps the down vote indicates I didn't make it well, that monitoring and alerts are extremely important for any startup. They admit as much in the article, had they known sooner there wouldn't have been any down time. It isn't an excuse to say it got lost in the crush (not that they are trying to use it as an excuse, they're being very upfront and admitting fault). We have neither the load or the team they do and we have comprehensive monitoring and alerts across our infrastructure as part of the SOE that each server gets. Perhaps we prioritized monitoring higher than them as we are a small team? And I bet they have prioritized monitoring now. Now where did I put the key to that barn door?
  - metageek 16y ago
    Now where did I put the key to that barn door? It's in the saddlebags.
  - binspace 16y ago
    Humans don't judge probability and risk very well. It's hard to forsee the future and problems. Hindsight is 20/20. My bag of cliches is exhausted. Maybe there were other things the engineers were working on that brought definite (as in 100%) benefit. Maybe there's a good product/consulting opportunity here. Everybody needs monitoring, but it takes time and resources. I agree, that they are focusing on it now, because it happened.
heresy 16y ago
If it's anything like here, any one engineer's task list is essentially a weighted list of fires to put out and technical debt to pay down, in addition to the features to be delivered next day / week. Guess more extensive monitoring just got bumped to the top :)
- hartror 16y ago
  If it is anything like any start up I have ever heard of :P My point is that it is stupidly important to have good monitoring in place and I am surprised that one of the best known startups had a gaping hole in theirs.
- lr 16y ago
  Which is exactly why the "don't optimize now" and/or "wait until you have users before you worry about capacity planning" crowd is simply clueless. You need to think about your data and how it will scale from day one!
sofuture 16y ago
2 cents an hour!
83457 16y ago
Do they have server admin on staff, or is the whole point of hosting in the cloud to not need them?
- dedward 16y ago
  Don't know about them - but there seems to be a somewhat scary, hopefully brief, undercurrent out there where business (startups, corporations, whatever) think they can work solely with a few developers and some outsourced services - nobody is really taking responsibility for overall architecture and growth. (again, NOT saying this applies to foursquare, I have no idea)

by hartror 16y ago

binspace 16y ago

Other things get prioritized.

hartror 16y ago
Like not having a massive downtime that gives their users a reason to try out some massive social network's offering that has already started eating their lunch? Oh wait no they didn't prioritize that. My point is, and perhaps the down vote indicates I didn't make it well, that monitoring and alerts are extremely important for any startup. They admit as much in the article, had they known sooner there wouldn't have been any down time. It isn't an excuse to say it got lost in the crush (not that they are trying to use it as an excuse, they're being very upfront and admitting fault). We have neither the load or the team they do and we have comprehensive monitoring and alerts across our infrastructure as part of the SOE that each server gets. Perhaps we prioritized monitoring higher than them as we are a small team? And I bet they have prioritized monitoring now. Now where did I put the key to that barn door?
- metageek 16y ago
  Now where did I put the key to that barn door? It's in the saddlebags.
- binspace 16y ago
  Humans don't judge probability and risk very well. It's hard to forsee the future and problems. Hindsight is 20/20. My bag of cliches is exhausted. Maybe there were other things the engineers were working on that brought definite (as in 100%) benefit. Maybe there's a good product/consulting opportunity here. Everybody needs monitoring, but it takes time and resources. I agree, that they are focusing on it now, because it happened.

heresy 16y ago

If it's anything like here, any one engineer's task list is essentially a weighted list of fires to put out and technical debt to pay down, in addition to the features to be delivered next day / week. Guess more extensive monitoring just got bumped to the top :)

hartror 16y ago
If it is anything like any start up I have ever heard of :P My point is that it is stupidly important to have good monitoring in place and I am surprised that one of the best known startups had a gaping hole in theirs.
lr 16y ago
Which is exactly why the "don't optimize now" and/or "wait until you have users before you worry about capacity planning" crowd is simply clueless. You need to think about your data and how it will scale from day one!

sofuture 16y ago

2 cents an hour!

83457 16y ago

Do they have server admin on staff, or is the whole point of hosting in the cloud to not need them?

dedward 16y ago
Don't know about them - but there seems to be a somewhat scary, hopefully brief, undercurrent out there where business (startups, corporations, whatever) think they can work solely with a few developers and some outsourced services - nobody is really taking responsibility for overall architecture and growth. (again, NOT saying this applies to foursquare, I have no idea)