4 ms·

Jeebus, what a way to turn a molehill into a mountain. @22cans: In fact @pmolyneux BEGGED the team to go home but they wouldn't have any of it :) https://twit

Jeebus, what a way to turn a molehill into a mountain. @22cans: In fact @pmolyneux BEGGED the team to go home but they wouldn't have any of it :) https://twitter.com/22Cans/status/268387590368030721 https://twitter.com/22Cans/status/268387590368030721 22 Cans is a tiny startup and they were expecting 10,000 concurrent users tops -- they're at 2 million and counting now, and they have to capitalize on that momentum and not piss off their obviously fanatical players. So they pulled an all-nighter to build and migrate from a single server to a clustered architecture, and hell yeah, if I was in that position I'd do that too. Seriously, is there another option? If you think the answer is "clock out at 5:01 PM and go home", you shouldn't be in a startup. Obviously pulling 36 hour shifts on a regular basis is not sustainable, but this is precisely the type of situation where it's warranted.

wallflower 14y ago
Keeping the servers up under heavy load is part of the startup archetype. You have to cross the Rubicon. > The servers melted. Systrom and Krieger worked a straight 24 hours to keep the app running—in that period 25,000 iPhone users downloaded the free service. http://www.forbes.com/sites/stevenbertoni/2012/08/01/instagrams-kevin-systrom-the-stanford-millionaire-machine-strikes-again/3/ http://www.forbes.com/sites/stevenbertoni/2012/08/01/instagr...
fatbird 14y ago
The question is why did they assume 10k concurrent users, or plan for that to be the cap handled under the current architecture? Any competent project manager should have said "let's pretend we're massively successful... how will it work then?" and have some contingency plan in place so that they're not trying "experimental" fixes. The flaw here isn't the heroic efforts, it's that heroic efforts were needed. In the days of cloud based architectures, it's a lot easier to be ready for massive horizontal scaling than it used to be, so much so that being unready is malpractice.
- Negitivefrags 14y ago
  Assuming massive success is a stupid move. The majority of the time you are not that successful. The ground is littered by projects that spent vast sums of money (and more importantly time) designing for load they never needed to handle before release. In about a month my company will be releasing an online game. We think we will get about 15-20k concurrent users in the first few days, but we will be testing the realm for as high as 50k concurrent users using load testing bots. What will happen if we suddenly get 100k or 200k concurrent users? Even throwing more hardware at the problem might not solve it immediately if there is some new bottleneck we haven't found before. Testing the realm for more than 50k concurrents and then spending engineering effort required to make it work will consume time and effort that we can't afford to spend. All for an outcome that is unlikely to occur.
  - fatbird 14y ago
    Assuming massive success is stupid, but testing for it seems prudent. Given how easy it is to spin up EC2 instances generate bot-load, even just to see what'll happen or to find the collapse point, seems so straightforward as to be routine. And as the commenter below observes, the Curiosity interaction model requires high server load for all interaction.
    - btilly 14y ago
      The problem with your theory is that there are lies, damned lies, and load tests. Any sensible app is going to have caching layers. A simple load test is going to just be a really good exercise of the caching layers. By contrast a complex load test with lots of calls to do really random stuff is going to stress you out in entirely unreasonable ways. And finding a realistic balance of traffic is difficult unless you've simply sampled real traffic - which you can't do until after you've generated some.
      - bowyakka 14y ago
        This! I cant agree with this more, unless you approach me with an experiment that would put most science to shame I do not value your use of Jmeter/Tsung/Siege/Rando-Python-Script. If you test plan does not have pauses in it, and you are not engineering a nuclear reactor I probably wont even start the conversation. .... At $DAYJOB, when we were putting together a new search engine we spent some time upfront putting in a lot of tooling to worry about and test performance every step of the way. Some of this tooling was built around jmeter and focused on generating realistic test plans (which we could derive from the already operational existing search engine). It was a neat infrastructure it tracked everything by commit, captured all manner of metrics and used huge datasets, mined from the logs, built out with markov models. Everything we reported into an install of the software that runs http://speed.pypy.org http://speed.pypy.org, we thought we had it conquered. Yet it was so flawed! Data would get stale, models would break, caches invalidated datasets and much much more. It was funny at the same time I wrote a tool that we now know as darklaunch, this tool is stupid simple, it captures incoming traffic to the production system with pcap, and then (after a little mussing with it) sends it through to a always on test cluster. Yet its this tool that produces all the load, has all the weird user behavior, never gets out of sync and can run load, soak, endurance and impact tests. Guess which one we use to test out new software, or new features, or to tune a garbage collector in as close to production as possible ? I have not started Jmeter now in over a year, its been no great loss; sure we have microbenchmarks where they matter but for everything else you often hear the words "darklaunch it" Now for sure we have the luxury of having real traffic to use and abuse, but if you don't have real traffic then realise you are using a scrying device anyhow and I will bet you good money your assumptions are wrong.
- jpatokal 14y ago
  10,000 concurrent users within a day of release is massive success in today's application market. Engineering a system to handle an order of magnitude more traffic than you expect is sensible; engineering it to handle three orders of magnitude more than you expect would, virtually always, be ridiculously wasteful. Also, Curiosity is not your average horizontally scalable web app, because all the users are continuously interacting with the same data set. Scaling this up to 2 million concurrent users would be a challenge even with time on your hands.
  - fatbird 14y ago
    As you observe, Curiosity's interaction model implies continuous server load. 10K users actually seems low to me, given that.
  - seivan 14y ago
    Very off topic, but what are your thoughts on this; I built something alone for both the client (iOS with Cocos2d) and server (Ruby, MongoDb, Redis) that managed 3k users on release for a full day. The app was only for a day (Event app for an ad agency) This took three weeks to develop. The client had 3d graphics and a custom map coded and drawn from scratch.
- angrycoder 14y ago
  Because people make mistakes? Aren't people allowed to make mistakes anymore? Most software developers love solving problems. When presented with a tough problem they don't mind skipping out on a few dinners at home or not watching the latest sports ball game. It is just being a developer, not some kind of malpractice, or slave driving, or not having read enough mythical books about herding cats. Shit goes wrong sometimes, it is in some peoples nature to want to fix the problems.
Luc 14y ago
The point is that these kind of emergencies ('crunch time') always seem to happen in the games industry. It's _assumed_ that this will happen. Surely the game is not very great _unless_ there is a few months of crunch time!
vidarh 14y ago
That he "begged" them to go home is suspect. Is he not in charge? I've sent people home during crunch times before when they demonstrated they were becoming a liability even though they still wanted to keep working. You don't beg. You tell them that is how it is going to be. Leave or get escorted off the premises. > Seriously, is there another option? Yes. Send people home for some sleep and resume. Unless the people who stayed were carrying out purely menial tasks, odds are having them on the job 36 hours straight got far less work done than if they'd taken reasonable breaks. Sure, you can pull more than 8 hours now and again. Sure, you can even pull all nighters now and again. But over a 36 hour period the vast majority of peoples productivity at complex tasks will nosedive well before half way, and stay flat to even negative in jobs like software development, as you slow down other people and negatively impact communications once you get too tired. If you want the most amount of work done in 36 hours, you don't make people work the whole 36 hours.
- ghshephard 14y ago
  Actually, if you enter it well rested the first 36 hours (with a couple 5-10 minute catnaps) can be quite productive, with only minimal reduction in productivity. Particularly if you get into the "zone." It's after that 36 hours, that you need to get about 10-12 hours sleep to get caught up, and then have a few sane days. Before we do complex cut over onto a new software system, we usually make sure all of our software engineers are very well rested leading into a friday night, in case we've got a couple all nighters ahead of us. Anybody who tries to do back-back 36 hour death-marches with only 5-6 hours of sleep in between is setting themselves up for the seriously reduced productivity you've described. Nothing good ever seems to come of that. So - in this scenario, I think they did precisely what they should have, but they shouldn't push their luck.

wallflower 14y ago

Keeping the servers up under heavy load is part of the startup archetype. You have to cross the Rubicon. > The servers melted. Systrom and Krieger worked a straight 24 hours to keep the app running—in that period 25,000 iPhone users downloaded the free service. http://www.forbes.com/sites/stevenbertoni/2012/08/01/instagrams-kevin-systrom-the-stanford-millionaire-machine-strikes-again/3/ http://www.forbes.com/sites/stevenbertoni/2012/08/01/instagr...

fatbird 14y ago

The question is why did they assume 10k concurrent users, or plan for that to be the cap handled under the current architecture? Any competent project manager should have said "let's pretend we're massively successful... how will it work then?" and have some contingency plan in place so that they're not trying "experimental" fixes. The flaw here isn't the heroic efforts, it's that heroic efforts were needed. In the days of cloud based architectures, it's a lot easier to be ready for massive horizontal scaling than it used to be, so much so that being unready is malpractice.

Negitivefrags 14y ago
Assuming massive success is a stupid move. The majority of the time you are not that successful. The ground is littered by projects that spent vast sums of money (and more importantly time) designing for load they never needed to handle before release. In about a month my company will be releasing an online game. We think we will get about 15-20k concurrent users in the first few days, but we will be testing the realm for as high as 50k concurrent users using load testing bots. What will happen if we suddenly get 100k or 200k concurrent users? Even throwing more hardware at the problem might not solve it immediately if there is some new bottleneck we haven't found before. Testing the realm for more than 50k concurrents and then spending engineering effort required to make it work will consume time and effort that we can't afford to spend. All for an outcome that is unlikely to occur.
- fatbird 14y ago
  Assuming massive success is stupid, but testing for it seems prudent. Given how easy it is to spin up EC2 instances generate bot-load, even just to see what'll happen or to find the collapse point, seems so straightforward as to be routine. And as the commenter below observes, the Curiosity interaction model requires high server load for all interaction.
  - btilly 14y ago
    The problem with your theory is that there are lies, damned lies, and load tests. Any sensible app is going to have caching layers. A simple load test is going to just be a really good exercise of the caching layers. By contrast a complex load test with lots of calls to do really random stuff is going to stress you out in entirely unreasonable ways. And finding a realistic balance of traffic is difficult unless you've simply sampled real traffic - which you can't do until after you've generated some.
    - bowyakka 14y ago
      This! I cant agree with this more, unless you approach me with an experiment that would put most science to shame I do not value your use of Jmeter/Tsung/Siege/Rando-Python-Script. If you test plan does not have pauses in it, and you are not engineering a nuclear reactor I probably wont even start the conversation. .... At $DAYJOB, when we were putting together a new search engine we spent some time upfront putting in a lot of tooling to worry about and test performance every step of the way. Some of this tooling was built around jmeter and focused on generating realistic test plans (which we could derive from the already operational existing search engine). It was a neat infrastructure it tracked everything by commit, captured all manner of metrics and used huge datasets, mined from the logs, built out with markov models. Everything we reported into an install of the software that runs http://speed.pypy.org http://speed.pypy.org, we thought we had it conquered. Yet it was so flawed! Data would get stale, models would break, caches invalidated datasets and much much more. It was funny at the same time I wrote a tool that we now know as darklaunch, this tool is stupid simple, it captures incoming traffic to the production system with pcap, and then (after a little mussing with it) sends it through to a always on test cluster. Yet its this tool that produces all the load, has all the weird user behavior, never gets out of sync and can run load, soak, endurance and impact tests. Guess which one we use to test out new software, or new features, or to tune a garbage collector in as close to production as possible ? I have not started Jmeter now in over a year, its been no great loss; sure we have microbenchmarks where they matter but for everything else you often hear the words "darklaunch it" Now for sure we have the luxury of having real traffic to use and abuse, but if you don't have real traffic then realise you are using a scrying device anyhow and I will bet you good money your assumptions are wrong.
jpatokal 14y ago
10,000 concurrent users within a day of release is massive success in today's application market. Engineering a system to handle an order of magnitude more traffic than you expect is sensible; engineering it to handle three orders of magnitude more than you expect would, virtually always, be ridiculously wasteful. Also, Curiosity is not your average horizontally scalable web app, because all the users are continuously interacting with the same data set. Scaling this up to 2 million concurrent users would be a challenge even with time on your hands.
- fatbird 14y ago
  As you observe, Curiosity's interaction model implies continuous server load. 10K users actually seems low to me, given that.
- seivan 14y ago
  Very off topic, but what are your thoughts on this; I built something alone for both the client (iOS with Cocos2d) and server (Ruby, MongoDb, Redis) that managed 3k users on release for a full day. The app was only for a day (Event app for an ad agency) This took three weeks to develop. The client had 3d graphics and a custom map coded and drawn from scratch.
angrycoder 14y ago
Because people make mistakes? Aren't people allowed to make mistakes anymore? Most software developers love solving problems. When presented with a tough problem they don't mind skipping out on a few dinners at home or not watching the latest sports ball game. It is just being a developer, not some kind of malpractice, or slave driving, or not having read enough mythical books about herding cats. Shit goes wrong sometimes, it is in some peoples nature to want to fix the problems.

Luc 14y ago

The point is that these kind of emergencies ('crunch time') always seem to happen in the games industry. It's _assumed_ that this will happen. Surely the game is not very great _unless_ there is a few months of crunch time!

vidarh 14y ago

That he "begged" them to go home is suspect. Is he not in charge? I've sent people home during crunch times before when they demonstrated they were becoming a liability even though they still wanted to keep working. You don't beg. You tell them that is how it is going to be. Leave or get escorted off the premises. > Seriously, is there another option? Yes. Send people home for some sleep and resume. Unless the people who stayed were carrying out purely menial tasks, odds are having them on the job 36 hours straight got far less work done than if they'd taken reasonable breaks. Sure, you can pull more than 8 hours now and again. Sure, you can even pull all nighters now and again. But over a 36 hour period the vast majority of peoples productivity at complex tasks will nosedive well before half way, and stay flat to even negative in jobs like software development, as you slow down other people and negatively impact communications once you get too tired. If you want the most amount of work done in 36 hours, you don't make people work the whole 36 hours.

ghshephard 14y ago
Actually, if you enter it well rested the first 36 hours (with a couple 5-10 minute catnaps) can be quite productive, with only minimal reduction in productivity. Particularly if you get into the "zone." It's after that 36 hours, that you need to get about 10-12 hours sleep to get caught up, and then have a few sane days. Before we do complex cut over onto a new software system, we usually make sure all of our software engineers are very well rested leading into a friday night, in case we've got a couple all nighters ahead of us. Anybody who tries to do back-back 36 hour death-marches with only 5-6 hours of sleep in between is setting themselves up for the seriously reduced productivity you've described. Nothing good ever seems to come of that. So - in this scenario, I think they did precisely what they should have, but they shouldn't push their luck.