5 ms·

Assuming massive success is a stupid move. The majority of the time you are not that successful. The ground is littered by projects that spent vast sums of mone

Assuming massive success is a stupid move. The majority of the time you are not that successful. The ground is littered by projects that spent vast sums of money (and more importantly time) designing for load they never needed to handle before release. In about a month my company will be releasing an online game. We think we will get about 15-20k concurrent users in the first few days, but we will be testing the realm for as high as 50k concurrent users using load testing bots. What will happen if we suddenly get 100k or 200k concurrent users? Even throwing more hardware at the problem might not solve it immediately if there is some new bottleneck we haven't found before. Testing the realm for more than 50k concurrents and then spending engineering effort required to make it work will consume time and effort that we can't afford to spend. All for an outcome that is unlikely to occur.

fatbird 14y ago
Assuming massive success is stupid, but testing for it seems prudent. Given how easy it is to spin up EC2 instances generate bot-load, even just to see what'll happen or to find the collapse point, seems so straightforward as to be routine. And as the commenter below observes, the Curiosity interaction model requires high server load for all interaction.
- btilly 14y ago
  The problem with your theory is that there are lies, damned lies, and load tests. Any sensible app is going to have caching layers. A simple load test is going to just be a really good exercise of the caching layers. By contrast a complex load test with lots of calls to do really random stuff is going to stress you out in entirely unreasonable ways. And finding a realistic balance of traffic is difficult unless you've simply sampled real traffic - which you can't do until after you've generated some.
  - bowyakka 14y ago
    This! I cant agree with this more, unless you approach me with an experiment that would put most science to shame I do not value your use of Jmeter/Tsung/Siege/Rando-Python-Script. If you test plan does not have pauses in it, and you are not engineering a nuclear reactor I probably wont even start the conversation. .... At $DAYJOB, when we were putting together a new search engine we spent some time upfront putting in a lot of tooling to worry about and test performance every step of the way. Some of this tooling was built around jmeter and focused on generating realistic test plans (which we could derive from the already operational existing search engine). It was a neat infrastructure it tracked everything by commit, captured all manner of metrics and used huge datasets, mined from the logs, built out with markov models. Everything we reported into an install of the software that runs http://speed.pypy.org http://speed.pypy.org, we thought we had it conquered. Yet it was so flawed! Data would get stale, models would break, caches invalidated datasets and much much more. It was funny at the same time I wrote a tool that we now know as darklaunch, this tool is stupid simple, it captures incoming traffic to the production system with pcap, and then (after a little mussing with it) sends it through to a always on test cluster. Yet its this tool that produces all the load, has all the weird user behavior, never gets out of sync and can run load, soak, endurance and impact tests. Guess which one we use to test out new software, or new features, or to tune a garbage collector in as close to production as possible ? I have not started Jmeter now in over a year, its been no great loss; sure we have microbenchmarks where they matter but for everything else you often hear the words "darklaunch it" Now for sure we have the luxury of having real traffic to use and abuse, but if you don't have real traffic then realise you are using a scrying device anyhow and I will bet you good money your assumptions are wrong.
  - reidrac 14y ago
    I totally agree. There's nothing like reality and customers doing absurd things to break stuff you'd never imagined that could break.
- Negitivefrags 14y ago
  We will throw bots at the servers until they fall over, but at any point after 50k concurrents we will not devote engineering time to fixing it.
anigbrowl 14y ago
Test for the big numbers. With server virtualization you don't have to buy the hardware ahead of time so there's no reason not to do so. Even if it doesn't work at more than 50k currents, you may as well find out whre the failure points are ahead of time rather than when your getting squeezed up against them.
biot 14y ago
As long as you've put in enough consideration to avoid painting yourself into a corner with your system architecture so that you know how you'd scale the system should the need arise, you don't need to invest the engineering time up-front to actually build it. However, if you don't understand how your application is going to scale then you might be totally screwed by bad architectural decisions. For example, you might assume that the data for all concurrent users fits within RAM on a single server. It's one thing to plan for an 8GB server for 50K concurrent load and then have to scale the RAM to 16GB, 32GB, 64GB, 128GB, and so on. But if your RAM usage is O(n^2) for n concurrent users and both your concurrent user count and the average data stored per user increases such that you need hundreds of terabytes of RAM on a single server in order to continue to function, maybe putting in a modicum of up-front work would have paid off.
- Negitivefrags 14y ago
  I'm not in any way saying that you shouldn't plan for the future. We certainly have designed everything wherever possible not to get in the way of future scaling. It's just the problems you can't predict are what really get you. For example, in theory the first "designed in" limitation that our architecture has should kick in around 1.5 million concurrents. There is a service in the backend that, due to the way it is designed, could never possibly handle more concurrents than that without redesigning some pretty core internals. In reality though, when we get to 2 orders of magnitude less concurrents then that the realm starts to develop issues that we are currently resolving. So yes, don't design things that can't scale, but don't expect your designs to actually scale without serious engineering effort.

fatbird 14y ago

Assuming massive success is stupid, but testing for it seems prudent. Given how easy it is to spin up EC2 instances generate bot-load, even just to see what'll happen or to find the collapse point, seems so straightforward as to be routine. And as the commenter below observes, the Curiosity interaction model requires high server load for all interaction.

btilly 14y ago
The problem with your theory is that there are lies, damned lies, and load tests. Any sensible app is going to have caching layers. A simple load test is going to just be a really good exercise of the caching layers. By contrast a complex load test with lots of calls to do really random stuff is going to stress you out in entirely unreasonable ways. And finding a realistic balance of traffic is difficult unless you've simply sampled real traffic - which you can't do until after you've generated some.
- bowyakka 14y ago
  This! I cant agree with this more, unless you approach me with an experiment that would put most science to shame I do not value your use of Jmeter/Tsung/Siege/Rando-Python-Script. If you test plan does not have pauses in it, and you are not engineering a nuclear reactor I probably wont even start the conversation. .... At $DAYJOB, when we were putting together a new search engine we spent some time upfront putting in a lot of tooling to worry about and test performance every step of the way. Some of this tooling was built around jmeter and focused on generating realistic test plans (which we could derive from the already operational existing search engine). It was a neat infrastructure it tracked everything by commit, captured all manner of metrics and used huge datasets, mined from the logs, built out with markov models. Everything we reported into an install of the software that runs http://speed.pypy.org http://speed.pypy.org, we thought we had it conquered. Yet it was so flawed! Data would get stale, models would break, caches invalidated datasets and much much more. It was funny at the same time I wrote a tool that we now know as darklaunch, this tool is stupid simple, it captures incoming traffic to the production system with pcap, and then (after a little mussing with it) sends it through to a always on test cluster. Yet its this tool that produces all the load, has all the weird user behavior, never gets out of sync and can run load, soak, endurance and impact tests. Guess which one we use to test out new software, or new features, or to tune a garbage collector in as close to production as possible ? I have not started Jmeter now in over a year, its been no great loss; sure we have microbenchmarks where they matter but for everything else you often hear the words "darklaunch it" Now for sure we have the luxury of having real traffic to use and abuse, but if you don't have real traffic then realise you are using a scrying device anyhow and I will bet you good money your assumptions are wrong.
- reidrac 14y ago
  I totally agree. There's nothing like reality and customers doing absurd things to break stuff you'd never imagined that could break.
Negitivefrags 14y ago
We will throw bots at the servers until they fall over, but at any point after 50k concurrents we will not devote engineering time to fixing it.

anigbrowl 14y ago

Test for the big numbers. With server virtualization you don't have to buy the hardware ahead of time so there's no reason not to do so. Even if it doesn't work at more than 50k currents, you may as well find out whre the failure points are ahead of time rather than when your getting squeezed up against them.

biot 14y ago

As long as you've put in enough consideration to avoid painting yourself into a corner with your system architecture so that you know how you'd scale the system should the need arise, you don't need to invest the engineering time up-front to actually build it. However, if you don't understand how your application is going to scale then you might be totally screwed by bad architectural decisions. For example, you might assume that the data for all concurrent users fits within RAM on a single server. It's one thing to plan for an 8GB server for 50K concurrent load and then have to scale the RAM to 16GB, 32GB, 64GB, 128GB, and so on. But if your RAM usage is O(n^2) for n concurrent users and both your concurrent user count and the average data stored per user increases such that you need hundreds of terabytes of RAM on a single server in order to continue to function, maybe putting in a modicum of up-front work would have paid off.

Negitivefrags 14y ago
I'm not in any way saying that you shouldn't plan for the future. We certainly have designed everything wherever possible not to get in the way of future scaling. It's just the problems you can't predict are what really get you. For example, in theory the first "designed in" limitation that our architecture has should kick in around 1.5 million concurrents. There is a service in the backend that, due to the way it is designed, could never possibly handle more concurrents than that without redesigning some pretty core internals. In reality though, when we get to 2 orders of magnitude less concurrents then that the realm starts to develop issues that we are currently resolving. So yes, don't design things that can't scale, but don't expect your designs to actually scale without serious engineering effort.