5 ms·

> Not everything was smooth sailing at launch! When issues emerged around the game’s stability, Niantic and Google engineers braved each problem in sequence, wo

> Not everything was smooth sailing at launch! When issues emerged around the game’s stability, Niantic and Google engineers braved each problem in sequence, working quickly to create and deploy solutions. Google CRE worked hand-in-hand with Niantic to review every part of their architecture, tapping the expertise of core Google Cloud engineers and product managers — all against a backdrop of millions of new players pouring into the game. IMO This is the most valuable thing in this article. It essentially says what others are pointing out. You can't just press a button and have scale. It's not that easy. You have to tackle many layers. Considering they 50x'd their worst case scenario it would have only taken a few bad queries to fuck shit up.

kevan 10y ago
At 50x designed max load bad queries would be one of the easier problems to solve. We design systems all the time that have hard scaling limits, and it's only a problem when you operate the system past those limits. e.g. I wrote a service with ACID assumptions from the underlying database but now the largest ACID db box I can buy isn't big enough. Oops. There's a bunch of possible ways around that but they usually involve a nontrivial amount of engineering effort. I wouldn't expect any service to survive that much unplanned load. Maybe they could've estimated better, but how likely was it for the game to go viral? Worth sinking lots of dev time into and delaying launch? That's a hard question to answer without the benefit of hindsight.
- sp527 10y ago
  Honestly I was surprised they were able to scale as elastically as they did and we now know the reason why: they used Datastore instead of something like Postgres with sharding. That decision alone is most of the reason they made it. By contrast, my team doesn't use managed NoSQL services for our projects because they're simply too expensive to deploy before seeing considerable growth.
  - jshen 10y ago
    Datastore seems really cheap to me, am I missing something?
  - viraptor 10y ago
    They also have a system that's almost trivial to shard in any way they want, which would help even with postgres all around. Since there's no real interaction between users (only user-environment really), they can split databases by location, user groups, or some hashes as they want. Also the data they deal with is extremely cachable - as long as you play on one device.
    - itcmcgrath 10y ago
      There is some, when users battle gyms they are battling with other users Pokemon which would be from anywhere in the world. When the Pokemon on the Gym loses it needs to be removed in a transaction with an update on the gym. This takes the easiest form of sharding off the table, so it's not as simple as you make it seem.
      - ec109685 10y ago
        It doesn't really need to be a transaction since it is okay if it is eventually consistent (e.g. they have a cleanup job if a transaction is lost), that would be fine.
- late2part 10y ago
  This: "but now the largest ACID db box I can buy isn't big enough." The hard part about scaling is knowing when to invest in scale. The binomial choice is to build and throw away, or build so as to be scalable. So hard to predict.
user5994461 10y ago
> You can't just press a button and have scale Actually you can... if the system was designed for this purpose.
- bottled_poe 10y ago
  Technically true, but it doesn't make sense from a business perspective. Anticipating and accounting for every scaling issue costs a lot of money. Most businesses would decide that satisfying 90% of scenarios is sufficient.
  - user5994461 10y ago
    Business is irrelevant once you're sure that the service will go into production AND will have some amount of users. It's known that some things will break from time to time (e.g. servers, hard drives, networking). If you've designed the application poorly so that a single minor failure can snowball to the entire system, it's guaranteed that your site will fall apart periodically, costing you lots of money each time. At this point, the best technical AND business converge to "pick an option that can sustain [minor] breakages". Either you've done it like that from the start, or you're in the process of rewriting/redesigning your site.
    - bottled_poe 10y ago
      > Business is irrelevant once... Not really. In the case described they had accounted for some maximum number of users and instead got 50x that number. It is reasonable to design a system which handles just some planned number as long as there is a strategy in place to manage a situation beyond those limits. In this case the strategy was to apply engineering effort. That seems like a valid risk management plan to me.
- Arnt 10y ago
  All you need to do is follow the waterfall development process properly! Think hard, establish a perfect design in enough meetings, write a flawless spec, then implement it correctly and the button will work first time! Rrrrrright?
riquito 10y ago
> Considering they 50x'd their worst case scenario Nitpick: they 10x'd their worst case scenario

kevan 10y ago

At 50x designed max load bad queries would be one of the easier problems to solve. We design systems all the time that have hard scaling limits, and it's only a problem when you operate the system past those limits. e.g. I wrote a service with ACID assumptions from the underlying database but now the largest ACID db box I can buy isn't big enough. Oops. There's a bunch of possible ways around that but they usually involve a nontrivial amount of engineering effort. I wouldn't expect any service to survive that much unplanned load. Maybe they could've estimated better, but how likely was it for the game to go viral? Worth sinking lots of dev time into and delaying launch? That's a hard question to answer without the benefit of hindsight.

sp527 10y ago
Honestly I was surprised they were able to scale as elastically as they did and we now know the reason why: they used Datastore instead of something like Postgres with sharding. That decision alone is most of the reason they made it. By contrast, my team doesn't use managed NoSQL services for our projects because they're simply too expensive to deploy before seeing considerable growth.
- jshen 10y ago
  Datastore seems really cheap to me, am I missing something?
- viraptor 10y ago
  They also have a system that's almost trivial to shard in any way they want, which would help even with postgres all around. Since there's no real interaction between users (only user-environment really), they can split databases by location, user groups, or some hashes as they want. Also the data they deal with is extremely cachable - as long as you play on one device.
  - itcmcgrath 10y ago
    There is some, when users battle gyms they are battling with other users Pokemon which would be from anywhere in the world. When the Pokemon on the Gym loses it needs to be removed in a transaction with an update on the gym. This takes the easiest form of sharding off the table, so it's not as simple as you make it seem.
    - ec109685 10y ago
      It doesn't really need to be a transaction since it is okay if it is eventually consistent (e.g. they have a cleanup job if a transaction is lost), that would be fine.
late2part 10y ago
This: "but now the largest ACID db box I can buy isn't big enough." The hard part about scaling is knowing when to invest in scale. The binomial choice is to build and throw away, or build so as to be scalable. So hard to predict.

user5994461 10y ago

> You can't just press a button and have scale Actually you can... if the system was designed for this purpose.

bottled_poe 10y ago
Technically true, but it doesn't make sense from a business perspective. Anticipating and accounting for every scaling issue costs a lot of money. Most businesses would decide that satisfying 90% of scenarios is sufficient.
- user5994461 10y ago
  Business is irrelevant once you're sure that the service will go into production AND will have some amount of users. It's known that some things will break from time to time (e.g. servers, hard drives, networking). If you've designed the application poorly so that a single minor failure can snowball to the entire system, it's guaranteed that your site will fall apart periodically, costing you lots of money each time. At this point, the best technical AND business converge to "pick an option that can sustain [minor] breakages". Either you've done it like that from the start, or you're in the process of rewriting/redesigning your site.
  - bottled_poe 10y ago
    > Business is irrelevant once... Not really. In the case described they had accounted for some maximum number of users and instead got 50x that number. It is reasonable to design a system which handles just some planned number as long as there is a strategy in place to manage a situation beyond those limits. In this case the strategy was to apply engineering effort. That seems like a valid risk management plan to me.
Arnt 10y ago
All you need to do is follow the waterfall development process properly! Think hard, establish a perfect design in enough meetings, write a flawless spec, then implement it correctly and the button will work first time! Rrrrrright?

riquito 10y ago

> Considering they 50x'd their worst case scenario Nitpick: they 10x'd their worst case scenario