9 ms·

From a user point of view it did not go nearly as smoothly as it is made to sound in this post. I would love to see a follow-up post about what went wrong along

From a user point of view it did not go nearly as smoothly as it is made to sound in this post. I would love to see a follow-up post about what went wrong along the way, and whether the hype surge might have continued longer and stronger if more people picking it up for the first time would've had a smooth experience (rather than lots and lots of errors talking to the servers).

FreakyT 10y ago
Agreed—this line in particular stood out to me as an outright fabrication: > In response, Google CRE seamlessly provisioned extra capacity on behalf of Niantic to stay well ahead of their record-setting growth. Anyone who played that game within the first week of launch would certainly not use "seamless" to describe the experience.
- tarancato 10y ago
  Make that the first month...
- itcmcgrath 10y ago
  Who said it was a capacity issue? The very next paragraph: "Not everything was smooth sailing at launch! When issues emerged around..."
- Cidan 10y ago
  There's a difference between supplying the infrastructure for a service, and writing the software for a service. Just because Google was able to supply the infrastructure for Niantic, that doesn't mean their software was designed in such a way that they could handle the load.
  - kevincox 10y ago
    Yes, an example from the article showed that Niantic was running in a Kubernetes cluster that could only scale to ~1k nodes. So even if the resources were provided they couldn't add those nodes to their cluster. You could argue that Google was providing GCE and therefore GCE couldn't scale, but GCE is really just hosted Kubernetes and it's scaling limits were known in advance. Luckily GCP was able to push a quick version update and migrate the cluster but that took considerable time, coordination and engineering effort that couldn't be done "seamlessly". This is a clear situation where the software Niantic chose couldn't handle the resources that were available. It turns out a lot of the choices you made for your worst case capacity aren't necessarily adequate for 50x that amount :) (disclaimer I work at Google but not on GCP)
    - jsolson 10y ago
      Note: GCE - Google Compute Engine. VMs in the Cloud. GKE - Google Container Engine. Hosted Kubernetes. Runs on top of GCE. (I work on GCE)
      - kevincox 10y ago
        Oops, I do that all the time. (I'm thinking container engine).
cloudjacker 10y ago
> Game traffic travels Google’s private fiber network through most of its transit, delivering reliable, low-latency experiences for players worldwide. "reliable" "low-latency"
chimeracoder 10y ago
> From a user point of view it did not go nearly as smoothly as it is made to sound in this post. I think a large part of that was due to the non-Google login from Pokemon Trainer Club (which must have been handled by Nintendo's servers, AFAIK), as opposed to the Google OAuth. Both groups of users had problems, but people who used their pre-existing, Nintendo-issued logins had much more problems (and it took a lot longer for those to get fixed). There was a period of about a week or so during which time people who used the Pokemon Trainer Club accounts were still having just as much trouble logging in and staying logged in, but people who used their Google accounts were fine. I wouldn't be surprised if providing multiple login options made it harder for them to properly separate their login servers from their game servers, and if this coupling meant that millions of users (effectively) DDoSing the PTC servers ended up impacting the game's uptime more than simply doing all authentication themselves would have.
- ge0rg 10y ago
  Not just that, playing the game (once logged in) turned out to be hardly possible at certain times of the day, and the game freezes due to communication timeouts didn't help the situation either.
- natdempk 10y ago
  There were plenty of login issues with Google accounts as well for days following launch in the US. I think you could attribute issues primarily to the way they did load-balancing though. The app assigned you a server at login time, so if something happened to that server or it got overloaded, you were out of luck until you restarted the app and logged in again. I would guess this probably caused cascading failures of servers as thousands of users got booted from one server and then all put onto the next one in line when they realized their app had frozen and logged back in. Don't forget that at launch it was common for users to have to restart their application and re-login every time they caught a pokemon. A real dynamic load-balancing scheme would probably have solved their issues. Maybe Pokemon trainer club login went down, but there were still plenty of unrelated server issues for those using Google accounts as well.
  - 10y ago
vuldin 10y ago
I think it's possible that the disconnect between how the Google Cloud team describes what happened and the reality for users can be attributed to the fact that they are just one team in control of one aspect of the entire system. From their standpoint, taking into account their responsibilities, maybe it did go as smoothly they say. A similar post from Niantic would be very interesting.
Lewisham 10y ago
Look at this chart: https://4.bp.blogspot.com/-GLc5ve5_djc/V-ysZgW6uDI/AAAAAAAADIY/-tVg2iC0qQcWBwjc0vBVBxVcJ9Yhbij6gCLcB/s1600/niantic-1.png https://4.bp.blogspot.com/-GLc5ve5_djc/V-ysZgW6uDI/AAAAAAAAD... The worst case scenario was 5x, which was a factor of 10 off. If you are that far off when you do your capacity planning, you can be pretty sure you've got problems throughout your entire stack.
- roninb 10y ago
  There is literally no way anyone could have expected it to do this well. Nintendo has steered clear of mobile gaming because they expected all of their IP to flop in the mobile world. That said, what you said is still true. If you're thinking you're going to have 100k users, you might be willing to allow a lot more data to be transmitted and/or processed than if you had 10m users. Just looking at tracker alone, having to transmit and measure distance between a dozen points and ordering them is a lot more work than checking whether a dozen coordinates are within a range and listing them in any order.
  - jandrese 10y ago
    > Nintendo has steered clear of mobile gaming because they expected all of their IP to flop in the mobile world. I don't believe this for a second. Nintendo knows where its goldmines are. IMHO it is far more likely that they steered clear of mobile because they didn't want to cannibalize their Game Boy sales. It's a classic case of a big corporation being slow to react to change or even trying to stop it because they hold a dominant position in the old system. Nintendo would much prefer if "mobile gaming" still meant Game Boys but the world has already made their decision and they can't stop it. They can't ignore the sales figures on what would have otherwise been an obscure goofy spinoff fitness app on the Game Boy. They've ignored the mobile market for too long and now there is pent up demand. Without Pokemon Go I doubt you would have seen the sudden scramble to develop and release a Mario game for iPhone.
    - merb 10y ago
      well Pokemon Go compared to the good old Pokemon Silver/Gold is just crap and that was a Gameboy Color Game.
ultramancool 10y ago
Yeah, they talked about numbers of bugs fixed, but those surely seem to have impacted the user experience. My big worry is that if I were to consider deploying to such a service, that I would encounter similar bugs, but not have the political pull to get them fixed and simply have to perform hacky fixes around them myself.
- itcmcgrath 10y ago
  If you encountered bugs that only occur because your the largest ever, I'm sure you'll have the political pull :)

FreakyT 10y ago

Agreed—this line in particular stood out to me as an outright fabrication: > In response, Google CRE seamlessly provisioned extra capacity on behalf of Niantic to stay well ahead of their record-setting growth. Anyone who played that game within the first week of launch would certainly not use "seamless" to describe the experience.

tarancato 10y ago
Make that the first month...
itcmcgrath 10y ago
Who said it was a capacity issue? The very next paragraph: "Not everything was smooth sailing at launch! When issues emerged around..."
Cidan 10y ago
There's a difference between supplying the infrastructure for a service, and writing the software for a service. Just because Google was able to supply the infrastructure for Niantic, that doesn't mean their software was designed in such a way that they could handle the load.
- kevincox 10y ago
  Yes, an example from the article showed that Niantic was running in a Kubernetes cluster that could only scale to ~1k nodes. So even if the resources were provided they couldn't add those nodes to their cluster. You could argue that Google was providing GCE and therefore GCE couldn't scale, but GCE is really just hosted Kubernetes and it's scaling limits were known in advance. Luckily GCP was able to push a quick version update and migrate the cluster but that took considerable time, coordination and engineering effort that couldn't be done "seamlessly". This is a clear situation where the software Niantic chose couldn't handle the resources that were available. It turns out a lot of the choices you made for your worst case capacity aren't necessarily adequate for 50x that amount :) (disclaimer I work at Google but not on GCP)
  - jsolson 10y ago
    Note: GCE - Google Compute Engine. VMs in the Cloud. GKE - Google Container Engine. Hosted Kubernetes. Runs on top of GCE. (I work on GCE)
    - kevincox 10y ago
      Oops, I do that all the time. (I'm thinking container engine).

cloudjacker 10y ago

> Game traffic travels Google’s private fiber network through most of its transit, delivering reliable, low-latency experiences for players worldwide. "reliable" "low-latency"

chimeracoder 10y ago

> From a user point of view it did not go nearly as smoothly as it is made to sound in this post. I think a large part of that was due to the non-Google login from Pokemon Trainer Club (which must have been handled by Nintendo's servers, AFAIK), as opposed to the Google OAuth. Both groups of users had problems, but people who used their pre-existing, Nintendo-issued logins had much more problems (and it took a lot longer for those to get fixed). There was a period of about a week or so during which time people who used the Pokemon Trainer Club accounts were still having just as much trouble logging in and staying logged in, but people who used their Google accounts were fine. I wouldn't be surprised if providing multiple login options made it harder for them to properly separate their login servers from their game servers, and if this coupling meant that millions of users (effectively) DDoSing the PTC servers ended up impacting the game's uptime more than simply doing all authentication themselves would have.

ge0rg 10y ago
Not just that, playing the game (once logged in) turned out to be hardly possible at certain times of the day, and the game freezes due to communication timeouts didn't help the situation either.
natdempk 10y ago
There were plenty of login issues with Google accounts as well for days following launch in the US. I think you could attribute issues primarily to the way they did load-balancing though. The app assigned you a server at login time, so if something happened to that server or it got overloaded, you were out of luck until you restarted the app and logged in again. I would guess this probably caused cascading failures of servers as thousands of users got booted from one server and then all put onto the next one in line when they realized their app had frozen and logged back in. Don't forget that at launch it was common for users to have to restart their application and re-login every time they caught a pokemon. A real dynamic load-balancing scheme would probably have solved their issues. Maybe Pokemon trainer club login went down, but there were still plenty of unrelated server issues for those using Google accounts as well.
- 10y ago

vuldin 10y ago

I think it's possible that the disconnect between how the Google Cloud team describes what happened and the reality for users can be attributed to the fact that they are just one team in control of one aspect of the entire system. From their standpoint, taking into account their responsibilities, maybe it did go as smoothly they say. A similar post from Niantic would be very interesting.

Lewisham 10y ago

Look at this chart: https://4.bp.blogspot.com/-GLc5ve5_djc/V-ysZgW6uDI/AAAAAAAADIY/-tVg2iC0qQcWBwjc0vBVBxVcJ9Yhbij6gCLcB/s1600/niantic-1.png https://4.bp.blogspot.com/-GLc5ve5_djc/V-ysZgW6uDI/AAAAAAAAD... The worst case scenario was 5x, which was a factor of 10 off. If you are that far off when you do your capacity planning, you can be pretty sure you've got problems throughout your entire stack.

roninb 10y ago
There is literally no way anyone could have expected it to do this well. Nintendo has steered clear of mobile gaming because they expected all of their IP to flop in the mobile world. That said, what you said is still true. If you're thinking you're going to have 100k users, you might be willing to allow a lot more data to be transmitted and/or processed than if you had 10m users. Just looking at tracker alone, having to transmit and measure distance between a dozen points and ordering them is a lot more work than checking whether a dozen coordinates are within a range and listing them in any order.
- jandrese 10y ago
  > Nintendo has steered clear of mobile gaming because they expected all of their IP to flop in the mobile world. I don't believe this for a second. Nintendo knows where its goldmines are. IMHO it is far more likely that they steered clear of mobile because they didn't want to cannibalize their Game Boy sales. It's a classic case of a big corporation being slow to react to change or even trying to stop it because they hold a dominant position in the old system. Nintendo would much prefer if "mobile gaming" still meant Game Boys but the world has already made their decision and they can't stop it. They can't ignore the sales figures on what would have otherwise been an obscure goofy spinoff fitness app on the Game Boy. They've ignored the mobile market for too long and now there is pent up demand. Without Pokemon Go I doubt you would have seen the sudden scramble to develop and release a Mario game for iPhone.
  - merb 10y ago
    well Pokemon Go compared to the good old Pokemon Silver/Gold is just crap and that was a Gameboy Color Game.

ultramancool 10y ago

Yeah, they talked about numbers of bugs fixed, but those surely seem to have impacted the user experience. My big worry is that if I were to consider deploying to such a service, that I would encounter similar bugs, but not have the political pull to get them fixed and simply have to perform hacky fixes around them myself.

itcmcgrath 10y ago
If you encountered bugs that only occur because your the largest ever, I'm sure you'll have the political pull :)