12 ms·

A bunch of the comments are already pointing out the launch issues Pokemon Go had, and it's well known that a rep from AWS was also throwing jabs at them during

by jcastro 10y ago

A bunch of the comments are already pointing out the launch issues Pokemon Go had, and it's well known that a rep from AWS was also throwing jabs at them during launch for their issues. It would be naive for everyone to assume that a high traffic launch is all about the cloud underneath and only that. The article didn't mention any of the technical details of the Pokemon application itself, for all we know the infrastructure was humming nicely and the application itself didn't scale. Or the other way around, or a combination of both or one of the other of thousands of moving pieces it takes to launch something.

deanCommie 10y ago
It was actually the CTO of AWS, Werner Vogels. :)
- Someone1234 10y ago
  Here is an article about it: http://www.businessinsider.com/amazon-cto-makes-dig-at-pokemon-go-crash-2016-7 http://www.businessinsider.com/amazon-cto-makes-dig-at-pokem...
- toomuchtodo 10y ago
  https://twitter.com/werner/status/751511267702087680 https://twitter.com/werner/status/751511267702087680 https://www.reddit.com/r/pokemongo/comments/4rwrhf/amazons_chief_technology_officer_just_tweeted/ https://www.reddit.com/r/pokemongo/comments/4rwrhf/amazons_c... Werner actually posted as himself in that Reddit thread ^ "Guys, this was not intended as a snark, joke, rip or self-promotion. I just want people (including myself) to play given how much fun it is. We have some experience in running big software systems that need to scale and just want to help. Update for the skeptics: yes I am @werner, not Amazon PR. I have the same username on Instagram." EDIT: Unless you can prove the snark or malicious intent was there, to say otherwise is disingenuous.
  - brazzledazzle 10y ago
    Seems a bit like back peddling. He knows that Google has experience running big software systems that need to scale as well. We all know it. He also knows Alphabet is a part-owner of Niantic.
  - dekhn 10y ago
    There was no reason to tweet it publicly. After all, he can talk to Hanke any time he wants. Tweeting it like that was snark.
  - pbarnes_1 10y ago
    Except that... I'm pretty sure Google knows how to run big software systems... This is called back-peddling.
    - toomuchtodo 10y ago
      They have downtime/issues just like everyone else [1]. Shit is complicated. Shit breaks. No one is infallible. [1] https://status.cloud.google.com/summary https://status.cloud.google.com/summary
  - patrec 10y ago
    Yup, it's all about Werner just wanting to play Pokemon Go in a purely private capacity, and he naturally expects his company to help him with that. Nothing at all to do with PR or business.
spacehunt 10y ago
> it's well known that a rep from AWS was also throwing jabs at them during launch for their issues That's pretty low. FWIW Simcity 2013 is on AWS and the launch was far more disastrous. Doesn't prove anything.
- mpdehaan2 10y ago
  Programming architecture/problems in SC, yes. I think Pokemon scaled remarkably better than any app many of us have ever dealt with, seeing the crazy demand spikes. You would run into a lot of app issues, data choices, a lot of things that don't just get solved from simple uses of autoscaling.
  - surge 10y ago
    It scaled better than Twitter, it hit the same active user count (or close to it) in a month that took Twitter years to reach, and Twitter would sh*t the bed all the time with the infamous "fail whale" popping up for users constantly even with its steady increase in growth over time and time to adapt. Twitter was pioneering at reaching that scale, that many now use as guide/lessons learned now, so its forgiven, but how many companies see that much growth, that fast?
    - hamandcheese 10y ago
      And for just a little extra context, Twitter launched the same year as AWS.
    - rryan 10y ago
      To be fair -- Twitter is a much harder architectural problem (significant fan-out and fan-in problems to solve). Pokemon Go is less complicated in terms of the conceptual model. There are only a few pretty simple ways that different players can interact (gyms, lures on pokestops, that's about it).
      - chickenbane 10y ago
        On the other hand, Twitter is dealing with strings < 140 character. Pokemon Go is tracking all the trainers and Pokemon at all times, not to mention the gyms and lures.
0xdeadbeefbabe 10y ago
> The article didn't mention any of the technical details Like contending with Team Rocket
xorgar831 10y ago
Load testing is really hard, it would be interesting to see more research in this area, such as tooling and design patterns that flag bottlenecks that could be outages at larger scales.
- kevincox 10y ago
  Load testing wasn't really relevant here anyways. Even if they load tested their worst case it wouldn't have told them much about 50x that.
empath75 10y ago
the scaled from nothing to the size of facebook almost in a few weeks... i'd say it's pretty impressive even with the speedbumps.
- stepanhruda 10y ago
  Not from nothing, they had years of experience and infrastructure from Ingress.
  - verst 10y ago
    Ingress ran on App Engine. I was one of several people working with Niantic at the time to address application architecture issues to avoid App Engine anti-patterns.
    - julianpye 10y ago
      Interesting, can you share why it would not make sense to use App Engine here (or why Ingress ran into anti-patterns)?
      - ben_jones 10y ago
        Not GP but I recently assessed GAE for my company and ultimately decided to go with GKE instead. The reason was GAE had no support for websockets [1] and had a weird way of influencing almost all implementation details in your code base. It pushes towards full adoption of GCP's other products like CloudSQL, Memcache, Cloud Datastore, etc, because those products run so bloodly performantly you feel like a poor engineer not taking advantage of them. In short those "negatives" are actually perfectly reasonable when you consider what GAE was designed to be: A Google managed PaaS that handles all the hard parts for you IFF you follow the platforms conventions. The problem is that others see it as a drop-in for EC2 and get burned as they throw away their previous tool chains like Postgres, Redis, and the entire networking stack. [1]: http://stackoverflow.com/questions/4114902/how-can-i-run-a-websocket-server-on-top-of-gae http://stackoverflow.com/questions/4114902/how-can-i-run-a-w...
CorvusCrypto 10y ago
Upon first read I actually had the same thoughts as many and applied them to the google services as well as the application itself. However on reflection, yeah the services google provided were pretty impressive, especially after re-reading the following: "Google CRE worked hand-in-hand with Niantic to review every part of their architecture, tapping the expertise of core Google Cloud engineers and product managers" Architecture analysis may or may not be standard (I don't actually know since I've never had to deal with something like this) but that sounds great to me.
- kyrra 10y ago
  https://cloud.google.com/support/ https://cloud.google.com/support/ Gold and Platinum support can help with architectural analysis. I'm also guessing Google had a vested interest to help Pokemon Go do an analysis of their architecture. Doing so could help find any inefficiencies to lighten the load on their cloud servers.
- CobrastanJorji 10y ago
  One of the nice things about cloud in general is that it's self-serve for small customers (enter credit card info, a few seconds later you have some computers in the cloud), but as you scale up and start paying millions of dollars per month and sign up for the "Platinum" or "Enterprise" or "Premier" plans, software engineers start showing up at your door to help you design, launch, and fix your service. I've seen this with Google and Amazon, and I imagine it's probably also true for the other providers.
tarancato 10y ago
I remember the server application being written in Java. At least, it was served using Jetty. curl -i https://pgorelease.nianticlabs.com/plfe/531/rpc https://pgorelease.nianticlabs.com/plfe/531/rpc
balls187 10y ago
At the same time, Niantic was spun out of Google, so their choice of using Google Cloud is highly dubious.
- golfer 10y ago
  Why is it highly dubious?
  - balls187 10y ago
    By not mentioning that Niantic is made up of Googlers, and spun out of an internal project, the post makes it seem that Niantic organically and independently choose Google Cloud from among the top P/IaaS vendors. The omission of that fact may cause the reader to infer that Google Cloud Platform was chosen for it's technical merits, and not because the company was actually a Google team that spun out.
    - oldmanjay 10y ago
      Anyone responsible for meaningful technical decisions who relies on inference is getting what they've earned for themselves.
      - balls187 10y ago
        While that may be true, it doesn't make the article's claim any less disingenuous.
    - honkhonkpants 10y ago
      The niantic people basically came to Google and left together. The interchange was minimal.
  - dismantlethesun 10y ago
    In theory, the idea that it was a true 'choice' is suspect. They were a Google company previously, so they'd just use Google by default even if everyone on staff also was equally expert at AWS. That said... in reality, they likely had real reasons to use Google since they probably have a higher expertise with it that AWS.

by jcastro 10y ago

deanCommie 10y ago

It was actually the CTO of AWS, Werner Vogels. :)

Someone1234 10y ago
Here is an article about it: http://www.businessinsider.com/amazon-cto-makes-dig-at-pokemon-go-crash-2016-7 http://www.businessinsider.com/amazon-cto-makes-dig-at-pokem...
toomuchtodo 10y ago
https://twitter.com/werner/status/751511267702087680 https://twitter.com/werner/status/751511267702087680 https://www.reddit.com/r/pokemongo/comments/4rwrhf/amazons_chief_technology_officer_just_tweeted/ https://www.reddit.com/r/pokemongo/comments/4rwrhf/amazons_c... Werner actually posted as himself in that Reddit thread ^ "Guys, this was not intended as a snark, joke, rip or self-promotion. I just want people (including myself) to play given how much fun it is. We have some experience in running big software systems that need to scale and just want to help. Update for the skeptics: yes I am @werner, not Amazon PR. I have the same username on Instagram." EDIT: Unless you can prove the snark or malicious intent was there, to say otherwise is disingenuous.
- brazzledazzle 10y ago
  Seems a bit like back peddling. He knows that Google has experience running big software systems that need to scale as well. We all know it. He also knows Alphabet is a part-owner of Niantic.
- dekhn 10y ago
  There was no reason to tweet it publicly. After all, he can talk to Hanke any time he wants. Tweeting it like that was snark.
- pbarnes_1 10y ago
  Except that... I'm pretty sure Google knows how to run big software systems... This is called back-peddling.
  - toomuchtodo 10y ago
    They have downtime/issues just like everyone else [1]. Shit is complicated. Shit breaks. No one is infallible. [1] https://status.cloud.google.com/summary https://status.cloud.google.com/summary
- patrec 10y ago
  Yup, it's all about Werner just wanting to play Pokemon Go in a purely private capacity, and he naturally expects his company to help him with that. Nothing at all to do with PR or business.

spacehunt 10y ago

> it's well known that a rep from AWS was also throwing jabs at them during launch for their issues That's pretty low. FWIW Simcity 2013 is on AWS and the launch was far more disastrous. Doesn't prove anything.

mpdehaan2 10y ago
Programming architecture/problems in SC, yes. I think Pokemon scaled remarkably better than any app many of us have ever dealt with, seeing the crazy demand spikes. You would run into a lot of app issues, data choices, a lot of things that don't just get solved from simple uses of autoscaling.
- surge 10y ago
  It scaled better than Twitter, it hit the same active user count (or close to it) in a month that took Twitter years to reach, and Twitter would sh*t the bed all the time with the infamous "fail whale" popping up for users constantly even with its steady increase in growth over time and time to adapt. Twitter was pioneering at reaching that scale, that many now use as guide/lessons learned now, so its forgiven, but how many companies see that much growth, that fast?
  - hamandcheese 10y ago
    And for just a little extra context, Twitter launched the same year as AWS.
  - rryan 10y ago
    To be fair -- Twitter is a much harder architectural problem (significant fan-out and fan-in problems to solve). Pokemon Go is less complicated in terms of the conceptual model. There are only a few pretty simple ways that different players can interact (gyms, lures on pokestops, that's about it).
    - chickenbane 10y ago
      On the other hand, Twitter is dealing with strings < 140 character. Pokemon Go is tracking all the trainers and Pokemon at all times, not to mention the gyms and lures.

0xdeadbeefbabe 10y ago

> The article didn't mention any of the technical details Like contending with Team Rocket

xorgar831 10y ago

Load testing is really hard, it would be interesting to see more research in this area, such as tooling and design patterns that flag bottlenecks that could be outages at larger scales.

kevincox 10y ago
Load testing wasn't really relevant here anyways. Even if they load tested their worst case it wouldn't have told them much about 50x that.

empath75 10y ago

the scaled from nothing to the size of facebook almost in a few weeks... i'd say it's pretty impressive even with the speedbumps.

stepanhruda 10y ago
Not from nothing, they had years of experience and infrastructure from Ingress.
- verst 10y ago
  Ingress ran on App Engine. I was one of several people working with Niantic at the time to address application architecture issues to avoid App Engine anti-patterns.
  - julianpye 10y ago
    Interesting, can you share why it would not make sense to use App Engine here (or why Ingress ran into anti-patterns)?
    - ben_jones 10y ago
      Not GP but I recently assessed GAE for my company and ultimately decided to go with GKE instead. The reason was GAE had no support for websockets [1] and had a weird way of influencing almost all implementation details in your code base. It pushes towards full adoption of GCP's other products like CloudSQL, Memcache, Cloud Datastore, etc, because those products run so bloodly performantly you feel like a poor engineer not taking advantage of them. In short those "negatives" are actually perfectly reasonable when you consider what GAE was designed to be: A Google managed PaaS that handles all the hard parts for you IFF you follow the platforms conventions. The problem is that others see it as a drop-in for EC2 and get burned as they throw away their previous tool chains like Postgres, Redis, and the entire networking stack. [1]: http://stackoverflow.com/questions/4114902/how-can-i-run-a-websocket-server-on-top-of-gae http://stackoverflow.com/questions/4114902/how-can-i-run-a-w...

CorvusCrypto 10y ago

Upon first read I actually had the same thoughts as many and applied them to the google services as well as the application itself. However on reflection, yeah the services google provided were pretty impressive, especially after re-reading the following: "Google CRE worked hand-in-hand with Niantic to review every part of their architecture, tapping the expertise of core Google Cloud engineers and product managers" Architecture analysis may or may not be standard (I don't actually know since I've never had to deal with something like this) but that sounds great to me.

kyrra 10y ago
https://cloud.google.com/support/ https://cloud.google.com/support/ Gold and Platinum support can help with architectural analysis. I'm also guessing Google had a vested interest to help Pokemon Go do an analysis of their architecture. Doing so could help find any inefficiencies to lighten the load on their cloud servers.
CobrastanJorji 10y ago
One of the nice things about cloud in general is that it's self-serve for small customers (enter credit card info, a few seconds later you have some computers in the cloud), but as you scale up and start paying millions of dollars per month and sign up for the "Platinum" or "Enterprise" or "Premier" plans, software engineers start showing up at your door to help you design, launch, and fix your service. I've seen this with Google and Amazon, and I imagine it's probably also true for the other providers.

tarancato 10y ago

I remember the server application being written in Java. At least, it was served using Jetty. curl -i https://pgorelease.nianticlabs.com/plfe/531/rpc https://pgorelease.nianticlabs.com/plfe/531/rpc

balls187 10y ago

At the same time, Niantic was spun out of Google, so their choice of using Google Cloud is highly dubious.

golfer 10y ago
Why is it highly dubious?
- balls187 10y ago
  By not mentioning that Niantic is made up of Googlers, and spun out of an internal project, the post makes it seem that Niantic organically and independently choose Google Cloud from among the top P/IaaS vendors. The omission of that fact may cause the reader to infer that Google Cloud Platform was chosen for it's technical merits, and not because the company was actually a Google team that spun out.
  - oldmanjay 10y ago
    Anyone responsible for meaningful technical decisions who relies on inference is getting what they've earned for themselves.
    - balls187 10y ago
      While that may be true, it doesn't make the article's claim any less disingenuous.
  - honkhonkpants 10y ago
    The niantic people basically came to Google and left together. The interchange was minimal.
- dismantlethesun 10y ago
  In theory, the idea that it was a true 'choice' is suspect. They were a Google company previously, so they'd just use Google by default even if everyone on staff also was equally expert at AWS. That said... in reality, they likely had real reasons to use Google since they probably have a higher expertise with it that AWS.