7 ms·

Update on Maintaining and Running the Pokémon GO Service

63 points by pratheekrebala 10y ago

pingec 10y ago
They banned third party clients from accessing their API. From the chart it looks like third party API clients were generating two thirds of total traffic. That is a lot. On the other hand, this has sparkled a huge community effort to reverse, understand and bypass the implemented protection/fingerprinting. https://www.reddit.com/r/pokemongodev/comments/4w1cvr/pokemongo_current_api_status/ https://www.reddit.com/r/pokemongodev/comments/4w1cvr/pokemo...
- elthran 10y ago
  I'd be really interesting the see that graph extended to to left - what was the traffic like before the 3 paws "bug" was first introduced. I definitely feel that a significant amount of the drive to use the 3rd party services was due to the fact that you couldn't realistically hunt down any of the pokemon that show up on your radar.
  - agar 10y ago
    Yes, the lack of a functional Nearby feature - either integrated or with a third party app - really affects motivation while playing. I was out walking today and saw an unknown Pokemon silhouette in my Nearby list. All I could do was shrug and hope I'd run into it (I didn't). Previously I would have gone significantly out of my way to find it, adding mileage to my walk - which I understood to be one of the design objectives of the game.
    - xorcist 10y ago
      Not only must the three steps have been buggy, the whole "nearby" list is useless and/or wrong. Two friends walking together, with the same exact GPS position on the maps, can have vastly different nearby-lists, sometimes without even one entry in common. How is that even possible?
      - aleksei 10y ago
        It could be that, since the nearby listing holds only 9 entries, that there are more Pokemon nearby and the query to display them is unordered.
  - curiousgal 10y ago
    http://i.imgur.com/RLc4i9V.png http://i.imgur.com/RLc4i9V.png
- Nanite 10y ago
  That's assuming the the y-axis starts at zero, it also lacks numbers for the "spatial queries". This graph doesn't actually tell us anything, besides that there was a drop in traffic after blocking third party access.
- masklinn 10y ago
  > From the chart it looks like third party API clients were generating two thirds of total traffic. That is a lot. Around the time they banned API access they removed distance evaluation from the software (after having had it broken for a few weeks), initially the game included three distance ranges shown as steps in the UI, there's no distance information left whatsoever now.
- maxsilver 10y ago
  > From the chart it looks like third party API clients were generating two thirds of total traffic. That is a lot. True, but I don't know if that's a fair metric. The Third Party API clients were the only effective way to play the game (because unlike real Pokemon games, "Go" focuses almost exclusively on the act of finding and catching, instead of training. And finding and catching doesn't actually work in their own client) Effectively, Ninantic has "gotten traffic under control" by banning all non-beginner-level players and play strategies. It's like if YouTube decided to "manage traffic" by removing every video from PewDiePie.
  - spotman 10y ago
    Your analogy might be fair, but to continue it all the way, it would be like them turning off ability for others to upload pewdiepie videos so that they have the resources to give him his own you tube branded channel. Which we know, hasn't happened yet, but it's an important part of the direction they are going.
cosarara97 10y ago
I'd like to see some numbers in that graph.
codingmyway 10y ago
Seems like they started authenticating a key (unknown6) that was previously being sent but not getting authenticated against. A lot of bot writers are trying to break that authentication. Given that people are paying others to 'walk' their pokemon there is real money in faking GPS movement in the API.
- JorgeGT 10y ago
  Or go middle way and spoof the GPS radio signal near the device: http://www.rtl-sdr.com/cheating-at-pokemon-go-with-a-hackrf-and-gps-spoofing/ http://www.rtl-sdr.com/cheating-at-pokemon-go-with-a-hackrf-...
  - curiousgal 10y ago
    That is extremely overpowered, all you need is a rooted device with Xposed framework and a GPS spoofing app.
    - JorgeGT 10y ago
      I was (half-)joking :)
  - methou 10y ago
    This is illegal in most countries and may end up in a criminal court.
yomly 10y ago
Godspeed Niantic dev-team!!! Really despise the negative press you're getting. Don't think anyone in their wildest dreams would have thought PoGo would have taken off as well as it has and now they are stuck between a rock and a hard place: Do they suffer the rage of entire countries, and potentially their backers by delaying launching in territories to fix bugs? OR do they suffer the rage of existing users by delaying bug fixes to keep to their original schedule by launching new territories? All the while, they are crippled by scaling issues they never dreamt of having and managing cheating/additional traffic from the meta-ecosystem of trackers. Have no doubt the team are working doubletime and overtime as it is, a shame that redditers are unable to see this. So keep it up Niantic - thanks for bringing us the AR killer app and for showing that there are still opportunities for huge apps to launch![0] [0] https://www.appannie.com/insights/mobile-strategy/pokemon-go-an-opportunity-not-a-threat/ https://www.appannie.com/insights/mobile-strategy/pokemon-go... PS if you guys need more devs, do let me know :P EDIT: downvotes - really? The post scriptum was obviously a cute joke. Or are the downvotes towards the Niantic support?
- gambiting 10y ago
  Uhmmm I guess the downvotes are coming because no one believes that Niantic couldn't predict that the first mobile game(if we forget Pokemon Shuffle for a second) in one of the biggest franchises in the world would be this popular, and I feel like it's quite a popular opinion that the app was released at least 6 months too early, it's poorly optimized and full of bugs. Not to mention that Niantic cannot handle PR, they don't even have a Pokemon GO-dedicated rep yet, they are looking for one. Personally I know several people who played this game every day after it came out, now this number fell down to zero, after a while there isn't much to do and the recent updates made the game worse. I have stopped playing after few days because every other catch the app would freeze and I would have to restart the whole thing. Plus the fact that it's obliterating the battery just by being open is unacceptable - obviously it's just a personal opinion so take it as you will.
  - xorcist 10y ago
    It's also unfortunate that the game handles failures so badly. It hangs and behaves strangely in a number of ways instead of showing an error message.
curiousgal 10y ago
That graph is a joke. http://i.imgur.com/HC9PEx0.png http://i.imgur.com/HC9PEx0.png
Fiahil 10y ago
How about putting some actual efforts into proper engineering before launch, stop blaming third parties for your 6h+ downtime and proving ban effectiveness with a graph missing its Y axis? The only edge Niantic have over their fanbase is their gis database. Lack of communication was appalling, execution and reactivity too. I'm surprised nobody started a "private server" implementation of the game, yet.
- threeseed 10y ago
  You seem borderline vitriolic about this. Nobody including Niantic would ever have dreamed that Pokemon Go would be as huge a sensation as it has been. Nobody. And they have smartly realised that this opportunity is potentially fleeting and so they need to capitalise as quick as possible i.e. grow into other regions. To expect them to do accurate capacity planning is ridiculous. Likewise it is ridiculous to assume that they can react any quicker than they can. It's very hard to scale quickly whilst still maintaining a high level of uptime.
  - TeMPOraL 10y ago
    This still doesn't explain the graph without scale on Y axis and especially one without a clear 0 label. This is like stats 101, and not doing so implies either utter incompetence or intentional attempt to mislead the audience. There is no reason whatsoever to omit the scale and 0 label on a graph if the data actually supports the argued thesis, so most of the time lack of those elements means you're being lied to.
    - ipince 10y ago
      More likely, the nominal values of the y-axis are confidential, and they just want to show the % reduction in traffic. I do assume the crossing with the x-axis is actually at 0.
      - TeMPOraL 10y ago
        You assume. The graph itself doesn't justify that assumption. If you want to lie to someone using a graph, this is exactly the assumption you're hoping they'll make while reading it. Not labeling axis and not starting the graph at 0 are two most basic techniques from the book of Dark Patterns. Of course this could be an innocent mistake made by Niantic. Could be. But given that PR statements in general tend to be manipulative, I'm not willing to trust that. If they didn't want to reveal their resource capacity, they could have labeled the Y-axis with percentages.
dineshp2 10y ago
Looking at the title, I was expecting them to share the juicy details of how they handled the huge amount of traffic, the hacking attempts and the engineering problems they are facing. Instead I was met with some sort of PR attempt to explain the downtime.
- oddevan 10y ago
  Not necessarily downtime, but why they "killed" a very popular third-party tool to show real-time locations of Pokemon.
  - brianwawok 10y ago
    I mean the tool basically defeated the point of the game.
    - nix0n 10y ago
      Really? You think wandering around was the point of the game, as opposed to actually catching the Pokémon?
      - ohnomrbill 10y ago
        Depends on how you look at the game. The point of the game for users is to catch pokemon. The point of the game for Niantic is to have users wander around, so they can make money via sponsored locations. A winning strategy must accommodate both.
      - brianwawok 10y ago
        I mean, if you reverse engineer an api for "where is pokemon X", you may as well just reverse engineer your GPS device for "where am I standing" to move your in game character to right where the pokemon is. Then you can do a for loop and catch all the pokemon in 30 seconds. Actually catching the pokemon is a 0 skill item. Its literally dragging your finger across the screen. The entire game is the exploring the real world part.
    - 10y ago
cloudjacker 10y ago
Spatial graph, no scale
deleted 10y ago
[deleted]
discardorama 10y ago
They don't cache images (of Pokestops) client-side. This has got to be one of the bigger inefficiencies of this client. Aggressive client-side caching can relieve a lot of load on the server.
- aetherson 10y ago
  Well, perhaps. But presumably the images from the pokestops can also be delivered by a CDN with a simple key lookup, and can be outside all of their spatial databases and primary logic servers. The graph they showed was of spatial queries. That's not going to be affected by lookups of images for pokestops.
- tvanantwerp 10y ago
  This was my biggest surprise when I started playing the game. There are two pokestops outside my office which were the ones I most often visited. And every single time, it had to load from scratch despite absolutely zero changes in image or information. Can't understand why they wouldn't cache that on my phone. Little things like that add up.
- jeremiep 10y ago
  That's only on the network side. Playing the game you get a feel that the entire codebase is a cesspool. Everyone I talk to who's also played Ingress sees history repeating itself. They've experienced the same bugs, lack of communication, shutting down services making the game better instead of improving said game and whatnot. I really, really wish Niantic fixes its technical, management and communication problems. The game has so much potential but at this point I lost almost all hope they'll be able fully exploit it.

They banned third party clients from accessing their API. From the chart it looks like third party API clients were generating two thirds of total traffic. That is a lot. On the other hand, this has sparkled a huge community effort to reverse, understand and bypass the implemented protection/fingerprinting. https://www.reddit.com/r/pokemongodev/comments/4w1cvr/pokemongo_current_api_status/ https://www.reddit.com/r/pokemongodev/comments/4w1cvr/pokemo...

elthran 10y ago
I'd be really interesting the see that graph extended to to left - what was the traffic like before the 3 paws "bug" was first introduced. I definitely feel that a significant amount of the drive to use the 3rd party services was due to the fact that you couldn't realistically hunt down any of the pokemon that show up on your radar.
- agar 10y ago
  Yes, the lack of a functional Nearby feature - either integrated or with a third party app - really affects motivation while playing. I was out walking today and saw an unknown Pokemon silhouette in my Nearby list. All I could do was shrug and hope I'd run into it (I didn't). Previously I would have gone significantly out of my way to find it, adding mileage to my walk - which I understood to be one of the design objectives of the game.
  - xorcist 10y ago
    Not only must the three steps have been buggy, the whole "nearby" list is useless and/or wrong. Two friends walking together, with the same exact GPS position on the maps, can have vastly different nearby-lists, sometimes without even one entry in common. How is that even possible?
    - aleksei 10y ago
      It could be that, since the nearby listing holds only 9 entries, that there are more Pokemon nearby and the query to display them is unordered.
- curiousgal 10y ago
  http://i.imgur.com/RLc4i9V.png http://i.imgur.com/RLc4i9V.png
Nanite 10y ago
That's assuming the the y-axis starts at zero, it also lacks numbers for the "spatial queries". This graph doesn't actually tell us anything, besides that there was a drop in traffic after blocking third party access.
masklinn 10y ago
> From the chart it looks like third party API clients were generating two thirds of total traffic. That is a lot. Around the time they banned API access they removed distance evaluation from the software (after having had it broken for a few weeks), initially the game included three distance ranges shown as steps in the UI, there's no distance information left whatsoever now.
maxsilver 10y ago
> From the chart it looks like third party API clients were generating two thirds of total traffic. That is a lot. True, but I don't know if that's a fair metric. The Third Party API clients were the only effective way to play the game (because unlike real Pokemon games, "Go" focuses almost exclusively on the act of finding and catching, instead of training. And finding and catching doesn't actually work in their own client) Effectively, Ninantic has "gotten traffic under control" by banning all non-beginner-level players and play strategies. It's like if YouTube decided to "manage traffic" by removing every video from PewDiePie.
- spotman 10y ago
  Your analogy might be fair, but to continue it all the way, it would be like them turning off ability for others to upload pewdiepie videos so that they have the resources to give him his own you tube branded channel. Which we know, hasn't happened yet, but it's an important part of the direction they are going.

cosarara97 10y ago

I'd like to see some numbers in that graph.

codingmyway 10y ago

Seems like they started authenticating a key (unknown6) that was previously being sent but not getting authenticated against. A lot of bot writers are trying to break that authentication. Given that people are paying others to 'walk' their pokemon there is real money in faking GPS movement in the API.

JorgeGT 10y ago
Or go middle way and spoof the GPS radio signal near the device: http://www.rtl-sdr.com/cheating-at-pokemon-go-with-a-hackrf-and-gps-spoofing/ http://www.rtl-sdr.com/cheating-at-pokemon-go-with-a-hackrf-...
- curiousgal 10y ago
  That is extremely overpowered, all you need is a rooted device with Xposed framework and a GPS spoofing app.
  - JorgeGT 10y ago
    I was (half-)joking :)
- methou 10y ago
  This is illegal in most countries and may end up in a criminal court.

yomly 10y ago

Godspeed Niantic dev-team!!! Really despise the negative press you're getting. Don't think anyone in their wildest dreams would have thought PoGo would have taken off as well as it has and now they are stuck between a rock and a hard place: Do they suffer the rage of entire countries, and potentially their backers by delaying launching in territories to fix bugs? OR do they suffer the rage of existing users by delaying bug fixes to keep to their original schedule by launching new territories? All the while, they are crippled by scaling issues they never dreamt of having and managing cheating/additional traffic from the meta-ecosystem of trackers. Have no doubt the team are working doubletime and overtime as it is, a shame that redditers are unable to see this. So keep it up Niantic - thanks for bringing us the AR killer app and for showing that there are still opportunities for huge apps to launch![0] [0] https://www.appannie.com/insights/mobile-strategy/pokemon-go-an-opportunity-not-a-threat/ https://www.appannie.com/insights/mobile-strategy/pokemon-go... PS if you guys need more devs, do let me know :P EDIT: downvotes - really? The post scriptum was obviously a cute joke. Or are the downvotes towards the Niantic support?

gambiting 10y ago
Uhmmm I guess the downvotes are coming because no one believes that Niantic couldn't predict that the first mobile game(if we forget Pokemon Shuffle for a second) in one of the biggest franchises in the world would be this popular, and I feel like it's quite a popular opinion that the app was released at least 6 months too early, it's poorly optimized and full of bugs. Not to mention that Niantic cannot handle PR, they don't even have a Pokemon GO-dedicated rep yet, they are looking for one. Personally I know several people who played this game every day after it came out, now this number fell down to zero, after a while there isn't much to do and the recent updates made the game worse. I have stopped playing after few days because every other catch the app would freeze and I would have to restart the whole thing. Plus the fact that it's obliterating the battery just by being open is unacceptable - obviously it's just a personal opinion so take it as you will.
- xorcist 10y ago
  It's also unfortunate that the game handles failures so badly. It hangs and behaves strangely in a number of ways instead of showing an error message.

curiousgal 10y ago

That graph is a joke. http://i.imgur.com/HC9PEx0.png http://i.imgur.com/HC9PEx0.png

Fiahil 10y ago

How about putting some actual efforts into proper engineering before launch, stop blaming third parties for your 6h+ downtime and proving ban effectiveness with a graph missing its Y axis? The only edge Niantic have over their fanbase is their gis database. Lack of communication was appalling, execution and reactivity too. I'm surprised nobody started a "private server" implementation of the game, yet.

threeseed 10y ago
You seem borderline vitriolic about this. Nobody including Niantic would ever have dreamed that Pokemon Go would be as huge a sensation as it has been. Nobody. And they have smartly realised that this opportunity is potentially fleeting and so they need to capitalise as quick as possible i.e. grow into other regions. To expect them to do accurate capacity planning is ridiculous. Likewise it is ridiculous to assume that they can react any quicker than they can. It's very hard to scale quickly whilst still maintaining a high level of uptime.
- TeMPOraL 10y ago
  This still doesn't explain the graph without scale on Y axis and especially one without a clear 0 label. This is like stats 101, and not doing so implies either utter incompetence or intentional attempt to mislead the audience. There is no reason whatsoever to omit the scale and 0 label on a graph if the data actually supports the argued thesis, so most of the time lack of those elements means you're being lied to.
  - ipince 10y ago
    More likely, the nominal values of the y-axis are confidential, and they just want to show the % reduction in traffic. I do assume the crossing with the x-axis is actually at 0.
    - TeMPOraL 10y ago
      You assume. The graph itself doesn't justify that assumption. If you want to lie to someone using a graph, this is exactly the assumption you're hoping they'll make while reading it. Not labeling axis and not starting the graph at 0 are two most basic techniques from the book of Dark Patterns. Of course this could be an innocent mistake made by Niantic. Could be. But given that PR statements in general tend to be manipulative, I'm not willing to trust that. If they didn't want to reveal their resource capacity, they could have labeled the Y-axis with percentages.

dineshp2 10y ago

Looking at the title, I was expecting them to share the juicy details of how they handled the huge amount of traffic, the hacking attempts and the engineering problems they are facing. Instead I was met with some sort of PR attempt to explain the downtime.

oddevan 10y ago
Not necessarily downtime, but why they "killed" a very popular third-party tool to show real-time locations of Pokemon.
- brianwawok 10y ago
  I mean the tool basically defeated the point of the game.
  - nix0n 10y ago
    Really? You think wandering around was the point of the game, as opposed to actually catching the Pokémon?
    - ohnomrbill 10y ago
      Depends on how you look at the game. The point of the game for users is to catch pokemon. The point of the game for Niantic is to have users wander around, so they can make money via sponsored locations. A winning strategy must accommodate both.
    - brianwawok 10y ago
      I mean, if you reverse engineer an api for "where is pokemon X", you may as well just reverse engineer your GPS device for "where am I standing" to move your in game character to right where the pokemon is. Then you can do a for loop and catch all the pokemon in 30 seconds. Actually catching the pokemon is a 0 skill item. Its literally dragging your finger across the screen. The entire game is the exploring the real world part.
  - 10y ago

cloudjacker 10y ago

Spatial graph, no scale

deleted 10y ago

[deleted]

discardorama 10y ago

They don't cache images (of Pokestops) client-side. This has got to be one of the bigger inefficiencies of this client. Aggressive client-side caching can relieve a lot of load on the server.

aetherson 10y ago
Well, perhaps. But presumably the images from the pokestops can also be delivered by a CDN with a simple key lookup, and can be outside all of their spatial databases and primary logic servers. The graph they showed was of spatial queries. That's not going to be affected by lookups of images for pokestops.
tvanantwerp 10y ago
This was my biggest surprise when I started playing the game. There are two pokestops outside my office which were the ones I most often visited. And every single time, it had to load from scratch despite absolutely zero changes in image or information. Can't understand why they wouldn't cache that on my phone. Little things like that add up.
jeremiep 10y ago
That's only on the network side. Playing the game you get a feel that the entire codebase is a cesspool. Everyone I talk to who's also played Ingress sees history repeating itself. They've experienced the same bugs, lack of communication, shutting down services making the game better instead of improving said game and whatnot. I really, really wish Niantic fixes its technical, management and communication problems. The game has so much potential but at this point I lost almost all hope they'll be able fully exploit it.