4 ms·

Just as glad I didn't get a ticket ($20? sure. Scalper prices? Heck no) and waste the time. This whole thing is going to turn into a real fiasco for some time d

Just as glad I didn't get a ticket ($20? sure. Scalper prices? Heck no) and waste the time. This whole thing is going to turn into a real fiasco for some time due to the number of people who bought resold passes (so the original seller sold it and now gets their original $20 back), people who traveled (I know many got hotels, not sure how many flew in), and questions of how they're going to get that in-game currency to people (reportedly some are still in line to get in, while others are waiting in a not-quite-as-long line just to get out). Each person attending has a uniquely-numbered wristband pass, and apparently on entry they're getting an envelope with a patch and a unique-to-them QR code which needs to be scanned after spinning a special "Pokestop" and effectively checks you into the event. Since the game is crashing basically at startup for most folks, that makes it a bit of a challenge just to get in. Still, I suspect the in-game credit is going to be based on people who checked in with that code since the ordering process didn't involve providing a game login. Also, I've seen some folks questioning why they didn't have WiFi set up, but I'm pretty sure that the way WiFi works would make that almost impossible anyway (2.4 would be a bad joke no matter what, 5GHz might be possible over most/all available channels, APs that can handle hundreds or thousands of simultaneous connections are rare and probably require major advance planning, AP broadcast power would probably have to be quite low to allow large numbers of APs in a physically small area, etc.).

maxerickson 9y ago
People in a stadium aren't going to be quite as focused on their phones, but lots of people in a tight space isn't an unusual occurrence for networks.
- fencepost 9y ago
  ~15k people in a small section of Grant Park all trying to play a somewhat data-intensive online game that requires constant communication with servers is a VERY different activity stream than people in a stadium. Might be closer to think of those people in a stadium all trying to stream video of the event.
  - wfunction 9y ago
    Is the game that data-intensive? What data is there to send/receive that you're comparing it with streaming video?
    - sigmar 9y ago
      how many people are live streaming a concert at a given moment? maybe 5-10 on periscope, in addition to maybe 2x that number on facebook. definitely not 15k.
    - Schampu 9y ago
      The game uses google's protobuf protocol which makes communication more efficient and the game updates (I can only tell from <= 0.35 analysis) all 10-30 seconds like loading pokemon spawns and map entities of the current map area. The most expensive data in 0.35 was: - Pokemon models (They were dynamically loaded but cached then) - Pokestop images (Server sends image url and client searches for a chached version) - Syncing the player state with the client (Inventory, rewards, egg states..) - Updating the map (Future pokemon spawns, pokestops/gyms in view..) If a player is idle and doesn't move, just keeps farming a single pokestop and battles sometimes, then it should be relatively low cost since the game doesn't use a realtime communication (It's actually HTTPS based lol).
      - RugnirViking 9y ago
        The pokemon models are loaded over the network? Whatever for? Seems like a big waste of precious data
        dimillian 9y ago
        So you can add/update them without doing an .ipa/.apk update ? Seems really convenient, and a one time deal if it's cached.
        Schampu 9y ago
- ericabiz 9y ago
  Stadiums have issues with this too. I was at Marlins Park for the All-Star series. During the Home Run Derby I had press access, which had special Wifi credentials on a completely separate network. It was down for the entire game. A lot of hotspots popped up in the press area. The during the actual All-Star game I tried the regular stadium Wifi. It was down for most of the game. T-Mobile was up for most of the game, so I was still able to publish photos and read people's reactions. A good thing as T-Mo was the main sponsor for the games.
  - azernik 9y ago
    It's a challenge, but a very well-characterized one that all enterprise-grade access points are built to handle. Lots of access points, with radio power turned way down to prevent neighbor interference, and channels selected to avoid overlap with neighboring APs. Shameless plug - my ex-employer Meraki is particularly good at supplying these systems for events, since they have automatic channel-selection and good tooling for selecting power levels, and let you set up configurations at your leisure that the APs will use as soon as they get an internet connection and fetch the configs from Meraki servers.
aeleos 9y ago
There are definitely companies have the kind of wifi infrastructure you are talking about who offer rentals to the people who organize these kinds of events. Here is an example of one[0]. Wifi is often deployed at large events (PAX East for example) and it is more then feasible if the company is willing to pay the money for another company to do it. However, their server infrastructure is another question entirely. In today's age, making scale-able network services with very little downtime is not something that requires a dedicated networking team and hundreds of millions of dollars. I can't imagine they have very many good excuses to why they continue to allow this to happen other than that they don't value investment into their infrastructure enough (despite being a company completely dependent on the internet) The problems that they are experiencing are ones that have caused problems and had solutions created for them for years. The just seem to be unwilling to properly invest in the solutions to fix them, which are cheaper than ever (considering the complexity of the issues in the grand scheme of things). [0] https://tradeshowinternet.com/solutions/event-organizers https://tradeshowinternet.com/solutions/event-organizers
- fencepost 9y ago
  They may also be having load-balancer issues assuming they have front-end servers that are checking whether accounts are supposed to be directed to a specific set of servers. Assuming that they normally load-balance based on geographic location, the check for specialized server connections may be something that they never actually load-tested before after confirming that it worked for their staff in testing. I asked earlier and have asked again whether anyone at the location but not officially attending is having better luck connecting, that would help to narrow it down between carrier issues, Niantic server issues and Niantic server-for-this-event issues Edit: They expanded the area for special/regional spawns to 2 miles around Grant Park but still only for people who "checked in" at the park, and people who've left are reporting much better luck when they're away from the park. That may indicate that most of the issue was insufficient networking.
ploxiln 9y ago
A company I used to work for specialized in exactly this case - high density WiFi. They have an 8 radio / 24 antenna AP: https://www.xirrus.com/product/xr-4000-series/ https://www.xirrus.com/product/xr-4000-series/ You can use 6 20 MHz channels in the 5GHz band and 1 or 2 channels in the 2.4 GHz band, on each of these multi-APs, and you can get much smaller "cell sizes" (low power signal, smaller signal area) than cellular providers. You can bump clients from busier channels on to less busy ones to force better balance. WiFi drivers for clients and APs are a big pile of bugs - the resilience (retries at lower rates, chip-timeout-resets, etc) covers a lot of them up. You can get more consistent throughput and latency by digging in and fixing these bugs (and working around bugs in most clients). Most good WiFi APs are debugged until they work pretty well for the "typical" home or office use-cases. But the Xirrus APs (and some competitors to some degree) are debugged for the "high density" use-case - multiple hundreds of users per multi-AP, and maybe 8 to 30 of these multi-APs in a large conference hall or stadium. You have to know what you're doing. It is expensive. Niantic has a hugely popular service. They're making money. But they never seem to know how to handle the massive scale that was predictable after the first month or so. They sold tickets, they knew how many were coming. There are people who know how to handle that. Niantic just didn't feel like doing the research to find out how or who to handle it. Their systems fall over again and again and again.

maxerickson 9y ago

People in a stadium aren't going to be quite as focused on their phones, but lots of people in a tight space isn't an unusual occurrence for networks.

fencepost 9y ago
~15k people in a small section of Grant Park all trying to play a somewhat data-intensive online game that requires constant communication with servers is a VERY different activity stream than people in a stadium. Might be closer to think of those people in a stadium all trying to stream video of the event.
- wfunction 9y ago
  Is the game that data-intensive? What data is there to send/receive that you're comparing it with streaming video?
  - sigmar 9y ago
    how many people are live streaming a concert at a given moment? maybe 5-10 on periscope, in addition to maybe 2x that number on facebook. definitely not 15k.
  - Schampu 9y ago
    The game uses google's protobuf protocol which makes communication more efficient and the game updates (I can only tell from <= 0.35 analysis) all 10-30 seconds like loading pokemon spawns and map entities of the current map area. The most expensive data in 0.35 was: - Pokemon models (They were dynamically loaded but cached then) - Pokestop images (Server sends image url and client searches for a chached version) - Syncing the player state with the client (Inventory, rewards, egg states..) - Updating the map (Future pokemon spawns, pokestops/gyms in view..) If a player is idle and doesn't move, just keeps farming a single pokestop and battles sometimes, then it should be relatively low cost since the game doesn't use a realtime communication (It's actually HTTPS based lol).
    - RugnirViking 9y ago
      The pokemon models are loaded over the network? Whatever for? Seems like a big waste of precious data
      - dimillian 9y ago
        So you can add/update them without doing an .ipa/.apk update ? Seems really convenient, and a one time deal if it's cached.
        Schampu 9y ago
ericabiz 9y ago
Stadiums have issues with this too. I was at Marlins Park for the All-Star series. During the Home Run Derby I had press access, which had special Wifi credentials on a completely separate network. It was down for the entire game. A lot of hotspots popped up in the press area. The during the actual All-Star game I tried the regular stadium Wifi. It was down for most of the game. T-Mobile was up for most of the game, so I was still able to publish photos and read people's reactions. A good thing as T-Mo was the main sponsor for the games.
- azernik 9y ago
  It's a challenge, but a very well-characterized one that all enterprise-grade access points are built to handle. Lots of access points, with radio power turned way down to prevent neighbor interference, and channels selected to avoid overlap with neighboring APs. Shameless plug - my ex-employer Meraki is particularly good at supplying these systems for events, since they have automatic channel-selection and good tooling for selecting power levels, and let you set up configurations at your leisure that the APs will use as soon as they get an internet connection and fetch the configs from Meraki servers.

aeleos 9y ago

There are definitely companies have the kind of wifi infrastructure you are talking about who offer rentals to the people who organize these kinds of events. Here is an example of one[0]. Wifi is often deployed at large events (PAX East for example) and it is more then feasible if the company is willing to pay the money for another company to do it. However, their server infrastructure is another question entirely. In today's age, making scale-able network services with very little downtime is not something that requires a dedicated networking team and hundreds of millions of dollars. I can't imagine they have very many good excuses to why they continue to allow this to happen other than that they don't value investment into their infrastructure enough (despite being a company completely dependent on the internet) The problems that they are experiencing are ones that have caused problems and had solutions created for them for years. The just seem to be unwilling to properly invest in the solutions to fix them, which are cheaper than ever (considering the complexity of the issues in the grand scheme of things). [0] https://tradeshowinternet.com/solutions/event-organizers https://tradeshowinternet.com/solutions/event-organizers

fencepost 9y ago
They may also be having load-balancer issues assuming they have front-end servers that are checking whether accounts are supposed to be directed to a specific set of servers. Assuming that they normally load-balance based on geographic location, the check for specialized server connections may be something that they never actually load-tested before after confirming that it worked for their staff in testing. I asked earlier and have asked again whether anyone at the location but not officially attending is having better luck connecting, that would help to narrow it down between carrier issues, Niantic server issues and Niantic server-for-this-event issues Edit: They expanded the area for special/regional spawns to 2 miles around Grant Park but still only for people who "checked in" at the park, and people who've left are reporting much better luck when they're away from the park. That may indicate that most of the issue was insufficient networking.

ploxiln 9y ago

A company I used to work for specialized in exactly this case - high density WiFi. They have an 8 radio / 24 antenna AP: https://www.xirrus.com/product/xr-4000-series/ https://www.xirrus.com/product/xr-4000-series/ You can use 6 20 MHz channels in the 5GHz band and 1 or 2 channels in the 2.4 GHz band, on each of these multi-APs, and you can get much smaller "cell sizes" (low power signal, smaller signal area) than cellular providers. You can bump clients from busier channels on to less busy ones to force better balance. WiFi drivers for clients and APs are a big pile of bugs - the resilience (retries at lower rates, chip-timeout-resets, etc) covers a lot of them up. You can get more consistent throughput and latency by digging in and fixing these bugs (and working around bugs in most clients). Most good WiFi APs are debugged until they work pretty well for the "typical" home or office use-cases. But the Xirrus APs (and some competitors to some degree) are debugged for the "high density" use-case - multiple hundreds of users per multi-AP, and maybe 8 to 30 of these multi-APs in a large conference hall or stadium. You have to know what you're doing. It is expensive. Niantic has a hugely popular service. They're making money. But they never seem to know how to handle the massive scale that was predictable after the first month or so. They sold tickets, they knew how many were coming. There are people who know how to handle that. Niantic just didn't feel like doing the research to find out how or who to handle it. Their systems fall over again and again and again.