11 ms·
Pokémon Go Fest attendees to get refunds as technical issues break the event
- vortico 9y agoThis game has been so much of a trainwreck, I don't see why Game Freaks licensed Niantic to use their IP.
- sddfd 9y agoNiantic seems to have expertise in Mobile Gaming from ingress. Which other company has? If your comment is based on this event, it seems unfair: Who else has successfully done an event with similar bandwidth requirements before? Also, my understanding is that the cellular network is the problem, but you might correct me if I'm wrong. Besides, Blizzard had spectacular problems in the past with WOW launch events, simply because no one had ever done something of that /scale/ before.
- gambiting 9y agoIt sounds like they planned an event for 20k people without telling mobile networks they are doing it. For festivals mobile networks usually deploy extra capacity to handle larger than usual number of people. It sounds like this could be handled by large number of local WiFi spots as well.
- pilsetnieks 9y agoOr the mobile networks fucked up and underestimated the network load.
- TrickyRick 9y agoYeah, I would have gone with the wifi option as well if I was setting up an event like this.
- wfunction 9y agoHow do you just "deploy extra capacity"? Do they install new antennas just for an event?
- dungle6 9y agoDeployment of cell sites at events like stadiums for sporting events and outdoor music festivals is pretty common for years and years. Unless you're a hermit you almost have certainly seen a COW at one point or another. Cf Wikipedia mobile cell sites, especially COW.
- wfunction 9y agoI've seen them around alright, just not realized how common or easy-to-set-up they might be for an event like this, or how they work from an inter-provider standpoint. Do they not cost a whole ton for the organization setting up the event, and wouldn't every provider (T-Mobile, AT&T, etc.) be required to set one up? It seems like such a massive headache and cost for something like a Pokemon-catching crowd that might not even generate much revenue(?) that I'm confused how everyone is saying "just deploy extra capacity". Like do providers just follow crowded events for free and put antennas wherever people are as if it's no big deal?
- deleted 9y ago[deleted]
- deleted 9y ago[deleted]
- dungle6 9y ago20k is actually not that many people as far as these things go. Over 50k in that confined stadium area and you really start hitting some hard limits with current cell tech. 20k should have been no problem with careful planning.
- sigmar 9y ago>20k is actually not that many people as far as these things go 20k people in close proximity uploading and downloading constantly without breaks? What specific event is that comparable to? Music festivals don't have all attendees constantly using their data network.
- TillE 9y agoThe point is, there are many places in the world that draw large crowds into a tiny area on a weekly basis, and the cell networks will fall over and die without special measures. Did they do anything at all to anticipate that load? It seems not. "We tried and failed" is understandable, but simply inviting 20k people and expecting them to play a mobile game is monumentally foolish.
- 40acres 9y agoHas it? I read recently that they still had around 70m MAU.
- FullMtlAlcoholc 9y agoThe integration of ltheir location api with AR was reasonably well done at the time. I agree though, Niantic has no business making games. I try to imagine a world in which they license their AR SDK
- mrisoli 9y agoYou think the AR was well done? Serious question, not trying to flame or anything, but I disagree. For me, the AR is pretty much just overlaying a Pokemon graphic on top of camera footage, there's very little interaction to the camera input. In fact, after playing for a bit, I turned off the AR because of loading times.
- sand500 9y agoThats not the AR tho, the true AR aspect is tying it up with the real world.
- FullMtlAlcoholc 9y agoYou're talking more about the graphics than the actual AR experience. It was tied to the real world as mentioned in the sibling comment. I imagine they accumulated a lot of data from Ingress. More often than not, the gyms and pokemon hotspots were located at some point of interest and/or place that had heavy foot traffic, increasing the probability that you'd run into other players. It was a global sensation for a reason and despite it's extensive flaws, it was one of the funnest gaming experiences I've ever had. Not because of what happened in the game, which admittedly was beyond terrible at every level from core gameplay to polish and UI (I've seen and worked on better better at 24 hr gamejams. I have no idea why Nintendo didn't step in and take over development after the craze happened) The gyms closest to me were located in a popular shopping mall that had no shortage of players. It led to a few new meatspace friends for me. There was something meta-magical about seeing someone walking down the street glued to their phone while you're doing the same, both of you realizing that you're trying to catch some rare pokemon. Often enough, it led to some friendly social encounter, providing just a glimpse of what AR gaming could be. It also inspired the dev community to make all sorts of apps and mods that addressed many of the games shortcomings.I left after Niantic dismissed the dev community because the game could've been so much more with the power of the crowd. Instead, Niantic preferred to keep the game shitty so that they could extract profits from the parents of 8 year olds. I was doubly disappointed when it was revealed that the Switch, a mobile game console, would have no GPS chip. Nintendo could have pioneered AR gameplay like they did with 3D gaming. Maybe that's expecting too much from a company that doesn't even get online gameplay. It only lasted two weeks, but that's the standard shelf life of most games that aren't endless grindfests.
- hkmurakami 9y agoBecause ingress was an enormous success and a cultural phenomenon in Japan.
- yoodenvranx 9y agoNot only in Japan but also in Germany and the US
- monksy 9y agoThe weather in Chicago since 5pm till early this morning has been filled with storms. Storms were also threatening this afternoon, but surprisingly they didn't come. I had to cancel a beach meetup that I was looking forward do over this.
- kermittd 9y agoThe event looked interesting and the game is a great concept. Sorry to hear about this fail as well.
- kayoone 9y agoEven at big music festivals, mobile networks are usually totally overloaded just by the sheer amount of people logged in at the same time, without even doing anything. Even recieving an SMS can take minutes. I don't assume they would do an event like this without letting providers know in advance to upgrade capacity, but it seems like this.
- iDemonix 9y agoI've always wondered why cellular is affected by this, anyone have a link?
- TrickyRick 9y agoThere's a limited amount of "space" in the air, ie not everyone can be transmitting at the same time. This is fixed using a few different techniques depending on the cell network but for GSM something called TDMA (Time division multiple access) is used. This basically gives your phone a certain amount of time to broadcast on before it should leave the airwaves for someone else to transmit on. Too many people in one location means not enough time for everyone to transmit.
- ghthor 9y agoThis is the main reason yes. No amount of backhaul can change this. There are 2 solutions to this problem. 1) Distributing a web of much smaller cells, lower power so much less devices are connected per node. 2) DIDO pCell technology developed by Artemis. All antennas transmit and the waveforms from each broadcast interfere around each receiver to create a valid signal for that signal device. Both systems require loads of calculations by the broadcasters to either manage handoffs from each mini cell, or to calalate valid distributed broadcasts per each receiver. This type of event would have been a great opportunity for Artemis to show off their tech.
- vvanders 9y agoThat's not really quite accurate on modern networks. Most of them use CDMA which allows many devices to transmit/receive at the same time, assuming that they can modulate their power with respect to distance from the tower/etc(which is a requirement for the "xor" to cancel out). That said there's still a bunch of other stuff you have to sort out if you want to support that many devices in a small space.
- fencepost 9y agoJust as glad I didn't get a ticket ($20? sure. Scalper prices? Heck no) and waste the time. This whole thing is going to turn into a real fiasco for some time due to the number of people who bought resold passes (so the original seller sold it and now gets their original $20 back), people who traveled (I know many got hotels, not sure how many flew in), and questions of how they're going to get that in-game currency to people (reportedly some are still in line to get in, while others are waiting in a not-quite-as-long line just to get out). Each person attending has a uniquely-numbered wristband pass, and apparently on entry they're getting an envelope with a patch and a unique-to-them QR code which needs to be scanned after spinning a special "Pokestop" and effectively checks you into the event. Since the game is crashing basically at startup for most folks, that makes it a bit of a challenge just to get in. Still, I suspect the in-game credit is going to be based on people who checked in with that code since the ordering process didn't involve providing a game login. Also, I've seen some folks questioning why they didn't have WiFi set up, but I'm pretty sure that the way WiFi works would make that almost impossible anyway (2.4 would be a bad joke no matter what, 5GHz might be possible over most/all available channels, APs that can handle hundreds or thousands of simultaneous connections are rare and probably require major advance planning, AP broadcast power would probably have to be quite low to allow large numbers of APs in a physically small area, etc.).
- maxerickson 9y agoPeople in a stadium aren't going to be quite as focused on their phones, but lots of people in a tight space isn't an unusual occurrence for networks.
- fencepost 9y ago~15k people in a small section of Grant Park all trying to play a somewhat data-intensive online game that requires constant communication with servers is a VERY different activity stream than people in a stadium. Might be closer to think of those people in a stadium all trying to stream video of the event.
- wfunction 9y ago
- intopieces 9y agoAnyone who has ever tried to send a photo at Soldier Field during any large event could have predicted the crippling of the network that happened here. Maybe I missed it in the article, but did the event organizers not set up an ad-hoc network for this?
- Hydraulix989 9y agoThis is a situation where a mesh network topology (like Firechat) would be necessary.
- azernik 9y agoMesh is unfortunate for these situations; the limiting factor at these scales isn't infrastructure size, but EM spectrum. The accepted solution is mandating players go onto WiFi, where you can get them to much lower emitted power by deploying enough APs, close enough to any player location, that no clients need to shout to get their traffic heard.
- dboreham 9y agoSigh. Except in battlefield and deep off grid situations mesh is always a bad idea.
- Thriptic 9y agoAgreed, this should have been totally predictable based on Bears games, lolla, etc.
- yoodenvranx 9y agoNiantic organized a lot of big Ingress events for the last 3 years so they _must_ have known about these problems but somehow they still managed to fuck it up.
- bergie 9y agoThe Ingress events have worked as long as players are distributed around a city. When everybody was in a single location (for example a shard), stuff would fail
- free_everybody 9y ago>When Niantic’s John Hanke took the stage, he was greeted by an audience a few thousand deep, many of them chanting “FIX YOUR GAME” or “WE CAN’T PLAY!” Can you imagine how stressful this would be? It's one thing to get disappointing feedback data on bugs, UX, etc. Standing in front of thousands of angry people screaming at you to fix your software this instant? Yikes!!
- capkutay 9y agoI remember Google Cloud was doing a ton of press about how Niantic was using their platform for Pokemon Go. I wonder if that will backfire for them now.
- dmoy 9y agoThese are completely unrelated, no? One is about scaling the servers and stuff behind the scenes, this thing is about cellular data networks failing and dying.
- cptcobalt 9y agoHumans just aren't made for that sort of feedback. I couldn't imagine 10 or 100 people yelling genuine critical feedback like this at me -- let alone thousands. I imagine he didn't have a great day.
- ShabbosGoy 9y agoCould they have used a mesh network to help load balance connections to WiFi APs?
- NightlyDev 9y agoI guess that's no suprise? Downtime with every single event som far, so why should this one be anything different? WiFi is a thing and should he able to handle it just fine. Heck, they could even block anything but PGO and get away with low bandwidth too. When voulenteers can manage to get WiFi working for thousands of nerds at indoor LANs... Let's just say that Niantic should be able to avoid this with a little bit of planning. This is the company running one of the most unstable and buggy games I have ever played, so I would expect no less. At least the game is very fun for a lot of people... When it works.
- make3 9y agoDon't large lans use.. LAN ? (Ethernet). I could be wrong, but as many desktops still don't have wifi and you know where people are and it's so much more stable large scale.. it would make sense
- NightlyDev 9y agoOfc ethernet is used, but WiFi is often used for phones and additional devices like game consoles as it's mostly 1 port/person. The gathering and Dreamhack are two good examples of events with WiFi scaling to many thousands of connected devices without much problems. More or less every AP provider has solutions for this.
- yoodenvranx 9y agoAs a long term Ingress player i am not surprised by this _at all_. Niantics management has proven itself again and again and again as being completely incompetent at everything they do. (They even to manage to ban their own official event photographer after he had GPS problems while traveling to Paris). I used to be angry at them for ruining Ingress (one of the best games ever made) but nowadays I am just amazed at the level of sheer incompetence. They managed to learn _nothing_ from 4 years of Ingress and I think now it's time that the whole upper management is fired so that the company can start fresh.
- lini 9y agoThe scale of this event is bigger than any Ingress anomaly until now. Also, all players seem to be concentrated into a smaller area than Ingress events, which further complicates things. I think the main issue they could not control today is that Verizon ran out of bandwidth and people simply couldn't connect. Another difference with Ingress events - the Go Fest is a closed venue with security and ticket validation at the gate. In Ingress, you usually validate your ticket beforehand and don't have to worry about queues in the hours before the actual event stats.
- grogenaut 9y agoDid they buy extra coverage for the event? I've been to football games and mardi gras and they don't have issues with coverage and that's a lot more than 20k people even if they're not all playing it should be similar unless PMG is really really bad with bandwidth.
- dEnigma 9y agoNiantec just posted an update[0]. These are the steps they will be taking: All registered attendees will soon receive an email with instructions on how to receive a full refund for the cost of their ticket. These instructions will be sent to the email addresses associated with your Pokémon GO account. All registered attendees will receive $100 in PokéCoins in their Pokémon GO account. Special Pokémon, Eggs, and check-in PokéStops appearing during Pokémon GO Fest have had their range increased to a two mile radius surrounding Grant Park through Monday morning, July 24. These Pokémon and Eggs will only be visible to Pokémon GO Fest attendees who validated the QR code they received when they entered Pokémon GO Fest. Attendees who were unable to validate their QR code during the event can do so through the special PokéStops through Monday morning. All registered attendees will have the Legendary Pokémon, Lugia, added to their account. [0]http://pokemongolive.com/en/post/pokemongofestupdate http://pokemongolive.com/en/post/pokemongofestupdate
- kendallpark 9y ago> All registered attendees will have the Legendary Pokémon, Lugia, added to their account. Lugia? That’s some damage control right there.
- mizzao 9y agoBeg for forgiveness and give out free stuff, that generally makes people much happier :)
- cloudkj 9y agoAre there any Niantic employees active here on HN? As an engineer, I would be very interested in hearing some first hand accounts of some of the technical problems Niantic's engineers have had to deal with, not necessarily just for the Go Fest, but for the game in general. To the extent that they're allowed to talk about it, it would be very interesting to get an AMA style take on some of the challenges.
- davidmurdoch 9y agoAs the lead engineer on an up coming game of a similar scale, YES! This would be so awesome!
- em3rgent0rdr 9y agoThe silver lining is that all these people would would only interact with each other virtually then had a perfect opportunity to talk to each other face-to-face, fwiw.
- jbmorgado 9y agoWhat surprised me in this article is that people still gather to play Pokemon Go. I personally never got into the game, while my cousin was totally into it, but where I live (which is a medium city for Europe), the fad has totally passed in less than 6 months after launch.