6 ms·
For all the flak people usually give the telemetry systems, this bug sounds like it could be made trivial if devs could push custom reporting to users. "When th
by viraptor 7y ago
For all the flak people usually give the telemetry systems, this bug sounds like it could be made trivial if devs could push custom reporting to users. "When the companion dies, send me last few events and a minimal world-state". Having ready data collection/reporting when things go weird is super helpful, whether it's an own, server side software, or a remote client.
- aaronbrethorst 7y agoIt doesn't sound like that would've worked here. One reason it was so hard to pin down is that it was impossible to tell when the bug actually happened -- all of the cases we had were essentially "hey something bad happened in the last ten hours and now my quest is broken" (5/18)
- cjbprime 7y agoBut they knew that the proximate cause of the quest failure was a dead companion, so adding logging around companion death would have figured it out.
- rgoulter 7y agoThe tweets seem to also say the companions could die in battle, which would lead to a lot of noise.
- reificator 7y agoThey can get to a "downed" state, but they cannot die. Depending on how early they realized it was limited to companions on the ship, with a specific state (the source is not clear on how early they determined that to be) they could log events that damage a player when in that specific state. That would have gotten them to fall damage, at which point they might be able to make the leap to ladders. If not, they could then focus their logging around changes in elevation when in that state. But that's backward reasoning from the already known end state, so it's not fair to apply in hindsight. It's also not clear how much log data they can ingest, whether end-user telemetry was feasible, etc...
- thrower123 7y agoOn the hardest difficulty level, companions do die often, as they aren't very smart and are quite squishy.
- rgoulter 7y agoThe debugging is to reconcile "Observation: quests are failing because companions die" which contradicts "My understanding of the game rules: companions can't die". One of the ways the companions don't die is that the companions are revived after combat, and it's only non-companions that are 'really dead'. The other way is "could die from fall damage on the ship, but there's nothing that high". Turns out the flaw in the logic was that the NPCs can reach great height on the ship. But the flaw in the logic just as easily could have been in the combat. (Or some other way that the tweeter didn't mention). In any case, it's really fun to read the "aha! got it" moment.
- thaumasiotes 7y agoTheir non-logging-based investigation got them to the point that they were aware the problem occurred when a companion fell from a great height while the player was on their ship: > The only place in the game when a companion is present but not in the active party is when the player is on their ship > Eventually we figured out that "undamageable" does not mean "invulnerable" -- they can't take damage from attacks but can still get hurt from other things > One of those things: falling a great distance > The problem with that is that there are no spots in the player's ship that are high enough to result in a lethal fall > So now we had to figure out how companions were mysteriously ending up way above the level > I looked into tons of theories This does look like logging would have caught it. They knew exactly what they were looking for: companions reaching strange heights while the player is on their ship.
- aneutron 7y agoYou generally do not enable the level of logging that I imagine would be necessary to find this bug, on the enduser version of the software. I could be wrong.
- netsharc 7y agoSurely it could be enabled with a runtime flag?
- stouset 7y agoYou would need to know to enable it before your companion dies, and thus before the bug is even encountered.
- gmueckl 7y agoAs another comment noted, you would log into a ring buffer in memory constantly and only dump it after the event in question happened. But without an initial state at the start od the log, this may not be enough.
- girvo 7y agoI wonder if that level of logging would have a performance impact, especially on consoles?
- wyldfire 7y ago"logging"/telemetry can be flight-data-recorder style like XRay uses [1]. This can be limited to a smaller window because it writes to a circular buffer, then again you could fit a lot of entries in a ~512MB buffer. [1] https://llvm.org/docs/XRay.html#flight-data-recorder-mode https://llvm.org/docs/XRay.html#flight-data-recorder-mode
- deleted 7y ago[deleted]
- SkyPuncher 7y agoThey knew what to observe for, but they couldn't figure out how to reproduce that.
- madrox 7y agoThis is why it's important to log everything and keep it around forever!
- CDSlice 7y agoDo people really get worked up over that kind of telemetry? I know I wouldn't. I would even be OK with much more telemetry around most actions in the game as long as it doesn't snoop on anything outside of what I do in the game.
- MereInterest 7y agoYes. I don't know what is being sent back, or what information it has gathered. If a program does not require internet access for its basic functionality, and it then accesses the internet without explicitly asking me for permission, it should be viewed with extreme suspicion.
- reificator 7y agoI'm a fan of pure gameplay telemetry, pure error telemetry, etc... I specifically remember being amazed at the heatmaps that Bungie shared about data in Halo 2 and thinking that I needed to learn how to do that, and it was going to change how development was done forever. But it's difficult to police good telemetry and easy to change silently. Bad actors have continually given justifiable reasons for data collection, sometimes even with a pinky-swear that they'll not use it for evil. But the terms can be updated by the party collecting the data at any time they desire, with little recourse for the user other than to stop using that product or service. So no, I'm not worked up over that kind of telemetry, but I'm still going to block it. It doesn't make sense not to.
- e1ven 7y agoThis was my favorite game of 2019, and if it had mandatory telemetry like that I would not have played it. If it were opt-in, it could be used well however, similar to what the Telltale games had done. 65% of people made choice A. 35% of people made choice B. They have a sort of proxy for that with Achievements however.
- deleted 7y ago[deleted]
- deogeo 7y agoNo need for telemetry. Keep a local log (of what you would be sending via telemetry), and when the player decides to file a bug, inform them that'll include sending the log, with the option to cancel/file bug without a log. Many bug reporting systems already work this way, such as the one used by Firefox, I think. Just because something can be done by violating user privacy, that doesn't mean it's the only way, or even that alternatives are difficult.
- sdenton4 7y agoEh, that gets you things big enough that someone bothers to send a bug report. Startup performance, mis-clicks, etc, will be missed. Not to mention ragequits that don't produce a bug report... The problem is that devs reeeeally need a mix of very specific data for debugging, and very aggregated data for prioritization... There's all kinds of cool differential privacy tools for doing things like this, but no way for users to know or trust that they do what's claimed on the tin.
- boarnoah 7y agoSubnautica did it really well [short GDC talk: https://www.youtube.com/watch?v=Urx7WQE6NY0 https://www.youtube.com/watch?v=Urx7WQE6NY0 ]. I've been floating the idea of setting up some log shipper plugins for Unity/Unreal to make a turnkey solution you could drop in and self host.