5 ms·

But they knew that the proximate cause of the quest failure was a dead companion, so adding logging around companion death would have figured it out.

rgoulter 7y ago
The tweets seem to also say the companions could die in battle, which would lead to a lot of noise.
- reificator 7y ago
  They can get to a "downed" state, but they cannot die. Depending on how early they realized it was limited to companions on the ship, with a specific state (the source is not clear on how early they determined that to be) they could log events that damage a player when in that specific state. That would have gotten them to fall damage, at which point they might be able to make the leap to ladders. If not, they could then focus their logging around changes in elevation when in that state. But that's backward reasoning from the already known end state, so it's not fair to apply in hindsight. It's also not clear how much log data they can ingest, whether end-user telemetry was feasible, etc...
  - thrower123 7y ago
    On the hardest difficulty level, companions do die often, as they aren't very smart and are quite squishy.
  - rgoulter 7y ago
    The debugging is to reconcile "Observation: quests are failing because companions die" which contradicts "My understanding of the game rules: companions can't die". One of the ways the companions don't die is that the companions are revived after combat, and it's only non-companions that are 'really dead'. The other way is "could die from fall damage on the ship, but there's nothing that high". Turns out the flaw in the logic was that the NPCs can reach great height on the ship. But the flaw in the logic just as easily could have been in the combat. (Or some other way that the tweeter didn't mention). In any case, it's really fun to read the "aha! got it" moment.
  - thaumasiotes 7y ago
    Their non-logging-based investigation got them to the point that they were aware the problem occurred when a companion fell from a great height while the player was on their ship: > The only place in the game when a companion is present but not in the active party is when the player is on their ship > Eventually we figured out that "undamageable" does not mean "invulnerable" -- they can't take damage from attacks but can still get hurt from other things > One of those things: falling a great distance > The problem with that is that there are no spots in the player's ship that are high enough to result in a lethal fall > So now we had to figure out how companions were mysteriously ending up way above the level > I looked into tons of theories This does look like logging would have caught it. They knew exactly what they were looking for: companions reaching strange heights while the player is on their ship.
aneutron 7y ago
You generally do not enable the level of logging that I imagine would be necessary to find this bug, on the enduser version of the software. I could be wrong.
- netsharc 7y ago
  Surely it could be enabled with a runtime flag?
  - stouset 7y ago
    You would need to know to enable it before your companion dies, and thus before the bug is even encountered.
    - gmueckl 7y ago
      As another comment noted, you would log into a ring buffer in memory constantly and only dump it after the event in question happened. But without an initial state at the start od the log, this may not be enough.
- girvo 7y ago
  I wonder if that level of logging would have a performance impact, especially on consoles?
  - wyldfire 7y ago
    "logging"/telemetry can be flight-data-recorder style like XRay uses [1]. This can be limited to a smaller window because it writes to a circular buffer, then again you could fit a lot of entries in a ~512MB buffer. [1] https://llvm.org/docs/XRay.html#flight-data-recorder-mode https://llvm.org/docs/XRay.html#flight-data-recorder-mode

rgoulter 7y ago

The tweets seem to also say the companions could die in battle, which would lead to a lot of noise.

reificator 7y ago
They can get to a "downed" state, but they cannot die. Depending on how early they realized it was limited to companions on the ship, with a specific state (the source is not clear on how early they determined that to be) they could log events that damage a player when in that specific state. That would have gotten them to fall damage, at which point they might be able to make the leap to ladders. If not, they could then focus their logging around changes in elevation when in that state. But that's backward reasoning from the already known end state, so it's not fair to apply in hindsight. It's also not clear how much log data they can ingest, whether end-user telemetry was feasible, etc...
- thrower123 7y ago
  On the hardest difficulty level, companions do die often, as they aren't very smart and are quite squishy.
- rgoulter 7y ago
  The debugging is to reconcile "Observation: quests are failing because companions die" which contradicts "My understanding of the game rules: companions can't die". One of the ways the companions don't die is that the companions are revived after combat, and it's only non-companions that are 'really dead'. The other way is "could die from fall damage on the ship, but there's nothing that high". Turns out the flaw in the logic was that the NPCs can reach great height on the ship. But the flaw in the logic just as easily could have been in the combat. (Or some other way that the tweeter didn't mention). In any case, it's really fun to read the "aha! got it" moment.
- thaumasiotes 7y ago
  Their non-logging-based investigation got them to the point that they were aware the problem occurred when a companion fell from a great height while the player was on their ship: > The only place in the game when a companion is present but not in the active party is when the player is on their ship > Eventually we figured out that "undamageable" does not mean "invulnerable" -- they can't take damage from attacks but can still get hurt from other things > One of those things: falling a great distance > The problem with that is that there are no spots in the player's ship that are high enough to result in a lethal fall > So now we had to figure out how companions were mysteriously ending up way above the level > I looked into tons of theories This does look like logging would have caught it. They knew exactly what they were looking for: companions reaching strange heights while the player is on their ship.

aneutron 7y ago

You generally do not enable the level of logging that I imagine would be necessary to find this bug, on the enduser version of the software. I could be wrong.

netsharc 7y ago
Surely it could be enabled with a runtime flag?
- stouset 7y ago
  You would need to know to enable it before your companion dies, and thus before the bug is even encountered.
  - gmueckl 7y ago
    As another comment noted, you would log into a ring buffer in memory constantly and only dump it after the event in question happened. But without an initial state at the start od the log, this may not be enough.
girvo 7y ago
I wonder if that level of logging would have a performance impact, especially on consoles?
- wyldfire 7y ago
  "logging"/telemetry can be flight-data-recorder style like XRay uses [1]. This can be limited to a smaller window because it writes to a circular buffer, then again you could fit a lot of entries in a ~512MB buffer. [1] https://llvm.org/docs/XRay.html#flight-data-recorder-mode https://llvm.org/docs/XRay.html#flight-data-recorder-mode