4 ms·

It doesn't sound like that would've worked here. One reason it was so hard to pin down is that it was impossible to tell when the bug actually happened -- all

It doesn't sound like that would've worked here. One reason it was so hard to pin down is that it was impossible to tell when the bug actually happened -- all of the cases we had were essentially "hey something bad happened in the last ten hours and now my quest is broken" (5/18)

cjbprime 7y ago
But they knew that the proximate cause of the quest failure was a dead companion, so adding logging around companion death would have figured it out.
- rgoulter 7y ago
  The tweets seem to also say the companions could die in battle, which would lead to a lot of noise.
  - reificator 7y ago
    They can get to a "downed" state, but they cannot die. Depending on how early they realized it was limited to companions on the ship, with a specific state (the source is not clear on how early they determined that to be) they could log events that damage a player when in that specific state. That would have gotten them to fall damage, at which point they might be able to make the leap to ladders. If not, they could then focus their logging around changes in elevation when in that state. But that's backward reasoning from the already known end state, so it's not fair to apply in hindsight. It's also not clear how much log data they can ingest, whether end-user telemetry was feasible, etc...
    - thrower123 7y ago
      On the hardest difficulty level, companions do die often, as they aren't very smart and are quite squishy.
    - rgoulter 7y ago
      The debugging is to reconcile "Observation: quests are failing because companions die" which contradicts "My understanding of the game rules: companions can't die". One of the ways the companions don't die is that the companions are revived after combat, and it's only non-companions that are 'really dead'. The other way is "could die from fall damage on the ship, but there's nothing that high". Turns out the flaw in the logic was that the NPCs can reach great height on the ship. But the flaw in the logic just as easily could have been in the combat. (Or some other way that the tweeter didn't mention). In any case, it's really fun to read the "aha! got it" moment.
    - thaumasiotes 7y ago
      Their non-logging-based investigation got them to the point that they were aware the problem occurred when a companion fell from a great height while the player was on their ship: > The only place in the game when a companion is present but not in the active party is when the player is on their ship > Eventually we figured out that "undamageable" does not mean "invulnerable" -- they can't take damage from attacks but can still get hurt from other things > One of those things: falling a great distance > The problem with that is that there are no spots in the player's ship that are high enough to result in a lethal fall > So now we had to figure out how companions were mysteriously ending up way above the level > I looked into tons of theories This does look like logging would have caught it. They knew exactly what they were looking for: companions reaching strange heights while the player is on their ship.
- aneutron 7y ago
  You generally do not enable the level of logging that I imagine would be necessary to find this bug, on the enduser version of the software. I could be wrong.
  - netsharc 7y ago
    Surely it could be enabled with a runtime flag?
    - stouset 7y ago
      You would need to know to enable it before your companion dies, and thus before the bug is even encountered.
      - gmueckl 7y ago
        As another comment noted, you would log into a ring buffer in memory constantly and only dump it after the event in question happened. But without an initial state at the start od the log, this may not be enough.
  - girvo 7y ago
    I wonder if that level of logging would have a performance impact, especially on consoles?
    - wyldfire 7y ago
      "logging"/telemetry can be flight-data-recorder style like XRay uses [1]. This can be limited to a smaller window because it writes to a circular buffer, then again you could fit a lot of entries in a ~512MB buffer. [1] https://llvm.org/docs/XRay.html#flight-data-recorder-mode https://llvm.org/docs/XRay.html#flight-data-recorder-mode
deleted 7y ago
[deleted]
SkyPuncher 7y ago
They knew what to observe for, but they couldn't figure out how to reproduce that.
madrox 7y ago
This is why it's important to log everything and keep it around forever!

cjbprime 7y ago

But they knew that the proximate cause of the quest failure was a dead companion, so adding logging around companion death would have figured it out.

rgoulter 7y ago
The tweets seem to also say the companions could die in battle, which would lead to a lot of noise.
- reificator 7y ago
  They can get to a "downed" state, but they cannot die. Depending on how early they realized it was limited to companions on the ship, with a specific state (the source is not clear on how early they determined that to be) they could log events that damage a player when in that specific state. That would have gotten them to fall damage, at which point they might be able to make the leap to ladders. If not, they could then focus their logging around changes in elevation when in that state. But that's backward reasoning from the already known end state, so it's not fair to apply in hindsight. It's also not clear how much log data they can ingest, whether end-user telemetry was feasible, etc...
  - thrower123 7y ago
    On the hardest difficulty level, companions do die often, as they aren't very smart and are quite squishy.
  - rgoulter 7y ago
    The debugging is to reconcile "Observation: quests are failing because companions die" which contradicts "My understanding of the game rules: companions can't die". One of the ways the companions don't die is that the companions are revived after combat, and it's only non-companions that are 'really dead'. The other way is "could die from fall damage on the ship, but there's nothing that high". Turns out the flaw in the logic was that the NPCs can reach great height on the ship. But the flaw in the logic just as easily could have been in the combat. (Or some other way that the tweeter didn't mention). In any case, it's really fun to read the "aha! got it" moment.
  - thaumasiotes 7y ago
    Their non-logging-based investigation got them to the point that they were aware the problem occurred when a companion fell from a great height while the player was on their ship: > The only place in the game when a companion is present but not in the active party is when the player is on their ship > Eventually we figured out that "undamageable" does not mean "invulnerable" -- they can't take damage from attacks but can still get hurt from other things > One of those things: falling a great distance > The problem with that is that there are no spots in the player's ship that are high enough to result in a lethal fall > So now we had to figure out how companions were mysteriously ending up way above the level > I looked into tons of theories This does look like logging would have caught it. They knew exactly what they were looking for: companions reaching strange heights while the player is on their ship.
aneutron 7y ago
You generally do not enable the level of logging that I imagine would be necessary to find this bug, on the enduser version of the software. I could be wrong.
- netsharc 7y ago
  Surely it could be enabled with a runtime flag?
  - stouset 7y ago
    You would need to know to enable it before your companion dies, and thus before the bug is even encountered.
    - gmueckl 7y ago
      As another comment noted, you would log into a ring buffer in memory constantly and only dump it after the event in question happened. But without an initial state at the start od the log, this may not be enough.
- girvo 7y ago
  I wonder if that level of logging would have a performance impact, especially on consoles?
  - wyldfire 7y ago
    "logging"/telemetry can be flight-data-recorder style like XRay uses [1]. This can be limited to a smaller window because it writes to a circular buffer, then again you could fit a lot of entries in a ~512MB buffer. [1] https://llvm.org/docs/XRay.html#flight-data-recorder-mode https://llvm.org/docs/XRay.html#flight-data-recorder-mode

deleted 7y ago

[deleted]

SkyPuncher 7y ago

They knew what to observe for, but they couldn't figure out how to reproduce that.

madrox 7y ago

This is why it's important to log everything and keep it around forever!