5 ms·

You can have all the CI, staging, test, etc. If some bug after that process nulls the file, the rest doesn't matter

Jtsummers 2y ago
If a garbage file is pushed out, the program could have handled it by ignoring it. In this case, it did not and now we're (the collective IT industry) dealing with the consequences of one company that can't be bothered to validate its input (they aren't the only ones, but this is a particularly catastrophic demonstration of the importance of input validation).
- russdill 2y ago
  I'll agree that this appears to have been preventable. Whatever goes through CI should have a hash, deployment should validate that hash, and the deployment system itself should be rigorously tested to insure it breaks properly if the hash mismatches at some point in the process
fabian2k 2y ago
Those signature files should have a checksum, or even a digital signature. I mean even if it doesn't crash the entire computer, a flipped bit in there could still turn the entire thing against a harmless component of the system and lead to the same result.
- HL33tibCe7 2y ago
  What happens when your mechanism for checksumming doesn't work? What happens when your mechanism for installing after the checksum is validated doesn't work? It's just too early to tell what happened here. The likelihood is that it _was_ negligence. But we need a proper post-mortem to be able to determine one way or another.
LorenPechtel 2y ago
Yup. I had quite a battle with some sort of system bug (never fully traced) where I wrote valid data but what ended up on disk was all zero. It appeared to involve corrupted packets being accepted as valid. It doesn't matter how much you test if something down the line zeroes out your stuff.
Cerium 2y ago
What sort of sane system modifies the build output after testing? Our release process is more like: build and package, sign package, run CI tests on signed package, run manual tests on signed package, release signed package. The deployment process should check those signatures. A test process should by design be able to detect any copy errors between test and release in a safe way.
jononor 2y ago
The issue is not that a file with nulls was produced. It is that an invalid file (or any kind) can trigger a blue screen of death.

If a garbage file is pushed out, the program could have handled it by ignoring it. In this case, it did not and now we're (the collective IT industry) dealing with the consequences of one company that can't be bothered to validate its input (they aren't the only ones, but this is a particularly catastrophic demonstration of the importance of input validation).

russdill 2y ago
I'll agree that this appears to have been preventable. Whatever goes through CI should have a hash, deployment should validate that hash, and the deployment system itself should be rigorously tested to insure it breaks properly if the hash mismatches at some point in the process

fabian2k 2y ago

Those signature files should have a checksum, or even a digital signature. I mean even if it doesn't crash the entire computer, a flipped bit in there could still turn the entire thing against a harmless component of the system and lead to the same result.

HL33tibCe7 2y ago
What happens when your mechanism for checksumming doesn't work? What happens when your mechanism for installing after the checksum is validated doesn't work? It's just too early to tell what happened here. The likelihood is that it _was_ negligence. But we need a proper post-mortem to be able to determine one way or another.

LorenPechtel 2y ago

Yup. I had quite a battle with some sort of system bug (never fully traced) where I wrote valid data but what ended up on disk was all zero. It appeared to involve corrupted packets being accepted as valid. It doesn't matter how much you test if something down the line zeroes out your stuff.

Cerium 2y ago

What sort of sane system modifies the build output after testing? Our release process is more like: build and package, sign package, run CI tests on signed package, run manual tests on signed package, release signed package. The deployment process should check those signatures. A test process should by design be able to detect any copy errors between test and release in a safe way.

jononor 2y ago

The issue is not that a file with nulls was produced. It is that an invalid file (or any kind) can trigger a blue screen of death.