3 ms·

Is it just me or does it seem like this change simply wasn't tested beyond a simple unit test?

• sylens 2y ago
  My thoughts as well.
• acdha 2y ago
  The big thing I was wondering is what their coverage analysis is like. I can see a developer missing this in a hurry, but where’s the review or second-order analysis? Tons of projects with far less importance monitor branch coverage, use fuzz testing and path analysis tools, etc. and while it’s not trivial to test a kernel driver it’s not _that_ hard, especially when you have the resources of a company valued in the tens of billions which allegedly specializes in exactly this kind of work. The thing I’ve been thinking about are all of the assurances they made about SDLC, testing, secure development practices, etc. They have so many huge customers in regulated industries, government, etc. that they completed almost every certification in existence and seeing this really raises questions about how those assertions were reviewed.
• darylteo 2y ago
  I posit that there are multiple disparate teams involved. -- Team 1 tells Team 2 that the schema is updating. Team 2 updates their schema. Team 2 tests against updated schema (which would be a test file) All green in test. Team 1 doesn't actually follow the schema. Deployment fails.
  • armalis 2y ago
    It’s worse than that. They updated the schema, and tested it with previous data that does not exercise the new parameter. Tests are passing. When they go and actually use the new parameter, it crashes. The new schema was improperly tested (among a list of other failures).