3 ms·
If the audit was only run once as a exploratory measure, then you are correct, but that is not the case. If you run a audit intended to detect defects at the 1
by Veserv 3y ago
If the audit was only run once as a exploratory measure, then you are correct, but that is not the case. If you run a audit intended to detect defects at the 1 M$ level 10 times and literally every single time you find truckloads of defects then you have statistical evidence on the quality of your processes.
As a simpler example, suppose you did a firing test of a bulletproof vest and the bullet went right through. They patch that up and then you fire again and it goes right through. They do that 10 times and literally every single time it goes right through. You can conclude that those failures were not a fluke, their bulletproof vest is just garbage.
To use a slightly more complex variant, suppose they had bulletproof vest version 1 and during acceptance testing the bullet goes right through. They go back to the drawing board and produce bulletproof vest version 2 claiming it is bulletproof and then the bullet goes right through again. They do this 10 times and literally every time they claimed it was bulletproof and every time the bullet goes right through. You can conclude that their engineering or validation processes are garbage and, additionally, you can only trust them as far as you can throw them. Until they conclusively demonstrate robust protection against bullets in representative tests their products and "expertise" should be ignored.
The thing in the cybersecurity industry is that they fail the audit every time across all fields across all products across all systems. This is not a one-off case where a company has never done it before and they failed once so it might be a fluke or they can improve. This is everybody doing thousands of audits each and literally every single one of them has never once demonstrated a defect rate better than the low M$ range no matter how many changes or "improvements" they make. And the entire time they have been issuing deceptive marketing making people think they can. That is about as clear cut statistical evidence of industry-wide quality control failures as you can find in any field.
Just for reference, this is extremely abnormal in other industries. Only in cybersecurity and software are poor quality control and validation processes so rampant that obvious global process failures are viewed as just the way things are done. Yes, some parts of the software industry are even worse, but even the "best" parts are laughably inadequate. Engineering is not about subjective evaluations like better, it is about objective evaluations like fit for purpose. And commercial IT cybersecurity fails that with a resounding no.
- klabb3 3y agoThanks for following up. I agree, however say Google and Apple have very few of these despite being disproportionately targeted. Doesn’t mean they are doing enough. But it sure as hell makes a difference. In ~15 years my gmail has never been leaked or hacked. That’s better than having an unprotected DB leaking every customers credit score, like we’ve seen. Certainly this audit-, certification- and checklist based security is a joke. Security isn’t a patch-work. It’s not separable from software development, and cannot be a separate process/workstream either (a thousand faint gasps ring out from middle managers across the enterprise world). Anyway, I’m rambling. I guess I wanted to say that I’m optimistic for the long term. We’ve already made some significant progress in the last 10 years, for the better. I expect that to continue slowly.