4 ms·

Checklist are what Danial is complaining about. NVD saw the vuln, thought "Oh, I think that fits in the 'authentication bypass' checklist entry", and ranked it

Checklist are what Danial is complaining about. NVD saw the vuln, thought "Oh, I think that fits in the 'authentication bypass' checklist entry", and ranked it as high severity. Danial is saying that no, the curl developers don't need a dumb checklist because they know the vulnerability and code, they know it's not severe. Checklists are getting in the way of reality. Checklists are a tool you use to organize a process, to organize information that exists. They're not a good tool for gaining that knowledge in the first place. You get good estimates not by having a checklist that says "Oh, if it's in javascript, the checklist says add 3 weeks, if it's in fortran, the checklist says add 6 weeks", you get good at estimating by actually understanding the exact work that needs to be done. Unfortunately, no one ever understands the actual work that needs to be done. You get good security ratings by understanding the vulnerability, not by having a checklist that says "if there's memory corruption, it's severe".

turtleyacht 3y ago
Thank-you. Yes, checklists that obscure nuance or try to shortcut understanding are bad. Checklists created in the absence of context, or even worse, without the same folks to evaluate them (developers themselves!) are next to useless. Regarding estimation, yes--I agree those examples are bad. I do know you could come up with effective ones too, though. Checklists should reflect the domain, some reasonable path that a person of training could use. Maybe one of the criteria of a good checklist should also include, "This checklist is terrible and should be thrown out; in the absence of Not Applicable as a choice, or even Too Nuanced; Time Ran Out choices, here is an alternative: _____." I don't think NVD should use one checklist, but maybe they could use it as part of an overall toolkit, one that also includes--time permitting--a detailed investigation of the subtleties behind a less severe "authentication bypass" and a more severe one. (After all, in agile estimation, points are relative to previous story estimates, given the same tuple of [Team, Product, Sprint, Story].) I do feel it is unfair to be dinged by a separate org. Maybe high severity ought to be reserved for when there is mutual agreement without optics. Because to NVD it's just another item, but to cURL org it impacts their metrics too. On the other hand, there may be bad actors trying to game that. > You get good security ratings by understanding the vulnerability Yes. Maybe NVD needs more visibility by rating certain vulnerabilities as "High Severity (Contested: Low)," if only to underscore their limited resources. Everyone knows how important cURL is, and how unhelpful it is to attract developer ire. Maybe we cannot label a vulnerability on a single axis alone. I wish I could describe the problem: a proof where a mapping of multiple attributes cannot be described by any one in the same set. > Checklists... use to organize a process... organize information... not a good tool for gaining that knowledge in the first place I agree. Even an outline, 5 Whys, fishbone, or lots of other tools. Run the branch locally, try to repro the bug. Lots of ways. But if NVD's reason-for-being is to assess things, not necessarily to fix them, that may also need to be considered in their findings. At the end, before assigning the final severity, go through one (or more) checklists to ensure nothing is missed, and to shake out contradictions or inconsistencies. It would also be good to allow a response window for developers. If it's not as bad as all that, use that response. If it reeks of damage control, stick to your guns.

turtleyacht 3y ago

Thank-you. Yes, checklists that obscure nuance or try to shortcut understanding are bad. Checklists created in the absence of context, or even worse, without the same folks to evaluate them (developers themselves!) are next to useless. Regarding estimation, yes--I agree those examples are bad. I do know you could come up with effective ones too, though. Checklists should reflect the domain, some reasonable path that a person of training could use. Maybe one of the criteria of a good checklist should also include, "This checklist is terrible and should be thrown out; in the absence of Not Applicable as a choice, or even Too Nuanced; Time Ran Out choices, here is an alternative: _____." I don't think NVD should use one checklist, but maybe they could use it as part of an overall toolkit, one that also includes--time permitting--a detailed investigation of the subtleties behind a less severe "authentication bypass" and a more severe one. (After all, in agile estimation, points are relative to previous story estimates, given the same tuple of [Team, Product, Sprint, Story].) I do feel it is unfair to be dinged by a separate org. Maybe high severity ought to be reserved for when there is mutual agreement without optics. Because to NVD it's just another item, but to cURL org it impacts their metrics too. On the other hand, there may be bad actors trying to game that. > You get good security ratings by understanding the vulnerability Yes. Maybe NVD needs more visibility by rating certain vulnerabilities as "High Severity (Contested: Low)," if only to underscore their limited resources. Everyone knows how important cURL is, and how unhelpful it is to attract developer ire. Maybe we cannot label a vulnerability on a single axis alone. I wish I could describe the problem: a proof where a mapping of multiple attributes cannot be described by any one in the same set. > Checklists... use to organize a process... organize information... not a good tool for gaining that knowledge in the first place I agree. Even an outline, 5 Whys, fishbone, or lots of other tools. Run the branch locally, try to repro the bug. Lots of ways. But if NVD's reason-for-being is to assess things, not necessarily to fix them, that may also need to be considered in their findings. At the end, before assigning the final severity, go through one (or more) checklists to ensure nothing is missed, and to shake out contradictions or inconsistencies. It would also be good to allow a response window for developers. If it's not as bad as all that, use that response. If it reeks of damage control, stick to your guns.