3 ms·

Sounds like the company realised they can't solve the issue in 90 days. Betting a combination of infrastructure scale problems, terrible tech, no-longer-buildin

Sounds like the company realised they can't solve the issue in 90 days. Betting a combination of infrastructure scale problems, terrible tech, no-longer-building old solutions, no maintenance fee's built in and contractors who hate them. So they pulled the only lever they had left, which was the lawyers. Same time, RedThreat's email was kinda (maybe rightly) hostile. Read from the other side it's basically "You have 90 days to work (/maybe pay) me before you start hearing your name on TickTok under the label 'wanna hack the city?'". "Work with your team" leaves a ton of negotiating opportunity for a company that obviously does this for a living and expects to make money somewhere.

samtho 2y ago
The 90 day window is an industry standard for zero-days, how the author worded it is neither here nor there. 90 days is ample time for even a half-functioning organization to address the issue in some way. I agree with Red Threat’s decision to not show their hand in the first email. The altruistic take on this is that they do not want the email to fall upon deaf ears (or even a bad actor within a company) and would prefer to have a channel of communication open with the security team before outlining the details. The biggest problem when faced with a zero-day is that it’s unknown who else knows about it. This helps the the company’s security team justify the work due to the fire lit under the company to take action - especially if their corporate structure does not allow for more “elective” fixes.
- mike_hearn 2y ago
  90 days isn't really an industry standard. It's what Google unilaterally decided upon when they chose to staff up Project Zero and reflects their assumptions being, as they are, a company that grew up on the web. One of those assumptions is that you have fully automated test processes and the ability to remotely update any/all installs of your software within a week or so, which in turn implies that users aren't involved in the decision about whether to upgrade or not. It also assumes you can do this as often as you want. This is a strong set of assumptions that happens to be true for Google but isn't true of SCADA shops. Even if they make a patched firmware, actually rolling it out would require a lot of work by their customers and of course maybe the same guy finds another security bug after 80 days and the whole thing starts again. Given the unclear threat model here (how does one get access to the networks that these are attached to? could you just hotwire the lights themselves and bypass the controller?) it's also not really obvious how you'd classify reports. If there's a bypassable HTTP login page that's clearly an exploit but customers may not care if they trust the underlying VPNs/firewalls/air gaps. If there's unauthenticated SNMP access by design then is it even intended to be secure against malicious network access at all? In many cases these devices will be reachable from the public internet, and in some of those cases it will be intentional. But is the security bug there on the controller or in the network setup that allows that access? It's probably easier to properly firewall off the controllers than continuously patch all the controller firmwares themselves, especially as the latter done wrong could easily enable hackers to perform a worse-than-Crowdstrike level takedown of all controllers simultaneously. It's really not clear that the model that works well for web browsers will ever work well for infrastructure. We just saw an awesome demo of what can go wrong when rapidly hot-patching security updates into critical infrastructure computers goes wrong.
  - wolrah 2y ago
    > 90 days isn't really an industry standard. It's what Google unilaterally decided upon when they chose to staff up Project Zero Every published vulnerability disclosure policy I've found in a quick look around has a 90 or 120 day timer involved somewhere. There may be some variation in exact details but there's not significant disagreement in the industry aside from those who don't want any firm time limits at all. Also to be specific Google's policy is actually 90+30, you have 90 days to release a patch and as long as you do that details will be withheld an additional 30 days from the release of the patch. There is also an option for a 14 day grace period on the patch release if a vendor has been working in good faith and Google has reason to believe they will actually get it done in that time. > and reflects their assumptions being, as they are, a company that grew up on the web. One of those assumptions is that you have fully automated test processes and the ability to remotely update any/all installs of your software within a week or so, If you are competent software developers in 2024 you have fully automated test processes and if your things are expected to be connected to the internet you should have the ability to remotely update them. These are not things that anyone has any excuse to not understand. I am well aware that OT equipment vendors are often absolute horror shows from a software development standpoint, that's not an excuse for anything. If they can't do things right they deserve everything that happens to them and their customers should hold them accountable for the inevitable result. > which in turn implies that users aren't involved in the decision about whether to upgrade or not. It also assumes you can do this as often as you want. This is a strong set of assumptions that happens to be true for Google but isn't true of SCADA shops. It's an assumption that has to be true of anything connected to the open internet. If a bad actor discovers the exploit and starts using it against exposed systems you won't have a choice but to patch it yesterday, at which point a 90+30 deadline will feel like all the time in the world. The simple answer of course is if for whatever reason you can not patch the thing in such a timeframe it should never be connected to the internet and connections between internet-connected systems and the private network should be severely restricted and heavily monitored for unusual activity. At that point you don't care about whether exploit details are public because you know every single person who could potentially implement it. > Given the unclear threat model here (how does one get access to the networks that these are attached to? The article also mentions this, but you answer your own question a paragraph later. > In many cases these devices will be reachable from the public internet, and in some of those cases it will be intentional. And again the fact is if it is on the internet it must be rapidly patchable. If short-notice patching is not viable for your use case then don't put it on the internet. Very simple, no exceptions. > But is the security bug there on the controller or in the network setup that allows that access? Yes. If the intentional access controls can be bypassed that's a bug in the controller, but if the OT device is accessible to the general internet in the first place that's a bug in the network setup. > It's probably easier to properly firewall off the controllers than continuously patch all the controller firmwares themselves, especially as the latter done wrong could easily enable hackers to perform a worse-than-Crowdstrike level takedown of all controllers simultaneously. Also yes. > It's really not clear that the model that works well for web browsers will ever work well for infrastructure. The model is "if it is exposed to the internet and there is a remotely exploitable vulnerability known it must be either patched or not exposed to the internet anymore". It doesn't matter what the thing is, be it browsers, infrastructure, medical, etc. Either take it off the internet or be ready and willing to patch it on short notice. > We just saw an awesome demo of what can go wrong when rapidly hot-patching security updates into critical infrastructure computers goes wrong. I'd argue that was more of a demo of what will go wrong when you don't have automated testing and why you should always have staged deployment when doing things at scale. That said, I'll return to the same point, how much infrastructure that wasn't connected to the internet was affected? Every system that was affected was allowed to download software from the internet controlled by a third party. This discussion reminds me of the recent discussion around Entrust where a lot of the excuses were around certs being used in places where they could not be easily rotated, which led to the obvious question of "what would these users have done in the event of a key compromise?" having no good answers. When you're using internet infrastructure you need to be able to move quickly from time to time.
- burningChrome 2y ago
  I work in accessibility as an engineer. Now that digital web properties are included the ADA, there are law firms out there essentially doing the same thing. They are actively scanning the internet to find companies who have accessibility issues which are primarily widgets or overlays. Then they're emailing them the issues and "allowing" them 90 days to correct the issues, otherwise, they will be sued. There's been like a 400% increase in these suits over the last two years because you can go after a large company and even if they fix one issue, you can find another issue and sue or threaten them on that one as well. My co-workers think its great because of the pressure to solve these issues that really do need to be fixed. But like in this instance, its a fine line between doing something positive and extorting money for yourself or in our case, a law firm.

samtho 2y ago

The 90 day window is an industry standard for zero-days, how the author worded it is neither here nor there. 90 days is ample time for even a half-functioning organization to address the issue in some way. I agree with Red Threat’s decision to not show their hand in the first email. The altruistic take on this is that they do not want the email to fall upon deaf ears (or even a bad actor within a company) and would prefer to have a channel of communication open with the security team before outlining the details. The biggest problem when faced with a zero-day is that it’s unknown who else knows about it. This helps the the company’s security team justify the work due to the fire lit under the company to take action - especially if their corporate structure does not allow for more “elective” fixes.

mike_hearn 2y ago
90 days isn't really an industry standard. It's what Google unilaterally decided upon when they chose to staff up Project Zero and reflects their assumptions being, as they are, a company that grew up on the web. One of those assumptions is that you have fully automated test processes and the ability to remotely update any/all installs of your software within a week or so, which in turn implies that users aren't involved in the decision about whether to upgrade or not. It also assumes you can do this as often as you want. This is a strong set of assumptions that happens to be true for Google but isn't true of SCADA shops. Even if they make a patched firmware, actually rolling it out would require a lot of work by their customers and of course maybe the same guy finds another security bug after 80 days and the whole thing starts again. Given the unclear threat model here (how does one get access to the networks that these are attached to? could you just hotwire the lights themselves and bypass the controller?) it's also not really obvious how you'd classify reports. If there's a bypassable HTTP login page that's clearly an exploit but customers may not care if they trust the underlying VPNs/firewalls/air gaps. If there's unauthenticated SNMP access by design then is it even intended to be secure against malicious network access at all? In many cases these devices will be reachable from the public internet, and in some of those cases it will be intentional. But is the security bug there on the controller or in the network setup that allows that access? It's probably easier to properly firewall off the controllers than continuously patch all the controller firmwares themselves, especially as the latter done wrong could easily enable hackers to perform a worse-than-Crowdstrike level takedown of all controllers simultaneously. It's really not clear that the model that works well for web browsers will ever work well for infrastructure. We just saw an awesome demo of what can go wrong when rapidly hot-patching security updates into critical infrastructure computers goes wrong.
- wolrah 2y ago
  > 90 days isn't really an industry standard. It's what Google unilaterally decided upon when they chose to staff up Project Zero Every published vulnerability disclosure policy I've found in a quick look around has a 90 or 120 day timer involved somewhere. There may be some variation in exact details but there's not significant disagreement in the industry aside from those who don't want any firm time limits at all. Also to be specific Google's policy is actually 90+30, you have 90 days to release a patch and as long as you do that details will be withheld an additional 30 days from the release of the patch. There is also an option for a 14 day grace period on the patch release if a vendor has been working in good faith and Google has reason to believe they will actually get it done in that time. > and reflects their assumptions being, as they are, a company that grew up on the web. One of those assumptions is that you have fully automated test processes and the ability to remotely update any/all installs of your software within a week or so, If you are competent software developers in 2024 you have fully automated test processes and if your things are expected to be connected to the internet you should have the ability to remotely update them. These are not things that anyone has any excuse to not understand. I am well aware that OT equipment vendors are often absolute horror shows from a software development standpoint, that's not an excuse for anything. If they can't do things right they deserve everything that happens to them and their customers should hold them accountable for the inevitable result. > which in turn implies that users aren't involved in the decision about whether to upgrade or not. It also assumes you can do this as often as you want. This is a strong set of assumptions that happens to be true for Google but isn't true of SCADA shops. It's an assumption that has to be true of anything connected to the open internet. If a bad actor discovers the exploit and starts using it against exposed systems you won't have a choice but to patch it yesterday, at which point a 90+30 deadline will feel like all the time in the world. The simple answer of course is if for whatever reason you can not patch the thing in such a timeframe it should never be connected to the internet and connections between internet-connected systems and the private network should be severely restricted and heavily monitored for unusual activity. At that point you don't care about whether exploit details are public because you know every single person who could potentially implement it. > Given the unclear threat model here (how does one get access to the networks that these are attached to? The article also mentions this, but you answer your own question a paragraph later. > In many cases these devices will be reachable from the public internet, and in some of those cases it will be intentional. And again the fact is if it is on the internet it must be rapidly patchable. If short-notice patching is not viable for your use case then don't put it on the internet. Very simple, no exceptions. > But is the security bug there on the controller or in the network setup that allows that access? Yes. If the intentional access controls can be bypassed that's a bug in the controller, but if the OT device is accessible to the general internet in the first place that's a bug in the network setup. > It's probably easier to properly firewall off the controllers than continuously patch all the controller firmwares themselves, especially as the latter done wrong could easily enable hackers to perform a worse-than-Crowdstrike level takedown of all controllers simultaneously. Also yes. > It's really not clear that the model that works well for web browsers will ever work well for infrastructure. The model is "if it is exposed to the internet and there is a remotely exploitable vulnerability known it must be either patched or not exposed to the internet anymore". It doesn't matter what the thing is, be it browsers, infrastructure, medical, etc. Either take it off the internet or be ready and willing to patch it on short notice. > We just saw an awesome demo of what can go wrong when rapidly hot-patching security updates into critical infrastructure computers goes wrong. I'd argue that was more of a demo of what will go wrong when you don't have automated testing and why you should always have staged deployment when doing things at scale. That said, I'll return to the same point, how much infrastructure that wasn't connected to the internet was affected? Every system that was affected was allowed to download software from the internet controlled by a third party. This discussion reminds me of the recent discussion around Entrust where a lot of the excuses were around certs being used in places where they could not be easily rotated, which led to the obvious question of "what would these users have done in the event of a key compromise?" having no good answers. When you're using internet infrastructure you need to be able to move quickly from time to time.
burningChrome 2y ago
I work in accessibility as an engineer. Now that digital web properties are included the ADA, there are law firms out there essentially doing the same thing. They are actively scanning the internet to find companies who have accessibility issues which are primarily widgets or overlays. Then they're emailing them the issues and "allowing" them 90 days to correct the issues, otherwise, they will be sued. There's been like a 400% increase in these suits over the last two years because you can go after a large company and even if they fix one issue, you can find another issue and sue or threaten them on that one as well. My co-workers think its great because of the pressure to solve these issues that really do need to be fixed. But like in this instance, its a fine line between doing something positive and extorting money for yourself or in our case, a law firm.