7 ms·

Things one could do that would increase security without hindering business goals: - PATCHING OF KNOWN CRITICAL SECURITY VULNERABILITIES IMMEDIATELY. This is

Things one could do that would increase security without hindering business goals: - PATCHING OF KNOWN CRITICAL SECURITY VULNERABILITIES IMMEDIATELY. This is an architectural decision! If you cannot put out a hotfix within months of a vulnerability being made public, YOUR APPLICATION & DEPLOYMENT DESIGNS ARE BROKEN. - WAF rules for common attacks and known CVEs - Least privilege enforced architecturally via stored procedures or microservices or something else so that a vulnerability in one application (the online dispute portal in this case) can't lead to a dump of everyone's PII. - Data should be encrypted and encryption keys should be stored apart from both the application and the data. Upon startup, the application should make an authenticated request to read the encryption keys into memory. This significantly raises the bar to dumping sensitive data. - Monitoring for connections transferring anomalous amounts of data That's just off the top of my head. All of these can be designed into a system and still maintain 100% uptime.

wavegeek 9y ago
> PATCHING OF KNOWN CRITICAL SECURITY VULNERABILITIES IMMEDIATELY. I worked in system admin for many years. If anyone was actually foolish enough to do this, they would be dealing with multiple major outages every year. > for every complex problem there is a solution that is simple, obvious, and wrong.
- georgebarnett 9y ago
  "There might be an outage" is the stupidest reason I have ever heard to advocate not applying security patches. There's plenty of companies that are able to regularly patch and upgrade software, incident free. If you team isn't able to do this, they need to fix their process instead of putting their head in the sand.
  - wavegeek 9y ago
    > stupid I am not defending the totality of EFX's process by any means. I am just pointing out that "install every high severity patch right away" is not as easy as it looks. Others have pointed out this patch was not available for all current versions, for example. So, to install the patch you need to upgrade. Oh, that breaks <dependency>. So we need to upgrade <other thing>. But that regresses a feature we use .... The major issue with EFX was the fragility of their overall architecture. A single weakness should not be enough to lay the whole DBMS open.
    - speedplane 9y ago
      The tech industry has to move more towards declarative systems, rather than procedural ones. Docker, Kubernetes and similar tech can auto-update in the background without any downtime. It's impossible to maintain thousands of VMs without some sort of declarative system.
      - qaq 9y ago
        Each thing you are listing increases complexity and attack surface.
        speedplane 9y ago
        It actually decreases complexity. Updating 1000 VM servers using these technologies can be made automatic. True, it provides an additional attack surface, but the gains of having everything always patched are well worth it.
        qaq 9y ago
        Everything does not include your actual app, so yes it's convenient but doesn't really help that much
        speedplane 9y ago
        I'd argue that convenience is a security feature. The easier something is to do, the fewer mistakes you or your team will make. Stand by my initial post, declarative systems will become more and more popular and will improve security.
      - thehardsphere 9y ago
        What is the non-tech industry supposed to do?
    - georgebarnett 9y ago
      There are great reasons for not installing patches right away. "There might be an outage" is not one of them.
  - edwhitesell 9y ago
    I don't recall being at, or working with, a company where this wasn't _the_ reason patches were delayed at least some period if time. Companies are driven by profit; unknown patches can negatively affect profit and are usually treated as such. Alternatively, there are sometimes political reasons. A director may not want to risk an outage for fear of being dressed down by their VP. Any change to a production environment comes with risks, sometimes having a security risk is judged to be lower risk than making a change. I've seen this from small startups to multi-billion dollar companies. Ignoring these factors is also putting your head in the sand.
    - dopamean 9y ago
      > Alternatively, there are sometimes political reasons. A director may not want to risk an outage for fear of being dressed down by their VP. This is such a weird thing to me. I've never worked anywhere like this and so I have a hard time imagining it really. Wouldn't that same director get more than dressed down by the VP if on their watch a _known security vulnerability_ was ignored and the system was breached? I just can't understand this way of thinking.
      - thehardsphere 9y ago
        It's easy to understand once you get just how simplistic it is. The VP will deliver a dressing down if the company loses money and the blame can be placed on the director. The cost of an outage is very easy to quantify (revenue per minute the system is down), and the probability that something will go wrong while applying the patch is also somewhat easy to predict, and usually greater than zero. The director will be blamed with certainty for the outage, since he approved it. The cost of a security breach is difficult to quantify; it depends on what gets breached and how bad. Note here, I say breach, not vulnerability. Even if there is a known security vulnerability, it's not immediately obvious in all systems what the consequence will be; there may be other mitigations in place outside of software that reduce the potential damage, or there may be unknown vulnerabilities that are exploitable due to the known vulnerability that make would make a breach worse. The lack of certainty about the consequences means it's also possible for the director to avoid blame if the breach is minor ("how was I supposed to know that other team is still using MD5?"). If there is no breach, then there is nothing for the director to be blamed for. Given that the director would like to avoid being dressed down, director will be more inclined to delay patching over possibly causing an outage, because the costs of an outage are easy to predict and he will take all blame for it. The breach may never happen and even if it does, it may cost him personally less than the outage. If this still seems weird, it might be because you are someone who views patching as an easy thing to do, because you probably work for a software company. Software companies are used to managing changing software, and have all kinds of practices around minimizing the risks of doing so. Non-software companies typically find patching to be hard and costly because their core business is something else; changes can disturb the "something else."
- cookiecaper 9y ago
  While you may sort of have a point for low-severity issues (if you patch every one ASAP, you are constantly restarting; should be tested in a group in stage and released to prod on a regular timeline), this was a 10.0 severity CVE and it is _definitely_ the kind of issue that you tolerate some downtime for. Check out the CVE, this could be exploited by anyone with a basic knowledge of cURL in 60 seconds or less. The vulnerability was just a matter of sending an arbitrary command in an HTTP header.
- notzorbo3 9y ago
  You need to do some testing of course. "IMMEDIATELY" doesn't mean within 5 seconds. If you can't properly apply security updates in a reasonable time frame without it leading to major outages, something is extremely wrong with either the architecture, the chosen software or the QA and deployment strategy. > without it leading to major outages I think I would have preferred an outage over leaking millions of american's private information.
  - wavegeek 9y ago
    > I think I would have preferred an outage over leaking millions of american's private information. With full 100% hindsight vision, of course.
    - muninn_ 9y ago
      No you don't need hindsight for that. Protecting the data is more important than anything else. It's negligent to not do so.
    - mi100hael 9y ago
      It's not the first time this has happened to a company, and it surely won't be the last. We've already had full 100% hindsight vision a dozen times over, and plenty of companies have chosen to ignore it.
jldugger 9y ago
> This is an architectural decision It's also a business decision. You need to invest staff time in testing and automating said updates, and will inevitably lead to more outages. Thus less profits, directly contrary to your assertion. Managers have trade-off between a small but relatively predictable cost, and a large but presumed highly improbable cost. This is something that neither data nor human intuition are particularly good with. Best case scenario insurance lets you convert your large risk into a small predictable cost, and the premium difference decides whether you stay on the CVE treadmill or not. But its not like actuaries have a lot of data to model to the point of accurately imposing a cost per day of unpatched CVE. Worse, many places have competitors, and if they're choosing to take on risks in the name of lower prices and higher market share, the customers of your secured app walk out the door. Not only do you need to invest in security, you need the competition to as well. I have a few ideas on how to achieve this, but they're not historically palatable to the HN crowd.
- siculars 9y ago
  I think it's worth it when the alternative are multi billion dollar lawsuits and government regulation.
- jacques_chester 9y ago
  > It's also a business decision. You need to invest staff time in testing and automating said updates, and will inevitably lead to more outages. There are many vendors attacking this exact problem from many directions, including my employers[0], but pretty much every platform vendor has something here. I do agree that automation and risk are at the root of the tension between security and stability. It's a feedback loop that can run in two directions. If you embrace stability, the loop becomes "no changes ever" and security becomes much more difficult to achieve in practice. If you embrace security, then at first you face instability -- but embracing rapidity and building out tooling for it makes you both more secure and more stable in the long run. > But its not like actuaries have a lot of data to model to the point of accurately imposing a cost per day of unpatched CVE. I'd be interested to know if this is actually true. I recently got interested in the FAIR risk modelling approach. It requires a bunch of legwork, so in practice it won't catch on without determination to use it or something like it. Usefully it breaks risk down into more easily-estimated, easily-tracked subcomponents[1]. "Probability" becomes "Loss Event Frequency", which further breaks down into Threat Event Frequency and Vulnerability, these break down further in turn. Similarly, the usually-vague "Impact" becomes Primary Loss and Secondary Loss, each of which has 6 specific types that can be estimated. Those estimates can be based on available data, whether your own or industry statistics. > Worse, many places have competitors, and if they're choosing to take on risks in the name of lower prices and higher market share, the customers of your secured app walk out the door. [0] I work for Pivotal. We've been building towards the vision of "Rotate, Repair, Repave" in Cloud Foundry: https://builttoadapt.io/the-three-r-s-of-enterprise-security-rotate-repave-and-repair-f64f6d6ba29d https://builttoadapt.io/the-three-r-s-of-enterprise-security... [1] https://cdn2.hubspot.net/hubfs/1616664/The%20FAIR%20Model_FINAL_Web%20Only.pdf?t=1505484555066 https://cdn2.hubspot.net/hubfs/1616664/The%20FAIR%20Model_FI...
gaius 9y ago
- PATCHING OF KNOWN CRITICAL SECURITY VULNERABILITIES IMMEDIATELY. Immediately? No real company can do this. 1-2 weeks after the announcement is the best you'll see, even with something of the magnitude and publicity of Heartbleed. I'm not saying that's how it should be but that's how it is. Most organisation's it's quarterly! All of these can be designed into a system and still maintain 100% uptime Sure. If you were designing from a clean slate right now. The vast majority of organisations in the world aren't. And believe me: in 10 or 20 years your new design would have revealed flaws too, and look laughably primitive to the practitioners of the day. What's the solution then? An architecture in which no single compromise can compromise the entire system. That's getting harder and harder to do as the landscape homogenises around Lintel.
- notzorbo3 9y ago
  > No real company can do this. 1-2 weeks after the announcement is the best you'll see You don't need to patch your software right away, but you should have other strategies for mitigation such as a WAP in front of your services that you can configure to block malicious traffic when you identify a remote code execution vulnerability.
  - gaius 9y ago
    Or two, by two different vendors, ideally sharing no code whatsoever.
- grogenaut 9y ago
  release: https://nvd.nist.gov/vuln/detail/CVE-2014-0160 https://nvd.nist.gov/vuln/detail/CVE-2014-0160 (04/07/2014) patch: https://aws.amazon.com/security/security-bulletins/heartbleed-bug-update/ https://aws.amazon.com/security/security-bulletins/heartblee... (4/8/2914) IIRC: 12 or 18 hours rolled out to all of aws ELB (main part affected). https://usn.ubuntu.com/usn/usn-2165-1/ https://usn.ubuntu.com/usn/usn-2165-1/ Patch was available at time of vulnerability release for current and LTS releases of ubuntu. apt-get update. But you know... That's just too hard.
  - 9y ago
agiamas 9y ago
seriously, FFS these are guidelines that we use for not-remotely-that-sensitive-data and equifax that literally... "You had one job to do, protect our data" Can't follow? This is just business being a) ignorant b) incompetent c) careless one or more of the above applied for a hack like this to happen. It's really sad that even ycombinator crowd doesn't get it, given that the crowd here is significantly more technical and sensitive to these issues than the average Equifax Joe..
- thehardsphere 9y ago
  No, we get cases A, B, and C, because those cases are really easy for anyone to understand. What you're missing is that some of the "ycombinator crowd" have worked with enough "average Equifax Joes" enough to know that the problems faced by "Equifax Joe" are not always solvable by merely being knowledgeable, competent, and careful.

wavegeek 9y ago

> PATCHING OF KNOWN CRITICAL SECURITY VULNERABILITIES IMMEDIATELY. I worked in system admin for many years. If anyone was actually foolish enough to do this, they would be dealing with multiple major outages every year. > for every complex problem there is a solution that is simple, obvious, and wrong.

georgebarnett 9y ago
"There might be an outage" is the stupidest reason I have ever heard to advocate not applying security patches. There's plenty of companies that are able to regularly patch and upgrade software, incident free. If you team isn't able to do this, they need to fix their process instead of putting their head in the sand.
- wavegeek 9y ago
  > stupid I am not defending the totality of EFX's process by any means. I am just pointing out that "install every high severity patch right away" is not as easy as it looks. Others have pointed out this patch was not available for all current versions, for example. So, to install the patch you need to upgrade. Oh, that breaks <dependency>. So we need to upgrade <other thing>. But that regresses a feature we use .... The major issue with EFX was the fragility of their overall architecture. A single weakness should not be enough to lay the whole DBMS open.
  - speedplane 9y ago
    The tech industry has to move more towards declarative systems, rather than procedural ones. Docker, Kubernetes and similar tech can auto-update in the background without any downtime. It's impossible to maintain thousands of VMs without some sort of declarative system.
    - qaq 9y ago
      Each thing you are listing increases complexity and attack surface.
      - speedplane 9y ago
        It actually decreases complexity. Updating 1000 VM servers using these technologies can be made automatic. True, it provides an additional attack surface, but the gains of having everything always patched are well worth it.
        qaq 9y ago
        Everything does not include your actual app, so yes it's convenient but doesn't really help that much
        speedplane 9y ago
        I'd argue that convenience is a security feature. The easier something is to do, the fewer mistakes you or your team will make. Stand by my initial post, declarative systems will become more and more popular and will improve security.
    - thehardsphere 9y ago
      What is the non-tech industry supposed to do?
  - georgebarnett 9y ago
    There are great reasons for not installing patches right away. "There might be an outage" is not one of them.
- edwhitesell 9y ago
  I don't recall being at, or working with, a company where this wasn't _the_ reason patches were delayed at least some period if time. Companies are driven by profit; unknown patches can negatively affect profit and are usually treated as such. Alternatively, there are sometimes political reasons. A director may not want to risk an outage for fear of being dressed down by their VP. Any change to a production environment comes with risks, sometimes having a security risk is judged to be lower risk than making a change. I've seen this from small startups to multi-billion dollar companies. Ignoring these factors is also putting your head in the sand.
  - dopamean 9y ago
    > Alternatively, there are sometimes political reasons. A director may not want to risk an outage for fear of being dressed down by their VP. This is such a weird thing to me. I've never worked anywhere like this and so I have a hard time imagining it really. Wouldn't that same director get more than dressed down by the VP if on their watch a _known security vulnerability_ was ignored and the system was breached? I just can't understand this way of thinking.
    - thehardsphere 9y ago
      It's easy to understand once you get just how simplistic it is. The VP will deliver a dressing down if the company loses money and the blame can be placed on the director. The cost of an outage is very easy to quantify (revenue per minute the system is down), and the probability that something will go wrong while applying the patch is also somewhat easy to predict, and usually greater than zero. The director will be blamed with certainty for the outage, since he approved it. The cost of a security breach is difficult to quantify; it depends on what gets breached and how bad. Note here, I say breach, not vulnerability. Even if there is a known security vulnerability, it's not immediately obvious in all systems what the consequence will be; there may be other mitigations in place outside of software that reduce the potential damage, or there may be unknown vulnerabilities that are exploitable due to the known vulnerability that make would make a breach worse. The lack of certainty about the consequences means it's also possible for the director to avoid blame if the breach is minor ("how was I supposed to know that other team is still using MD5?"). If there is no breach, then there is nothing for the director to be blamed for. Given that the director would like to avoid being dressed down, director will be more inclined to delay patching over possibly causing an outage, because the costs of an outage are easy to predict and he will take all blame for it. The breach may never happen and even if it does, it may cost him personally less than the outage. If this still seems weird, it might be because you are someone who views patching as an easy thing to do, because you probably work for a software company. Software companies are used to managing changing software, and have all kinds of practices around minimizing the risks of doing so. Non-software companies typically find patching to be hard and costly because their core business is something else; changes can disturb the "something else."
cookiecaper 9y ago
While you may sort of have a point for low-severity issues (if you patch every one ASAP, you are constantly restarting; should be tested in a group in stage and released to prod on a regular timeline), this was a 10.0 severity CVE and it is _definitely_ the kind of issue that you tolerate some downtime for. Check out the CVE, this could be exploited by anyone with a basic knowledge of cURL in 60 seconds or less. The vulnerability was just a matter of sending an arbitrary command in an HTTP header.
notzorbo3 9y ago
You need to do some testing of course. "IMMEDIATELY" doesn't mean within 5 seconds. If you can't properly apply security updates in a reasonable time frame without it leading to major outages, something is extremely wrong with either the architecture, the chosen software or the QA and deployment strategy. > without it leading to major outages I think I would have preferred an outage over leaking millions of american's private information.
- wavegeek 9y ago
  > I think I would have preferred an outage over leaking millions of american's private information. With full 100% hindsight vision, of course.
  - muninn_ 9y ago
    No you don't need hindsight for that. Protecting the data is more important than anything else. It's negligent to not do so.
  - mi100hael 9y ago
    It's not the first time this has happened to a company, and it surely won't be the last. We've already had full 100% hindsight vision a dozen times over, and plenty of companies have chosen to ignore it.

jldugger 9y ago

> This is an architectural decision It's also a business decision. You need to invest staff time in testing and automating said updates, and will inevitably lead to more outages. Thus less profits, directly contrary to your assertion. Managers have trade-off between a small but relatively predictable cost, and a large but presumed highly improbable cost. This is something that neither data nor human intuition are particularly good with. Best case scenario insurance lets you convert your large risk into a small predictable cost, and the premium difference decides whether you stay on the CVE treadmill or not. But its not like actuaries have a lot of data to model to the point of accurately imposing a cost per day of unpatched CVE. Worse, many places have competitors, and if they're choosing to take on risks in the name of lower prices and higher market share, the customers of your secured app walk out the door. Not only do you need to invest in security, you need the competition to as well. I have a few ideas on how to achieve this, but they're not historically palatable to the HN crowd.

siculars 9y ago
I think it's worth it when the alternative are multi billion dollar lawsuits and government regulation.
jacques_chester 9y ago
> It's also a business decision. You need to invest staff time in testing and automating said updates, and will inevitably lead to more outages. There are many vendors attacking this exact problem from many directions, including my employers[0], but pretty much every platform vendor has something here. I do agree that automation and risk are at the root of the tension between security and stability. It's a feedback loop that can run in two directions. If you embrace stability, the loop becomes "no changes ever" and security becomes much more difficult to achieve in practice. If you embrace security, then at first you face instability -- but embracing rapidity and building out tooling for it makes you both more secure and more stable in the long run. > But its not like actuaries have a lot of data to model to the point of accurately imposing a cost per day of unpatched CVE. I'd be interested to know if this is actually true. I recently got interested in the FAIR risk modelling approach. It requires a bunch of legwork, so in practice it won't catch on without determination to use it or something like it. Usefully it breaks risk down into more easily-estimated, easily-tracked subcomponents[1]. "Probability" becomes "Loss Event Frequency", which further breaks down into Threat Event Frequency and Vulnerability, these break down further in turn. Similarly, the usually-vague "Impact" becomes Primary Loss and Secondary Loss, each of which has 6 specific types that can be estimated. Those estimates can be based on available data, whether your own or industry statistics. > Worse, many places have competitors, and if they're choosing to take on risks in the name of lower prices and higher market share, the customers of your secured app walk out the door. [0] I work for Pivotal. We've been building towards the vision of "Rotate, Repair, Repave" in Cloud Foundry: https://builttoadapt.io/the-three-r-s-of-enterprise-security-rotate-repave-and-repair-f64f6d6ba29d https://builttoadapt.io/the-three-r-s-of-enterprise-security... [1] https://cdn2.hubspot.net/hubfs/1616664/The%20FAIR%20Model_FINAL_Web%20Only.pdf?t=1505484555066 https://cdn2.hubspot.net/hubfs/1616664/The%20FAIR%20Model_FI...

gaius 9y ago

- PATCHING OF KNOWN CRITICAL SECURITY VULNERABILITIES IMMEDIATELY. Immediately? No real company can do this. 1-2 weeks after the announcement is the best you'll see, even with something of the magnitude and publicity of Heartbleed. I'm not saying that's how it should be but that's how it is. Most organisation's it's quarterly! All of these can be designed into a system and still maintain 100% uptime Sure. If you were designing from a clean slate right now. The vast majority of organisations in the world aren't. And believe me: in 10 or 20 years your new design would have revealed flaws too, and look laughably primitive to the practitioners of the day. What's the solution then? An architecture in which no single compromise can compromise the entire system. That's getting harder and harder to do as the landscape homogenises around Lintel.

notzorbo3 9y ago
> No real company can do this. 1-2 weeks after the announcement is the best you'll see You don't need to patch your software right away, but you should have other strategies for mitigation such as a WAP in front of your services that you can configure to block malicious traffic when you identify a remote code execution vulnerability.
- gaius 9y ago
  Or two, by two different vendors, ideally sharing no code whatsoever.
grogenaut 9y ago
release: https://nvd.nist.gov/vuln/detail/CVE-2014-0160 https://nvd.nist.gov/vuln/detail/CVE-2014-0160 (04/07/2014) patch: https://aws.amazon.com/security/security-bulletins/heartbleed-bug-update/ https://aws.amazon.com/security/security-bulletins/heartblee... (4/8/2914) IIRC: 12 or 18 hours rolled out to all of aws ELB (main part affected). https://usn.ubuntu.com/usn/usn-2165-1/ https://usn.ubuntu.com/usn/usn-2165-1/ Patch was available at time of vulnerability release for current and LTS releases of ubuntu. apt-get update. But you know... That's just too hard.
- 9y ago

agiamas 9y ago

seriously, FFS these are guidelines that we use for not-remotely-that-sensitive-data and equifax that literally... "You had one job to do, protect our data" Can't follow? This is just business being a) ignorant b) incompetent c) careless one or more of the above applied for a hack like this to happen. It's really sad that even ycombinator crowd doesn't get it, given that the crowd here is significantly more technical and sensitive to these issues than the average Equifax Joe..

thehardsphere 9y ago
No, we get cases A, B, and C, because those cases are really easy for anyone to understand. What you're missing is that some of the "ycombinator crowd" have worked with enough "average Equifax Joes" enough to know that the problems faced by "Equifax Joe" are not always solvable by merely being knowledgeable, competent, and careful.