8 ms·

U.S. government probes VPN hack within federal agencies, races to find clues

153 points by c1c2c3 5y ago

waihtis 5y ago
> The U.S. plans to address some of these systemic issues with an upcoming executive order that will require agencies to identify their most critical software and promote a “bill of materials” that demands a certain level of digital security across products sold to the government. Interesting, no mention of any requirements towards software manufacturers themselves. If you think about it, this will further incentivize poor-quality software as responsibility of vulnerability response is now being laid on the product owner.
- thinkmassive 5y ago
  The government has no authority to demand a software bill of materials (SBOM) from everyone who publishes software. Imposing this requirement on their own agencies is enforceable because there's software that can generate an SBOM, at least from container images. Then the agencies will have to choose software that meets compliance requirements, so they're the ones putting pressure on their chosen vendors. It follows logically that a vendor who wants a better chance of being chosen for more government contracts will make it easy to obtain SBOMs for their software.
  - WaitWaitWha 5y ago
    >The government has no authority to demand a software bill of materials (SBOM) from everyone who publishes software. Speaking from US Gov perspective - if the company is part of a contract (and ~40% of the Gov are contractors), Gov certainly can. They can put nearly anything (legal) into the RFP/Q. Even if they do not say "give us your BoM", they can wrap it in requirements that in essence delivers the same exact result. That said, it is Gov mistake to ask for the BoM. They will do little with it in a timely fashion, and lack the expertise to identify risks, and lack the resources to go after it. The best contracts are the ones where the rules and parameters are set for the contractor, (i.e. no untested software, no foreign influence, no this, no that, must have this and that), and auditing of the compliance.
    - jcrites 5y ago
      They could build in a requirement that the software has undergone penetration testing by a security firm, and that a copy of the penetration testing report along with any mitigations applied to the software be provided. I've never even heard of the software the government is using. Why aren't they using Cisco AnyConnect like literally every other company I've worked for who has a VPN?
      - Jtsummers 5y ago
        Not all agencies, but the US gov't does use Cisco AnyConnect and pretty much everything they use for IT is COTS these days.
        systematical 5y ago
        Federal contractors as well.
      - Bluecobra 5y ago
        Pulse Secure is pretty well regarded (or maybe was better regarded when it was a Juniper product). AnyConnect has had had will have its fair share of vulnerabilities as well. A few years ago I had to update the firmware our ASAs like four times in a year due to new vulnerabilities. Any commercial product you pick is going to have new vulnerabilities and you just need to stay on top of it.
- joe_the_user 5y ago
  This theme keeps coming up. Some cohort of HN is upset that software manufacturers aren't directly required to produce "secure software" [1] I would suggest people look at a very foundational essay on this [2]. Key quote: "Security is a process, not a product. Products provide some protection, but the only way to effectively do business in an insecure world is to put processes in place that recognize the inherent insecurity in the products. " How many times do we have to learn this? [1] In quotes 'cause "secure software" does not exist. In two different ways; software always has bugs and using a piece of software incorrectly makes a secure system insecure. [2] https://www.schneier.com/essays/archives/2000/04/the_process_of_secur.html https://www.schneier.com/essays/archives/2000/04/the_process...
  - g_p 5y ago
    I think there's a couple of issues at play here. Firstly there's the information asymmetry for non-technical users - they don't think of themselves as buying security, they think of themselves as buying a remote access solution. They therefore don't see this as a process, but instead as a product or solution. That means they're surprised and caught unaware when something goes wrong. The second issue is that people creating the software aren't themselves thinking about security, because the customer isn't buying security, or comparing security. And how do you measure or quantify or observe security? There's no commercial incentive to invest a month in hardening a product against attack, unless that month of engineering effort sees more sales and revenues. And since the people who buy are satisfied by slideware and specification sheets for security, nothing changes. I think we need a whole change to how we buy software, hardware, and solutions in general, to see this change. The underlying economics don't incentivise secure products, in fact they actively discourage them.
  - closeparen 5y ago
    Part of the process of security is to design and code with consideration towards potential vulnerabilities. Organizations that ought to care care about security, and even spend a lot of money on security, also buy a lot of bug-riddled crap to run on their “secure” networks. Firewall rules and Group Policies can’t fix everything.
  - TheOtherHobbes 5y ago
- indymike 5y ago
  If I were a federal contractor, wanting to make more from my cost plus contraction, what better way than generating text files full of dependencies that will cause billable meetings to discuss why we should be ok with some old insecure library being used... that will always end with even more billable work to update the old, insecure library. Even better would be if I had to incur some billable time and cost on certifications. Cost plus FTW (unless you are a taxpayer).
  - systematical 5y ago
    Luckily not all federal contractors think like this. Some would report this behavior and some of us would be quite happy to report it. There are those that still believe in doing the right thing, value tax payer dollars, and want to deliver for the American people. If only more of us wanted to completely rip out the rot.
    - indymike 5y ago
      > If only more of us wanted to completely rip out the rot. Even if you report it, this kind of behavior is so normal, I'd be shocked if it does anything other than create more billable project management hours for the company you are protesting.
- er4hn 5y ago
  > If you think about it, this will further incentivize poor-quality software as responsibility of vulnerability response is now being laid on the product owner. Not really, this is more about transparency of all components and letting people downstream be aware that there is an issue and either fix it, mitigate it, or raise the issue upstream. My guess is that this is related to Allan Friedman's SBOM work at NTIA (sorry - this is not the most up to date link: https://www.csiac.org/podcast/software-bill-of-materials-sbom/ https://www.csiac.org/podcast/software-bill-of-materials-sbo... ) The problem that keeps on getting hit time and time again is that both end users and product manufacturers do not know everything that is in their system. Consider the case of say, an MRI machine. What OS is it running and how up to date is it? If the end user has an SBOM they can better evaluate that and demand fixes if there are known issues. Likewise if the MRI manufacturer is good at making MRIs, but not so much at knowing if their version of Windows on the MRI is out of date, the SBOM for the MRI can be analyzed to automatically flag problems. You can regulate all you want about "There must be no open issues" and plenty of certifications for the Fed government do have that language. The problem this answers is forcing a listing of every component so that "Sorry I didn't know OpenSSH v.1.2.3 is out of date" or "I had no idea we were running Windows 95 on this hardware" are no longer valid excuses.
- uzakov 5y ago
  If anyone is interested to read more about Software Bill of Materials and how you can implement it check out OWASP Dependency Track project - https://owasp.org/www-project-dependency-track/ https://owasp.org/www-project-dependency-track/
olyjohn 5y ago
Wasn't Pulse Secure VPN the one that required an ActiveX control and IE in order to "secure" your system on Windows? I mean, when I see that kind of shit, I kind of assume the vendor sells some shit software.
- reaperducer 5y ago
  Just yesterday I was on a federal government web site (FWS, BLM, or some similar agency), and it popped up a window saying that the web site doesn't work in Safari, and I should use IE10.
  - 55555 5y ago
    There are major federal government websites that have office hours. The EIN application website only works between 9am and 5pm. I like to imagine that it's this way because there's no webapp, there's just a team of people who get all the HTTP requests and manually respond to them. This would also explain its speed.
    - KMag 5y ago
      If they're only paying support staff 9 to 5, it might make sense to shut it down outside of supported hours, even if it would probably still work, especially if they're not positive that it won't do something odd like start issuing duplicates if the back-end DB is down.
rossdavidh 5y ago
Prediction: at some point (if it isn't already happening as we speak), the government insistence on "we need to be able to hack into any software if it's important" will collide with "we need to be able to keep foreign powers out of our software", and there will be bitter internal fights about it, both sides claiming national security interests.
- legutierr 5y ago
  Here's a good discussion of that very same debate that is happening right now. https://www.lawfareblog.com/lawfare-podcast-nicole-perlroth-cyberweapons-arms-race https://www.lawfareblog.com/lawfare-podcast-nicole-perlroth-...
- sbierwagen 5y ago
  Bruce Schneier has been complaining about this tradeoff for more than a decade: https://www.schneier.com/blog/archives/2014/05/disclosing_vs_h.html https://www.schneier.com/blog/archives/2014/05/disclosing_vs... >The NSA can play either defense or offense. It can either alert the vendor and get a still-secret vulnerability fixed, or it can hold on to it and use it to eavesdrop on foreign computer systems. Both are important US policy goals, but the NSA has to choose which one to pursue. By fixing the vulnerability, it strengthens the security of the Internet against all attackers: other countries, criminals, hackers. By leaving the vulnerability open, it is better able to attack others on the Internet. But each use runs the risk of the target government learning of, and using for itself, the vulnerability — or of the vulnerability becoming public and criminals starting to use it. Unsurprisingly, the NSA often chooses to keep zerodays for their own use.
  - jl2718 5y ago
    If only it was just one agency from one country plinking holes in things...
  - NortySpock 5y ago
    Makes me wonder if we should have a white-hat government org that notifies big corporations or software projects about critical vulnerabilities in their code.
baybal2 5y ago
Do people know that Fortinet is pretty much a de-facto Chinese company?
- uzakov 5y ago
  Any links on that? Only notable incident was when they apparently sold intentionally mislabeled Chinese-made equipment to U.S. government end users. https://en.wikipedia.org/wiki/Fortinet#cite_note-37 https://en.wikipedia.org/wiki/Fortinet#cite_note-37
  - freeflight 5y ago
    I think they are referring to the fact how Fortinet was founded by two Chinese born brothers. Tho Ken Xie is also a Stanford graduate and has had US citizenship for decades.
    - baybal2 5y ago
      Not only that, but them having the bulk of their staff in China.
      - uzakov 5y ago
        > not only that By this logic does it not mean that Google is a Russian/Soviet company? Sergey Brin was born and spent some years in Russia, also Google has staff in Russia(and China in this regard)
        baybal2 5y ago
        He was born there, but he did not remain as a citizen of that country, and not been going back, and forth in line of business. A situation is different from where a double citizen operates a sales office of a company from his home country
        uzakov 5y ago
ComodoHacker 5y ago
This new arms race can eventually lead us to militarization of the whole economy. Almost every business operation will cost 40% more than now because of security costs. Security doesn't scale well and can't be commoditized (until we get AGI I guess). You can't just outsource it to Google or other megacorp. That would be an insane waste of resources.

153 points by c1c2c3 5y ago

> The U.S. plans to address some of these systemic issues with an upcoming executive order that will require agencies to identify their most critical software and promote a “bill of materials” that demands a certain level of digital security across products sold to the government. Interesting, no mention of any requirements towards software manufacturers themselves. If you think about it, this will further incentivize poor-quality software as responsibility of vulnerability response is now being laid on the product owner.

thinkmassive 5y ago
The government has no authority to demand a software bill of materials (SBOM) from everyone who publishes software. Imposing this requirement on their own agencies is enforceable because there's software that can generate an SBOM, at least from container images. Then the agencies will have to choose software that meets compliance requirements, so they're the ones putting pressure on their chosen vendors. It follows logically that a vendor who wants a better chance of being chosen for more government contracts will make it easy to obtain SBOMs for their software.
- WaitWaitWha 5y ago
  >The government has no authority to demand a software bill of materials (SBOM) from everyone who publishes software. Speaking from US Gov perspective - if the company is part of a contract (and ~40% of the Gov are contractors), Gov certainly can. They can put nearly anything (legal) into the RFP/Q. Even if they do not say "give us your BoM", they can wrap it in requirements that in essence delivers the same exact result. That said, it is Gov mistake to ask for the BoM. They will do little with it in a timely fashion, and lack the expertise to identify risks, and lack the resources to go after it. The best contracts are the ones where the rules and parameters are set for the contractor, (i.e. no untested software, no foreign influence, no this, no that, must have this and that), and auditing of the compliance.
  - jcrites 5y ago
    They could build in a requirement that the software has undergone penetration testing by a security firm, and that a copy of the penetration testing report along with any mitigations applied to the software be provided. I've never even heard of the software the government is using. Why aren't they using Cisco AnyConnect like literally every other company I've worked for who has a VPN?
    - Jtsummers 5y ago
      Not all agencies, but the US gov't does use Cisco AnyConnect and pretty much everything they use for IT is COTS these days.
      - systematical 5y ago
        Federal contractors as well.
    - Bluecobra 5y ago
      Pulse Secure is pretty well regarded (or maybe was better regarded when it was a Juniper product). AnyConnect has had had will have its fair share of vulnerabilities as well. A few years ago I had to update the firmware our ASAs like four times in a year due to new vulnerabilities. Any commercial product you pick is going to have new vulnerabilities and you just need to stay on top of it.
joe_the_user 5y ago
This theme keeps coming up. Some cohort of HN is upset that software manufacturers aren't directly required to produce "secure software" [1] I would suggest people look at a very foundational essay on this [2]. Key quote: "Security is a process, not a product. Products provide some protection, but the only way to effectively do business in an insecure world is to put processes in place that recognize the inherent insecurity in the products. " How many times do we have to learn this? [1] In quotes 'cause "secure software" does not exist. In two different ways; software always has bugs and using a piece of software incorrectly makes a secure system insecure. [2] https://www.schneier.com/essays/archives/2000/04/the_process_of_secur.html https://www.schneier.com/essays/archives/2000/04/the_process...
- g_p 5y ago
  I think there's a couple of issues at play here. Firstly there's the information asymmetry for non-technical users - they don't think of themselves as buying security, they think of themselves as buying a remote access solution. They therefore don't see this as a process, but instead as a product or solution. That means they're surprised and caught unaware when something goes wrong. The second issue is that people creating the software aren't themselves thinking about security, because the customer isn't buying security, or comparing security. And how do you measure or quantify or observe security? There's no commercial incentive to invest a month in hardening a product against attack, unless that month of engineering effort sees more sales and revenues. And since the people who buy are satisfied by slideware and specification sheets for security, nothing changes. I think we need a whole change to how we buy software, hardware, and solutions in general, to see this change. The underlying economics don't incentivise secure products, in fact they actively discourage them.
- closeparen 5y ago
  Part of the process of security is to design and code with consideration towards potential vulnerabilities. Organizations that ought to care care about security, and even spend a lot of money on security, also buy a lot of bug-riddled crap to run on their “secure” networks. Firewall rules and Group Policies can’t fix everything.
- TheOtherHobbes 5y ago
indymike 5y ago
If I were a federal contractor, wanting to make more from my cost plus contraction, what better way than generating text files full of dependencies that will cause billable meetings to discuss why we should be ok with some old insecure library being used... that will always end with even more billable work to update the old, insecure library. Even better would be if I had to incur some billable time and cost on certifications. Cost plus FTW (unless you are a taxpayer).
- systematical 5y ago
  Luckily not all federal contractors think like this. Some would report this behavior and some of us would be quite happy to report it. There are those that still believe in doing the right thing, value tax payer dollars, and want to deliver for the American people. If only more of us wanted to completely rip out the rot.
  - indymike 5y ago
    > If only more of us wanted to completely rip out the rot. Even if you report it, this kind of behavior is so normal, I'd be shocked if it does anything other than create more billable project management hours for the company you are protesting.
er4hn 5y ago
> If you think about it, this will further incentivize poor-quality software as responsibility of vulnerability response is now being laid on the product owner. Not really, this is more about transparency of all components and letting people downstream be aware that there is an issue and either fix it, mitigate it, or raise the issue upstream. My guess is that this is related to Allan Friedman's SBOM work at NTIA (sorry - this is not the most up to date link: https://www.csiac.org/podcast/software-bill-of-materials-sbom/ https://www.csiac.org/podcast/software-bill-of-materials-sbo... ) The problem that keeps on getting hit time and time again is that both end users and product manufacturers do not know everything that is in their system. Consider the case of say, an MRI machine. What OS is it running and how up to date is it? If the end user has an SBOM they can better evaluate that and demand fixes if there are known issues. Likewise if the MRI manufacturer is good at making MRIs, but not so much at knowing if their version of Windows on the MRI is out of date, the SBOM for the MRI can be analyzed to automatically flag problems. You can regulate all you want about "There must be no open issues" and plenty of certifications for the Fed government do have that language. The problem this answers is forcing a listing of every component so that "Sorry I didn't know OpenSSH v.1.2.3 is out of date" or "I had no idea we were running Windows 95 on this hardware" are no longer valid excuses.
uzakov 5y ago
If anyone is interested to read more about Software Bill of Materials and how you can implement it check out OWASP Dependency Track project - https://owasp.org/www-project-dependency-track/ https://owasp.org/www-project-dependency-track/

olyjohn 5y ago

Wasn't Pulse Secure VPN the one that required an ActiveX control and IE in order to "secure" your system on Windows? I mean, when I see that kind of shit, I kind of assume the vendor sells some shit software.

reaperducer 5y ago
Just yesterday I was on a federal government web site (FWS, BLM, or some similar agency), and it popped up a window saying that the web site doesn't work in Safari, and I should use IE10.
- 55555 5y ago
  There are major federal government websites that have office hours. The EIN application website only works between 9am and 5pm. I like to imagine that it's this way because there's no webapp, there's just a team of people who get all the HTTP requests and manually respond to them. This would also explain its speed.
  - KMag 5y ago
    If they're only paying support staff 9 to 5, it might make sense to shut it down outside of supported hours, even if it would probably still work, especially if they're not positive that it won't do something odd like start issuing duplicates if the back-end DB is down.

rossdavidh 5y ago

Prediction: at some point (if it isn't already happening as we speak), the government insistence on "we need to be able to hack into any software if it's important" will collide with "we need to be able to keep foreign powers out of our software", and there will be bitter internal fights about it, both sides claiming national security interests.

legutierr 5y ago
Here's a good discussion of that very same debate that is happening right now. https://www.lawfareblog.com/lawfare-podcast-nicole-perlroth-cyberweapons-arms-race https://www.lawfareblog.com/lawfare-podcast-nicole-perlroth-...
sbierwagen 5y ago
Bruce Schneier has been complaining about this tradeoff for more than a decade: https://www.schneier.com/blog/archives/2014/05/disclosing_vs_h.html https://www.schneier.com/blog/archives/2014/05/disclosing_vs... >The NSA can play either defense or offense. It can either alert the vendor and get a still-secret vulnerability fixed, or it can hold on to it and use it to eavesdrop on foreign computer systems. Both are important US policy goals, but the NSA has to choose which one to pursue. By fixing the vulnerability, it strengthens the security of the Internet against all attackers: other countries, criminals, hackers. By leaving the vulnerability open, it is better able to attack others on the Internet. But each use runs the risk of the target government learning of, and using for itself, the vulnerability — or of the vulnerability becoming public and criminals starting to use it. Unsurprisingly, the NSA often chooses to keep zerodays for their own use.
- jl2718 5y ago
  If only it was just one agency from one country plinking holes in things...
- NortySpock 5y ago
  Makes me wonder if we should have a white-hat government org that notifies big corporations or software projects about critical vulnerabilities in their code.

baybal2 5y ago

Do people know that Fortinet is pretty much a de-facto Chinese company?

uzakov 5y ago
Any links on that? Only notable incident was when they apparently sold intentionally mislabeled Chinese-made equipment to U.S. government end users. https://en.wikipedia.org/wiki/Fortinet#cite_note-37 https://en.wikipedia.org/wiki/Fortinet#cite_note-37
- freeflight 5y ago
  I think they are referring to the fact how Fortinet was founded by two Chinese born brothers. Tho Ken Xie is also a Stanford graduate and has had US citizenship for decades.
  - baybal2 5y ago
    Not only that, but them having the bulk of their staff in China.
    - uzakov 5y ago
      > not only that By this logic does it not mean that Google is a Russian/Soviet company? Sergey Brin was born and spent some years in Russia, also Google has staff in Russia(and China in this regard)
      - baybal2 5y ago
        He was born there, but he did not remain as a citizen of that country, and not been going back, and forth in line of business. A situation is different from where a double citizen operates a sales office of a company from his home country
        uzakov 5y ago

ComodoHacker 5y ago

This new arms race can eventually lead us to militarization of the whole economy. Almost every business operation will cost 40% more than now because of security costs. Security doesn't scale well and can't be commoditized (until we get AGI I guess). You can't just outsource it to Google or other megacorp. That would be an insane waste of resources.