7 ms·

I work in security/systems/ops/etc. and fundamentally disagree with this premise. I understand the author is saying "it's not that easy" and I agree completely

I work in security/systems/ops/etc. and fundamentally disagree with this premise. I understand the author is saying "it's not that easy" and I agree completely with that, but I don't agree that it means you're doing your job well. Unfortunately the vast majority of people do their jobs poorly. The entire industry is set-up to support people doing their job poorly and to make doing your job well hard. If I deploy digital signage the only network access it should have is whitelisted to my servers' IP addresses and it should only accept updates that are signed and connections that have been established with certificate pinning. This makes it nearly impossible for a remote attacker to mess with it. Look at the security industry that has exploded from the rise of IoT. There's signage out there (replace with any other IoT/SCADA/deployed device) with open ports and default passwords, I guarantee it. IoT is just a computer, but it's also a computer that you neglect even more than the servers/virtual machines you're already running poorly. People don't want to accept this, or even might be affronted by this. There are some places doing things well - but it's the vast minorities of companies out there, because you are not incentivised to do things well. "Best practises" or following instructions from vendors does not mean you are doing things well. It means you are doing just enough that a vendor can be bothered to support. Which in a lot of cases is unfettered network access.

hypeatei 2y ago
For IoT in particular, you hit a crossroads where the embedded devs haven't really dealt with advanced security concepts so you kinda have to micromange the implementation. And, in small teams it's hard to justify the overhead of managing x509 certs and all the processes that come along with it. Just my personal experience.
- szundi 2y ago
  Yeah you know, just roll out our MVP, let's see where the business goes with it, and then we'll fix it. Whaat? Budget of fixing it is 2x of the product itself? Hm. Let's have meetings over meetings to postpone the decision until the next one, indefinitely - we cannot really make the decision not to do it of course.
- throw0101a 2y ago
  > And, in small teams it's hard to justify the overhead of managing x509 certs and all the processes that come along with it. Just my personal experience. If you're using (say) Python in your client code, call SSLSocket.getpeercert() and check if your company's domain is in the subjectAltName: * https://docs.python.org/3/library/ssl.html#ssl.SSLSocket.getpeercert https://docs.python.org/3/library/ssl.html#ssl.SSLSocket.get... You can ensure it is a valid cert from a valid public CA (like Let's Encrypt) instead of doing your own private CA (which you would specify with SSLContext.load_verify_locations()).
  - karmarepellent 2y ago
    I think parent refers to the infrastructure that is required to (automatically?) sign certificates by an internal CA and managing the distribution of those certificates. I don't think verification is the issue.
    - hypeatei 2y ago
      This is correct. You have to consider every step from when the device is manufactured to when something goes catastrophically wrong in the field. All the internal documentation and tools so Joe from support can help customers and Bob in manufacturing can provision devices on his own all while maintaining controls around that process so nothing is getting leaked or abused.
- kwhitefoot 2y ago
  Surely white lists and certificate pinning are not advanced security concepts?
  - pizzafeelsright 2y ago
    Yes, they are. Perhaps not in a practical or educational sense but in the real world, of people with non-cryptographic or security related jobs, a certificate is a PITA that goes beyond the functional requirements. I have seen many insecure building automation systems that are maintained by reclassified HVAC technicians. The movies about hackers taking over an elevator are entirely accurate.
    - CapstanRoller 2y ago
      The hassles of cert pinning, etc. should not be laid at the feet of the customer/integrator/whatever. Regardless of whether that person is an HVAC tech who learned about serial ports & busybox yesterday or is a seasoned expert with Ghidra & Wireshark & binwalk. Companies are being incredibly lazy (at our expense), and the author states this obliquely: >virtually the entire software landscape has been designed with the assumption of internet connectivity
      - shadowgovt 2y ago
        The issue is the alternative does not scale. It's not that companies are being lazy at our expense; it's that nobody wants to pick up the bill. If you write something to work against an online system, the fact it is online implies it adheres to some standard that you can work with, so solving the problem for one online client creates an artifact that is likely applicable to many clients. Air-gapped systems drift. They get bespoke. They get very out of date. So you have the two practical problems of labor: (a) the product created solves the problem here, today, but nobody else benefits from repurposing that solution and (b) the developer isn't gaining as many transferrable skills for the next gig, and they know it, and so the developers who are willing to do the air-gapped work are harder to find and more expensive. (I believe this is also the reason you see air-gap a lot more often in government security and banks: they can afford to retain talent past the current project with the certitude there will be more projects in the future).
- CapstanRoller 2y ago
  OK, that's fine. Not everyone has to know everything. So why aren't their employers investing in educating their devs & PMs about security? (rhetorical - we all know why)
- skywhopper 2y ago
  Sure, for “small teams”. Does that apply to the companies with huge impact from this issue? Is Delta Airlines IT run by a small team? I hope not.
karmarepellent 2y ago
> I understand the author is saying "it's not that easy" and I agree completely with that, but I don't agree that it means you're doing your job well. Could you elaborate what you mean by this? It seems to me that your comment just highlights another set of problems that should (in theory) motivate people to think more clearly about the ways their system communicates with the internet. I don't see where you disagree with the blog author. Or are you saying that it's fundamentally impossible to improve security in internet-connected systems because people are not equipped to do so?
- flumpcakes 2y ago
  There's no reason for digital signage inside an airport to be connected to the internet (or running enterprise security software either). The author seemingly doesn't agree with this. Hospital computers should not be connected to the internet. If you are receiving real-time updates directly from a vendor, you are connected to the internet. Ideally updates should come from a central source internally to the organisation that has been vetted and approved by the organisation itself. Clearly CrowdStrike knows this and that's why they offer N, N-1, N-2 updates for their falcon sensor. It's easier to remote into a box and just pull updates from the internet though. Granted I have not had dozens of jobs, but the only place I have worked where security was treated as the first class issue that it is, (and this type of CrowdStrike incident probably wouldn't have happened), is at one of the largest financial services companies in the world. And it did not hamper development, it actually improved it because you couldn't make stupid mistakes like relying on externally hosted CDN content for your backend app. But for people that don't do their job well, it's a pain. "Hey why doesn't my docker image build on a prod machine, why can't I download from docker hub?"
  - karmarepellent 2y ago
    Just to add to this: Apart from the security aspect of hosting your software dependencies internally, it also gives you the added benefit of better availability and performance. As you mentioned in your other comment however, this presumes a certain mindset in people where they are willing to plan upfront and are mindful of the dependencies their software needs. As you say, just pulling whatever from Docker hub is certainly easier. Internally hosted repositories also allow you to pull and install updates at your own pace, possibly days after they have been released upstream. So if a patch is borked you won't be affected.
slumberlust 2y ago
There is some legislation with actual teeth getting implemented to help force vendors to incorporate decent security standards in all these edge deveices: https://www.withersworldwide.com/en-gb/insight/read/new-uk-legislation-on-security-of-iot-devices https://www.withersworldwide.com/en-gb/insight/read/new-uk-l...
zippergz 2y ago
A sign connected to the internet but with IP whitelists and cryptographic checks is still CONNECTED TO THE INTERNET. Yeah, it's way safer than the same sign with ports open to the world and no authentication, but you can't treat it as "not connected to the internet." You still have to worry about networking bugs, cryptographic vulnerabilities, configuration errors, and other issues that can allow remote attackers to exploit the system. If you want to make the point, you have to give an example of something that's literally not connected to the internet, not one that's simply locked down better.
- Spivak 2y ago
  The number of people who are willing and able to build their own disconnected network is vanishingly small which is the author's point. When deploying "edge" computing like signage which demands remote administration telling your customers anything other than "get it connected to the internet and we'll handle it from there" isn't going to go over well. "Sorry you can't deploy our signs because we haven't deployed our custom LoRa towers in your area" is just gonna get laughs.
  - throwaway173738 2y ago
    That’s not necessarily true. Otherwise Pricer wouldn’t have a viable business.

hypeatei 2y ago

For IoT in particular, you hit a crossroads where the embedded devs haven't really dealt with advanced security concepts so you kinda have to micromange the implementation. And, in small teams it's hard to justify the overhead of managing x509 certs and all the processes that come along with it. Just my personal experience.

szundi 2y ago
Yeah you know, just roll out our MVP, let's see where the business goes with it, and then we'll fix it. Whaat? Budget of fixing it is 2x of the product itself? Hm. Let's have meetings over meetings to postpone the decision until the next one, indefinitely - we cannot really make the decision not to do it of course.
throw0101a 2y ago
> And, in small teams it's hard to justify the overhead of managing x509 certs and all the processes that come along with it. Just my personal experience. If you're using (say) Python in your client code, call SSLSocket.getpeercert() and check if your company's domain is in the subjectAltName: * https://docs.python.org/3/library/ssl.html#ssl.SSLSocket.getpeercert https://docs.python.org/3/library/ssl.html#ssl.SSLSocket.get... You can ensure it is a valid cert from a valid public CA (like Let's Encrypt) instead of doing your own private CA (which you would specify with SSLContext.load_verify_locations()).
- karmarepellent 2y ago
  I think parent refers to the infrastructure that is required to (automatically?) sign certificates by an internal CA and managing the distribution of those certificates. I don't think verification is the issue.
  - hypeatei 2y ago
    This is correct. You have to consider every step from when the device is manufactured to when something goes catastrophically wrong in the field. All the internal documentation and tools so Joe from support can help customers and Bob in manufacturing can provision devices on his own all while maintaining controls around that process so nothing is getting leaked or abused.
kwhitefoot 2y ago
Surely white lists and certificate pinning are not advanced security concepts?
- pizzafeelsright 2y ago
  Yes, they are. Perhaps not in a practical or educational sense but in the real world, of people with non-cryptographic or security related jobs, a certificate is a PITA that goes beyond the functional requirements. I have seen many insecure building automation systems that are maintained by reclassified HVAC technicians. The movies about hackers taking over an elevator are entirely accurate.
  - CapstanRoller 2y ago
    The hassles of cert pinning, etc. should not be laid at the feet of the customer/integrator/whatever. Regardless of whether that person is an HVAC tech who learned about serial ports & busybox yesterday or is a seasoned expert with Ghidra & Wireshark & binwalk. Companies are being incredibly lazy (at our expense), and the author states this obliquely: >virtually the entire software landscape has been designed with the assumption of internet connectivity
    - shadowgovt 2y ago
      The issue is the alternative does not scale. It's not that companies are being lazy at our expense; it's that nobody wants to pick up the bill. If you write something to work against an online system, the fact it is online implies it adheres to some standard that you can work with, so solving the problem for one online client creates an artifact that is likely applicable to many clients. Air-gapped systems drift. They get bespoke. They get very out of date. So you have the two practical problems of labor: (a) the product created solves the problem here, today, but nobody else benefits from repurposing that solution and (b) the developer isn't gaining as many transferrable skills for the next gig, and they know it, and so the developers who are willing to do the air-gapped work are harder to find and more expensive. (I believe this is also the reason you see air-gap a lot more often in government security and banks: they can afford to retain talent past the current project with the certitude there will be more projects in the future).
CapstanRoller 2y ago
OK, that's fine. Not everyone has to know everything. So why aren't their employers investing in educating their devs & PMs about security? (rhetorical - we all know why)
skywhopper 2y ago
Sure, for “small teams”. Does that apply to the companies with huge impact from this issue? Is Delta Airlines IT run by a small team? I hope not.

karmarepellent 2y ago

> I understand the author is saying "it's not that easy" and I agree completely with that, but I don't agree that it means you're doing your job well. Could you elaborate what you mean by this? It seems to me that your comment just highlights another set of problems that should (in theory) motivate people to think more clearly about the ways their system communicates with the internet. I don't see where you disagree with the blog author. Or are you saying that it's fundamentally impossible to improve security in internet-connected systems because people are not equipped to do so?

flumpcakes 2y ago
There's no reason for digital signage inside an airport to be connected to the internet (or running enterprise security software either). The author seemingly doesn't agree with this. Hospital computers should not be connected to the internet. If you are receiving real-time updates directly from a vendor, you are connected to the internet. Ideally updates should come from a central source internally to the organisation that has been vetted and approved by the organisation itself. Clearly CrowdStrike knows this and that's why they offer N, N-1, N-2 updates for their falcon sensor. It's easier to remote into a box and just pull updates from the internet though. Granted I have not had dozens of jobs, but the only place I have worked where security was treated as the first class issue that it is, (and this type of CrowdStrike incident probably wouldn't have happened), is at one of the largest financial services companies in the world. And it did not hamper development, it actually improved it because you couldn't make stupid mistakes like relying on externally hosted CDN content for your backend app. But for people that don't do their job well, it's a pain. "Hey why doesn't my docker image build on a prod machine, why can't I download from docker hub?"
- karmarepellent 2y ago
  Just to add to this: Apart from the security aspect of hosting your software dependencies internally, it also gives you the added benefit of better availability and performance. As you mentioned in your other comment however, this presumes a certain mindset in people where they are willing to plan upfront and are mindful of the dependencies their software needs. As you say, just pulling whatever from Docker hub is certainly easier. Internally hosted repositories also allow you to pull and install updates at your own pace, possibly days after they have been released upstream. So if a patch is borked you won't be affected.

slumberlust 2y ago

There is some legislation with actual teeth getting implemented to help force vendors to incorporate decent security standards in all these edge deveices: https://www.withersworldwide.com/en-gb/insight/read/new-uk-legislation-on-security-of-iot-devices https://www.withersworldwide.com/en-gb/insight/read/new-uk-l...

zippergz 2y ago

A sign connected to the internet but with IP whitelists and cryptographic checks is still CONNECTED TO THE INTERNET. Yeah, it's way safer than the same sign with ports open to the world and no authentication, but you can't treat it as "not connected to the internet." You still have to worry about networking bugs, cryptographic vulnerabilities, configuration errors, and other issues that can allow remote attackers to exploit the system. If you want to make the point, you have to give an example of something that's literally not connected to the internet, not one that's simply locked down better.

Spivak 2y ago
The number of people who are willing and able to build their own disconnected network is vanishingly small which is the author's point. When deploying "edge" computing like signage which demands remote administration telling your customers anything other than "get it connected to the internet and we'll handle it from there" isn't going to go over well. "Sorry you can't deploy our signs because we haven't deployed our custom LoRa towers in your area" is just gonna get laughs.
- throwaway173738 2y ago
  That’s not necessarily true. Otherwise Pricer wouldn’t have a viable business.