4 ms·

I'd much rather store HIPAA data on a server in my office or closet than worry I got all the IAM settings right. And if I fire someone, security makes sure the

I'd much rather store HIPAA data on a server in my office or closet than worry I got all the IAM settings right. And if I fire someone, security makes sure they can't get in the building. You cannot say the same about the cloud. Yes, I know you can do cloud security right, but on prem security is just harder to mess up.

walnutclosefarm 4y ago
More than half of security penetrations in our institution (A medical center - research - med school complex) over the 8 years I worked there, ending in 2020, came through the research arm, even though Research accounted for no more than 10% of enabled servers in the infrastructure. And we're talking APT penetrations. They weren't looking for HIPAA data (although I used that example in my original post), they were looking for a path to permanent presence in our network in order to mine research. So, why did they come through Research? Because a PI buys some equipment - servers or other network enabled stuff - puts a grad student or post doc in charge of it, and enjoys his or her cheap compute. But that student or post doc is not an infrastructure expert, and most definitely doesn't understand enterprise security. Next thing you know, we've got an APT owned server on the inside of the network. (And none of that is counting the ones where the post doc is a foreign national who actually intends to use their position to compromise their employer. Had that happen too.) There are a some computational scientists who actually do understand this stuff, but they're rare. Being on the cloud does not inherently fix this oroblem, but to fix it you have to be on institutionally, professionally managed infrastructure, and once you are, the cost differential between owned infrastructure and well negotiated, managed cloud infrastructure becomes much more nuanced.
- funstuff007 4y ago
  I appreciate your perspective, but it seems the security team should be watching for reverse proxies, tunnels, and other firewall anomalies for on-prem hardware just as a normal course of biz. And if a PI installs a self-managed server, that really should not gum up the works. All that being said, I have never worked at a place (or in a dept) whose threat profile made APT a real thing.
  - walnutclosefarm 4y ago
    Obviously we do all those things. But you NEVER want an outside, and particularly APT owned machine inside your network. They can be well hidden and still do very real damage. You're fortunate if APTs don't consider you a worthy target. They are no joke, and in most cases are playing a long game, more interested in penetration, persistent presence, and quiet theft of information, than in doing anything you'd notice - until they aren't. We had one who burned an asset that had been cultivated on our network for a couple of years to make a hard press play for information about a very particular high profile patient immediately after that patient had been seen (the fact that the patient was seen was public information). But with over 20,000 servers and 300,000 total nodes on the network - some of which cannot be fully patched because they run software someone has to have access to but which won't run on the newest versions of OSs or databases - you still don't know what else they've burrowed into. A big tech company supplier to our organization had a very telling incident that ended up being detected on our side of the connection, where a brief mistake by network admins opened a channel through their layers of protection for their buid pipeline. In the minutes it was open, an APT detected the access (likely because they already owned something internal), and inserted code into their certified OS build pipeline, which we ended up with in our institution.
    - funstuff007 4y ago
      Is an APT-owned server a significantly different risk than an APT-owned desktop? They are both inside your house.
      - walnutclosefarm 4y ago
        Probably not hugely more significant, but there are differences. A workstation should be segmented and limited in the range of nodes it can communicate with, if you're running your network properly. A server will likely be in a segment that has much broader access to it, unless you're doing micro-segmentation, and doing it well. By construction, a server set up by a workgroup team outside your core IT server administration staff is unlikely to be properly segmented. And if you're doing traffic analysis to look for rogue behavior, it's harder to spot from something that profiles as a server, because, again, you expect a server to have lots of contacts within the network. Counterbalancing that, if it profiles as a server, you should be more suspicious of any outbound activity.

walnutclosefarm 4y ago

More than half of security penetrations in our institution (A medical center - research - med school complex) over the 8 years I worked there, ending in 2020, came through the research arm, even though Research accounted for no more than 10% of enabled servers in the infrastructure. And we're talking APT penetrations. They weren't looking for HIPAA data (although I used that example in my original post), they were looking for a path to permanent presence in our network in order to mine research. So, why did they come through Research? Because a PI buys some equipment - servers or other network enabled stuff - puts a grad student or post doc in charge of it, and enjoys his or her cheap compute. But that student or post doc is not an infrastructure expert, and most definitely doesn't understand enterprise security. Next thing you know, we've got an APT owned server on the inside of the network. (And none of that is counting the ones where the post doc is a foreign national who actually intends to use their position to compromise their employer. Had that happen too.) There are a some computational scientists who actually do understand this stuff, but they're rare. Being on the cloud does not inherently fix this oroblem, but to fix it you have to be on institutionally, professionally managed infrastructure, and once you are, the cost differential between owned infrastructure and well negotiated, managed cloud infrastructure becomes much more nuanced.

funstuff007 4y ago
I appreciate your perspective, but it seems the security team should be watching for reverse proxies, tunnels, and other firewall anomalies for on-prem hardware just as a normal course of biz. And if a PI installs a self-managed server, that really should not gum up the works. All that being said, I have never worked at a place (or in a dept) whose threat profile made APT a real thing.
- walnutclosefarm 4y ago
  Obviously we do all those things. But you NEVER want an outside, and particularly APT owned machine inside your network. They can be well hidden and still do very real damage. You're fortunate if APTs don't consider you a worthy target. They are no joke, and in most cases are playing a long game, more interested in penetration, persistent presence, and quiet theft of information, than in doing anything you'd notice - until they aren't. We had one who burned an asset that had been cultivated on our network for a couple of years to make a hard press play for information about a very particular high profile patient immediately after that patient had been seen (the fact that the patient was seen was public information). But with over 20,000 servers and 300,000 total nodes on the network - some of which cannot be fully patched because they run software someone has to have access to but which won't run on the newest versions of OSs or databases - you still don't know what else they've burrowed into. A big tech company supplier to our organization had a very telling incident that ended up being detected on our side of the connection, where a brief mistake by network admins opened a channel through their layers of protection for their buid pipeline. In the minutes it was open, an APT detected the access (likely because they already owned something internal), and inserted code into their certified OS build pipeline, which we ended up with in our institution.
  - funstuff007 4y ago
    Is an APT-owned server a significantly different risk than an APT-owned desktop? They are both inside your house.
    - walnutclosefarm 4y ago
      Probably not hugely more significant, but there are differences. A workstation should be segmented and limited in the range of nodes it can communicate with, if you're running your network properly. A server will likely be in a segment that has much broader access to it, unless you're doing micro-segmentation, and doing it well. By construction, a server set up by a workgroup team outside your core IT server administration staff is unlikely to be properly segmented. And if you're doing traffic analysis to look for rogue behavior, it's harder to spot from something that profiles as a server, because, again, you expect a server to have lots of contacts within the network. Counterbalancing that, if it profiles as a server, you should be more suspicious of any outbound activity.