11 ms·

So what defenses should an organization employ to prevent these types of attacks? From this non-technical article, it looks like they penetrated employees' com

by chubot 12y ago

So what defenses should an organization employ to prevent these types of attacks? From this non-technical article, it looks like they penetrated employees' computers and used their credentials, which makes sense because it's probably the weakest link. It reminds me the philosophy/motivation behind Qubes OS [1]: there is no server security without client security. What are banks running on employee computers these days? I'm guessing Windows. Do they have anything beyond what typical corporate IT does to Windows machines (install virus checkers, auto updates, most users don't have root)? Clearly that's not sufficient. It sounds like you want some kind of strict compartmentalization like Qubes. There's probably no reason that an e-mail client like Outlook needs to share any state with whatever app they used to manage accounts. Besides perhaps sharing a clipboard for cutting and pasting a tiny amount of info. The machines probably need secure boot and attestation of the root file system state too. It's pretty bad that in this attack and I think in the Anthem case that attackers were inside their network for such a long period without detection. I also remember a DEFCON talk where a penetration tester said the hardest site he ever worked on was where they had a strict "star" network topology. None of the computers in the enterprise could talk to each other or even see each other. All communication had to be proxied through a central hub, which would audit all the connections. Do any banks do that now? Is there any reason they couldn't in practice? I imagine that there isn't really a need for two tellers in the same office to be sharing files directly with each other. Let alone tellers in different offices. I've never worked at a bank do I have no idea what their networks are like. Possibly there would be some uptime concerns with a centralized system like that. I'm just brainstorming and wondering if anyone has direct work-related experience. [1] https://qubes-os.org/ https://qubes-os.org/

aidos 12y ago
It's just staggering. I know it's silly to think that banks would be better than anyone else, but good lord, malware running on machines capable of transferring millions of dollars that's able to send out video feeds from the network without anyone noticing?! Your various IT/Security teams should be absolutely ashamed. And then the banks don't even have to stand up and admit their incompetence publicly; that's a total disgrace. That's the state of corporate security I guess. I've dealt with corporate IT departments over the years where they put these "processes" in place to mitigate these security issues but it's all a load of rubbish. Filling in forms to tick boxes so that everyone can go home happy pretending there's security going on, when really their network is a leaky sieve. At one point I saw a release by a 3rd party supplier to a large corporate system that included privilege escalation, blatantly, at the start of a T-SQL script. It was done because the IT department refused to carry out the action on request via the official channel but it was work that needed to be done to complete a project. The 3rd party knew the admins would just be running scripts as SA so they escalated their own account to do what they needed to do later. I know it's silly to be so frustrated about it, but we've all dealt with crappy banking systems for years, with totally insane security measures; meanwhile hackers can just walk away with millions using a bit of malware.
- chubot 12y ago
  So are you saying that bank IT is no better than corporate IT? They don't have any special software or policies? (like the star network thing I mentioned) I would honestly expect it to be a bit better than average. I suppose there are many different types of banks and they all vary. Let's just consider your chain banks like Wells Fargo or BoA, since I'm sure somebody around here has worked at one of those places.
  - aidos 12y ago
    To be honest, I have no experience - so I can't say, if the description in the article is at all accurate though, they're in a pretty bad way: "The cybercriminals sent their victims infected emails — a news clip or message that appeared to come from a colleague — as bait. When the bank employees clicked on the email, they inadvertently downloaded malicious code. That allowed the hackers to crawl across a bank’s network" There must be plenty of people on HN with experience in this field, so it'll be interesting to hear their take on it. My (very ranting/rambling) point was that I've seen other large organisations pretending to do security (and probably believing it themselves), where it's really just security theatre.
  - busterarm 12y ago
    I used to work in financial IT. Across the industry it's generally better than corporate IT. A lot better. However, it varies widely by sector. Companies with trading floors or that interact regularly with traders have the best IT practices in the industry. Banking conglomerates are kind of messy - they combine IT operations for each business and never change anything and the systems don't cooperate. I remember one such company's backup procedures. At that point they were made up of 13 separate large (regional/national) banks. They were trying to standardize the backup procedures between all the banks and run them from a centralized system. At the time that I got there, the nightly backup process failed every single day for over a year and a half. I didn't even get a computer or working logins to be able to do any work for nearly a month. Anyway, getting it to work involved getting the people responsible for the backups at each individual bank's IT group to get their system to cooperate. All of them knew that this would be putting them out of a job at the completion of the project, so there was tons of resistance and it usually took a week of calling peoples' bosses to get the work done. This was also in the middle of forced relocations for most of them. Most of the folks responsible for the work quit. It was really ugly.
contingencies 12y ago
So what defenses should an organization employ to prevent these types of attacks? Infrastructure architect at a major Bitcoin exchange here. It's about defense in depth. Processes. An architecture level stance like "do not trust the client, the server, the network, the data center, the hardware provider, or any particular stage within those three elements". Each element validates the other. An alarm raised by inappropriate behavior at any point will shut down an entire instance, cell, or data center before allowing an attacker a foothold. The only way to realistically take such a stance without going broke or becoming functionally paralyzed is infrastructure level automation beyond what is common in the industry. Hence, cue for meaningful cloud infrastructure management systems spanning private and arbitrary third party infrastructure. Docker-level stuff is about 1/2 way, what we really need is a few degrees of abstraction beyond that.
- chubot 12y ago
  So, I'm specifically asking about protecting employees' machines. My reading of the article is that the attackers got a foothold on employees' machines and credentials, and just piggybacked their malicious transactions along with normal transactions. In that case, it doesn't matter how much security you have in your data center. Employees need access to central systems to do their job, so client security is paramount. For instance, for a bank or Bitcoin exchange, I think it would relevant how many client operating systems can access your crown jewels. I think if you're just using Mac or Windows with antivirus or whatever, there's already a pretty low upper bound on your client security and thus your overall system security. What I'm wondering if anybody is deploying some kind of custom client OS similar in spirit to Qubes OS, or a build of Chromium OS or Android, which have application sandboxing beyond what stock Linux, Mac or Windows have. Also, I would imagine that each teller has their own credentials, and the bank should have policies about the transaction rate / total for a single teller. It sounds like the attackers would have to compromise multiple employee accounts to steal that much money. So you also want to protect employees machines from each other as much as possible (not just "outside" attackers). I'm guessing that a Bitcoin Exchange doesn't have that many employees, since the whole industry is new. You probably have people just accessing stuff with their personal MacBooks or whatever, and that's fine for now (there are bigger risks). But when you start to have 100, 1000, 10,000 employees capable of doing financial damage, then I think this type of thing will start to matter more. EDIT: Actually I remember one large deposit I made required three people at a bank to approve it. The teller said, "Wait my boss has to approve this." Then the boss said, "Wait my boss has to approve this". So they are probably using the presence of three credentials and credentials at a sufficient employee level to authorize large transactions. So I take it the attackers would have to target employees with those credentials. But that can cause problems for customers -- e.g. if the branch manager isn't around, you might not be able to do what you wanted. To some degree, they are using meat space protocols to mitigate risk that their software systems can't handle.
belorn 12y ago
The best defense an organization can employ, is to make departments/managers/people economical liable. This result in insurance being bought, budgets assign to risk management, and practical prevention mechanism being implemented. No organization like being attacked, but any defensive measure that cost money will always be balanced to the potential loss, risk, and convenience of employees. If the risk feels low, the potential loss minimum (worst case, government will intervene), and employees inconvenience high from employing effective security schemes, then no such efforts tend to be used.
- DominikR 12y ago
  I'd like to see the sysadmin or programmer that is willing to take the loss if someone hacks the network (or an app) of his employer and steals a few hundred million dollars.
  - ams6110 12y ago
    Professional Engineers (mechanical, civil, etc.) are exposed to liability for the buildings, bridges, etc. they approve.
    - smnrchrds 12y ago
      But we are not liable as long as we follow standards, e.g. building codes. And it's easily verifiable by the government, the employer and the engineer himself whether the standards are being complied with. Until you have similar standards for software development, I cannot see how such liability shift could work. This is one of the reasons I tend to avoid using the phrase software engineering. It's so different from traditional engineering that it feels incorrect to put it in the same category.
      - TwoBit 12y ago
        It's not enough to put standards in the software development. Users can misuse software regardless of how well it's written. Same as if you build a bridge and users overload it.
dguido 12y ago
The weakest link was that the computer with access to $10 million+ had access to the general web and was running a general purpose operating system at all. You don't need Qubes to secure this situation. You could use an iPad/Chromebook or a filtering proxy (whitelisted websites) and either would be sufficient.
- TwoBit 12y ago
  That seems to be the fundamental engineering flaw here. Also, their email system shouldn't allow executable attachments. The last company I worked at completely stopped all such virus infections by killing all executable attachments.
  - dguido 12y ago
    That works too! Why any of this software was running on a machine that can transfer $10mil+ is beyond me.
ryanlol 12y ago
PLEASE do not link to qubes in any security related discussion, the devs are known for their incompetence and making some rather hilarious public claims[1]. Not only that, Qubes uses a vulnerable git version from several years ago so practically anyone could go backdoor it if they cared. [1] http://en.wikipedia.org/wiki/Blue_Pill_%28software%29 http://en.wikipedia.org/wiki/Blue_Pill_%28software%29
deleted 12y ago
[deleted]
xorcist 12y ago
You are never safe from targeted attacks but that doesn't mean you should run exotic architectures or give up. There's no reason not to stop carpet bomb attacks. Most of the big known breaches have resulted from those. Don't run Outlook and don't autorun USB. That should stop most automated attacks, including all the big known ones that breached large companies such as RSA and Google. To stop the rest, don't surf from sensitive machines, and require two factor auth such as Yubikeys or RSA dongles to log in to them. Compartmentalize sensitive information on separate machines and networks, and externalise sanity checks of data transactions where possible.
burneraccount 12y ago
Six years ago I was an intern at a Wall Street firm for a year. The firm that I worked for used an account system that was built in the early 90s and relied on all employees learning special terminal commands to access anything. I can't really go into detail for fear of being sued, but suffice to say the system was archaic. I was amazed that a multi billion dollar company relied so heavily on and invested so little in something essential to the business. The IT department seemed to use the following logic to justify it: the system served its purpose, the legacy employees already knew how to use it and the developers who made it were long gone thus it was cheaper and easier to just leave it be. While my firm had plenty of developers that could rewrite the system from scratch, their attention was devoted solely to money making endeavors like trading platforms and client facing projects. As for the "Enterprise Architecture Group" (ie the developer department) that I interned in, the big problem was the heavy reliance on third party development companies. While the firm wanted to hire more developers, simply put very few developers want to work for banks (it's funny though that people in finance would have killed to work at the firm). It would take 6 months to a year on average to fill a developer position and they would have to pay a big premium over the average dev salary with a large yearly bonus. In order to keep up with all the various projects, they would pay third party development/consulting companies millions to come in and create apps. While this allowed the firm to get the necessary apps "done", it created the most crazy spaghetti architecture you could ever imagine. All these different apps were built using different companies/languages/platforms/technologies then thrown together in a big mish mash of iframes and duct tape. The fact that any of them were able to communicate with each other at all was a miracle. I don't actually blame the developers themselves for this, they would constantly voice their concerns while the completely clueless department head/"architects"/project managers/business analysts would shoot them down. They would say things like "I understand your concerns but Super Consultancy X says that they would do it and it will only take 12 to 18 months!! They are even available to help support the app once its finished!!". Security and use experience were not even on the company's radar, only making money.

by chubot 12y ago

aidos 12y ago

It's just staggering. I know it's silly to think that banks would be better than anyone else, but good lord, malware running on machines capable of transferring millions of dollars that's able to send out video feeds from the network without anyone noticing?! Your various IT/Security teams should be absolutely ashamed. And then the banks don't even have to stand up and admit their incompetence publicly; that's a total disgrace. That's the state of corporate security I guess. I've dealt with corporate IT departments over the years where they put these "processes" in place to mitigate these security issues but it's all a load of rubbish. Filling in forms to tick boxes so that everyone can go home happy pretending there's security going on, when really their network is a leaky sieve. At one point I saw a release by a 3rd party supplier to a large corporate system that included privilege escalation, blatantly, at the start of a T-SQL script. It was done because the IT department refused to carry out the action on request via the official channel but it was work that needed to be done to complete a project. The 3rd party knew the admins would just be running scripts as SA so they escalated their own account to do what they needed to do later. I know it's silly to be so frustrated about it, but we've all dealt with crappy banking systems for years, with totally insane security measures; meanwhile hackers can just walk away with millions using a bit of malware.

chubot 12y ago
So are you saying that bank IT is no better than corporate IT? They don't have any special software or policies? (like the star network thing I mentioned) I would honestly expect it to be a bit better than average. I suppose there are many different types of banks and they all vary. Let's just consider your chain banks like Wells Fargo or BoA, since I'm sure somebody around here has worked at one of those places.
- aidos 12y ago
  To be honest, I have no experience - so I can't say, if the description in the article is at all accurate though, they're in a pretty bad way: "The cybercriminals sent their victims infected emails — a news clip or message that appeared to come from a colleague — as bait. When the bank employees clicked on the email, they inadvertently downloaded malicious code. That allowed the hackers to crawl across a bank’s network" There must be plenty of people on HN with experience in this field, so it'll be interesting to hear their take on it. My (very ranting/rambling) point was that I've seen other large organisations pretending to do security (and probably believing it themselves), where it's really just security theatre.
- busterarm 12y ago
  I used to work in financial IT. Across the industry it's generally better than corporate IT. A lot better. However, it varies widely by sector. Companies with trading floors or that interact regularly with traders have the best IT practices in the industry. Banking conglomerates are kind of messy - they combine IT operations for each business and never change anything and the systems don't cooperate. I remember one such company's backup procedures. At that point they were made up of 13 separate large (regional/national) banks. They were trying to standardize the backup procedures between all the banks and run them from a centralized system. At the time that I got there, the nightly backup process failed every single day for over a year and a half. I didn't even get a computer or working logins to be able to do any work for nearly a month. Anyway, getting it to work involved getting the people responsible for the backups at each individual bank's IT group to get their system to cooperate. All of them knew that this would be putting them out of a job at the completion of the project, so there was tons of resistance and it usually took a week of calling peoples' bosses to get the work done. This was also in the middle of forced relocations for most of them. Most of the folks responsible for the work quit. It was really ugly.

contingencies 12y ago

So what defenses should an organization employ to prevent these types of attacks? Infrastructure architect at a major Bitcoin exchange here. It's about defense in depth. Processes. An architecture level stance like "do not trust the client, the server, the network, the data center, the hardware provider, or any particular stage within those three elements". Each element validates the other. An alarm raised by inappropriate behavior at any point will shut down an entire instance, cell, or data center before allowing an attacker a foothold. The only way to realistically take such a stance without going broke or becoming functionally paralyzed is infrastructure level automation beyond what is common in the industry. Hence, cue for meaningful cloud infrastructure management systems spanning private and arbitrary third party infrastructure. Docker-level stuff is about 1/2 way, what we really need is a few degrees of abstraction beyond that.

chubot 12y ago
So, I'm specifically asking about protecting employees' machines. My reading of the article is that the attackers got a foothold on employees' machines and credentials, and just piggybacked their malicious transactions along with normal transactions. In that case, it doesn't matter how much security you have in your data center. Employees need access to central systems to do their job, so client security is paramount. For instance, for a bank or Bitcoin exchange, I think it would relevant how many client operating systems can access your crown jewels. I think if you're just using Mac or Windows with antivirus or whatever, there's already a pretty low upper bound on your client security and thus your overall system security. What I'm wondering if anybody is deploying some kind of custom client OS similar in spirit to Qubes OS, or a build of Chromium OS or Android, which have application sandboxing beyond what stock Linux, Mac or Windows have. Also, I would imagine that each teller has their own credentials, and the bank should have policies about the transaction rate / total for a single teller. It sounds like the attackers would have to compromise multiple employee accounts to steal that much money. So you also want to protect employees machines from each other as much as possible (not just "outside" attackers). I'm guessing that a Bitcoin Exchange doesn't have that many employees, since the whole industry is new. You probably have people just accessing stuff with their personal MacBooks or whatever, and that's fine for now (there are bigger risks). But when you start to have 100, 1000, 10,000 employees capable of doing financial damage, then I think this type of thing will start to matter more. EDIT: Actually I remember one large deposit I made required three people at a bank to approve it. The teller said, "Wait my boss has to approve this." Then the boss said, "Wait my boss has to approve this". So they are probably using the presence of three credentials and credentials at a sufficient employee level to authorize large transactions. So I take it the attackers would have to target employees with those credentials. But that can cause problems for customers -- e.g. if the branch manager isn't around, you might not be able to do what you wanted. To some degree, they are using meat space protocols to mitigate risk that their software systems can't handle.

belorn 12y ago

The best defense an organization can employ, is to make departments/managers/people economical liable. This result in insurance being bought, budgets assign to risk management, and practical prevention mechanism being implemented. No organization like being attacked, but any defensive measure that cost money will always be balanced to the potential loss, risk, and convenience of employees. If the risk feels low, the potential loss minimum (worst case, government will intervene), and employees inconvenience high from employing effective security schemes, then no such efforts tend to be used.

DominikR 12y ago
I'd like to see the sysadmin or programmer that is willing to take the loss if someone hacks the network (or an app) of his employer and steals a few hundred million dollars.
- ams6110 12y ago
  Professional Engineers (mechanical, civil, etc.) are exposed to liability for the buildings, bridges, etc. they approve.
  - smnrchrds 12y ago
    But we are not liable as long as we follow standards, e.g. building codes. And it's easily verifiable by the government, the employer and the engineer himself whether the standards are being complied with. Until you have similar standards for software development, I cannot see how such liability shift could work. This is one of the reasons I tend to avoid using the phrase software engineering. It's so different from traditional engineering that it feels incorrect to put it in the same category.
    - TwoBit 12y ago
      It's not enough to put standards in the software development. Users can misuse software regardless of how well it's written. Same as if you build a bridge and users overload it.

dguido 12y ago

The weakest link was that the computer with access to $10 million+ had access to the general web and was running a general purpose operating system at all. You don't need Qubes to secure this situation. You could use an iPad/Chromebook or a filtering proxy (whitelisted websites) and either would be sufficient.

TwoBit 12y ago
That seems to be the fundamental engineering flaw here. Also, their email system shouldn't allow executable attachments. The last company I worked at completely stopped all such virus infections by killing all executable attachments.
- dguido 12y ago
  That works too! Why any of this software was running on a machine that can transfer $10mil+ is beyond me.

ryanlol 12y ago

PLEASE do not link to qubes in any security related discussion, the devs are known for their incompetence and making some rather hilarious public claims[1]. Not only that, Qubes uses a vulnerable git version from several years ago so practically anyone could go backdoor it if they cared. [1] http://en.wikipedia.org/wiki/Blue_Pill_%28software%29 http://en.wikipedia.org/wiki/Blue_Pill_%28software%29

deleted 12y ago

[deleted]

xorcist 12y ago

You are never safe from targeted attacks but that doesn't mean you should run exotic architectures or give up. There's no reason not to stop carpet bomb attacks. Most of the big known breaches have resulted from those. Don't run Outlook and don't autorun USB. That should stop most automated attacks, including all the big known ones that breached large companies such as RSA and Google. To stop the rest, don't surf from sensitive machines, and require two factor auth such as Yubikeys or RSA dongles to log in to them. Compartmentalize sensitive information on separate machines and networks, and externalise sanity checks of data transactions where possible.

burneraccount 12y ago

Six years ago I was an intern at a Wall Street firm for a year. The firm that I worked for used an account system that was built in the early 90s and relied on all employees learning special terminal commands to access anything. I can't really go into detail for fear of being sued, but suffice to say the system was archaic. I was amazed that a multi billion dollar company relied so heavily on and invested so little in something essential to the business. The IT department seemed to use the following logic to justify it: the system served its purpose, the legacy employees already knew how to use it and the developers who made it were long gone thus it was cheaper and easier to just leave it be. While my firm had plenty of developers that could rewrite the system from scratch, their attention was devoted solely to money making endeavors like trading platforms and client facing projects. As for the "Enterprise Architecture Group" (ie the developer department) that I interned in, the big problem was the heavy reliance on third party development companies. While the firm wanted to hire more developers, simply put very few developers want to work for banks (it's funny though that people in finance would have killed to work at the firm). It would take 6 months to a year on average to fill a developer position and they would have to pay a big premium over the average dev salary with a large yearly bonus. In order to keep up with all the various projects, they would pay third party development/consulting companies millions to come in and create apps. While this allowed the firm to get the necessary apps "done", it created the most crazy spaghetti architecture you could ever imagine. All these different apps were built using different companies/languages/platforms/technologies then thrown together in a big mish mash of iframes and duct tape. The fact that any of them were able to communicate with each other at all was a miracle. I don't actually blame the developers themselves for this, they would constantly voice their concerns while the completely clueless department head/"architects"/project managers/business analysts would shoot them down. They would say things like "I understand your concerns but Super Consultancy X says that they would do it and it will only take 12 to 18 months!! They are even available to help support the app once its finished!!". Security and use experience were not even on the company's radar, only making money.