5 ms·

Morgan Stanley didn't wipe their hard drives before giving them to a third party

90 points by Benlights 4y ago

baobabKoodaa 4y ago
The real mistake in this trainwreck was that Morgan Stanley didn't encrypt their hard drives.
- kube-system 4y ago
  Many devices that a business might throw away in 2015 may not have been devices that supported full disk encryption out of the box.
  - kadoban 4y ago
    I've been using fde since like 2005 or so and I'm not even a bank. Seems like they don't have much excuse?
    - kube-system 4y ago
      That's at least two years before even the most forward thinking offices started using FDE on PCs. Bitlocker came out in 07.
      - myself248 4y ago
        Yeah, MS was ultra behind. Scramdisk came out in 1998 or 99 as I recall. I wasn't working in IT so I have no idea what corporate policy was like at the time, but it was highly recommended in hacker circles. It can't have been that hard.
        Alupis 4y ago
        Rather, things like Scramdisk were ahead of their time and nearly exclusively for enthusiasts and security gurus. In the early 2000's, any sort of encryption was a non-trivial burden on already slow (by today's standards) systems. Plus the whole export encryption fiasco and more. I'd say FDE didn't really take off until your mobile devices started to offer it by default, and make it easy enough that regular users don't ever need to think about it. Now pretty much all operating systems support FDE "out of the box". Saying folks should have been running FDE back in the early 2000's is just absurd, really.
      - xoa 4y ago
        Yeah 2005 for FDE would be pretty early adopter territory. On the Mac side Apple launched FileVault version 1 with 10.3 in 2003, but that only encrypted user home directories (IIRC it effectively was an attempt at transparently running a home directory off an encrypted disk image). Actual FDE came with FileVault 2 and 10.7 Lion, which wasn't until 2011. Though at the same time while we've gotten used to banks lagging horribly on tech, given their resources and the sensitivity of the information they deal with an argument can be made that they should be leading not lagging and that cost cutting and lack of leadership interest aren't great excuses for delays. I do think by 2015 yeah that was getting kind of bad. On the other hand, the penalty wasn't much ($35m in 2022 would be worth a lot less to them working back 7 years). It might still have been cheaper to setup FDE back then. Optimistically, there may be Morgan Stanley clients well off enough to mount real private lawsuits or at least take quite a lot of money elsewhere if they're irritated enough, so while this penalty alone might not be much of a lesson about PII perhaps they'll still come to regret it a little :\.
      - kadoban 4y ago
        Yeah I got into it early via Truecrypt and dmcrypt in linux.
    - luma 4y ago
      Are you running an EMC CLARiiON array with the export encryption option licensed? What works on your desktop isn't really comparable to what Morgan Stanley had on the datacenter floor 10 years ago.
      - Sebb767 4y ago
        Yeah, but they don't have millions of dollars available and they don't store personal information of millions of people. Morgan Stanley is and was clearly able to afford this license and if you need to handle data this sensitive, you must meet the necessary precautions.
        spydum 4y ago
        Just because the company has money, doesn't mean the department or line of business does. I didn't read the report but wouldn't surprise me if this was some acquisition infra (it would explain why they hired a moving company)
        Sebb767 4y ago
        Building a nuclear reactor without shielding is not okay just because the business department responsible for the construction doesn't have it in their budget this year. If you can't afford to encrypt this amount of data, you can't afford to store it, end of story. > I didn't read the report but wouldn't surprise me if this was some acquisition infra (it would explain why they hired a moving company) I didn't read it yet, either, but this seems unlikely - why would an acquisition have so much customer data in their DC? And if they had so much data, why didn't they encrypt it beforehand? Anyway, in the end, it was still Morgan Stanley that hired the moving company, so they f-ed up either way.
      - kadoban 4y ago
        Yes what they have should probably be fancier. I don't think this argues against my point?
  - Li7h 4y ago
    >Moreover, during this process, MSSB also learned that the local devices being decommissioned had been equipped with encryption capability, but that the firm had failed to activate the encryption software for years.
- Sebb767 4y ago
  Fully agreed. People like to complain about disk encryption just being a security theater checkbox in a cloud or datacenter environment, but this is exactly why it is needed.
- nikau 4y ago
  Worth nothing that Morgan Stanley essentially acquired Smith Barney a few years before this issue, so likely procedures were not synced up or ignored due to the integration work going on.
billybuckwheat 4y ago
>MSSB hired a moving and storage company with no experience or expertise in data destruction services to decommission thousands of hard drives and servers Guess the smartest people in the room weren't in the IT department ... Wonder if they chose that moving and storage company because they were a cheaper option.
duxup 4y ago
I used to visit the data centers of some very large financial institutions. The SoP at those places was that hard drives from the data center NEVER left the building except through a device that destroyed them…. Their security guards were really into checking for them and etc. It was a pretty common rule across those banks and etc at that time, and that was quite a while ago.
- hangonhn 4y ago
  Same. When I used to work at a hedge the standard procedure was to zero them out first. Then retain the hard drives in a closet. Then someone could come periodically to physically destroy them.
  - toomuchtodo 4y ago
    How was the process governed so that drives actually were wiped before going out the door? That’s really the challenge, the humans managing the kit are the weakest link. I do like the comment about drives only able to depart the premises through a shredder.
    - hangonhn 4y ago
      I think it was just governed by some IT policy which was basically, "no bare hard drives leaves the IT room". The people who literally shredded the hard drives would give us the literal bits back. It's kind of nutty but information is the life blood of hedge funds.
- cm2187 4y ago
  At the same time I heard several stories of people copying the files onto their desktop hard drives and leaving the building with them when lehman went bust.
  - duxup 4y ago
    I didn't see any indication that the hard drives from the data center policy had anything to do with drone's laptops coming and going. To be clear in one building there were a few thousand people working. When I visited myself and maybe a dozen or two dozen other people in the building had access to the data center. Cameras everywhere, appointment verification, IDs, man traps and all. I'd visit and go up to the doors and passers by would stop to watch "he's going inside..." Whatever a random drone was doing with their laptop, that's a whole other issue / policy. It was even more fun at military sites. NOTHING non essential ever left. You, your ID (they held it), your clothing, glasses... that was all that came out, your laptop and any spare parts were left behind every time. If you went to the very special sites... you also made sure nothing was in your car that you didn't want to lose.
snarfy 4y ago
> "Today’s action sends a clear message to financial institutions that they must take seriously their obligation to safeguard such data.” $35 million fine for 15 million customer's PII. The 'clear message' is that a customer's PII is worth about $2. Meanwhile the customers are on the hook for fraud monitoring in perpetuity.
- routerl 4y ago
  "Punishable by fine" means "legal if you can afford it". Until living, breathing, actual people face real consequences for this kind of thing, any enforcement actions are just theater.
  - cyanydeez 4y ago
    For corporations it means, "make sure there's a line item for the fines" At least humans are mostly controlled by ethics and morals. Corporations, not so much.
- cyanydeez 4y ago
  2$ for enforcement. Isn't like annually worth like $10.
- smabie 4y ago
  Could you sell each customers PII for more than $2 on darkweb? I kinda doubt it, seems like fair fine to me.
  - psd1 4y ago
    lolwut. If I cut open your £180k Aston Martin with an angle grinder to steal a pair of sunglasses that I sell in a pub for £10, should my fine be £11?
  - ShowalkKama 4y ago
    depends on what data we are talking about but yea, it can easily go for 5-10$ (highly depends on the information we are talking about) especially for financial data
rizza 4y ago
Opinions are my own As someone who works for a large financial institution, THIS SHOULD NEVER HAVE HAPPENED! This could be deeply flawed security and controls processes, a culture of not my problem, their tech leadership being incompetent, or CFO driving CIO/CTO decision making. Either way this is not the sign of a healthy company and the rot likely runs much deeper. You dont make this kind of mistake in this industry at a firm of that size.
advisedwang 4y ago
I wonder if the "moving and storage company with no experience or expertise in data destruction" was owned by a relative or friend of a Morgan Stanley exec.
notaccept 4y ago
This is completey unacceptable - not wiping your hard dicks before going to a party, that too a third one? Stop the madness. Not cool at all.

90 points by Benlights 4y ago

baobabKoodaa 4y ago

The real mistake in this trainwreck was that Morgan Stanley didn't encrypt their hard drives.

kube-system 4y ago
Many devices that a business might throw away in 2015 may not have been devices that supported full disk encryption out of the box.
- kadoban 4y ago
  I've been using fde since like 2005 or so and I'm not even a bank. Seems like they don't have much excuse?
  - kube-system 4y ago
    That's at least two years before even the most forward thinking offices started using FDE on PCs. Bitlocker came out in 07.
    - myself248 4y ago
      Yeah, MS was ultra behind. Scramdisk came out in 1998 or 99 as I recall. I wasn't working in IT so I have no idea what corporate policy was like at the time, but it was highly recommended in hacker circles. It can't have been that hard.
      - Alupis 4y ago
        Rather, things like Scramdisk were ahead of their time and nearly exclusively for enthusiasts and security gurus. In the early 2000's, any sort of encryption was a non-trivial burden on already slow (by today's standards) systems. Plus the whole export encryption fiasco and more. I'd say FDE didn't really take off until your mobile devices started to offer it by default, and make it easy enough that regular users don't ever need to think about it. Now pretty much all operating systems support FDE "out of the box". Saying folks should have been running FDE back in the early 2000's is just absurd, really.
    - xoa 4y ago
      Yeah 2005 for FDE would be pretty early adopter territory. On the Mac side Apple launched FileVault version 1 with 10.3 in 2003, but that only encrypted user home directories (IIRC it effectively was an attempt at transparently running a home directory off an encrypted disk image). Actual FDE came with FileVault 2 and 10.7 Lion, which wasn't until 2011. Though at the same time while we've gotten used to banks lagging horribly on tech, given their resources and the sensitivity of the information they deal with an argument can be made that they should be leading not lagging and that cost cutting and lack of leadership interest aren't great excuses for delays. I do think by 2015 yeah that was getting kind of bad. On the other hand, the penalty wasn't much ($35m in 2022 would be worth a lot less to them working back 7 years). It might still have been cheaper to setup FDE back then. Optimistically, there may be Morgan Stanley clients well off enough to mount real private lawsuits or at least take quite a lot of money elsewhere if they're irritated enough, so while this penalty alone might not be much of a lesson about PII perhaps they'll still come to regret it a little :\.
    - kadoban 4y ago
      Yeah I got into it early via Truecrypt and dmcrypt in linux.
  - luma 4y ago
    Are you running an EMC CLARiiON array with the export encryption option licensed? What works on your desktop isn't really comparable to what Morgan Stanley had on the datacenter floor 10 years ago.
    - Sebb767 4y ago
      Yeah, but they don't have millions of dollars available and they don't store personal information of millions of people. Morgan Stanley is and was clearly able to afford this license and if you need to handle data this sensitive, you must meet the necessary precautions.
      - spydum 4y ago
        Just because the company has money, doesn't mean the department or line of business does. I didn't read the report but wouldn't surprise me if this was some acquisition infra (it would explain why they hired a moving company)
        Sebb767 4y ago
        Building a nuclear reactor without shielding is not okay just because the business department responsible for the construction doesn't have it in their budget this year. If you can't afford to encrypt this amount of data, you can't afford to store it, end of story. > I didn't read the report but wouldn't surprise me if this was some acquisition infra (it would explain why they hired a moving company) I didn't read it yet, either, but this seems unlikely - why would an acquisition have so much customer data in their DC? And if they had so much data, why didn't they encrypt it beforehand? Anyway, in the end, it was still Morgan Stanley that hired the moving company, so they f-ed up either way.
    - kadoban 4y ago
      Yes what they have should probably be fancier. I don't think this argues against my point?
- Li7h 4y ago
  >Moreover, during this process, MSSB also learned that the local devices being decommissioned had been equipped with encryption capability, but that the firm had failed to activate the encryption software for years.
Sebb767 4y ago
Fully agreed. People like to complain about disk encryption just being a security theater checkbox in a cloud or datacenter environment, but this is exactly why it is needed.
nikau 4y ago
Worth nothing that Morgan Stanley essentially acquired Smith Barney a few years before this issue, so likely procedures were not synced up or ignored due to the integration work going on.

billybuckwheat 4y ago

>MSSB hired a moving and storage company with no experience or expertise in data destruction services to decommission thousands of hard drives and servers Guess the smartest people in the room weren't in the IT department ... Wonder if they chose that moving and storage company because they were a cheaper option.

duxup 4y ago

I used to visit the data centers of some very large financial institutions. The SoP at those places was that hard drives from the data center NEVER left the building except through a device that destroyed them…. Their security guards were really into checking for them and etc. It was a pretty common rule across those banks and etc at that time, and that was quite a while ago.

hangonhn 4y ago
Same. When I used to work at a hedge the standard procedure was to zero them out first. Then retain the hard drives in a closet. Then someone could come periodically to physically destroy them.
- toomuchtodo 4y ago
  How was the process governed so that drives actually were wiped before going out the door? That’s really the challenge, the humans managing the kit are the weakest link. I do like the comment about drives only able to depart the premises through a shredder.
  - hangonhn 4y ago
    I think it was just governed by some IT policy which was basically, "no bare hard drives leaves the IT room". The people who literally shredded the hard drives would give us the literal bits back. It's kind of nutty but information is the life blood of hedge funds.
cm2187 4y ago
At the same time I heard several stories of people copying the files onto their desktop hard drives and leaving the building with them when lehman went bust.
- duxup 4y ago
  I didn't see any indication that the hard drives from the data center policy had anything to do with drone's laptops coming and going. To be clear in one building there were a few thousand people working. When I visited myself and maybe a dozen or two dozen other people in the building had access to the data center. Cameras everywhere, appointment verification, IDs, man traps and all. I'd visit and go up to the doors and passers by would stop to watch "he's going inside..." Whatever a random drone was doing with their laptop, that's a whole other issue / policy. It was even more fun at military sites. NOTHING non essential ever left. You, your ID (they held it), your clothing, glasses... that was all that came out, your laptop and any spare parts were left behind every time. If you went to the very special sites... you also made sure nothing was in your car that you didn't want to lose.

snarfy 4y ago

> "Today’s action sends a clear message to financial institutions that they must take seriously their obligation to safeguard such data.” $35 million fine for 15 million customer's PII. The 'clear message' is that a customer's PII is worth about $2. Meanwhile the customers are on the hook for fraud monitoring in perpetuity.

routerl 4y ago
"Punishable by fine" means "legal if you can afford it". Until living, breathing, actual people face real consequences for this kind of thing, any enforcement actions are just theater.
- cyanydeez 4y ago
  For corporations it means, "make sure there's a line item for the fines" At least humans are mostly controlled by ethics and morals. Corporations, not so much.
cyanydeez 4y ago
2$ for enforcement. Isn't like annually worth like $10.
smabie 4y ago
Could you sell each customers PII for more than $2 on darkweb? I kinda doubt it, seems like fair fine to me.
- psd1 4y ago
  lolwut. If I cut open your £180k Aston Martin with an angle grinder to steal a pair of sunglasses that I sell in a pub for £10, should my fine be £11?
- ShowalkKama 4y ago
  depends on what data we are talking about but yea, it can easily go for 5-10$ (highly depends on the information we are talking about) especially for financial data

rizza 4y ago

Opinions are my own As someone who works for a large financial institution, THIS SHOULD NEVER HAVE HAPPENED! This could be deeply flawed security and controls processes, a culture of not my problem, their tech leadership being incompetent, or CFO driving CIO/CTO decision making. Either way this is not the sign of a healthy company and the rot likely runs much deeper. You dont make this kind of mistake in this industry at a firm of that size.

advisedwang 4y ago

I wonder if the "moving and storage company with no experience or expertise in data destruction" was owned by a relative or friend of a Morgan Stanley exec.

notaccept 4y ago

This is completey unacceptable - not wiping your hard dicks before going to a party, that too a third one? Stop the madness. Not cool at all.