5 ms·

Fannie Mae Logic Bomb Would Have Caused Weeklong Shutdown

28 points by albertni 18y ago

mixmax 18y ago
This shows how fragile large systems really are.
- palish 18y ago
  If a grumpy Google engineer managed to cause their Chubby service (synchronized file write service) to start corrupting random files, then that might have a big impact on the integrity of end-user data. My point is, even a well-designed system is vulnerable when someone has access to its internals. Give me a system and a day, and I could probably think of a few subtle (or not-so-subtle) ways to break it for the purpose of inflicting damage.
  - dasil003 18y ago
    And as long as you don't rely on security-by-pageful-of-blank-lines, you just may get away with it too!
    - palish 18y ago
      I looked in the indictment for any details about the code he posted. No such luck; on a tangent, this is confusing. From the article: "A logic bomb ... would have decimated all 4,000 servers at the company, causing millions of dollars in damage" From the indictment: "The defendant ... did knowingly cause and attempt to cause ... damage without authorization to a protected computer, and by such conduct caused and would, if completed, have caused loss to Fannie Mae during any one year period aggregating at least $5,000 in value." So... 1) Why are they charging him with "attempting to cause at least $5,000 in damage" when the true damage would allegedly have been in the millions? 2) It's hard to believe "[the code] would have decimated all 4,000 servers at the company". Let's consider the worst case scenario, which is that all 4,000 computers run the infected script at 9AM with administrator privileges. Would something like 'rm -rf /' really decimate the server? If they kept backups of each server, could the IT guys simply swap in old backups? ... Now that I've talked it out, it seems likely that really would knock the server farm offline for about a week. It would be difficult to swap in backups of 4,000 servers in a timely fashion. That's a lot of boxen.
      - eli 18y ago
        Because that's the name of the crime. Attempting to cause damage under $5000 is probably a lesser charge. "Millions" is indeed at least $5000, no?
        palish 18y ago
        Certainly. It just seemed strange.
      - whatusername 18y ago
        4000 boxen? Just restoring from backups? There's a quick way to completely fail any RTO you might have set.. 1 week would give you 2.5 minutes for each box.. (obviously you could do things in parallel - but not all at once) If you had to pull backups off tape - then you'd be limited by number of tape drives - 100 maybe? If you were pulling it all in off a SAN/VTL then you've still got a whole pile of bottlenecks - disk speed, fabric, network, etc.. Remember: Backups are not a DR solution. (and conversely - a hot-site or equivalent is not a backup solution!)
        lallysingh 18y ago
        DR?
        parenthesis 18y ago
        Disaster Recovery, presumably.
      - sriramk 18y ago
        Check out the FBI affidavit at http://blog.wired.com/27bstroke6/files/fannie_complaint.pdf http://blog.wired.com/27bstroke6/files/fannie_complaint.pdf. The accused seems to have done the following - SSHed in with his user id and gained root access to a dev server. The DHCP address for the client IP was last assigned to his laptop - Created a cron job that ran a script. The script checked whether it was January 31st, 2009. If so, it did the following - Disable internal monitoring systems to disable alerts - Create a list of all servers,walk through them and disable logisn and clear out logs - Wipe out data by overwriting with zeros - Uninstall software and turn off the machines - Clear itself out and zero out the root filesystem The 'smoking gun' seems tenuous at best- the person accused seems to use similar naming conventions for his personal temp files (the .x, .y format) which I agree is unconventional. I think the real smoking gun is the fact that his laptop and his login was used
rudyfink 18y ago
In a bit of a shadow of Office Space, the alleged bomb planter's employer was named OmniTech.
- ngvrnd 18y ago
  Damn. You beat me to it.
- deleted 18y ago
  [deleted]
- anonemouse 18y ago
  I wonder if the reporter meant OmniTI, http://omniti.com/ http://omniti.com/.
johns 18y ago
"Despite Makwana's termination, Makwana's computer access was not immediately terminated" Wow.
- ramchip 18y ago
  I don't find it that surprising... usually you'd hire people you can trust in the first place. And the guy might want to pick up his stray files before leaving. It sounds harsh to me to cut off everything before even talking to the guy.
tptacek 18y ago
You know, the really nasty attackers aren't going to waste time overwriting data with zeroes.
metaguri 18y ago
This is why when an investment bank fires a trader, the security guard is there to pick him up after the meeting with his boss ends. They mail you the stuff from your desk in boxes.
sh1mmer 18y ago
I'm curious if Wired's account is to be taken as the literal truth (I know, bear with me). Why is someone being fired for a single scripting error? What the hell is QA for? More-over how stupid does Fanny Mae management be to fire someone who can write and deploy logic bomb that meticulous in half a day? Let alone let any of their staff have that level of data center access. I'm sure we aren't seeing the full picture here.
- rbanffy 18y ago
  "More-over how stupid does Fanny Mae management be to fire someone who can write and deploy logic bomb that meticulous in half a day? Let alone let any of their staff have that level of data center access." Among their lack of oversight problems, this seems to be one of the smallest.
  - sh1mmer 18y ago
    Or management oversights is symptomatic of Fanny Mae generally.
deleted 18y ago
[deleted]

28 points by albertni 18y ago

mixmax 18y ago

This shows how fragile large systems really are.

palish 18y ago
If a grumpy Google engineer managed to cause their Chubby service (synchronized file write service) to start corrupting random files, then that might have a big impact on the integrity of end-user data. My point is, even a well-designed system is vulnerable when someone has access to its internals. Give me a system and a day, and I could probably think of a few subtle (or not-so-subtle) ways to break it for the purpose of inflicting damage.
- dasil003 18y ago
  And as long as you don't rely on security-by-pageful-of-blank-lines, you just may get away with it too!
  - palish 18y ago
    I looked in the indictment for any details about the code he posted. No such luck; on a tangent, this is confusing. From the article: "A logic bomb ... would have decimated all 4,000 servers at the company, causing millions of dollars in damage" From the indictment: "The defendant ... did knowingly cause and attempt to cause ... damage without authorization to a protected computer, and by such conduct caused and would, if completed, have caused loss to Fannie Mae during any one year period aggregating at least $5,000 in value." So... 1) Why are they charging him with "attempting to cause at least $5,000 in damage" when the true damage would allegedly have been in the millions? 2) It's hard to believe "[the code] would have decimated all 4,000 servers at the company". Let's consider the worst case scenario, which is that all 4,000 computers run the infected script at 9AM with administrator privileges. Would something like 'rm -rf /' really decimate the server? If they kept backups of each server, could the IT guys simply swap in old backups? ... Now that I've talked it out, it seems likely that really would knock the server farm offline for about a week. It would be difficult to swap in backups of 4,000 servers in a timely fashion. That's a lot of boxen.
    - eli 18y ago
      Because that's the name of the crime. Attempting to cause damage under $5000 is probably a lesser charge. "Millions" is indeed at least $5000, no?
      - palish 18y ago
        Certainly. It just seemed strange.
    - whatusername 18y ago
      4000 boxen? Just restoring from backups? There's a quick way to completely fail any RTO you might have set.. 1 week would give you 2.5 minutes for each box.. (obviously you could do things in parallel - but not all at once) If you had to pull backups off tape - then you'd be limited by number of tape drives - 100 maybe? If you were pulling it all in off a SAN/VTL then you've still got a whole pile of bottlenecks - disk speed, fabric, network, etc.. Remember: Backups are not a DR solution. (and conversely - a hot-site or equivalent is not a backup solution!)
      - lallysingh 18y ago
        DR?
        parenthesis 18y ago
        Disaster Recovery, presumably.
    - sriramk 18y ago
      Check out the FBI affidavit at http://blog.wired.com/27bstroke6/files/fannie_complaint.pdf http://blog.wired.com/27bstroke6/files/fannie_complaint.pdf. The accused seems to have done the following - SSHed in with his user id and gained root access to a dev server. The DHCP address for the client IP was last assigned to his laptop - Created a cron job that ran a script. The script checked whether it was January 31st, 2009. If so, it did the following - Disable internal monitoring systems to disable alerts - Create a list of all servers,walk through them and disable logisn and clear out logs - Wipe out data by overwriting with zeros - Uninstall software and turn off the machines - Clear itself out and zero out the root filesystem The 'smoking gun' seems tenuous at best- the person accused seems to use similar naming conventions for his personal temp files (the .x, .y format) which I agree is unconventional. I think the real smoking gun is the fact that his laptop and his login was used

rudyfink 18y ago

In a bit of a shadow of Office Space, the alleged bomb planter's employer was named OmniTech.

ngvrnd 18y ago
Damn. You beat me to it.
deleted 18y ago
[deleted]
anonemouse 18y ago
I wonder if the reporter meant OmniTI, http://omniti.com/ http://omniti.com/.

johns 18y ago

"Despite Makwana's termination, Makwana's computer access was not immediately terminated" Wow.

ramchip 18y ago
I don't find it that surprising... usually you'd hire people you can trust in the first place. And the guy might want to pick up his stray files before leaving. It sounds harsh to me to cut off everything before even talking to the guy.

tptacek 18y ago

You know, the really nasty attackers aren't going to waste time overwriting data with zeroes.

metaguri 18y ago

This is why when an investment bank fires a trader, the security guard is there to pick him up after the meeting with his boss ends. They mail you the stuff from your desk in boxes.

sh1mmer 18y ago

I'm curious if Wired's account is to be taken as the literal truth (I know, bear with me). Why is someone being fired for a single scripting error? What the hell is QA for? More-over how stupid does Fanny Mae management be to fire someone who can write and deploy logic bomb that meticulous in half a day? Let alone let any of their staff have that level of data center access. I'm sure we aren't seeing the full picture here.

rbanffy 18y ago
"More-over how stupid does Fanny Mae management be to fire someone who can write and deploy logic bomb that meticulous in half a day? Let alone let any of their staff have that level of data center access." Among their lack of oversight problems, this seems to be one of the smallest.
- sh1mmer 18y ago
  Or management oversights is symptomatic of Fanny Mae generally.

deleted 18y ago

[deleted]