7 ms·

Hacked. A Short Story

147 points by skazka16 12y ago

olalonde 12y ago
Stories like this is what makes me believe immutable infrastructure is the future.
- phazmatis 12y ago
  Exactly. Does your business model care if your server is running apache or nginx? No? Then why the hell isn't that automated, baked, and tamper-proofed? We'll know we've arrived when the only feasible attack is sneaking malicious code into a git repo and hoping no one notices.
- curiously 12y ago
  curious to know how you can make a linux distro immutable from any modification.
  - stonogo 12y ago
    That's probably why it's the future and not the present.
  - CiaranMcNulty 12y ago
    By regularly redeploying or re-imaging. Exploits can only live for a certain timescale + you can lock down nearly every entry point (no need for SSH access for instance).
    - anon4 12y ago
      How do you redeploy without some sort of remote access?
      - acveilleux 12y ago
        You use cloud VM API and images that use something like chef or salt to configure themselves. Basically, you destroy a VM and spin up a new one with one of your images, it will then use chef or salt to pull in its config. SSH isn't needed then.
  - pjc50 12y ago
    Mount the root filesystem read-only? Boot from CD?
  - kalleboo 12y ago
    Toggle the Read-Only window on the floppy disk
  - ay 12y ago
    Build a custom LiveISO. There are plenty of tutorials, but it is boring to follow them manually, especially if you need to make a couple of tweaks to ISO or update it. So I put together a couple of scripts to automate the process: https://github.com/ayourtch/iso-livecd https://github.com/ayourtch/iso-livecd
    - Sir_Cmpwn 12y ago
      Get root, create a ramdisk, copy /usr/bin/* into it, and remount it under /usr/bin/*, and now you can do as you please.
- song 12y ago
  And that's why I'm quite excited by the concept of Coreos and the way they do updates.
Romkinson 12y ago
Read at habrahabr months ago. Great story
- x4m 12y ago
  Actually 3 years ago. But they credit original source. http://habrahabr.ru/post/132668/ http://habrahabr.ru/post/132668/
derFunk 12y ago
What's really important is that you never should keep a compromised system like this running, even if you think you found all modifications the attacker did. You probably didn't. So save your configs and set this machine up from scratch.
- adwf 12y ago
  I was gonna say much the same thing. Once a machine has been compromised, you've got to format and re-image it. The machine in the article could well still have a rootkit installed. Not to mention he doesn't seem to know the initial attack vector, so any security updates are meaningless if someones credentials have been stolen. Finally, if you're really paranoid, you also have to flash the bios in the machine too as there are rootkits that can potentially survive formatting...
  - swombat 12y ago
    I'd add that you don't even need to have the machine actually be compromised. If you think it might be compromised, if you have reason to believe that perhaps something bad happened, you probably should re-image it. There may be smoke without a fire, but it's not worth the risk, is it?
  - juliangregorian 12y ago
    This seems like a great reason to virtualize everything possible.
    - 0xCMP 12y ago
      True, but even this isn't always perfect. I was told recently that Xen shares the kernel memory space among VMs so if once is compromised at the kernel level all of them could be compromised. This person dislikes using Amazon for this reason because he can't be sure his box isn't popped through another VM.
      - deleted 12y ago
        [deleted]
ludwigvan 12y ago
He did not ask for money? Why on Earth? Dear fellow developers, know your importance and always ask for the work you have done.
- Luyt 12y ago
  He didn't configure exim4, for which he was hired. He wasn't hired to investigate the compromise, which he did do.
  - song 12y ago
    Still, he provided a service that was important for the customer. And, the customer wanted him to configure exim4 because a lot of the emails sent were marked as spam. The compromise was most likely the biggest reason for that.
    - acveilleux 12y ago
      Sometimes it's like that. He could try and get the customer to pay him for this but he might lose on the long term if they've been a recurring customer for a long time.
      - shaftoe 12y ago
        Let's be honest. The moment he discovered an attacker he should have been on the phone with the client explaining the situation and asking how to proceed. If the client wanted him to shut down and clean up the hack, he should be charging for that. If not, he should have done nothing. Transparency resolves most problems.
- tikl1 12y ago
  It's complicated to just do something you were not supposed to and then go with: "look I know I had to do that but instead I did that so please give me [amount you never aggreed to pay]". What he could have done on the other hand is: inform the client of the attack and propose him to get rid of it for a certain fee...
ronnier 12y ago
Offtopic: blog headers are becoming larger by the year it seems. Have there been any studies to see what such large headers do to readership?
- fmavituna 12y ago
  Ironically new standard wider monitor screen ratios make you see nothing before scrolling, I guess we need to "above the fold" concept for web dev.
  - LukeB_UK 12y ago
    "There is no fold" by Luke Wroblewski: https://storify.com/xiwcx/there-is-no-fold-1 https://storify.com/xiwcx/there-is-no-fold-1
- ProAm 12y ago
  Splash pages are back [1] [1] http://www.cosmonautdreams.com/html/blog/websites/readability/ux/2014/01/06/Are-Splash-Pages-Back.html http://www.cosmonautdreams.com/html/blog/websites/readabilit...
  - juliangregorian 12y ago
    I rather prefer these to splash screens, they evoke the opposing page to the beginning of a magazine article; splash screens feel more like studio logo animations before a film, only much less rewarding. Your source's UX qualifications also seem a bit... dubious.
    - ProAm 12y ago
      > Your source's UX qualifications also seem a bit... dubious. Just a developer and sys admin, my qualifications come purely from usage. I only noticed the trend of web development going back to what it was in the early 2000's where landing pages & splash screens were all the rage. I hate reading any blog on medium or that follows this format to me it detracts from the content and is visually irritating.
jwildeboer 12y ago
AFAICS the system wasn't updated for a year? Well, that's just plain stupid.

147 points by skazka16 12y ago

olalonde 12y ago

Stories like this is what makes me believe immutable infrastructure is the future.

phazmatis 12y ago
Exactly. Does your business model care if your server is running apache or nginx? No? Then why the hell isn't that automated, baked, and tamper-proofed? We'll know we've arrived when the only feasible attack is sneaking malicious code into a git repo and hoping no one notices.
curiously 12y ago
curious to know how you can make a linux distro immutable from any modification.
- stonogo 12y ago
  That's probably why it's the future and not the present.
- CiaranMcNulty 12y ago
  By regularly redeploying or re-imaging. Exploits can only live for a certain timescale + you can lock down nearly every entry point (no need for SSH access for instance).
  - anon4 12y ago
    How do you redeploy without some sort of remote access?
    - acveilleux 12y ago
      You use cloud VM API and images that use something like chef or salt to configure themselves. Basically, you destroy a VM and spin up a new one with one of your images, it will then use chef or salt to pull in its config. SSH isn't needed then.
- pjc50 12y ago
  Mount the root filesystem read-only? Boot from CD?
- kalleboo 12y ago
  Toggle the Read-Only window on the floppy disk
- ay 12y ago
  Build a custom LiveISO. There are plenty of tutorials, but it is boring to follow them manually, especially if you need to make a couple of tweaks to ISO or update it. So I put together a couple of scripts to automate the process: https://github.com/ayourtch/iso-livecd https://github.com/ayourtch/iso-livecd
  - Sir_Cmpwn 12y ago
    Get root, create a ramdisk, copy /usr/bin/* into it, and remount it under /usr/bin/*, and now you can do as you please.
song 12y ago
And that's why I'm quite excited by the concept of Coreos and the way they do updates.

Romkinson 12y ago

Read at habrahabr months ago. Great story

x4m 12y ago
Actually 3 years ago. But they credit original source. http://habrahabr.ru/post/132668/ http://habrahabr.ru/post/132668/

derFunk 12y ago

What's really important is that you never should keep a compromised system like this running, even if you think you found all modifications the attacker did. You probably didn't. So save your configs and set this machine up from scratch.

adwf 12y ago
I was gonna say much the same thing. Once a machine has been compromised, you've got to format and re-image it. The machine in the article could well still have a rootkit installed. Not to mention he doesn't seem to know the initial attack vector, so any security updates are meaningless if someones credentials have been stolen. Finally, if you're really paranoid, you also have to flash the bios in the machine too as there are rootkits that can potentially survive formatting...
- swombat 12y ago
  I'd add that you don't even need to have the machine actually be compromised. If you think it might be compromised, if you have reason to believe that perhaps something bad happened, you probably should re-image it. There may be smoke without a fire, but it's not worth the risk, is it?
- juliangregorian 12y ago
  This seems like a great reason to virtualize everything possible.
  - 0xCMP 12y ago
    True, but even this isn't always perfect. I was told recently that Xen shares the kernel memory space among VMs so if once is compromised at the kernel level all of them could be compromised. This person dislikes using Amazon for this reason because he can't be sure his box isn't popped through another VM.
    - deleted 12y ago
      [deleted]

ludwigvan 12y ago

He did not ask for money? Why on Earth? Dear fellow developers, know your importance and always ask for the work you have done.

Luyt 12y ago
He didn't configure exim4, for which he was hired. He wasn't hired to investigate the compromise, which he did do.
- song 12y ago
  Still, he provided a service that was important for the customer. And, the customer wanted him to configure exim4 because a lot of the emails sent were marked as spam. The compromise was most likely the biggest reason for that.
  - acveilleux 12y ago
    Sometimes it's like that. He could try and get the customer to pay him for this but he might lose on the long term if they've been a recurring customer for a long time.
    - shaftoe 12y ago
      Let's be honest. The moment he discovered an attacker he should have been on the phone with the client explaining the situation and asking how to proceed. If the client wanted him to shut down and clean up the hack, he should be charging for that. If not, he should have done nothing. Transparency resolves most problems.
tikl1 12y ago
It's complicated to just do something you were not supposed to and then go with: "look I know I had to do that but instead I did that so please give me [amount you never aggreed to pay]". What he could have done on the other hand is: inform the client of the attack and propose him to get rid of it for a certain fee...

ronnier 12y ago

Offtopic: blog headers are becoming larger by the year it seems. Have there been any studies to see what such large headers do to readership?

fmavituna 12y ago
Ironically new standard wider monitor screen ratios make you see nothing before scrolling, I guess we need to "above the fold" concept for web dev.
- LukeB_UK 12y ago
  "There is no fold" by Luke Wroblewski: https://storify.com/xiwcx/there-is-no-fold-1 https://storify.com/xiwcx/there-is-no-fold-1
ProAm 12y ago
Splash pages are back [1] [1] http://www.cosmonautdreams.com/html/blog/websites/readability/ux/2014/01/06/Are-Splash-Pages-Back.html http://www.cosmonautdreams.com/html/blog/websites/readabilit...
- juliangregorian 12y ago
  I rather prefer these to splash screens, they evoke the opposing page to the beginning of a magazine article; splash screens feel more like studio logo animations before a film, only much less rewarding. Your source's UX qualifications also seem a bit... dubious.
  - ProAm 12y ago
    > Your source's UX qualifications also seem a bit... dubious. Just a developer and sys admin, my qualifications come purely from usage. I only noticed the trend of web development going back to what it was in the early 2000's where landing pages & splash screens were all the rage. I hate reading any blog on medium or that follows this format to me it detracts from the content and is visually irritating.

jwildeboer 12y ago

AFAICS the system wasn't updated for a year? Well, that's just plain stupid.