6 ms·

China Mafia-Style Hack Attack Drives California Firm to Brink

118 points by ssclafani 14y ago

wildranter 14y ago
Why do we do business with those sleazy bags? Just to save a few bucks on our next gadget? No. I guess we have to thank the Wallstreet bonus whores for dropping this f-bomb on us. Not even Bin Laden would've done better. Thanks boys, you rock!
- bruceboughton 14y ago
  That's a mighty accurate username.
  - wildranter 14y ago
    Thanks I guess. It's not that hard living up to that these days.
danso 14y ago
Ugh...this article reads like a movie about hacking. The story breathlessly conveys how elite Shanghai hackers toyed with a company's lifeblood, wreaking such prolific havoc that the owner literally crawled under the server building to see if a bug had been planted...but the ultimate culprit may have been... > Examining the script that controlled the payment processing function in November that year, he noticed that a single character was missing from the string -- an apostrophe. That was enough to cause the page to time out, rather than to complete the credit card transaction. Customers were leaving in frustration Am I right in thinking that this was all hack via SQL injection?
- FishonaSpear 14y ago
  Definitely took them some time to write all that down so pretty. Sounds like classic Chinese hacking though, this doesn't sound like the work of real pro's.
  - danso 14y ago
    I get that not every programmer can be well versed in cyber security, but how is it that apparently no one at this software firm apparently practices sound scientific reasoning? Is it possible to go through enough schooling in commuter science and not be able to diagnose a hack with logical reasoning before assuming Neo and the Matrix are real?
    - pyre 14y ago
      > schooling in commuter science I think that you found the issue right there. ;-)
      - danso 14y ago
        Ugh, I think someone just hacked my keyboard...the timing is just too convenient
        yen223 14y ago
        Must be those Chinese hackers!
        pyre 14y ago
        Just don't start fiddling around in your crawl-space looking for bugs! </pun-intended>
- frozenport 14y ago
  I think you are missing the point. China will steal your software and higher people who aim to destroy your lively-hood and life.
  - danso 14y ago
    No, I think you're missing the point. If it's true that rival nations will attempt to hack us for whatever reason, then it benefits us all to have a better understanding of basic cybersecurity than seeing foreign hackers as the Hand of God. Instead of examining the geopolitical problems here, this article takes us through a terrifying cyberstorm whipped up by mystical superhackers when the real perpetrators might as well have been script kiddies. The problem is that if cybersecurity continues to be framed in this fashion, then all that shit that HN continually complains about -- security theater (via homeland security) and draconian Internet laws (remember SOPA) -- will continue to be status quo.
    - saurik 14y ago
      (I think frozenport was telling a joke.)
    - damian2000 14y ago
      Script kiddies or not who cares - the effect on the small business was still effective. Like you say, its another wake up call to us all to have a better knowledge of security.
- damian2000 14y ago
  It seems like it was via an attachment in a phishing email that one of the employees clicked on... "Milburn contacted Matthew Thomlinson, a Microsoft Corp. (MSFT) threat expert for help. Thomlinson found the malware had downloaded software that burrowed into the company’s Microsoft operating system, automatically uploading more tools the hackers could use to control the network remotely."
deleted 14y ago
[deleted]
anigbrowl 14y ago
Milburn, after all, had built Solid Oak’s network himself. “I thought they might be able to get around some IT guy, but there’s no way they were going to get around me,” he says. /Facepalm. I was going to write that at least he got a bunch of free publicity from the article, but on the other hand the takeaway is that product is probably no more secure than his servers.
- dr_doom 14y ago
  I liked "He taught himself how to write code, and eventually mastered complex Internet software protocols." Seems like a lot of the issues would of been solved by moving to AWS. At least they would of still been making money.
  - anigbrowl 14y ago
    Likely. I don't want to be too hard on the guy because I can recognize a bit of myself in him, but this is a great example of why you should concentrate on your core competency and pay someone else to do the stuff you're not expert at.
  - ceol 14y ago
    If they were able to get into Google, something tells me they would have been able to break into AWS. Whether they would waste an AWS 0-day on some random stubborn American is another story.
fkdjs 14y ago
Hopefully that teaches him to rely on windows.
- 69_years_and 14y ago
  And that's what I was thinking as I read this - a guy/team who can write net filtering software would surely know what side is up. I was waiting for the part where he set up a GNU/Linix box and just rebuilt his site and perhaps directing his support email to/via GMail, heck how long would that take to setup, a week and cost far less than the lost sales and it would keep the engine running till a more permanent setup was devised. Or just clean installed to all his Windows machines. So many options and he choose the hard one. There has to be more to this story.
pyre 14y ago
> high-tech spies and digital combatants seek to gain > a brass-knuckle advantage in the global economy Digital combatants? Does enlistment in the PRC army now include light cycle[1] training? [1]: http://en.wikipedia.org/wiki/Tron_(franchise)#Light_cycles http://en.wikipedia.org/wiki/Tron_(franchise)#Light_cycles
groby_b 14y ago
"n April 2010, during a 6:30 a.m. check of his servers -- by then part of his daily routine -- Milburn stumbled on a folder buried in an obscure Microsoft directory, one that’s normally unused. What he found inside startled him. The file contained the encrypted versions of all eight passwords in his system -- the keys to the entire network. The hackers could use the passwords to control just about anything he could, from web servers to e-mail." Ah. He just decrypted the encrypted versions of his passwords by looking at them. I see. And he "stumbled upon" a directory during his routine 6:30am check. Really? He's manually inspecting servers every morning at 6:30am? This entire story is several levels of /headdesk.
- willvarfar 14y ago
  If your servers were under constant attack, yes checking your servers every morning first thing could well be your routine. You also assume he immediately recognised them. It could well have been someone one who later told him what he'd found. Sloppy of the attackers to leave it in plain sight.
- TazeTSchnitzel 14y ago
  Well, maybe he's just using Xenix...
- mahmud 14y ago
  Maybe he wants to beat the morning rush on his daily drive to the data-center~?
- deleted 14y ago
  [deleted]
- eatfish 14y ago
  >...decrypted the encrypted versions of his passwords by looking at them. I see. I think this means the hashed passwords. Microsoft have had several vulnerabilities in this area. Just entering a password to browse a server caused Windows (prior to Vista) to store the LM hash. An LM hash can be reversed in a matter of seconds.
jakejake 14y ago
It sounds a bit like the site owner was playing wack-a-mole for three years when he should have just wiped out everything. Once you have an intrusion, unfortunately you can't trust anything that a machine has access to and you should start again from a totally clean install. If this was a huge corporate network I could understand that may be impossible. But for a small shop with a few people, there's no reason not to go around and just wipe out every machine in the company. If this was a government sanctioned job, though, I imagine they could have stepped things up if the script-kiddie stuff didn't work. Why bring out the nuclear weapons if bottle rockets are doing the job just fine.
robomartin 14y ago
I'll preface this by saying that it is all too easy to criticize from the outside and without all of the relevant information and history. Once I got to the point in the article where it was obvious that he was under some form of a coordinated attack I almost wanted to send myself back in time to be able to go over to his office and yell "AWS! AWS!" in his ear. In addition to that, it seems it should have been obvious that bringing a team of experts to help secure the network (or transition it away) and fight the fight would have been the smartest idea. I would think that the quoted losses of some $58K per month would have covered this just fine. And, perhaps what is more important, it is likely that the technology fight wouldn't have lasted three years. The other thing that struck me --again, don't have perfect data, don't know all the facts-- is the apparent lack of help from the likes of the FBI. You would think that they'd be there in some sustained fashion to help out. This fellow was out of his league and paid a dear price for it. Hopefully the settlement compensated for some of it. It sounds like he might have ended-up with stress related health issues which are no laughing matter. It'll be really sad if the Internet becomes just another weapon of war. That should not be permitted. How? Not sure. Is it too late?
- javajosh 14y ago
  >It'll be really sad if the Internet becomes just another weapon of war. Anything of value can be turned into a weapon of war. Think about what a weapon is - it is a tool to disrupt something delicate. A missile destroys a plane. A knife stops a heart. A virus stops a computer. It really concerns me that a private US citizen was personally attacked by a sovereign nation. It seems to me that the US has an important duty to protect it's people, whether the attack is virtual or physical.
- dmschulman 14y ago
  They attackers didn't just cause downtime for his website. They stole his entire livelyhood and software creation.
ajays 14y ago
I stopped reading at this point: Commercial hacker hunters -- who refer to the team as the Comment group, for the hidden program code they use known as “comments” ..... /facepalm
- sanswork 14y ago
  Why? It's seriously what they do. The user installs the malware, it goes to a normal looking web page checks the html comments and receives it's commands. On the face of it everything looks fine to the user but the malware knows then what to do.
ThisIBereave 14y ago
The HN reaction to this story is pretty sad, I have to say. Yes, journalists are poor at providing technical coverage. That's not really their job. Yes, this guy could have created a more secure network system, but if a government hacking group comes after you, you will likely not do any better.
benmccann 14y ago
A lot of the comments about this presume that the victim could have avoided such trouble with better security knowledge. It's worth noting that the Chinese have hacked Google, Yahoo, Adobe, Lockheed Martin, and over a hundred other companies, which have many more security resources available to them than Solid Oak does. No small business in America would ever be able to defend itself from a sustained attack backed by the resources of one of the most economically powerful nations on earth. For example, does anyone here think they could defend against years of attacks from the NSA?
- wildranter 14y ago
  It doesn't matter. The main problem here is not hacking. It is piracy as the Chinese government stole their software. I wonder what would happen if Anonymous decided to target China over matter like this. Sure would be impossible to bring a government like that to their knees. But at least the public would gain some awareness on this matter.
  - deeqkah 14y ago
    Kill that noise right now. Before you grabbed that little war drum, if you thought of the consequences you really wouldn't want to start such a conflict in the first place. The quick, unorganized retaliatory response from an anonymous actor is usually such that it puts innocent services and people directly into the crossfire. Not to mention an anonymous moniker allows anyone to act on the behalf of anyone.
    - josefresco 14y ago
      Who said it has to be quick and unorganized? While officially the US can't 'engage' China in this way, another NGO could start and perform long term campaigns against Chinese corporations and assets similar to what is detailed in this article. I'm also not naive enough to think that this hasn't already begun.
      - pav3l 14y ago
cnlwsu 14y ago
from the Analysis that the article claims identified "thousands of lines copied of code" (sounds like they copied a blacklist): We found evidence that a number of these blacklists have been taken from the American-made filtering program CyberSitter. In particular, we found an encrypted configuration file, wfileu.dat, that references these blacklists with download URLs at CyberSitter's site. We also found a setup file, xstring.s2g, that appears to date these blacklists to 2006. Finally, csnews.dat is an encrypted 2004 news bulletin by CyberSitter. We conjecture that this file was accidentally included because it has the same file extension as the filters.
toyg 14y ago
Apart from the unverifiable security blunders (this is a Bloomberg article, not a Full Disclosure post), I have to say I can't feel very sympathetic towards somebody in the censorship business being harassed by fellow censors.
sown 14y ago
I hope tptacek chimes in. It sounds like a terrifying situation.
louischatriot 14y ago
The scary thing is indeed that most software companies do not have security experts nor the means to hire ones and are basically defenseless against such attacks. Kind of reminiscent of patent trolls. tldr version of the article: http://tldr.io/tldrs/50b6e4acbb22039977000f5b http://tldr.io/tldrs/50b6e4acbb22039977000f5b
rbanffy 14y ago
So... for how long did the guy keep credit card processing running on compromised servers? And it took them months to realize that?
clayrichardson 14y ago
I'm going to try to suggest a couple of solutions regarding problems encountered in the article that people might find to be useful to know. First, some pointers (non-exhaustive). For workstations: - Antivirus - Firewall - Disk encryption - Password complexity requirements - Updates For servers: - Key-based authentication - Updates - Firewall rules on host: explicit ingress/egress - Run services as non-privileged users in a chroot jail Firewall rules for network: explicit ingress/egress Yes, that's two firewalls for your hosting infrastructure. Attackers would have to escalate privileges from a service user to a root user in order to modify the firewall rules, at which point they'll have access to the other components of your internal network. To get outbound connectivity, they'd have to gain root access to the external firewall as well. Keep in mind they can still upload data if they can execute arbitrary commands as the web service user. Now, onto more specific items mentioned in the article. >realizing only later that the e-mail address was a couple of letters off. Tell your employees to only accept cryptographically signed email from you. S/MIME is simple to implement, and works across multiple platforms and clients. There is even support for it in iOS, Android, and Mail.app! http://en.wikipedia.org/wiki/S/MIME http://en.wikipedia.org/wiki/S/MIME You can get certificates here: http://www.symantec.com/verisign/digital-id http://www.symantec.com/verisign/digital-id You can even get free ones if you're on a budget! (although they do say for personal use, implement at your own risk if you're a business!) http://www.comodo.com/home/email-security/free-email-certificate.php http://www.comodo.com/home/email-security/free-email-certifi... You can do a Google search on how to get these certificates installed in your mail and mobile clients of choice. >clicked on the attachment Use Google Apps. I'm not sure what the success rate of Gmail's 0-day detection is with attachments, but I'd be willing to bet that they're better if not as equally effective as the average desktop antivirus email scanner. >Microsoft operating system The article doesn't say if they used any sort of antivirus or firewall protection on their workstations. If you absolutely have to have Windows somewhere in your business, you need to have a decent antivirus and firewall solution running. Also, make sure all of the latest updates are applied in a timely manner. IIRC, you can set Domain Policies from Active Directory to do this sort of thing for you. >automatically uploading more tools the hackers could use to control the network remotely. This could have been addressed in a couple of ways. (non-exhaustive list) 1. If the malware was listening for incoming connections, explicit ingress (aka no port forwarding) with a NAT (most residential routers) would have made it more difficult, as another layer must be evaded to connect to the internal machines. 2. If the malware was using a connect-back shell, explicit egress (aka outbound) traffic would make it difficult to get a shell on a compromised machine. As soon as you realize you're under attack, you should go into hardcore lockdown paranoid mode, and only allow network ranges that are absolutely necessary for business operation. >Then the company’s e-mail servers began shutting down, sometimes two or three times a week, slowing e-mail traffic, the main way the company provides customer service Hosting an email service is a ridiculously difficult task security-wise. Postfix and Sendmail both have a history of security vulnerabilities, and the only way to properly host an email server is to harden a *nix system to the extreme, and always be paranoid. But that's hard and most people aren't in the business of hosting email services. That said, I'm pretty sure Google Apps has a great cost/benefit factor when you look at ease of implementation and security, because the Gmail team is more likely better at server hardening than you are. Oh, and make sure you backup your email and support PIN, in case anything goes awry. >Similar problems began plaguing the web servers -- a bigger problem since web sales of CYBERsitter supply more than half of Solid Oak’s revenue. I would have to recommend putting CloudFlare in front of your servers, as (through their blog posts about several incidents) they have demonstrated an ability to successfully defend against a large number of attacks. This also has the benefit of providing a proxy, masking your backend server IPs, which I'll talk about more later. >Similar problems began plaguing the web servers >figure out how the hackers might be behind it. Bro, http://en.wikipedia.org/wiki/Host-based_intrusion_detection_system http://en.wikipedia.org/wiki/Host-based_intrusion_detection_... >But the agency shed almost no light on the situation, he says, and he was never told if the material was useful. I'm surprised they didn't offer any advice on how to mitigate attacks such as these. At the very least, some pointers similar to the ones I'm writing now would probably gain some sort of ground. >alternating between four different cell phones from three different carriers. This seems a little extreme, if not excessive. I've used burners (temporary pre-paid cellphones, disposed of after a duration of use) before when I attended DEFCON (hackers like to do sketchy things with RF) but never more than one at a time. I'm sure you could probably use something like Google Voice with multiple Google Accounts or Twilio for temporary numbers while not exposing your actual number. >constantly had to reboot servers >couldn’t trace the source of the network problems If you're going to have to do it more than twice, it's probably best to automate it. If reboots are required due to a crash, you should probably identify the cause of the crash (go through logs, etc.) and attempt to rectify it. If you seem to be way in over your head, bringing in an expert should definitely be on your mind. >to find that his commercial-grade SonicWALL firewall had failed >He spent a good part of the next day on the phone with the manufacturer, who was stumped. If you have any sort of hosting infrastructure, and your expertise isn't network administration or security, I'd recommend getting (full time, or contracting) a certified network administrator to help you get your networking shit together. If I were to pick equipment, I'd probably go with the big dogs, Cisco or Juniper. >He began writing his own software to monitor the connections his computers were making to outside networks, looking for tell-tale signs of the hackers at work. Bro, http://en.wikipedia.org/wiki/Network_intrusion_detection_system http://en.wikipedia.org/wiki/Network_intrusion_detection_sys... Snort: http://en.wikipedia.org/wiki/Snort_(software) http://en.wikipedia.org/wiki/Snort_(software) >servers >obscure Microsoft directory Contrary to popular (or at least some peoples') beliefs, it is possible to harden a Windows server, you just have to really know your shit and be able to tolerate all the graphical clicky stuff. >all eight passwords If you're running a business, chances are you have more than 8 accounts. 1. Use unique passwords for each service/account. 2. Store them encrypted, in a password manager. 3. Use keys/certificates whenever possible. 4. Use two-factor authentication whenever possible. >The folder was gone two days later, he says, and in its place were several pieces of software he didn’t recognize. You generally don't want to leave a compromised server in production rotation. Take it off the network and perform forensics to determine how it got compromised, and how to prevent that in the future. Make sure to wipe the server and reconfigure it before placing it back on the network. >Net losses averaged $58,000 a month (aside) I'd fix the CRAP out of your security problems for that much a month. >A hacker could certainly edit the script and break it so it wouldn’t work This means they have access to your web servers, and/or to your deployment process/service. One of the ways you can mitigate direct access to your web boxes is to use some sort of load balancer (HAProxy) or web application firewall (CloudFlare). Then only allow connections to your web boxes from the proxy, as well as a specified IP address to allow for maintenance. >That would be a great way to do it without calling attention to the fact that they were in the system. If you choose to use git as a tool to deploy your code, you can continuously monitor the codebase for modifications. I'm quite tired and I think I'm rambling at this point. I hope this is clear enough for some people to get something out of. I'll probably come back later and make some edits for clarity/correctness.

118 points by ssclafani 14y ago

Why do we do business with those sleazy bags? Just to save a few bucks on our next gadget? No. I guess we have to thank the Wallstreet bonus whores for dropping this f-bomb on us. Not even Bin Laden would've done better. Thanks boys, you rock!

bruceboughton 14y ago
That's a mighty accurate username.
- wildranter 14y ago
  Thanks I guess. It's not that hard living up to that these days.

danso 14y ago

Ugh...this article reads like a movie about hacking. The story breathlessly conveys how elite Shanghai hackers toyed with a company's lifeblood, wreaking such prolific havoc that the owner literally crawled under the server building to see if a bug had been planted...but the ultimate culprit may have been... > Examining the script that controlled the payment processing function in November that year, he noticed that a single character was missing from the string -- an apostrophe. That was enough to cause the page to time out, rather than to complete the credit card transaction. Customers were leaving in frustration Am I right in thinking that this was all hack via SQL injection?

FishonaSpear 14y ago
Definitely took them some time to write all that down so pretty. Sounds like classic Chinese hacking though, this doesn't sound like the work of real pro's.
- danso 14y ago
  I get that not every programmer can be well versed in cyber security, but how is it that apparently no one at this software firm apparently practices sound scientific reasoning? Is it possible to go through enough schooling in commuter science and not be able to diagnose a hack with logical reasoning before assuming Neo and the Matrix are real?
  - pyre 14y ago
    > schooling in commuter science I think that you found the issue right there. ;-)
    - danso 14y ago
      Ugh, I think someone just hacked my keyboard...the timing is just too convenient
      - yen223 14y ago
        Must be those Chinese hackers!
      - pyre 14y ago
        Just don't start fiddling around in your crawl-space looking for bugs! </pun-intended>
frozenport 14y ago
I think you are missing the point. China will steal your software and higher people who aim to destroy your lively-hood and life.
- danso 14y ago
  No, I think you're missing the point. If it's true that rival nations will attempt to hack us for whatever reason, then it benefits us all to have a better understanding of basic cybersecurity than seeing foreign hackers as the Hand of God. Instead of examining the geopolitical problems here, this article takes us through a terrifying cyberstorm whipped up by mystical superhackers when the real perpetrators might as well have been script kiddies. The problem is that if cybersecurity continues to be framed in this fashion, then all that shit that HN continually complains about -- security theater (via homeland security) and draconian Internet laws (remember SOPA) -- will continue to be status quo.
  - saurik 14y ago
    (I think frozenport was telling a joke.)
  - damian2000 14y ago
    Script kiddies or not who cares - the effect on the small business was still effective. Like you say, its another wake up call to us all to have a better knowledge of security.
damian2000 14y ago
It seems like it was via an attachment in a phishing email that one of the employees clicked on... "Milburn contacted Matthew Thomlinson, a Microsoft Corp. (MSFT) threat expert for help. Thomlinson found the malware had downloaded software that burrowed into the company’s Microsoft operating system, automatically uploading more tools the hackers could use to control the network remotely."

deleted 14y ago

[deleted]

anigbrowl 14y ago

Milburn, after all, had built Solid Oak’s network himself. “I thought they might be able to get around some IT guy, but there’s no way they were going to get around me,” he says. /Facepalm. I was going to write that at least he got a bunch of free publicity from the article, but on the other hand the takeaway is that product is probably no more secure than his servers.

dr_doom 14y ago
I liked "He taught himself how to write code, and eventually mastered complex Internet software protocols." Seems like a lot of the issues would of been solved by moving to AWS. At least they would of still been making money.
- anigbrowl 14y ago
  Likely. I don't want to be too hard on the guy because I can recognize a bit of myself in him, but this is a great example of why you should concentrate on your core competency and pay someone else to do the stuff you're not expert at.
- ceol 14y ago
  If they were able to get into Google, something tells me they would have been able to break into AWS. Whether they would waste an AWS 0-day on some random stubborn American is another story.

fkdjs 14y ago

Hopefully that teaches him to rely on windows.

69_years_and 14y ago
And that's what I was thinking as I read this - a guy/team who can write net filtering software would surely know what side is up. I was waiting for the part where he set up a GNU/Linix box and just rebuilt his site and perhaps directing his support email to/via GMail, heck how long would that take to setup, a week and cost far less than the lost sales and it would keep the engine running till a more permanent setup was devised. Or just clean installed to all his Windows machines. So many options and he choose the hard one. There has to be more to this story.

pyre 14y ago

> high-tech spies and digital combatants seek to gain > a brass-knuckle advantage in the global economy Digital combatants? Does enlistment in the PRC army now include light cycle[1] training? [1]: http://en.wikipedia.org/wiki/Tron_(franchise)#Light_cycles http://en.wikipedia.org/wiki/Tron_(franchise)#Light_cycles

groby_b 14y ago

"n April 2010, during a 6:30 a.m. check of his servers -- by then part of his daily routine -- Milburn stumbled on a folder buried in an obscure Microsoft directory, one that’s normally unused. What he found inside startled him. The file contained the encrypted versions of all eight passwords in his system -- the keys to the entire network. The hackers could use the passwords to control just about anything he could, from web servers to e-mail." Ah. He just decrypted the encrypted versions of his passwords by looking at them. I see. And he "stumbled upon" a directory during his routine 6:30am check. Really? He's manually inspecting servers every morning at 6:30am? This entire story is several levels of /headdesk.

willvarfar 14y ago
If your servers were under constant attack, yes checking your servers every morning first thing could well be your routine. You also assume he immediately recognised them. It could well have been someone one who later told him what he'd found. Sloppy of the attackers to leave it in plain sight.
TazeTSchnitzel 14y ago
Well, maybe he's just using Xenix...
mahmud 14y ago
Maybe he wants to beat the morning rush on his daily drive to the data-center~?
deleted 14y ago
[deleted]
eatfish 14y ago
>...decrypted the encrypted versions of his passwords by looking at them. I see. I think this means the hashed passwords. Microsoft have had several vulnerabilities in this area. Just entering a password to browse a server caused Windows (prior to Vista) to store the LM hash. An LM hash can be reversed in a matter of seconds.

jakejake 14y ago

It sounds a bit like the site owner was playing wack-a-mole for three years when he should have just wiped out everything. Once you have an intrusion, unfortunately you can't trust anything that a machine has access to and you should start again from a totally clean install. If this was a huge corporate network I could understand that may be impossible. But for a small shop with a few people, there's no reason not to go around and just wipe out every machine in the company. If this was a government sanctioned job, though, I imagine they could have stepped things up if the script-kiddie stuff didn't work. Why bring out the nuclear weapons if bottle rockets are doing the job just fine.

robomartin 14y ago

I'll preface this by saying that it is all too easy to criticize from the outside and without all of the relevant information and history. Once I got to the point in the article where it was obvious that he was under some form of a coordinated attack I almost wanted to send myself back in time to be able to go over to his office and yell "AWS! AWS!" in his ear. In addition to that, it seems it should have been obvious that bringing a team of experts to help secure the network (or transition it away) and fight the fight would have been the smartest idea. I would think that the quoted losses of some $58K per month would have covered this just fine. And, perhaps what is more important, it is likely that the technology fight wouldn't have lasted three years. The other thing that struck me --again, don't have perfect data, don't know all the facts-- is the apparent lack of help from the likes of the FBI. You would think that they'd be there in some sustained fashion to help out. This fellow was out of his league and paid a dear price for it. Hopefully the settlement compensated for some of it. It sounds like he might have ended-up with stress related health issues which are no laughing matter. It'll be really sad if the Internet becomes just another weapon of war. That should not be permitted. How? Not sure. Is it too late?

javajosh 14y ago
>It'll be really sad if the Internet becomes just another weapon of war. Anything of value can be turned into a weapon of war. Think about what a weapon is - it is a tool to disrupt something delicate. A missile destroys a plane. A knife stops a heart. A virus stops a computer. It really concerns me that a private US citizen was personally attacked by a sovereign nation. It seems to me that the US has an important duty to protect it's people, whether the attack is virtual or physical.
dmschulman 14y ago
They attackers didn't just cause downtime for his website. They stole his entire livelyhood and software creation.

ajays 14y ago

I stopped reading at this point: Commercial hacker hunters -- who refer to the team as the Comment group, for the hidden program code they use known as “comments” ..... /facepalm

sanswork 14y ago
Why? It's seriously what they do. The user installs the malware, it goes to a normal looking web page checks the html comments and receives it's commands. On the face of it everything looks fine to the user but the malware knows then what to do.

ThisIBereave 14y ago

The HN reaction to this story is pretty sad, I have to say. Yes, journalists are poor at providing technical coverage. That's not really their job. Yes, this guy could have created a more secure network system, but if a government hacking group comes after you, you will likely not do any better.

benmccann 14y ago

A lot of the comments about this presume that the victim could have avoided such trouble with better security knowledge. It's worth noting that the Chinese have hacked Google, Yahoo, Adobe, Lockheed Martin, and over a hundred other companies, which have many more security resources available to them than Solid Oak does. No small business in America would ever be able to defend itself from a sustained attack backed by the resources of one of the most economically powerful nations on earth. For example, does anyone here think they could defend against years of attacks from the NSA?

wildranter 14y ago
It doesn't matter. The main problem here is not hacking. It is piracy as the Chinese government stole their software. I wonder what would happen if Anonymous decided to target China over matter like this. Sure would be impossible to bring a government like that to their knees. But at least the public would gain some awareness on this matter.
- deeqkah 14y ago
  Kill that noise right now. Before you grabbed that little war drum, if you thought of the consequences you really wouldn't want to start such a conflict in the first place. The quick, unorganized retaliatory response from an anonymous actor is usually such that it puts innocent services and people directly into the crossfire. Not to mention an anonymous moniker allows anyone to act on the behalf of anyone.
  - josefresco 14y ago
    Who said it has to be quick and unorganized? While officially the US can't 'engage' China in this way, another NGO could start and perform long term campaigns against Chinese corporations and assets similar to what is detailed in this article. I'm also not naive enough to think that this hasn't already begun.
    - pav3l 14y ago

cnlwsu 14y ago

from the Analysis that the article claims identified "thousands of lines copied of code" (sounds like they copied a blacklist): We found evidence that a number of these blacklists have been taken from the American-made filtering program CyberSitter. In particular, we found an encrypted configuration file, wfileu.dat, that references these blacklists with download URLs at CyberSitter's site. We also found a setup file, xstring.s2g, that appears to date these blacklists to 2006. Finally, csnews.dat is an encrypted 2004 news bulletin by CyberSitter. We conjecture that this file was accidentally included because it has the same file extension as the filters.

toyg 14y ago

Apart from the unverifiable security blunders (this is a Bloomberg article, not a Full Disclosure post), I have to say I can't feel very sympathetic towards somebody in the censorship business being harassed by fellow censors.

sown 14y ago

I hope tptacek chimes in. It sounds like a terrifying situation.

louischatriot 14y ago

The scary thing is indeed that most software companies do not have security experts nor the means to hire ones and are basically defenseless against such attacks. Kind of reminiscent of patent trolls. tldr version of the article: http://tldr.io/tldrs/50b6e4acbb22039977000f5b http://tldr.io/tldrs/50b6e4acbb22039977000f5b

rbanffy 14y ago

So... for how long did the guy keep credit card processing running on compromised servers? And it took them months to realize that?

clayrichardson 14y ago

I'm going to try to suggest a couple of solutions regarding problems encountered in the article that people might find to be useful to know. First, some pointers (non-exhaustive). For workstations: - Antivirus - Firewall - Disk encryption - Password complexity requirements - Updates For servers: - Key-based authentication - Updates - Firewall rules on host: explicit ingress/egress - Run services as non-privileged users in a chroot jail Firewall rules for network: explicit ingress/egress Yes, that's two firewalls for your hosting infrastructure. Attackers would have to escalate privileges from a service user to a root user in order to modify the firewall rules, at which point they'll have access to the other components of your internal network. To get outbound connectivity, they'd have to gain root access to the external firewall as well. Keep in mind they can still upload data if they can execute arbitrary commands as the web service user. Now, onto more specific items mentioned in the article. >realizing only later that the e-mail address was a couple of letters off. Tell your employees to only accept cryptographically signed email from you. S/MIME is simple to implement, and works across multiple platforms and clients. There is even support for it in iOS, Android, and Mail.app! http://en.wikipedia.org/wiki/S/MIME http://en.wikipedia.org/wiki/S/MIME You can get certificates here: http://www.symantec.com/verisign/digital-id http://www.symantec.com/verisign/digital-id You can even get free ones if you're on a budget! (although they do say for personal use, implement at your own risk if you're a business!) http://www.comodo.com/home/email-security/free-email-certificate.php http://www.comodo.com/home/email-security/free-email-certifi... You can do a Google search on how to get these certificates installed in your mail and mobile clients of choice. >clicked on the attachment Use Google Apps. I'm not sure what the success rate of Gmail's 0-day detection is with attachments, but I'd be willing to bet that they're better if not as equally effective as the average desktop antivirus email scanner. >Microsoft operating system The article doesn't say if they used any sort of antivirus or firewall protection on their workstations. If you absolutely have to have Windows somewhere in your business, you need to have a decent antivirus and firewall solution running. Also, make sure all of the latest updates are applied in a timely manner. IIRC, you can set Domain Policies from Active Directory to do this sort of thing for you. >automatically uploading more tools the hackers could use to control the network remotely. This could have been addressed in a couple of ways. (non-exhaustive list) 1. If the malware was listening for incoming connections, explicit ingress (aka no port forwarding) with a NAT (most residential routers) would have made it more difficult, as another layer must be evaded to connect to the internal machines. 2. If the malware was using a connect-back shell, explicit egress (aka outbound) traffic would make it difficult to get a shell on a compromised machine. As soon as you realize you're under attack, you should go into hardcore lockdown paranoid mode, and only allow network ranges that are absolutely necessary for business operation. >Then the company’s e-mail servers began shutting down, sometimes two or three times a week, slowing e-mail traffic, the main way the company provides customer service Hosting an email service is a ridiculously difficult task security-wise. Postfix and Sendmail both have a history of security vulnerabilities, and the only way to properly host an email server is to harden a *nix system to the extreme, and always be paranoid. But that's hard and most people aren't in the business of hosting email services. That said, I'm pretty sure Google Apps has a great cost/benefit factor when you look at ease of implementation and security, because the Gmail team is more likely better at server hardening than you are. Oh, and make sure you backup your email and support PIN, in case anything goes awry. >Similar problems began plaguing the web servers -- a bigger problem since web sales of CYBERsitter supply more than half of Solid Oak’s revenue. I would have to recommend putting CloudFlare in front of your servers, as (through their blog posts about several incidents) they have demonstrated an ability to successfully defend against a large number of attacks. This also has the benefit of providing a proxy, masking your backend server IPs, which I'll talk about more later. >Similar problems began plaguing the web servers >figure out how the hackers might be behind it. Bro, http://en.wikipedia.org/wiki/Host-based_intrusion_detection_system http://en.wikipedia.org/wiki/Host-based_intrusion_detection_... >But the agency shed almost no light on the situation, he says, and he was never told if the material was useful. I'm surprised they didn't offer any advice on how to mitigate attacks such as these. At the very least, some pointers similar to the ones I'm writing now would probably gain some sort of ground. >alternating between four different cell phones from three different carriers. This seems a little extreme, if not excessive. I've used burners (temporary pre-paid cellphones, disposed of after a duration of use) before when I attended DEFCON (hackers like to do sketchy things with RF) but never more than one at a time. I'm sure you could probably use something like Google Voice with multiple Google Accounts or Twilio for temporary numbers while not exposing your actual number. >constantly had to reboot servers >couldn’t trace the source of the network problems If you're going to have to do it more than twice, it's probably best to automate it. If reboots are required due to a crash, you should probably identify the cause of the crash (go through logs, etc.) and attempt to rectify it. If you seem to be way in over your head, bringing in an expert should definitely be on your mind. >to find that his commercial-grade SonicWALL firewall had failed >He spent a good part of the next day on the phone with the manufacturer, who was stumped. If you have any sort of hosting infrastructure, and your expertise isn't network administration or security, I'd recommend getting (full time, or contracting) a certified network administrator to help you get your networking shit together. If I were to pick equipment, I'd probably go with the big dogs, Cisco or Juniper. >He began writing his own software to monitor the connections his computers were making to outside networks, looking for tell-tale signs of the hackers at work. Bro, http://en.wikipedia.org/wiki/Network_intrusion_detection_system http://en.wikipedia.org/wiki/Network_intrusion_detection_sys... Snort: http://en.wikipedia.org/wiki/Snort_(software) http://en.wikipedia.org/wiki/Snort_(software) >servers >obscure Microsoft directory Contrary to popular (or at least some peoples') beliefs, it is possible to harden a Windows server, you just have to really know your shit and be able to tolerate all the graphical clicky stuff. >all eight passwords If you're running a business, chances are you have more than 8 accounts. 1. Use unique passwords for each service/account. 2. Store them encrypted, in a password manager. 3. Use keys/certificates whenever possible. 4. Use two-factor authentication whenever possible. >The folder was gone two days later, he says, and in its place were several pieces of software he didn’t recognize. You generally don't want to leave a compromised server in production rotation. Take it off the network and perform forensics to determine how it got compromised, and how to prevent that in the future. Make sure to wipe the server and reconfigure it before placing it back on the network. >Net losses averaged $58,000 a month (aside) I'd fix the CRAP out of your security problems for that much a month. >A hacker could certainly edit the script and break it so it wouldn’t work This means they have access to your web servers, and/or to your deployment process/service. One of the ways you can mitigate direct access to your web boxes is to use some sort of load balancer (HAProxy) or web application firewall (CloudFlare). Then only allow connections to your web boxes from the proxy, as well as a specified IP address to allow for maintenance. >That would be a great way to do it without calling attention to the fact that they were in the system. If you choose to use git as a tool to deploy your code, you can continuously monitor the codebase for modifications. I'm quite tired and I think I'm rambling at this point. I hope this is clear enough for some people to get something out of. I'll probably come back later and make some edits for clarity/correctness.