15 ms·

The story around the Linode hack

357 points by foofoobar 13y ago

jameswyse 13y ago
There's more info about HTP5, including working mirrors of the files linked at the end of the linode document here: http://straylig.ht/zines/HTP5/ http://straylig.ht/zines/HTP5/
- dantiberian 13y ago
  HTP5 stands for Hack The Planet 5, the fifth issue of the zine linked in the parent comment.
blacktulip 13y ago
I am not familiar with the crackers' terms. So does this mean that name.com is not safe? All my domains are there...
- JonnieCache 13y ago
  The point is, when you're dealing with people of this level of supposed skill, they can just walk into pretty much any network. They're all vulnerable on some level if you're capable of actually breaking them yourself in novel ways. The only real solution is either to not depend on any single network, or to make it clear that you will simply kill anyone who troubles you. Or just remain innocuous enough that nobody will care to.
  - twistedpair 13y ago
    They had skilz, no doubt, but it does not appear the CF hack was the zen apex of hacking. It was a well known exploit you could drive a truck through. Writing Stuxnet, now that was mad skilz.
- sparkinson 13y ago
  Looking around there doesn't seem to be any news on a breach at name.com, official or un-official (aside from this of course). It does have me worried however.
robk 13y ago
That seems somewhat scary if they've compromised domain registrars and are intercepting login data from client sites that way.
- jrockway 13y ago
  It's more scary if they've compromised a SSL CA. A simple DNS attack won't stop your browser from displaying a broken certificate warning. (Though they can always not redirect from http to https and most users won't notice, sadly.)
  - blibble 13y ago
    it's very easy to get your own https cert once you control the dns for a domain, you just set up own nameserver that proxies requests to the original NS (except very specific ones, say those from Verisign), request your "domain control validation" https cert, and bam! valid https cert!
  - deleted 13y ago
    [deleted]
    - bstpierre 13y ago
      It said "not redirect from http to https" -- meaning that when someone requests http://example.com http://example.com they would normally be redirected by the site owner to https://example.com https://example.com, but the attacker could just leave the original request alone. The point is that most people wouldn't notice. HTTPS Everywhere or similar browser plugin would probably pop up an alert if this did happen.
      - graue 13y ago
        HTTP Strict Transport Security[1] is designed specifically to prevent that attack. Unfortunately, Linode's manager doesn't seem to use it: $ curl -I https://manager.linode.com/ HTTP/1.1 200 OK Server: nginx/0.7.65 Date: Tue, 07 May 2013 16:23:07 GMT Content-Type: text/html;charset=UTF-8 Connection: keep-alive Vary: Accept-Encoding If they were, there would be a line like this: $ curl -sI https://github.com | fgrep Strict Strict-Transport-Security: max-age=2592000 Which tells the browser, for the next 2592000 seconds (30 days), only request github.com over HTTPS, never HTTP. Not sure why Linode isn't using HSTS — lack of awareness? The super-old version of Nginx is also a little scary. [1] https://en.wikipedia.org/wiki/HTTP_Strict_Transport_Security https://en.wikipedia.org/wiki/HTTP_Strict_Transport_Security
ghshephard 13y ago
There is a lot of inside-baseball in this, but the one they keep talking about, is, "shred customer data" - as in, " Recognizing their situation, we instead told them that if they acknowledged HTP in their analysis, we'd go ahead and shred their customer data anyway." Do they honestly, for a single second, think that any LEA, corporation, or, well, anyone would believe that once the information was compromised, that there was no putting the genie back in the bottle? Also - I suspect there are probably disclosure laws that had to be followed by Linode anyways.
- deleted 13y ago
  [deleted]
- nathanb 13y ago
  What choice did they have? Comply, and trust the honor of black hat hackers? Or refuse, and have customers' data appearing on FTP and torrent sites within the day? As a Linode customer, I would rather them choose the latter. Not to try and sweep the incident under the rug (to your point about disclosure) but to prevent the data from being scraped by groups who exist solely for extracting credit card details from releases by groups like HTP (note the reference to "carders" in the article) and then being sold. (According to Linode's own post, the CC data were encrypted, meaning that it should be intractable to actually extract usable CC numbers from the data. But why would Linode not accept those terms, even if they believed that HTP were lying? At least it would give them until 1 May to get their house in order.)
runn1ng 13y ago
I... don't actually understand most of what they wrote there.
mappu 13y ago
Definitely worth reading the full zine, some scary stuff in there (including very readable python LFI-based exploits for unpatched MoinMoin and ColdFusion). Highlights: 1900+ days uptime on a sparc box somewhere in sourceforge.net, root on ICANN, root on Debian repositories..
- psutor 13y ago
  Link to full zine: http://straylig.ht/zines/HTP5/ http://straylig.ht/zines/HTP5/
  - cheapsteak 13y ago
    Any chance someone saved a copy? Link seems dead from here
- kokey 13y ago
  I've seen longer uptimes on Solaris sparc boxes, many times, when they had a board of battery replaced and they booted up with a reset clock which got corrected by NTP after boot.
tiredofcareer 13y ago
Some hopefully-helpful clarifications of the inside baseball talk from just the overview (I haven't read the full zine), enhanced with inside and general knowledge I've gained in my travels on this mortal coil: - HTP claims to have{, had} access to name.com, which Linode currently uses. This access enables an unauthorized party to update authoritative nameservers for your domain; i.e., if you host at Amazon, very likely your authoritative nameservers are Route 53 on your account. HTP would not have access to modify the zone directly through the registrar and would instead have to hijack the entire domain with a working, completely-transferred zone on their own nameservers. For this to go down entirely unnoticed is extraordinarily difficult. I won't say impossible, but damned close without a copy of the zone in hand and with Linode running AXFR disabled (you should be too). There are subzones of linode.com; they wouldn't have gotten them all, and it would have been noticed within minutes. - In order to attack SwiftIRC, to get back at some script kiddies DoS attacking them after their last release (because you know, that's a good target to burn registrar access on and all), HTP decided to backdoor SwiftIRC via their nameservers which are hosted at Linode. That's not the same as the registrar nameservers discussed above, but is instead the DNS data actually stored on a Linode on SwiftIRC's account. They do not hint what they were going to do with it once they had hijacked the nameservers, and I will not theorize. I could guess, though. - Before utilizing their registrar access (from the first bullet point) to hijack the linode.com zone and intercept manager logins silently by redirecting traffic via DNS -- also fairly difficult to pull off without a good linode.com certificate in hand, in terms of keeping the TLS session non-suspicious to a browser -- they instead discovered a zero-day in ColdFusion (Linode's stack) and got in that way. That's much quieter and much more likely to not be noticed. If we take the FBI's actions at HTP's word, the FBI was the only reason Linode was made aware of this outside of HTP's control; a DNS hijack would have been immediately noticed by Linode administrators. - Knowing what I know (let's leave it at that), a successful exploit on Linode's ColdFusion stack entails a database of Linodes, DNS, credit cards, e-mails, addresses, and keys to decrypt the actual card numbers, and a lot more data. You have to decide whether to take HTP at their word that they deleted credit cards. Consider your credit card and all prior credit cards compromised if it were in the system before April. - The access that HTP obtained does not, full stop, lead to root on Linode instances without at least one shutdown job or change of root password job showing up in your Linode's history that you did not ask for. Your Linode's root password is not stored in any Linode system aside from on your Linode itself. Your LISH password, as they say, is, and according to them is stored in plaintext; if you see things on your Linode's console (located under the Remote Access tab) that you did not type, that access was used upon you. If not, it wasn't. If you used the same root password on your Linode that you did for your LISH password, consider that password compromised. I'm suspicious of the claim that they rooted all those (assuming) customers without any of them noticing their Linode being rebooted to apply the new root password to allow HTP in, and I would read that as "potential access" instead of "access". They probably bounced some nmap.org servers to reset their root passwords -- a Linode system requirement -- without fyodor noticing. Which is interesting for a couple reasons. - Also, the access they obtained does not lead to root on the Linode host fleet itself, unless they are holding back some extra access they obtained such as a shared password between the ColdFusion stack and administrator credentials for Linode systems, which I consider unlikely for a couple reasons. With several days to get familiar with the architecture, HTP could have used their database write access to do things on the hosts, but it's a fairly limited set of things. Dumping Linode's database is bad, but root on their hosts is far, far worse, and by indications, I don't think they got it. - How does this relate to the Bitcoin hacks of yesteryear, you ask? The Bitcoin hackers probably got in the exact same way -- Linode hinted at a compromised admin credential, which is close enough to do everything HTP was able to do -- then shut down and reset root passwords on the Bitcoin Linodes they were after, which then gave them filesystem access. So ends clarifications, thus begins conclusions: - PAY ATTENTION WHEN YOUR SERVERS ARE REBOOTED WITHOUT YOUR COMMAND. - PAY ATTENTION WHEN YOUR SERVERS ARE REBOOTED WITHOUT YOUR COMMAND. - Linode added a feature that shoots you an e-mail when your Linode is manipulated in any way via jobs, such as resetting your Linode's root password (a la Bitcoin/HTP hacks). It's depressing they had to do this, but pay attention if you get the mail. External monitoring like Nagios that pages you when your server goes down is also a good idea, as long as it is hosted at another provider. - EDIT: After reading the zine, yet again, /CFIDE is the vector. There's no excuse for not hiding your administrative tools, generally the soft underbelly of the whole smash, from the Internet. None. It's one rewrite in nginx. Match /CFIDE<anything> from the public, redirect to /. Done. - EDIT: Again, after looking at how trivial the exploit was, it's probably time to reconsider using Adobe ColdFusion from a business continuity standpoint. Half Linode's fault for not hiding /CFIDE, half Adobe's fault for the engineering missteps that lead to this capability for a remote attacker. We should be just as hard on ColdFusion as we are on Rails. - SwiftIRC is a den of inquity, up there with EFnet; if you run a hosting provider, think twice about permitting SwiftIRC anywhere near you. To reiterate that, Linode was a casualty of someone going after SwiftIRC. Delink their nodes, cancel them, and kick them to the curb if you're interested in preserving your business. Not worth the money. Same with damned near all the IRC networks except OFTC and, to a far lesser extent, Freenode. There will always be targets but harboring SwiftIRC is probably a malicious-actor magnet. - Registrars (and CAs, though that's outside this discussion) are the weak point in the entire system. This is not the first time they have been shown to be so. Linode could be Fort Knox of digital security but if name.com falls over, it's all over; that's entirely outside of Linode's, and your, control. Currently, the registrar market is heavily profit-centric and, personally, I think people spend far too little on a domain in the general case. I would happily pay a registrar a lot more money -- hundreds a year or more -- if their offering were run competently, as it is fairly obvious name.com isn't. Compare your hosting bill to your registrar bill; what's wrong with that picture? - HTP is apparently fairly easy to troll into using valuable access for vengeance purposes. Shameful target selection and a burn of a good hack just to root SwiftIRC. That's like pissing in the ocean for a good time. - Linode got railroaded here and the general reaction by folks is a little overdone. You know that's true when even the hackers' overview of the hack specifically calls out people bitching about Linode security on Twitter. All it takes is one zero-day, and you will all be hit by one in your career, so cut Linode a little slack.
RoboTeddy 13y ago
Here's an attempt at an explanation/translation: HTP ("Hack The Planet") is a group that likes to break into things. Another (unnamed) group of people impersonated a third group of people ("ac1db1tch3z") and tried to cause trouble for HTP. The impersonators located HTP by examining one of HTP's botnets (a collection of compromised computers that are used to launch things like denial of service attacks). Botnets have to receive instructions (e.g., targets to attack) from somewhere, so it's likely that the impersonators followed the path taken by commands to the botnet, and found the network(s) that HTP uses to organize themselves. HTP realized this, and wanted to get back at the impersonators. They found out that the impersonators used an IRC channel (chat room) hosted on a network called SwiftIRC. If HTP could break into SwiftIRC (which is hosted on Linode), they could cause all sorts of trouble for the impersonators. So HTP decided to break into Linode, so they could break into SwiftIRC, so they could break into the group of impersonators. To break into Linode, HTP broke into their domain name registar (name.com). They planned to secretly take control of linode.com, and replace it with a version of linode.com would look and feel and work correctly, but had one additional feature -- it would collect the login information that people typed in. HTP probably hoped to gain the login for SwiftIRC directly, or collect the logins for Linode admins and obtain SwiftIRC's login from there. But, before they enacted the domain takeover (a maneuver that would likely be somewhat difficult to employ without being noticed), an HTP member discovered a new vulnerability in ColdFusion, the server software used by Linode. The ability to discover a new exploit on demand implies a high level of skill within the group. Using this exploit, HTP obtained direct access to Linode. They proceeded to gain access to SwiftIRC, as well as other sites hosted on Linode, including a well-known security site, nmap.org The FBI apparently had a mole in HTP, and they alerted Linode that HTP had access to nmap.org. This posed a bit of a problem for HTP: if it became public knowledge that they had obtained access to Linode, then perhaps they wouldn't have time to go after the impersonators using their newfound access to SwiftIRC. So, HTP tried to strong-arm Linode into staying quiet until May 1st. HTP had obtained the customer information and credit cards of all the Linode customers. HTP threatened to widely publish all this sensitive information if Linode didn't stay quiet. If Linode complied, then HTP would just delete all the info. Linode, though, was forced by the FBI to announce that they'd been broken into. HTP told Linode to just publicly acknowledge that HTP was the group that broke into Linode, and they'd delete the sensitive info. Linode did so (https://blog.linode.com/2013/04/16/security-incident-update/ https://blog.linode.com/2013/04/16/security-incident-update/). HTP conducted an internal investigation to determine which group member(s) were working with the FBI. HTP broke into the mole's computer and turned on their webcam, and saw an FBI employee looking over the shoulder of the mole. They kicked the mole out of the group, so the FBI doesn't have access to HTP anymore. (Remember, this is the story according to HTP.)
bestham 13y ago
So Much Drama in the HTP
- d23 13y ago
  It's kinda hard bein' L I N O D E
sp332 13y ago
Anyone care to speculate how likely this account is to be true?
- piggity 13y ago
  The only question I'm really interested in... And Linode isn't saying anything new
amitdugar 13y ago
Slightly OT, Is it possible to have a web application (using popular tech like RoR, PHP etc.) that cannot be cracked by anyone ?
- EvilLook 13y ago
  Only if the server is switched off and disconnected from the network.
  - amitdugar 13y ago
    Ha :) I guessed so .. It is impossible to make an un-crackable system .. not sure if that is a good thing or bad ...
    - ihsw 13y ago
      Every lock has a key. ~Hacker's motto
- dermotbrennan 13y ago
  It's ultimately unknowable but I think it's possible to have a very secure system. You just have to take a holistic approach and look at everything in the stack from top to bottom, keep on top of security updates and use the right security procedures. After all that you'll still never know whether it's secure or not...
- antihero 13y ago
  You can monitor exploit sites, but zero days are always possible and what will lead to serious hacks like this. So no, you can never be sure. The best thing is to keep your eye on the culture of the developers and how seriously they take security - for instance the Ruby on Rails developers ignored exploits/reports until people blew them wide open. Now, if some other hacker had known about that before the disclosure, they could have owned any RoR sites. From my experience, Django seems to be the best, and has not had any unfixed vulnerabilities for a while (though, due to it's complexity, it's completely possible that 0days exist). However, if I'm running Django sites and some do get owned, I can tell my boss/client/self/whatever that I did everything possible to prevent it happening. There is no such thing as 100% secure, however, it's fairly reasonable to be hardened to all but the most dedicated crackers. With an attack like HTP's, there's no fucking way anyone could have been expected to prevent, without running their entire own infrastructure, because they owned registras, Linode's LISH shell (so they get near-physical access to your Linode), and various other crap. If your boss were to fire you for getting owned in this attack, despite it preeetty much being zero of your own fault, they would be in the wrong (unless you have the resources to not depend on anyone).
rip747 13y ago
i'll never understand why way back in the day, someone thought that it would be a good idea to put all the scripts for the extension tags (like cfform) under the same parent directory (CFIDE) as the administrator.
antihero 13y ago
The ColdFusion hack...wow. How is CF engineered so badly? What person nowadays would still think to take paths of anything at all ever in the request parameters? I can sort of understand pre 2003 or something, but CF10 was released in 2012, for Pete's sakes. Also makes you wonder, if there are holes like this, how many more holes like this are there? Especially if this is a pattern across the system.
- druiid 13y ago
  There is a bunch of holes in CF like this. Look at their bug/security fix list for Coldfusion (Pretty much any version), and half of the security fixes are targeted to CFIDE based vulnerabilities. Any CF admin worth their salt disallows access to CFIDE as a matter of course.
orokusaki 13y ago
The ability for "hackers" to thrive is a necessary price to pay to secure our rights on the Internet. Trading freedom for security pays nothing, and never will. Let the FBI work their asses off to try to bust these people.
AndyKelley 13y ago
What I want to know is, what kind of hacker uses hard tabs in their zero day python script.
- peterwwillis 13y ago
  Hackers are [in general] shittier coders than a Windows server admin.
  - alan_cx 13y ago
    Yeah, dogs are shittier cats than .... cats.
- teach 13y ago
  You can have my hard tabs when you pry them from my cold, dead, RSI-crippled hands. :)
- meowface 13y ago
  It's fairly well-coded too, which makes it even stranger.
peterwwillis 13y ago
I can't think of a better classification for a terrorist than people who sit around all day working to destroy credibility of corporations and expose personal and financial information for the sake of their own fucked up moral code and amusement. It would be nice if we had internet role models. IRC is full of low-life degenerates who perpetuate the vitriol that reinforces this way of life as an acceptable pastime. If there were well respected hackers who spoke publicly against this kind of behavior it might make some people think twice. (Unfortunately, most well respected hackers used to be these kids before they got real jobs) HN is full of individuals who try to take the high road, versus the kind of anonymous internet idiocy that exists in nearly every forum and chatroom. I love this about HN. I wish more of the internet took it as an example.
- saraid216 13y ago
  IRC is a communication medium. Could we please not vilify it? It's like saying people who use burner phones are bad people.
  - peterwwillis 13y ago
    It's also a machine that generates dumbed-down conversation. The natural slant on IRC is away from intelligent discourse and toward a cross between texting and one-line jokes with friends. There's nothing inherently wrong with this. But it does foster negativity much of the time. I know hundreds of people who dedicate their lives to the drama and bullshit that is spawned solely by being in an IRC channel. If it went away, these people might just find something productive or positive to do with their lives.
    - saraid216 13y ago
      Yeah... I think you're hanging out with stupid people a bit overmuch. You can say the same thing about SMS, Facebook, Twitter, IM in general, email in general, MUDs and MMOs, telephones, telegraphs... Should all those go away, too?
    - diakritikal 13y ago
driverdan 13y ago
Some of their claims seem a bit far-fetched. Hacking name.com, Xinnet, MelbourneIT, and Moniker? That would be huge. Why haven't we heard more from them? > We identified which users on HTP were involved with the FBI, and promptly gained access to one of their cams. Not sure what they mean here. FBI camera? User's laptop camera? Either way this also seems far-fetched. If everything they said was actually true it's very impressive.
- GigabyteCoin 13y ago
  They claim to have accessed the mole's webcam. That's not very far-fetched at all.
- rashkov 13y ago
  Not so far fetched. They've posted hack logs of MelbourneIT, name.com, and Moniker in the same zine.
hexonexxon 13y ago
Nobody remembers the Linode bitcoin "hack" where it was assumed by bitcointalk that an admin was looting accounts? Im surprised anybody still uses them.
- tiredofcareer 13y ago
  It's discussed elsewhere in this thread, and bitcointalk assumed incorrectly.
kouiskas 13y ago
Looking at the hash found in the query HTP ran, 9gag's name.com password was "harry1" at the time of the exploit. It also tells us name.com stores the passwords as unsalted MySQL 4.1 PASSWORD() hashes...
orthecreedence 13y ago
First mistake: using Coldfusion. Second mistake: keeping it.
rth 13y ago
These kind of hacks improving the world. Thanks for to the hacks (not for stealing CCs or usernames) that they showed up again there is no f.cking security in the world.
arthulia 13y ago
The site appears to be down now, so... http://pastie.org/private/xedrpvi9lbcfwnz7wvb1a http://pastie.org/private/xedrpvi9lbcfwnz7wvb1a
rweir 13y ago
and today name.com emailed customers admitting they'd been pwned.
whoowy 13y ago
This story can make a movie

357 points by foofoobar 13y ago

There's more info about HTP5, including working mirrors of the files linked at the end of the linode document here: http://straylig.ht/zines/HTP5/ http://straylig.ht/zines/HTP5/

dantiberian 13y ago
HTP5 stands for Hack The Planet 5, the fifth issue of the zine linked in the parent comment.

blacktulip 13y ago

I am not familiar with the crackers' terms. So does this mean that name.com is not safe? All my domains are there...

JonnieCache 13y ago
The point is, when you're dealing with people of this level of supposed skill, they can just walk into pretty much any network. They're all vulnerable on some level if you're capable of actually breaking them yourself in novel ways. The only real solution is either to not depend on any single network, or to make it clear that you will simply kill anyone who troubles you. Or just remain innocuous enough that nobody will care to.
- twistedpair 13y ago
  They had skilz, no doubt, but it does not appear the CF hack was the zen apex of hacking. It was a well known exploit you could drive a truck through. Writing Stuxnet, now that was mad skilz.
sparkinson 13y ago
Looking around there doesn't seem to be any news on a breach at name.com, official or un-official (aside from this of course). It does have me worried however.

robk 13y ago

That seems somewhat scary if they've compromised domain registrars and are intercepting login data from client sites that way.

jrockway 13y ago
It's more scary if they've compromised a SSL CA. A simple DNS attack won't stop your browser from displaying a broken certificate warning. (Though they can always not redirect from http to https and most users won't notice, sadly.)
- blibble 13y ago
  it's very easy to get your own https cert once you control the dns for a domain, you just set up own nameserver that proxies requests to the original NS (except very specific ones, say those from Verisign), request your "domain control validation" https cert, and bam! valid https cert!
- deleted 13y ago
  [deleted]
  - bstpierre 13y ago
    It said "not redirect from http to https" -- meaning that when someone requests http://example.com http://example.com they would normally be redirected by the site owner to https://example.com https://example.com, but the attacker could just leave the original request alone. The point is that most people wouldn't notice. HTTPS Everywhere or similar browser plugin would probably pop up an alert if this did happen.
    - graue 13y ago
      HTTP Strict Transport Security[1] is designed specifically to prevent that attack. Unfortunately, Linode's manager doesn't seem to use it: $ curl -I https://manager.linode.com/ HTTP/1.1 200 OK Server: nginx/0.7.65 Date: Tue, 07 May 2013 16:23:07 GMT Content-Type: text/html;charset=UTF-8 Connection: keep-alive Vary: Accept-Encoding If they were, there would be a line like this: $ curl -sI https://github.com | fgrep Strict Strict-Transport-Security: max-age=2592000 Which tells the browser, for the next 2592000 seconds (30 days), only request github.com over HTTPS, never HTTP. Not sure why Linode isn't using HSTS — lack of awareness? The super-old version of Nginx is also a little scary. [1] https://en.wikipedia.org/wiki/HTTP_Strict_Transport_Security https://en.wikipedia.org/wiki/HTTP_Strict_Transport_Security

ghshephard 13y ago

There is a lot of inside-baseball in this, but the one they keep talking about, is, "shred customer data" - as in, " Recognizing their situation, we instead told them that if they acknowledged HTP in their analysis, we'd go ahead and shred their customer data anyway." Do they honestly, for a single second, think that any LEA, corporation, or, well, anyone would believe that once the information was compromised, that there was no putting the genie back in the bottle? Also - I suspect there are probably disclosure laws that had to be followed by Linode anyways.

deleted 13y ago
[deleted]
nathanb 13y ago
What choice did they have? Comply, and trust the honor of black hat hackers? Or refuse, and have customers' data appearing on FTP and torrent sites within the day? As a Linode customer, I would rather them choose the latter. Not to try and sweep the incident under the rug (to your point about disclosure) but to prevent the data from being scraped by groups who exist solely for extracting credit card details from releases by groups like HTP (note the reference to "carders" in the article) and then being sold. (According to Linode's own post, the CC data were encrypted, meaning that it should be intractable to actually extract usable CC numbers from the data. But why would Linode not accept those terms, even if they believed that HTP were lying? At least it would give them until 1 May to get their house in order.)

runn1ng 13y ago

I... don't actually understand most of what they wrote there.

mappu 13y ago

Definitely worth reading the full zine, some scary stuff in there (including very readable python LFI-based exploits for unpatched MoinMoin and ColdFusion). Highlights: 1900+ days uptime on a sparc box somewhere in sourceforge.net, root on ICANN, root on Debian repositories..

psutor 13y ago
Link to full zine: http://straylig.ht/zines/HTP5/ http://straylig.ht/zines/HTP5/
- cheapsteak 13y ago
  Any chance someone saved a copy? Link seems dead from here
kokey 13y ago
I've seen longer uptimes on Solaris sparc boxes, many times, when they had a board of battery replaced and they booted up with a reset clock which got corrected by NTP after boot.

tiredofcareer 13y ago

Some hopefully-helpful clarifications of the inside baseball talk from just the overview (I haven't read the full zine), enhanced with inside and general knowledge I've gained in my travels on this mortal coil: - HTP claims to have{, had} access to name.com, which Linode currently uses. This access enables an unauthorized party to update authoritative nameservers for your domain; i.e., if you host at Amazon, very likely your authoritative nameservers are Route 53 on your account. HTP would not have access to modify the zone directly through the registrar and would instead have to hijack the entire domain with a working, completely-transferred zone on their own nameservers. For this to go down entirely unnoticed is extraordinarily difficult. I won't say impossible, but damned close without a copy of the zone in hand and with Linode running AXFR disabled (you should be too). There are subzones of linode.com; they wouldn't have gotten them all, and it would have been noticed within minutes. - In order to attack SwiftIRC, to get back at some script kiddies DoS attacking them after their last release (because you know, that's a good target to burn registrar access on and all), HTP decided to backdoor SwiftIRC via their nameservers which are hosted at Linode. That's not the same as the registrar nameservers discussed above, but is instead the DNS data actually stored on a Linode on SwiftIRC's account. They do not hint what they were going to do with it once they had hijacked the nameservers, and I will not theorize. I could guess, though. - Before utilizing their registrar access (from the first bullet point) to hijack the linode.com zone and intercept manager logins silently by redirecting traffic via DNS -- also fairly difficult to pull off without a good linode.com certificate in hand, in terms of keeping the TLS session non-suspicious to a browser -- they instead discovered a zero-day in ColdFusion (Linode's stack) and got in that way. That's much quieter and much more likely to not be noticed. If we take the FBI's actions at HTP's word, the FBI was the only reason Linode was made aware of this outside of HTP's control; a DNS hijack would have been immediately noticed by Linode administrators. - Knowing what I know (let's leave it at that), a successful exploit on Linode's ColdFusion stack entails a database of Linodes, DNS, credit cards, e-mails, addresses, and keys to decrypt the actual card numbers, and a lot more data. You have to decide whether to take HTP at their word that they deleted credit cards. Consider your credit card and all prior credit cards compromised if it were in the system before April. - The access that HTP obtained does not, full stop, lead to root on Linode instances without at least one shutdown job or change of root password job showing up in your Linode's history that you did not ask for. Your Linode's root password is not stored in any Linode system aside from on your Linode itself. Your LISH password, as they say, is, and according to them is stored in plaintext; if you see things on your Linode's console (located under the Remote Access tab) that you did not type, that access was used upon you. If not, it wasn't. If you used the same root password on your Linode that you did for your LISH password, consider that password compromised. I'm suspicious of the claim that they rooted all those (assuming) customers without any of them noticing their Linode being rebooted to apply the new root password to allow HTP in, and I would read that as "potential access" instead of "access". They probably bounced some nmap.org servers to reset their root passwords -- a Linode system requirement -- without fyodor noticing. Which is interesting for a couple reasons. - Also, the access they obtained does not lead to root on the Linode host fleet itself, unless they are holding back some extra access they obtained such as a shared password between the ColdFusion stack and administrator credentials for Linode systems, which I consider unlikely for a couple reasons. With several days to get familiar with the architecture, HTP could have used their database write access to do things on the hosts, but it's a fairly limited set of things. Dumping Linode's database is bad, but root on their hosts is far, far worse, and by indications, I don't think they got it. - How does this relate to the Bitcoin hacks of yesteryear, you ask? The Bitcoin hackers probably got in the exact same way -- Linode hinted at a compromised admin credential, which is close enough to do everything HTP was able to do -- then shut down and reset root passwords on the Bitcoin Linodes they were after, which then gave them filesystem access. So ends clarifications, thus begins conclusions: - PAY ATTENTION WHEN YOUR SERVERS ARE REBOOTED WITHOUT YOUR COMMAND. - PAY ATTENTION WHEN YOUR SERVERS ARE REBOOTED WITHOUT YOUR COMMAND. - Linode added a feature that shoots you an e-mail when your Linode is manipulated in any way via jobs, such as resetting your Linode's root password (a la Bitcoin/HTP hacks). It's depressing they had to do this, but pay attention if you get the mail. External monitoring like Nagios that pages you when your server goes down is also a good idea, as long as it is hosted at another provider. - EDIT: After reading the zine, yet again, /CFIDE is the vector. There's no excuse for not hiding your administrative tools, generally the soft underbelly of the whole smash, from the Internet. None. It's one rewrite in nginx. Match /CFIDE<anything> from the public, redirect to /. Done. - EDIT: Again, after looking at how trivial the exploit was, it's probably time to reconsider using Adobe ColdFusion from a business continuity standpoint. Half Linode's fault for not hiding /CFIDE, half Adobe's fault for the engineering missteps that lead to this capability for a remote attacker. We should be just as hard on ColdFusion as we are on Rails. - SwiftIRC is a den of inquity, up there with EFnet; if you run a hosting provider, think twice about permitting SwiftIRC anywhere near you. To reiterate that, Linode was a casualty of someone going after SwiftIRC. Delink their nodes, cancel them, and kick them to the curb if you're interested in preserving your business. Not worth the money. Same with damned near all the IRC networks except OFTC and, to a far lesser extent, Freenode. There will always be targets but harboring SwiftIRC is probably a malicious-actor magnet. - Registrars (and CAs, though that's outside this discussion) are the weak point in the entire system. This is not the first time they have been shown to be so. Linode could be Fort Knox of digital security but if name.com falls over, it's all over; that's entirely outside of Linode's, and your, control. Currently, the registrar market is heavily profit-centric and, personally, I think people spend far too little on a domain in the general case. I would happily pay a registrar a lot more money -- hundreds a year or more -- if their offering were run competently, as it is fairly obvious name.com isn't. Compare your hosting bill to your registrar bill; what's wrong with that picture? - HTP is apparently fairly easy to troll into using valuable access for vengeance purposes. Shameful target selection and a burn of a good hack just to root SwiftIRC. That's like pissing in the ocean for a good time. - Linode got railroaded here and the general reaction by folks is a little overdone. You know that's true when even the hackers' overview of the hack specifically calls out people bitching about Linode security on Twitter. All it takes is one zero-day, and you will all be hit by one in your career, so cut Linode a little slack.

RoboTeddy 13y ago

Here's an attempt at an explanation/translation: HTP ("Hack The Planet") is a group that likes to break into things. Another (unnamed) group of people impersonated a third group of people ("ac1db1tch3z") and tried to cause trouble for HTP. The impersonators located HTP by examining one of HTP's botnets (a collection of compromised computers that are used to launch things like denial of service attacks). Botnets have to receive instructions (e.g., targets to attack) from somewhere, so it's likely that the impersonators followed the path taken by commands to the botnet, and found the network(s) that HTP uses to organize themselves. HTP realized this, and wanted to get back at the impersonators. They found out that the impersonators used an IRC channel (chat room) hosted on a network called SwiftIRC. If HTP could break into SwiftIRC (which is hosted on Linode), they could cause all sorts of trouble for the impersonators. So HTP decided to break into Linode, so they could break into SwiftIRC, so they could break into the group of impersonators. To break into Linode, HTP broke into their domain name registar (name.com). They planned to secretly take control of linode.com, and replace it with a version of linode.com would look and feel and work correctly, but had one additional feature -- it would collect the login information that people typed in. HTP probably hoped to gain the login for SwiftIRC directly, or collect the logins for Linode admins and obtain SwiftIRC's login from there. But, before they enacted the domain takeover (a maneuver that would likely be somewhat difficult to employ without being noticed), an HTP member discovered a new vulnerability in ColdFusion, the server software used by Linode. The ability to discover a new exploit on demand implies a high level of skill within the group. Using this exploit, HTP obtained direct access to Linode. They proceeded to gain access to SwiftIRC, as well as other sites hosted on Linode, including a well-known security site, nmap.org The FBI apparently had a mole in HTP, and they alerted Linode that HTP had access to nmap.org. This posed a bit of a problem for HTP: if it became public knowledge that they had obtained access to Linode, then perhaps they wouldn't have time to go after the impersonators using their newfound access to SwiftIRC. So, HTP tried to strong-arm Linode into staying quiet until May 1st. HTP had obtained the customer information and credit cards of all the Linode customers. HTP threatened to widely publish all this sensitive information if Linode didn't stay quiet. If Linode complied, then HTP would just delete all the info. Linode, though, was forced by the FBI to announce that they'd been broken into. HTP told Linode to just publicly acknowledge that HTP was the group that broke into Linode, and they'd delete the sensitive info. Linode did so (https://blog.linode.com/2013/04/16/security-incident-update/ https://blog.linode.com/2013/04/16/security-incident-update/). HTP conducted an internal investigation to determine which group member(s) were working with the FBI. HTP broke into the mole's computer and turned on their webcam, and saw an FBI employee looking over the shoulder of the mole. They kicked the mole out of the group, so the FBI doesn't have access to HTP anymore. (Remember, this is the story according to HTP.)

bestham 13y ago

So Much Drama in the HTP

d23 13y ago
It's kinda hard bein' L I N O D E

sp332 13y ago

Anyone care to speculate how likely this account is to be true?

piggity 13y ago
The only question I'm really interested in... And Linode isn't saying anything new

amitdugar 13y ago

Slightly OT, Is it possible to have a web application (using popular tech like RoR, PHP etc.) that cannot be cracked by anyone ?

EvilLook 13y ago
Only if the server is switched off and disconnected from the network.
- amitdugar 13y ago
  Ha :) I guessed so .. It is impossible to make an un-crackable system .. not sure if that is a good thing or bad ...
  - ihsw 13y ago
    Every lock has a key. ~Hacker's motto
dermotbrennan 13y ago
It's ultimately unknowable but I think it's possible to have a very secure system. You just have to take a holistic approach and look at everything in the stack from top to bottom, keep on top of security updates and use the right security procedures. After all that you'll still never know whether it's secure or not...
antihero 13y ago
You can monitor exploit sites, but zero days are always possible and what will lead to serious hacks like this. So no, you can never be sure. The best thing is to keep your eye on the culture of the developers and how seriously they take security - for instance the Ruby on Rails developers ignored exploits/reports until people blew them wide open. Now, if some other hacker had known about that before the disclosure, they could have owned any RoR sites. From my experience, Django seems to be the best, and has not had any unfixed vulnerabilities for a while (though, due to it's complexity, it's completely possible that 0days exist). However, if I'm running Django sites and some do get owned, I can tell my boss/client/self/whatever that I did everything possible to prevent it happening. There is no such thing as 100% secure, however, it's fairly reasonable to be hardened to all but the most dedicated crackers. With an attack like HTP's, there's no fucking way anyone could have been expected to prevent, without running their entire own infrastructure, because they owned registras, Linode's LISH shell (so they get near-physical access to your Linode), and various other crap. If your boss were to fire you for getting owned in this attack, despite it preeetty much being zero of your own fault, they would be in the wrong (unless you have the resources to not depend on anyone).

rip747 13y ago

i'll never understand why way back in the day, someone thought that it would be a good idea to put all the scripts for the extension tags (like cfform) under the same parent directory (CFIDE) as the administrator.

antihero 13y ago

The ColdFusion hack...wow. How is CF engineered so badly? What person nowadays would still think to take paths of anything at all ever in the request parameters? I can sort of understand pre 2003 or something, but CF10 was released in 2012, for Pete's sakes. Also makes you wonder, if there are holes like this, how many more holes like this are there? Especially if this is a pattern across the system.

druiid 13y ago
There is a bunch of holes in CF like this. Look at their bug/security fix list for Coldfusion (Pretty much any version), and half of the security fixes are targeted to CFIDE based vulnerabilities. Any CF admin worth their salt disallows access to CFIDE as a matter of course.

orokusaki 13y ago

The ability for "hackers" to thrive is a necessary price to pay to secure our rights on the Internet. Trading freedom for security pays nothing, and never will. Let the FBI work their asses off to try to bust these people.

AndyKelley 13y ago

What I want to know is, what kind of hacker uses hard tabs in their zero day python script.

peterwwillis 13y ago
Hackers are [in general] shittier coders than a Windows server admin.
- alan_cx 13y ago
  Yeah, dogs are shittier cats than .... cats.
teach 13y ago
You can have my hard tabs when you pry them from my cold, dead, RSI-crippled hands. :)
meowface 13y ago
It's fairly well-coded too, which makes it even stranger.

peterwwillis 13y ago

I can't think of a better classification for a terrorist than people who sit around all day working to destroy credibility of corporations and expose personal and financial information for the sake of their own fucked up moral code and amusement. It would be nice if we had internet role models. IRC is full of low-life degenerates who perpetuate the vitriol that reinforces this way of life as an acceptable pastime. If there were well respected hackers who spoke publicly against this kind of behavior it might make some people think twice. (Unfortunately, most well respected hackers used to be these kids before they got real jobs) HN is full of individuals who try to take the high road, versus the kind of anonymous internet idiocy that exists in nearly every forum and chatroom. I love this about HN. I wish more of the internet took it as an example.

saraid216 13y ago
IRC is a communication medium. Could we please not vilify it? It's like saying people who use burner phones are bad people.
- peterwwillis 13y ago
  It's also a machine that generates dumbed-down conversation. The natural slant on IRC is away from intelligent discourse and toward a cross between texting and one-line jokes with friends. There's nothing inherently wrong with this. But it does foster negativity much of the time. I know hundreds of people who dedicate their lives to the drama and bullshit that is spawned solely by being in an IRC channel. If it went away, these people might just find something productive or positive to do with their lives.
  - saraid216 13y ago
    Yeah... I think you're hanging out with stupid people a bit overmuch. You can say the same thing about SMS, Facebook, Twitter, IM in general, email in general, MUDs and MMOs, telephones, telegraphs... Should all those go away, too?
  - diakritikal 13y ago

driverdan 13y ago

Some of their claims seem a bit far-fetched. Hacking name.com, Xinnet, MelbourneIT, and Moniker? That would be huge. Why haven't we heard more from them? > We identified which users on HTP were involved with the FBI, and promptly gained access to one of their cams. Not sure what they mean here. FBI camera? User's laptop camera? Either way this also seems far-fetched. If everything they said was actually true it's very impressive.

GigabyteCoin 13y ago
They claim to have accessed the mole's webcam. That's not very far-fetched at all.
rashkov 13y ago
Not so far fetched. They've posted hack logs of MelbourneIT, name.com, and Moniker in the same zine.

hexonexxon 13y ago

Nobody remembers the Linode bitcoin "hack" where it was assumed by bitcointalk that an admin was looting accounts? Im surprised anybody still uses them.

tiredofcareer 13y ago
It's discussed elsewhere in this thread, and bitcointalk assumed incorrectly.

kouiskas 13y ago

Looking at the hash found in the query HTP ran, 9gag's name.com password was "harry1" at the time of the exploit. It also tells us name.com stores the passwords as unsalted MySQL 4.1 PASSWORD() hashes...

orthecreedence 13y ago

First mistake: using Coldfusion. Second mistake: keeping it.

rth 13y ago

These kind of hacks improving the world. Thanks for to the hacks (not for stealing CCs or usernames) that they showed up again there is no f.cking security in the world.

arthulia 13y ago

The site appears to be down now, so... http://pastie.org/private/xedrpvi9lbcfwnz7wvb1a http://pastie.org/private/xedrpvi9lbcfwnz7wvb1a

rweir 13y ago

and today name.com emailed customers admitting they'd been pwned.

whoowy 13y ago

This story can make a movie