12 ms·

Security incident update

318 points by asb 14y ago

cheald 14y ago
Good to see some discussion and acknowledgment from Linode, but their treatment of the credit card encryption is distressing. They don't address the claim that the private keys were on the servers' filesystems, and instead focus on them being password protected. If the password is brute-forced, then it's game over.
- tsahyt 14y ago
  My thoughts exactly. Then again, I'd expect them to use a rather strong password for sensitive data. Considering 10^12 tries per second (which is probably very far fetched) and a 64 character alphanumeric password, my credit card -- probably the whole galaxy -- will have long expired until the password can be expected to be cracked. At that point, it really depends on how strong their passphrase is. If it's a dictionary word, we're pretty much fucked.
- derefr 14y ago
  I think a lot of people are failing to understand how credit card processing works. The private key needs to live on some machine, if you actually want to have any CC requests processed at some point. If the private key itself isn't on the frontend/user-facing server, then instead, when the frontend node is hacked, the hacker will find a shared secret or signing key to auth the frontend node against the backend node that does process the CC requests. The hacker won't be able to take your CC number for themselves in this case, but they'll sure be able to put spurious/malicious charges on your account.
  - JonnieCache 14y ago
    I suppose it could have been kept only in memory, rather than just chilling on the filsystem in a file called private_key which is basically what happened AFAIK. It's surely within the boundaries of sense to load the key manually each time the app boots. Thinking about it though, that's basically the role of the key passphrase.
    - derefr 14y ago
      Right; this is a common experience with SSL/X.509 signing keys--they're encrypted by default, so any server you configure with your ssl.pem file will prompt for a passphrase on startup, and only keep the decrypted key in memory. That's why this line, that most SSL providers give in their instructions, is so dangerous: openssl rsa -in server.key.org -out server.key It strips this (rather useful) encryption, giving any successful attacker to your frontend server your private key, in exchange for letting the node come up non-interactively.
    - gingerlime 14y ago
      This is what ssh-agent does, and I could only hope Linode was using something similar. I think the statement suggests they did. > The private key is itself encrypted with passphrase encryption and the passphrase is not stored electronically So even if the hacker bragged about obtaining both the public and private keys, the private key was protected and therefore unusable without the passphrase. At this point, we can all hope linode used a very complex passphrase... My gut feeling is that they did.
      - joeyh 14y ago
        Storing a passphrase (or unlocked private key) in memory is storing it electronically. Once the attacker has root, they can dump the memory and access this information. The attacker claims to have gigabytes of data, which could well include a memory image (or swap partition image, if the program that stored this in memory didn't lock the memory to prevent it being swapped out).
        joeyh 14y ago
        Attacker claims to have done so in this irc log: http://pastebin.com/7WXRDyAg http://pastebin.com/7WXRDyAg > 07:52 < HTP> the CCrypter class of the linode application context was accessable from outside the wwwroot using undocumented ColdFusion methods. i was fully able to decrypt the ccs using the in-memory privkey that they supplied the password for. > 07:53 < HTP> coldfusion runs as a single process, and its memory can be accessed using the ColdFusion wrapper
  - rainforest 14y ago
    From their description it appears encrypted CC numbers were in the database amongst the other customer information. Sure, the data has to live somewhere, but the apparent situation of CC, customer, private key all accessible to the frontend looks sub-optimal.
    - gingerlime 14y ago
      there's a difference between obtaining the private key and the private key file, which according to Linode was protected with a (hopefully strong) passphrase...
      - rainforest 14y ago
        It might have been encrypted, but surely it must be exposed to their system somewhere to enable them to make charges? Is it feasible, if that was the case, that the attackers could use that to exfiltrate decrypted CC info? I suppose this wild speculation isn't helpful or conducive and waiting on more information might be a better idea. I think the reputation damage has already been done, judging by the comments on the previous post though.
        gingerlime 14y ago
        It depends on how they implemented it. Maybe there's an operator that logs on once a day and enters the passphrase, and then they use it to charge credit cards. So the private key is exposed only for a brief period. Another possibility is that the passphrase is entered when the machine is booted, and then the private key (not the passphrase) is loaded into memory. This is how e.g. ssh-agent[1] works. When this mechanism is used, you don't get access to the private key itself without direct memory access (which is possible, but not that easy to carry out, and would typically require root access and debug tools etc). However, the private key can be then used to carry out crypto operations. From the IRC chat logs, it looks like ryan got the private key file, but he didn't get the private key itself. At least he didn't seem to provide any solid proof to suggest he did. [1]https://en.wikipedia.org/wiki/Ssh-agent https://en.wikipedia.org/wiki/Ssh-agent
  - InclinedPlane 14y ago
    CC data however does NOT have to live on any machine that anyone except your merchant bank owns. If you are writing against billing APIs in the 21st century then you only need to store a token which you use for recurring billing. Not everyone does this, but it's a reasonably good practice, especially if you have had intrusions in the past.
    - shadowfox 14y ago
      Out of curiosity, is this token generation sufficiently general or is it tied to a specific bank/credit card provider? (Basically is it easy to switch providers without having all your customers enter their cc information again?)
      - nenolod 14y ago
        Unfortunately it is tied to the merchant. :(
  - deleted 14y ago
    [deleted]
- reidrac 14y ago
  I agree with you. Besides, how do they use the private key? Are they using a GPG agent o anything like that? Do they enter the password every time they need to use the private key? Having the private key in the web frontend sounds like a bad idea (as this incident has proved), they should use a backend system that is not accessible from the Internet. I don't know how they system works but the web frontend needs to encrypt CC numbers but I'm not sure it is required to decrypt them as that could be performed in a more restricted environment. Security incidents suck, but the good news is that they now have a chance to improve their security.
- zcam 14y ago
  Even then, attackers with half a brain (and malicious intent) would also have backdoored the payment page to harvest "fresh" credit cards numbers during the period they went unnoticed (possibly days/weeks). It's another good reason to leave the credit card handling to a reliable 3rd party imo.
- entropy_ 14y ago
  That doesn't make any sense. It's exactly like saying: "Oh sure, the credit card numbers were encrypted but if the encryption key is brute-forced then it's game over". The CC numbers were encrypted using a public key. The corresponding private key was itself encrypted using "[a] complex passphrase [that] is not stored electronically". They could be lying, but if they are not then this is perfectly valid security. Using this scheme, you can store new CC numbers any time you want using the public key. You'd have a regular billing cycle(say a month) with billings happening in a short time window(All my linode accounts get billed within minutes of each other so I think this is the case for linode). When you want to bill your customers you'd input the passphrase which wouldn't be stored anywhere, would be used to temporarily decrypt the CC numbers and bill everyone. Unless you are attacked in that window(or an attacker gains access before that window and can remain unnoticed until it happens, and even then they'd need root access to read the memory and not just access to the filesystem) then the CC numbers are pretty safe against any attack that leaks files/databases/etc...
  - thaumaturgy 14y ago
    > When you want to bill your customers you'd input the passphrase which wouldn't be stored anywhere, would be used to temporarily decrypt the CC numbers and bill everyone. I've had account changes billed to my card within minutes, even (and especially) in the middle of the night (when I do most of that stuff). I'm having a little bit of trouble imagining that someone's sitting in front of a terminal and typing a passphrase into a prompt every three to five minutes, all day long.
    - entropy_ 14y ago
      You make a good point
    - corford 14y ago
      Interesting. Whenever I make changes that affect how much I'll be billed, the amounts get added or removed and take effect when the card next gets charged (which happens once a month). At least that's how I remember it... going to dig through some old statements now to verify this as its been a while since I last changed anything that would alter what they charge me.
sturadnidge 14y ago
I really wish they would officially comment on the 'cover up' aspect. Security breaches happen, and are forgivable. But attempting to broker a 'silencing' deal with the intruders and hoping your customers will never be the wiser is not. All this update does is restore my faith in their ability to store my information correctly. It does nothing to reassure me that they won't try to cover anything up again.
- kbar13 14y ago
  As these situations, especially on the internet, come out as "he said, she said", I think it's probably more important to keep focused on what directly affects you.
  - sturadnidge 14y ago
    Sure, but in this case it wasn't Linode who initially came forward - it took a public announcement from the intruders for Linode to notify us.
    - rainforest 14y ago
      Devils advocate: preparing a press release from a company after plugging holes and auditing that makes the appropriate admissions and apologies probably takes more time than writing up a successful hit after attacking a page.
      - makomk 14y ago
        Linode didn't just take their time to announce this, just a few hours ago they were apparently telling customers definitively that their credit card information hadn't been accessed.
        kid0m4n 14y ago
        And that is "still" the case
- 14y ago
tsahyt 14y ago
The hacker has claimed the public and private key were both stored on the same machine and both accessible. The only protection left is the passphrase then. Various people have reported fraudulent activity on their CCs. Personally, I think Linode didn't mess this up as badly as many other companies. The statement took a while to appear but it's there. As a Linode customer myself I can neither report anything unusual on my box, nor activity on my CC. Thus far I'm somehow inclined to believe Linode when they say no CC information has been compromised. Either way, they are offering a great service and I'll stick with them. Worst case, I'll have to get a new CC.
- kbar13 14y ago
  > Various people have reported fraudulent activity on their CCs Most likely not even related
  - prawn 14y ago
    I've seen two mentioned in each of two recent Linode threads, potentially the same people. Have there been many more?
  - VLM 14y ago
    Well, lets think about it a bit. It turns out the numbers are ridiculously hard to find although I'm a pretty good searcher/infovore. Better info would be appreciated. Anyway the best I could find was two claims: There were 11 million american victims of identity theft in 2011 (outta 400 million) for a ratio of about 1 in 40 of the general population gets p0wned per year. Obviously not all ID theft is CC theft but I'd guess a lot of it is. Based on friends and family experience getting a CC number stolen every four decades of use passes the smell test so I will go with that. A quote from prweb.com found by google: "Started in 2003, Linode has grown to over 45,000 customers". I wonder what year that quote is from. Still "forty five thousand" is probably not totally ridiculous. It makes sense looking at their hostname scheme, ip space allocations as per whois, and some educated guesses. Lots of people including myself like linode. So assuming the average linode customer is the same as the average joe6pack then 45000/40/365 = about 3 linode customers should get p0wned per day regardless of any linode problem. To make my inner EE happy I'll call the "noise level" about 3/day and evaluate the SNR based on that noise level to see if there's a signal of p0wnership. Obviously the two reports over the course of a couple days is not 100% of all linode customers who got p0wned. But it does show that two reports doesn't really prove anything. Now, as a made up example, 900 reports over 3 days would be a HUGE indicator "something" happened. But at a predicted noise level of about ten or so over three days, two reported is down there in or below the noise. So, as a long term linode customer I'm not freaking out (yet). My CC does email me every time a charge is made, and nothing weird has been seen. All quiet on the western front.
jpallen 14y ago
I know Linode has been getting hammered for not being trustable or that they don't respond well when contacted, but in my experience they have been extremely responsive to any issues I have raised. I don't just mean technical issues either, we had what looked like a security breach on our account and it kicked of a small frenzy in their support staff and supervisory team to track down the problem. They figured out what had happened after a thorough investigation, gave us a full post mortem and promised they were reviewing procedures to make sure it couldn't happen again.
- ciaron 14y ago
  I can second this - Linode has always responded quickly and effectively in my experience.
  - andyhmltn 14y ago
    I third this. I've never had a problem. Most of my tickets have a response within 10 minutes and they always seem to have been open when anything has gone wrong on their end. Is anyone not thinking that the reason they didn't come out in public talking about this until now is because law enforcement was involved? Notifying the public could ruin the investigation.
    - threeseed 14y ago
      No. Because Linode has been hacked before and exactly the same situation is playing out. Major compromise and selective release of information. I really do wonder at what point people are going to realise that they aren't dealing with a company who cares about their interests.
- sveiss 14y ago
  I've always had an excellent experience when dealing with Linode one-on-one, via email or support ticket, but their one-to-many communication sometimes seems to be lacking. I'm glad they posted a more detailed update regarding this issue publicly.
- 14y ago
AlexMuir 14y ago
Hmm. I'm sort of re-assured, but what's happened to the claim that this was targeting a specific customer?
- keidian 14y ago
  It seems HTP went after nmap.org machines, so I'm guessing that woudl be the directed customer they picked?
  - scott_karana 14y ago
    Do you have a chatlog or citation? I'm not arguing, I'd love to read it! :)
    - lcampbell 14y ago
      Here's the writeup from seclists.org: http://seclists.org/nmap-dev/2013/q2/3 http://seclists.org/nmap-dev/2013/q2/3 It looks like seclists.org and nmap.org are hosted on different IP addresses, but both route to linode, and both have the same WHOIS information, so I think it's safe to say the writeup applies to both.
      - scott_karana 14y ago
        Thanks! By the sounds of their writeup, the attacks just visited the "highest profile" sites as per the Quora link to obfuscate their real attack.
ultimoo 14y ago
The update has quite a frank and an apologetic tone to it. Especially the concluding paragraph gives it a very empathetic touch. It must be truly tough for the ops folk at Linode to have suffered an attack due to a third party 0-Day exploit. It could happen to any of us really. On a side note, I am not sure of the "some occurrences of plaintext Lish passwords". Seems like quite a goofup on Linode's part.
- troels 14y ago
  It's not really a zero-day, if a patch has been out for a week though.
  - vrta 14y ago
    It was 0-day when they got hit. Sad thing is that they do not tell how much time has passed since the hack.
    - troels 14y ago
      Yeah, maybe you're right. The text is a bit unclear about that.
      - vrta 14y ago
        Yeah, they didn't tell us when they got hacked, but seclist did (3/31/2013): http://seclists.org/nmap-dev/2013/q2/3 http://seclists.org/nmap-dev/2013/q2/3
        mjs 14y ago
        3/31/2013 is the backup they're reverting to, not when they got hacked though, right? (The email refers to "downtime over the last week", so I guess the hack happened around 6th April or later.
    - codeulike 14y ago
      I'm not sure, their wording around it is a bit convoluted "... previously unknown zero-day vulnerability in Adobe’s ColdFusion application server. The vulnerabilities have only recently been addressed in Adobe’s APSB13-10 hotfix (CVE-2013-1387 and CVE-2013-1388) which was released less than a week ago." To me that reads like 'it was a former zero-day exploit", as all exploits are.
rainforest 14y ago
Should CC info even be stored in the customer database? I would have thought that information should be write only. Does PCI allow that to happen with only PKI in place?
- ultimoo 14y ago
  I agree to your position of not storing the CC in the consumer database. However, it cannot be in a write-only state since usually consumers have to billed periodically without having to ask for their number.
  - rainforest 14y ago
    Ah, my description wasn't the best, apologies. What I meant was the payment backend, a separate server, could only service write requests from the frontend. The backend itself would possess decryption keys to make charges, but the frontend could only ever ask it to bill a customer or change the information via a write-only API. Having said that, if the API were provided via ColdFusion it wouldn't have helped anyway.
    - singlow 14y ago
      That is in fact what was done in an application I worked on. We used ColdFusion for both the Front-End and the "Commerce Server" and the commerce server only took credit card information from the front-end and could not give it back. It was behind its own hardware firewall and any exploit in ColdFusion would have been hidden from the front end unless it could be exploited through the API methods we exposed through the gateway. To gain access to any other url endpoints you would need to access the server from within the firewall by connecting to a VPN gateway on a separate physical network that was only connected by a dedicated T1 to our office. We did this before there was PCI, because it made sense. The whole system has now been rewritten in dotNET. It would be interesting to see if I can find that old CF app. It ran solid for almost ten years on the same 3 NT4 servers handling tens of thousands of users per day. And it could have probably run on one server except that we isolated the commerce and MSSQL functions from the front-end app.
  - 14y ago
prawn 14y ago
To be honest, I'd rather have my credit card number in the wild than be worrying about my sites/servers being compromised. Obviously neither is ideal, however.
egeozcan 14y ago
It's about time we start seeing some OAuth bank account logins for payments.
citricsquid 14y ago
Must have been a coincidence my card was stolen then. At least I don't need to move away from Linode now, hurray.
espennilsen 14y ago
It seems seclists was affected http://seclists.org/nmap-dev/2013/q2/3 http://seclists.org/nmap-dev/2013/q2/3
scottjad 14y ago
What isn't in this post: - Any mention of the claim that they had made a deal with the hackers. No attempt to deny the claim. - Any mention of when the systems were compromised or when they discovered that they were compromised. The post starts with "Yesterday," mentions working "around the clock" and mentions Friday, giving the impression that this is all very recent, when according to the hacker Linode has known for two weeks.
- ScottWhigham 14y ago
  If it was your company and you had decided to make a deal with the hackers, would you publicize or publicly address that fact? My thinking is that doing so would incentivize others to do similar acts because you are an easy mark.
- cjh_ 14y ago
  I find it somewhat worrying that they haven't directly address the points of the hacker's claim. I am particularly interested to find out whether linode did enter into a deal with the hackers.
nfm 14y ago
"Despite the uselessness of these hashes"... Hrm.
neya 14y ago
Ahh, they still don't mention if even their ex-customers are affected by this. Now, I'm really paranoid! Let's just hope for the best...but when I read about 'storing passwords in plain text' I did get a little turned off. I know the Linode team is very technically strong, infact, they've aided me so many times. It hurts to see them make such a silly mistake :(
zhoutong 14y ago
EDIT 2: It seems that I have confused the "encryption" with normal hashing. So this comment is invalid if they are doing encryption properly. I wonder how hard it is to brute-force the credit card numbers. Given the last 4 digits, the Luhn algorithm and the starting digit of common cards (e.g. 4 for Visa), there are essentially only 10 digits left to crack. That's just 10 billion combinations. As part of the source code is exposed, most likely the encryption algorithm/sequence is already known. If the passphrase is shared among all credit cards and the encryption algorithm is a fast one, there's a fair chance that eventually these credit cards are going to be exposed all at once. Or am I missing anything? EDIT: Even if the encryption algorithm is as slow as 10,000 tries per second, exhaustively attempting every single credit card number possible is going to take 277 CPU-hours, or about a day in three quad-core machines.
- dchest 14y ago
  If they used encryption properly (random or unique IV depending on cipher mode of operation), you can't bruteforce credit card numbers (encrypted content), you can only try to bruteforce the key used for encryption. See http://en.wikipedia.org/wiki/Semantic_security http://en.wikipedia.org/wiki/Semantic_security and http://en.wikipedia.org/wiki/Ciphertext_indistinguishability http://en.wikipedia.org/wiki/Ciphertext_indistinguishability
  - zhoutong 14y ago
    Right. I guess I had confused it with hashing. If the credit cards were hashed then it would be relatively easy to crack. Edited my comment. Thanks!
alinspired 14y ago
very reassuring update for someone who didn't read the IRC logs with hacker and i think it will do the trick for most of linode users
antihero 14y ago
Who on earth uses ColdFusion any more? Surely when they re-built it a while back they could have used something more secure like Django?
- mekoka 14y ago
  I agree with questioning their choice of ColdFusion, but I have some reserves about your proposed alternative to use Django. I don't know how secure Django is, but I know it's big. And my perception is generally that anything that big is bound to have a few loose ends, if looked under the magnifier.
chris_wot 14y ago
"Along with the encrypted credit card, the last four digits are stored in clear text to assist in lookups and for display on things like your Account tab and payment receipt emails." I really don't see the need for that at all. What sort of credit card "lookup" are they doing exactly?
- lucaspiller 14y ago
  I guess it's mainly to display stuff. When you have multiple cards on Amazon you can choose which one to use, however without seeing the last four digits its pretty hard to tell which is which.
- mekoka 14y ago
  It's not for them, it's for users. It helps to know which of your CCs is going to be charged.
- UnoriginalGuy 14y ago
  It is fairly standard. Go look at your Amazon account, you'll see the last four digits.
abalone 14y ago
If the hackers got to the ColdFusion source code then it is quite possible they decrypted credit card numbers. The passphrase may not be stored in the database but presumably the decryption key is kept in memory by the app server while running. Otherwise it wouldn't be able to charge anything. And if the hackers could modify the ColdFusion source code, they can get to anything the app server can, including the decryption key. ColdFusion uses an interpreted language so injecting code is possible with just filesystem access. Linode says there's "no evidence" of decrypted card numbers but I think it's more a matter of no evidence.. yet. They also call salted hashes "useless" to hackers which betrays a rather unenlightened perspective on infosec.
- jaxbot 14y ago
  From what HTP said on IRC, the credit card numbers were decrypted within the application context, which would be consistent with what you said. But during the IRC conversation, HTP willingly gave out the last four digits of people's CC#s, but ignored any request for digits before that, which would be more consistent with what Linode is stating. It's all a very confusing world where nobody has a definite answer. Just 2 cents, don't act like I'm sharing a fact or anything
  - nenolod 14y ago
    Last 4 is all you need to wreak some serious havoc though. Service cancellations and whatnot...
  - abalone 14y ago
    There's no inconsistency. Linode said "Credit card numbers in our database are stored in encrypted format... Along with the encrypted credit card, the last four digits are stored in clear text..."
thaumaturgy 14y ago
They reset the passwords on Friday. Even assuming that they've been working for the last four days on discovering the extent of the intrusion, there's a pretty big gap between Friday's announcement: "This activity appears to have been a coordinated attempt to access the account of one of our customers. This customer is aware of this activity and we have determined its extent and impact. We have found no evidence that any Linode data of any other customer was accessed. In addition, we have found no evidence that payment information of any customer was accessed. ... Out of an abundance of caution, however, we have decided to implement a Linode Manager password reset," and their disclosure today. I find that I'm uncertain whether they would've released today's disclosure if it weren't for the IRC chat log getting passed around online. I don't get why they thought it would be prudent to reset everyone's manager passwords "out of an abundance of caution", but didn't bother to mention the possible leak of payment information. We don't have any information on the strength of the passphrase used (nor can they provide any, of course); we have no idea how feasible it might be to brute-force it. Even if the full credit card numbers are never broken, the last four digits are still available in the clear. That has been enough to give unauthorized persons access to accounts on other services in the past (see the Matt Honan thing from last year: http://www.wired.com/gadgetlab/2012/08/apple-amazon-mat-honan-hacking/all/ http://www.wired.com/gadgetlab/2012/08/apple-amazon-mat-hona...). Depending on what other information was stored in the database, customers may be at risk of having accounts compromised elsewhere using the billing information from Linode. Linode really should have included that information in Friday's announcement. Also, ColdFusion has had 103 different CVEs going back to 1999 (http://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=coldfusion http://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=coldfusion). Adobe doesn't really have a sterling record when it comes to this stuff. A nice example is the recent CVE-2013-0632 (http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2013-0632 http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2013-0632), a remote code execution exploit that was discovered already in use back on January 4th but didn't have a fix available until January 15th. Linode -- and their customers -- are a really big, juicy target for future CF vulnerabilities.
- corford 14y ago
  This is what Caker (founder of Linode IIRC?) has to say about the strength of the passphrase: "our private key is stored only in encrypted format. The passphrase is not guessable, sufficiently long and complex, not based on dictionary words, and not stored anywhere but in our heads." [1] [1] http://blog.linode.com/2013/04/16/security-incident-update/comment-page-1/#comment-419022 http://blog.linode.com/2013/04/16/security-incident-update/c...
codeulike 14y ago
Its not a "zero-day exploit" if there's been a patch out for it for N days, where N>0, right?
- acqq 14y ago
  Exactly. Zero-day were only if the attackers gained access before these N days. Did they?
mekoka 14y ago
The one thing that puzzles me above anything else about this whole thing is: "Linode uses ColdFusion". I mean, you have this Linux company that's pretty much an open source champion to us Linux guys and then... Adobe ColdFusion. Really? I know nothing about ColdFusion, don't get me wrong, I just find it an incredibly odd choice for a Linux company. That having been said, I think Linode just learnt a really unpleasant lesson and will become an even more secure provider for it. I believe that moving my sites away from them at this moment would be like selling stocks when they're about to go up.
- rjknight 14y ago
  Linode is a Linux company, but it's also a company founded in 2002-3. Back then, there weren't many great options for web programming and ColdFusion would have made a degree of sense (PHP 4.0 was released in 2000, and Ruby on Rails 1.0 didn't appear until 2004; Django was released in 2005). A modern, reasonably-secure web stack in those days really did mean "PHP with register_globals=Off". (Facebook and YouTube were both created later, and in PHP). It's a bit surprising that they haven't moved away from CF, but this is a good example of how hard it can be to transition away from legacy technologies. It's also not clear that security would have been improved by adopting, say, Rails, given that there has been no shortage in security issues there either.
  - hellweaver666 14y ago
    Also, if it works for them, it may not be financially viable to rewrite all of their code. For example, I work for a large company that uses Perl for all the backend stuff on our website. If we were to change to a more "modern" stack, we would have to stop and focus on that for a period of time instead of pushing the company forward and improving the customer experience etc. In the meantime, Perl works for us. It's not cool or sexy but it gets the job done. Probably a similar situation at Linode.
    - arc_of_descent 14y ago
      > In the meantime, Perl works for us. It's not cool or sexy > but it gets the job done. What is not cool about Perl? You don't need to transition to a new framework outside of Perl. You can upgrade your backend by investing your time in rewriting the Perl code to newer standards.
ig1 14y ago
I've canceled the card I used with Linode, the hacker may or may not have got access to the card number, but given the level of uncertainty I'd just rather cancel my card and get a new one reissued. I imagine my bank feels the same way. Remember that depending on where you're based / your agreement with your bank that you might lose any standard "protection" for any fraudulent transactions that occur after you learn your cards been compromised.
- qompiler 14y ago
  This should be the top comment and the reason why companies should disclose this information on time and more often. Just like you would block your card if you get robbed, you should block your card if someone steals your cc data.
- trebor 14y ago
  And for exactly that reason I've had my credit card reissued as well, even though no fraudulent charges have been made. The risk is just too great for me to feel safe with the old credit card number. Edit: Fixed spelling mistake.
- hvs 14y ago
  Agreed. As soon as I heard about this, I called my bank and canceled the card (no suspicious activity on it) and then regenerated 32 character passwords for all of my logins associated with Linode. An ounce of prevention and all that.
- kaiserama 14y ago
  I did the same thing. I had to read HN to even have a clue about this as their email correspondence/site gave no indication that this was even a possibility. I would assume a large majority of their clients don't read HN and would have no line of sight to do the same. It's unfortunate that this happened, but I'll be looking for another host in the near future.
mark_lee 14y ago
I don't know why so many guys here who obviously are not customers of linode care about this case so much, I've been a user of linode for 5 years, it's a company that you know you can trust, that's simply what I can tell from my experience.
- threeseed 14y ago
  I am a former customer who was burned the last time they were hacked and were less than truthful. Now it has happened again and nothing has changed. Clearly non existent security audits (cleartext passwords ?) and still lots of unanswered questions. It should be crystal clear by now that Linode is fundamentally untrustworthy.
  - VLM 14y ago
    Typical hackernews binary thinking. Trust is a spectrum not a binary. I'd trust them more than the 16 yr old waitress at the local restaurant who is legally judgment proof and as a juvenile frankly has no punishment related reason not to rip me off. Every time I pay with a CC at a restaurant and don't get my number stolen it pretty much restores my faith in humanity. Or I'd trust them more than the seedy psuedo-organized crime operator of my local gas station. Or the supermarket checker, again, juvenile and judgement proof. Would I date/marry/have kids with linode? Well probably not that much trust. Luckily rather than sharing precious bodily fluids with them, all I've been sharing is a 16 digit number which if lost can be replaced at no cost to myself other than some time. Where it gets fuzzy is something in the thousands of dollar range. Would I trust linode with the keys to my car? Well... it is insured, but the deductible is about a kilobuck because for me that's (more or less) pocket change...
    - pizzeys 14y ago
      > I'd trust them more than the 16 yr old waitress at the local restaurant who is legally judgment proof and as a juvenile frankly has no punishment related reason not to rip me off. Every time I pay with a CC at a restaurant and don't get my number stolen it pretty much restores my faith in humanity. Know what we did in the UK to solve this problem? Identified it as a weakness and switched to a system that removes this weakness. Linode aren't too good at that bit.
facorreia 14y ago
As a computer user, I'm thoroughly tired of Adobe's products constant stream of security vulnerabilities. Flash, Adobe Reader, ColdFusion. I avoid their products mostly for this reason. Flash, which I can't avoid completely, I have set up to only run when I say so, which I only do for a couple websites I trust. And why should a static document reader be able to run arbitrary code completely eludes me. Many companies should learn the lesson from Microsoft's huge shift years ago when they implemented the Trustworthy Computing Initiative. They have much larger targets (an operating system, a hypervisor, a database server, a full office suite, etc.) compared to a browser plugin or a document reader, and in my experience they're doing a far better job at keeping security at an acceptable level. Comparing IIS+ASP.NET security versus some frameworks such as Rails (another huge source of vulnerable web applications) is also enlightening.
- kijin 14y ago
  > why should a static document reader be able to run arbitrary code Because PDF was never meant to be a static format, and Adobe Reader was never meant to be a static document reader. However, since most PDFs are static anyway, I think it would be a very good idea for PDF reader softwares to treat every PDF as a static document by default, and only allow scripting after the user clicks Yes on a scary-looking warning.
  - facorreia 14y ago
    Correct. I wasn't clear, so let me clarify: I regard every PDF I read as a static document. That's how I see them and that's how I read them, with readers that treat them as static. I have no use for a PDF document that wants to acts as software and I wouldn't use a bug-nest of a piece of software to run it.
    - purephase 14y ago
      You're not the target market for those types of PDF's then. In large corporate environments, PDF's are roughly the equivalent of small web-apps that handle a lot of the data entry and sharing across the organization. I'm not a proponent of this approach, but it is what it is and it is the reason that Reader is such a bloated security-ridden nightmare.
spyder 14y ago
I think the only thing that could somewhat restore the trust in Linode would be a third-party security audit from a reputable company.
DigitalSea 14y ago
Hey Linode how about addressing the claims that the private and public keys were both stored in the same location on the compromised manager server. Is that true? And if so, why wasn't it revealed? Did they get access to any credit card information or was it truly just the last 4 digits used for lookup purposes? In my opinion these are the real questions us Linode customers want to know the answers to, sooner rather than later.
- FiloSottile 14y ago
  > Credit card numbers in our database are stored in encrypted format, using public and private key encryption. The private key is itself encrypted with passphrase encryption and the complex passphrase is not stored electronically.
- cmelbye 14y ago
  They did address that.
pc86 14y ago
I was going to post a TL;DR but the article is pretty short. Suffice it to say there are things in the database that should not be there. Encrypted credit card numbers along with the public and private keys? Plain text Lish passwords? I am curious why the fix for this specific vulnerability[0] wasn't implemented even though it's been out for a week now (obviously slightly less by the time they were affected). [0] http://www.adobe.com/support/security/bulletins/apsb13-10.html http://www.adobe.com/support/security/bulletins/apsb13-10.ht...
gr3yh47 14y ago
How can it be a 0-day vulnerability if it was addressed in a patch that came out more than a day ago? I understand patch cycles and I'm not hating on them for not having the latest update when it's been less than a week but still...
- singlow 14y ago
  It is a 0-day because the vulnerability was disclosed publicly before the patch was made available.
kbtr 14y ago
latest claims on #linode by HTP http://pastebin.com/7WXRDyAg http://pastebin.com/7WXRDyAg
tlrobinson 14y ago
"Despite the uselessness of these hashes, as you know we expired Linode Manager passwords on Friday." The uselessness of hashed passwords depends entirely on the hashing algorithm and the strength of the password.
moxie 14y ago
"There were occurrences of Lish passwords in clear text in our database." I think that this is what's most criminal about the whole event. It's probably safe to assume that "there were occurances of Lish passwords in clear text..." is a euphemism for "we stored all Lish passwords in cleartext." Simply invalidating these is probably not enough. It's very likely that people reused their root login passwords as their Lish passwords, meaning that Linodes running without firewalled sshd are potentially vulnerable. I hope Linode communicated who had "the occurrences" of compromised cleartext Lish passwords, so that those users could take appropriate action.
- sabat 14y ago
  The immediate, apparent, utter dishonesty of Linode in this incident had me closing my account yesterday—and watching my credit card's recent activity. It's bad enough that their security was awful; it's so much worse that they're trying to spin the incident instead of being 100% honest about it, and perhaps offering to compensate those who may have been wronged.
- peterwwillis 14y ago
  The lish passwords are the easiest to abuse immediately, but the larger impact is on all the user accounts with weak reused passwords. While all of the manager passwords were reset, each VM's root password was not. Once a hashed password is cracked their root password is vulnerable (if it was reused). And that's probably a lot more people than had exposed lish passwords. Everyone should go reset their virtual machine's root password if they reused them.
- yogo 14y ago
  Exactly! I'm pretty sure it was all Lish passwords. Yesterday the news was all focused on the credit card information but now it makes sense why Lish passwords were not working (when I was 100% sure I had it correct--Friday's email said nothing about lish iirc). Interestingly I even mentioned this as part of a ticket and they brushed it off as nothing... hmm.
- powertower 14y ago
  I've been a Linode customer since the start of 2007 and my biggest worry right now is this... > Credit card numbers in our database are stored in encrypted format, using public and private key encryption. The private key is itself encrypted with passphrase encryption and the complex passphrase is not stored electronically. By not providing further clarification that the private key was stored outside that compromised host in that update, you can read between the lines that they are trying hard not to address that particular issue. I really don't want to assume that the attacker can now brute force the private key's passphrase, and it might not take long if that "complex" passphrase was a word out of /usr/share/dict/words. Or worse, that the billing process was monitored via the compromised network, RAM, or a key logger - to get the unencrypted private key or it's passphrase.
jlongster 14y ago
I can't believe they don't address the claim that the public and private keys were stored in the same place. It makes me wonder if that's actually true, which knocks my trust down just a little bit more.
- deleted 14y ago
  [deleted]
hemancuso 14y ago
I wish PCI compliance required companies that take credit card info for any sort of recurring billing declare if they store your credit card information or if it is stored for them by their gateway/merchant account provider. There are a thousands of very small SAAS services out there that bill monthly which I assume would take advantage of gateway stored credit card info, then you find out that a comparatively larger more trustworthy player like Linode doesn't. Similarly it would be nice to also have them declare if they use something like Braintree's transparent redirect such that they never even see the card numbers.
darkarmani 14y ago
I'm assuming they mean these hashes are useless for the attacker, not for Linode users? "Linode Manager user passwords are not stored in our database, but their salted and cryptographically hashed representations are. Despite the uselessness of these hashes, as you know we expired Linode Manager passwords on Friday."
brandon272 14y ago
Linode exceeds technically in providing reliable VPS instances. Unfortunately, that's overshadowed by their lax approach toward security and their inability to be transparent when problems come up. This is not the first incidence where their lack of transparency has come under fire. This latest update dances around the issue and doesn't go far enough to explain what happened, whether or not your data is at risk, and what is being done to rectify it. If you are a Linode customer reading this, there is no indication as to whether or not you should cancel your credit card number. You, the customer, are left to decide on your own without all the information. Does Linode have an obligation to advise their customers to request new card numbers? It doesn't look as if they card information was stored in compliance with PCI-DSS and their systems were breached and private information was compromised. Is "the private key was also compromised but we don't think they can guess it, so let's just move on" good enough? Customers shouldn't have to read between the lines on security updates like this. I expect the companies I deal with to be up front and transparent, even if it makes them look bad. The reality here is that they messed up, not because they didn't patch a week-old exploit soon enough, but because of the poor security practices they had in place internally that allowed the hacker to get the information they did, and by not being up front with their customers to begin with. Any company can tell you that they take security seriously and that their customers best interests are their highest priority, but what ultimately matters are your actions. What are your security policies? What is your company culture toward security? How quickly do you announce problems and how transparent are you in those announcements?
- dangrossman 14y ago
  > It doesn't look as if they card information was stored in compliance with PCI-DSS Why do you say that? The requirements are basically to use strong encryption, only store those encrypted numbers on a system not accessible from the public internet, have auditable access controls and logs, and have a procedure for keeping systems up-to-date. The last 4 digits of the card are not subject to the same requirements as the full account number. All of that doesn't prevent the scenario that happened playing out -- someone gets into an internet-connected server, then can connect from there to the database server with the encrypted credit card numbers.
itsgettingcold 14y ago
This is deeply concerning for those of us who have business dependencies on linode. It certainly shakes confidence, at the very least.
andyhmltn 14y ago
This is really annoying for a number of reasons: 1) Well the obvious one is my card info is out there now 2) I really love Linode. Always respond to my tickets within 10 minutes, great prices and an awesome interface. It's such a shame that this has happened.

318 points by asb 14y ago

Good to see some discussion and acknowledgment from Linode, but their treatment of the credit card encryption is distressing. They don't address the claim that the private keys were on the servers' filesystems, and instead focus on them being password protected. If the password is brute-forced, then it's game over.

tsahyt 14y ago
My thoughts exactly. Then again, I'd expect them to use a rather strong password for sensitive data. Considering 10^12 tries per second (which is probably very far fetched) and a 64 character alphanumeric password, my credit card -- probably the whole galaxy -- will have long expired until the password can be expected to be cracked. At that point, it really depends on how strong their passphrase is. If it's a dictionary word, we're pretty much fucked.
derefr 14y ago
I think a lot of people are failing to understand how credit card processing works. The private key needs to live on some machine, if you actually want to have any CC requests processed at some point. If the private key itself isn't on the frontend/user-facing server, then instead, when the frontend node is hacked, the hacker will find a shared secret or signing key to auth the frontend node against the backend node that does process the CC requests. The hacker won't be able to take your CC number for themselves in this case, but they'll sure be able to put spurious/malicious charges on your account.
- JonnieCache 14y ago
  I suppose it could have been kept only in memory, rather than just chilling on the filsystem in a file called private_key which is basically what happened AFAIK. It's surely within the boundaries of sense to load the key manually each time the app boots. Thinking about it though, that's basically the role of the key passphrase.
  - derefr 14y ago
    Right; this is a common experience with SSL/X.509 signing keys--they're encrypted by default, so any server you configure with your ssl.pem file will prompt for a passphrase on startup, and only keep the decrypted key in memory. That's why this line, that most SSL providers give in their instructions, is so dangerous: openssl rsa -in server.key.org -out server.key It strips this (rather useful) encryption, giving any successful attacker to your frontend server your private key, in exchange for letting the node come up non-interactively.
  - gingerlime 14y ago
    This is what ssh-agent does, and I could only hope Linode was using something similar. I think the statement suggests they did. > The private key is itself encrypted with passphrase encryption and the passphrase is not stored electronically So even if the hacker bragged about obtaining both the public and private keys, the private key was protected and therefore unusable without the passphrase. At this point, we can all hope linode used a very complex passphrase... My gut feeling is that they did.
    - joeyh 14y ago
      Storing a passphrase (or unlocked private key) in memory is storing it electronically. Once the attacker has root, they can dump the memory and access this information. The attacker claims to have gigabytes of data, which could well include a memory image (or swap partition image, if the program that stored this in memory didn't lock the memory to prevent it being swapped out).
      - joeyh 14y ago
        Attacker claims to have done so in this irc log: http://pastebin.com/7WXRDyAg http://pastebin.com/7WXRDyAg > 07:52 < HTP> the CCrypter class of the linode application context was accessable from outside the wwwroot using undocumented ColdFusion methods. i was fully able to decrypt the ccs using the in-memory privkey that they supplied the password for. > 07:53 < HTP> coldfusion runs as a single process, and its memory can be accessed using the ColdFusion wrapper
- rainforest 14y ago
  From their description it appears encrypted CC numbers were in the database amongst the other customer information. Sure, the data has to live somewhere, but the apparent situation of CC, customer, private key all accessible to the frontend looks sub-optimal.
  - gingerlime 14y ago
    there's a difference between obtaining the private key and the private key file, which according to Linode was protected with a (hopefully strong) passphrase...
    - rainforest 14y ago
      It might have been encrypted, but surely it must be exposed to their system somewhere to enable them to make charges? Is it feasible, if that was the case, that the attackers could use that to exfiltrate decrypted CC info? I suppose this wild speculation isn't helpful or conducive and waiting on more information might be a better idea. I think the reputation damage has already been done, judging by the comments on the previous post though.
      - gingerlime 14y ago
        It depends on how they implemented it. Maybe there's an operator that logs on once a day and enters the passphrase, and then they use it to charge credit cards. So the private key is exposed only for a brief period. Another possibility is that the passphrase is entered when the machine is booted, and then the private key (not the passphrase) is loaded into memory. This is how e.g. ssh-agent[1] works. When this mechanism is used, you don't get access to the private key itself without direct memory access (which is possible, but not that easy to carry out, and would typically require root access and debug tools etc). However, the private key can be then used to carry out crypto operations. From the IRC chat logs, it looks like ryan got the private key file, but he didn't get the private key itself. At least he didn't seem to provide any solid proof to suggest he did. [1]https://en.wikipedia.org/wiki/Ssh-agent https://en.wikipedia.org/wiki/Ssh-agent
- InclinedPlane 14y ago
  CC data however does NOT have to live on any machine that anyone except your merchant bank owns. If you are writing against billing APIs in the 21st century then you only need to store a token which you use for recurring billing. Not everyone does this, but it's a reasonably good practice, especially if you have had intrusions in the past.
  - shadowfox 14y ago
    Out of curiosity, is this token generation sufficiently general or is it tied to a specific bank/credit card provider? (Basically is it easy to switch providers without having all your customers enter their cc information again?)
    - nenolod 14y ago
      Unfortunately it is tied to the merchant. :(
- deleted 14y ago
  [deleted]
reidrac 14y ago
I agree with you. Besides, how do they use the private key? Are they using a GPG agent o anything like that? Do they enter the password every time they need to use the private key? Having the private key in the web frontend sounds like a bad idea (as this incident has proved), they should use a backend system that is not accessible from the Internet. I don't know how they system works but the web frontend needs to encrypt CC numbers but I'm not sure it is required to decrypt them as that could be performed in a more restricted environment. Security incidents suck, but the good news is that they now have a chance to improve their security.
zcam 14y ago
Even then, attackers with half a brain (and malicious intent) would also have backdoored the payment page to harvest "fresh" credit cards numbers during the period they went unnoticed (possibly days/weeks). It's another good reason to leave the credit card handling to a reliable 3rd party imo.
entropy_ 14y ago
That doesn't make any sense. It's exactly like saying: "Oh sure, the credit card numbers were encrypted but if the encryption key is brute-forced then it's game over". The CC numbers were encrypted using a public key. The corresponding private key was itself encrypted using "[a] complex passphrase [that] is not stored electronically". They could be lying, but if they are not then this is perfectly valid security. Using this scheme, you can store new CC numbers any time you want using the public key. You'd have a regular billing cycle(say a month) with billings happening in a short time window(All my linode accounts get billed within minutes of each other so I think this is the case for linode). When you want to bill your customers you'd input the passphrase which wouldn't be stored anywhere, would be used to temporarily decrypt the CC numbers and bill everyone. Unless you are attacked in that window(or an attacker gains access before that window and can remain unnoticed until it happens, and even then they'd need root access to read the memory and not just access to the filesystem) then the CC numbers are pretty safe against any attack that leaks files/databases/etc...
- thaumaturgy 14y ago
  > When you want to bill your customers you'd input the passphrase which wouldn't be stored anywhere, would be used to temporarily decrypt the CC numbers and bill everyone. I've had account changes billed to my card within minutes, even (and especially) in the middle of the night (when I do most of that stuff). I'm having a little bit of trouble imagining that someone's sitting in front of a terminal and typing a passphrase into a prompt every three to five minutes, all day long.
  - entropy_ 14y ago
    You make a good point
  - corford 14y ago
    Interesting. Whenever I make changes that affect how much I'll be billed, the amounts get added or removed and take effect when the card next gets charged (which happens once a month). At least that's how I remember it... going to dig through some old statements now to verify this as its been a while since I last changed anything that would alter what they charge me.

sturadnidge 14y ago

I really wish they would officially comment on the 'cover up' aspect. Security breaches happen, and are forgivable. But attempting to broker a 'silencing' deal with the intruders and hoping your customers will never be the wiser is not. All this update does is restore my faith in their ability to store my information correctly. It does nothing to reassure me that they won't try to cover anything up again.

kbar13 14y ago
As these situations, especially on the internet, come out as "he said, she said", I think it's probably more important to keep focused on what directly affects you.
- sturadnidge 14y ago
  Sure, but in this case it wasn't Linode who initially came forward - it took a public announcement from the intruders for Linode to notify us.
  - rainforest 14y ago
    Devils advocate: preparing a press release from a company after plugging holes and auditing that makes the appropriate admissions and apologies probably takes more time than writing up a successful hit after attacking a page.
    - makomk 14y ago
      Linode didn't just take their time to announce this, just a few hours ago they were apparently telling customers definitively that their credit card information hadn't been accessed.
      - kid0m4n 14y ago
        And that is "still" the case
14y ago

tsahyt 14y ago

The hacker has claimed the public and private key were both stored on the same machine and both accessible. The only protection left is the passphrase then. Various people have reported fraudulent activity on their CCs. Personally, I think Linode didn't mess this up as badly as many other companies. The statement took a while to appear but it's there. As a Linode customer myself I can neither report anything unusual on my box, nor activity on my CC. Thus far I'm somehow inclined to believe Linode when they say no CC information has been compromised. Either way, they are offering a great service and I'll stick with them. Worst case, I'll have to get a new CC.

kbar13 14y ago
> Various people have reported fraudulent activity on their CCs Most likely not even related
- prawn 14y ago
  I've seen two mentioned in each of two recent Linode threads, potentially the same people. Have there been many more?
- VLM 14y ago
  Well, lets think about it a bit. It turns out the numbers are ridiculously hard to find although I'm a pretty good searcher/infovore. Better info would be appreciated. Anyway the best I could find was two claims: There were 11 million american victims of identity theft in 2011 (outta 400 million) for a ratio of about 1 in 40 of the general population gets p0wned per year. Obviously not all ID theft is CC theft but I'd guess a lot of it is. Based on friends and family experience getting a CC number stolen every four decades of use passes the smell test so I will go with that. A quote from prweb.com found by google: "Started in 2003, Linode has grown to over 45,000 customers". I wonder what year that quote is from. Still "forty five thousand" is probably not totally ridiculous. It makes sense looking at their hostname scheme, ip space allocations as per whois, and some educated guesses. Lots of people including myself like linode. So assuming the average linode customer is the same as the average joe6pack then 45000/40/365 = about 3 linode customers should get p0wned per day regardless of any linode problem. To make my inner EE happy I'll call the "noise level" about 3/day and evaluate the SNR based on that noise level to see if there's a signal of p0wnership. Obviously the two reports over the course of a couple days is not 100% of all linode customers who got p0wned. But it does show that two reports doesn't really prove anything. Now, as a made up example, 900 reports over 3 days would be a HUGE indicator "something" happened. But at a predicted noise level of about ten or so over three days, two reported is down there in or below the noise. So, as a long term linode customer I'm not freaking out (yet). My CC does email me every time a charge is made, and nothing weird has been seen. All quiet on the western front.

jpallen 14y ago

I know Linode has been getting hammered for not being trustable or that they don't respond well when contacted, but in my experience they have been extremely responsive to any issues I have raised. I don't just mean technical issues either, we had what looked like a security breach on our account and it kicked of a small frenzy in their support staff and supervisory team to track down the problem. They figured out what had happened after a thorough investigation, gave us a full post mortem and promised they were reviewing procedures to make sure it couldn't happen again.

ciaron 14y ago
I can second this - Linode has always responded quickly and effectively in my experience.
- andyhmltn 14y ago
  I third this. I've never had a problem. Most of my tickets have a response within 10 minutes and they always seem to have been open when anything has gone wrong on their end. Is anyone not thinking that the reason they didn't come out in public talking about this until now is because law enforcement was involved? Notifying the public could ruin the investigation.
  - threeseed 14y ago
    No. Because Linode has been hacked before and exactly the same situation is playing out. Major compromise and selective release of information. I really do wonder at what point people are going to realise that they aren't dealing with a company who cares about their interests.
sveiss 14y ago
I've always had an excellent experience when dealing with Linode one-on-one, via email or support ticket, but their one-to-many communication sometimes seems to be lacking. I'm glad they posted a more detailed update regarding this issue publicly.
14y ago

AlexMuir 14y ago

Hmm. I'm sort of re-assured, but what's happened to the claim that this was targeting a specific customer?

keidian 14y ago
It seems HTP went after nmap.org machines, so I'm guessing that woudl be the directed customer they picked?
- scott_karana 14y ago
  Do you have a chatlog or citation? I'm not arguing, I'd love to read it! :)
  - lcampbell 14y ago
    Here's the writeup from seclists.org: http://seclists.org/nmap-dev/2013/q2/3 http://seclists.org/nmap-dev/2013/q2/3 It looks like seclists.org and nmap.org are hosted on different IP addresses, but both route to linode, and both have the same WHOIS information, so I think it's safe to say the writeup applies to both.
    - scott_karana 14y ago
      Thanks! By the sounds of their writeup, the attacks just visited the "highest profile" sites as per the Quora link to obfuscate their real attack.

ultimoo 14y ago

The update has quite a frank and an apologetic tone to it. Especially the concluding paragraph gives it a very empathetic touch. It must be truly tough for the ops folk at Linode to have suffered an attack due to a third party 0-Day exploit. It could happen to any of us really. On a side note, I am not sure of the "some occurrences of plaintext Lish passwords". Seems like quite a goofup on Linode's part.

troels 14y ago
It's not really a zero-day, if a patch has been out for a week though.
- vrta 14y ago
  It was 0-day when they got hit. Sad thing is that they do not tell how much time has passed since the hack.
  - troels 14y ago
    Yeah, maybe you're right. The text is a bit unclear about that.
    - vrta 14y ago
      Yeah, they didn't tell us when they got hacked, but seclist did (3/31/2013): http://seclists.org/nmap-dev/2013/q2/3 http://seclists.org/nmap-dev/2013/q2/3
      - mjs 14y ago
        3/31/2013 is the backup they're reverting to, not when they got hacked though, right? (The email refers to "downtime over the last week", so I guess the hack happened around 6th April or later.
  - codeulike 14y ago
    I'm not sure, their wording around it is a bit convoluted "... previously unknown zero-day vulnerability in Adobe’s ColdFusion application server. The vulnerabilities have only recently been addressed in Adobe’s APSB13-10 hotfix (CVE-2013-1387 and CVE-2013-1388) which was released less than a week ago." To me that reads like 'it was a former zero-day exploit", as all exploits are.

rainforest 14y ago

Should CC info even be stored in the customer database? I would have thought that information should be write only. Does PCI allow that to happen with only PKI in place?

ultimoo 14y ago
I agree to your position of not storing the CC in the consumer database. However, it cannot be in a write-only state since usually consumers have to billed periodically without having to ask for their number.
- rainforest 14y ago
  Ah, my description wasn't the best, apologies. What I meant was the payment backend, a separate server, could only service write requests from the frontend. The backend itself would possess decryption keys to make charges, but the frontend could only ever ask it to bill a customer or change the information via a write-only API. Having said that, if the API were provided via ColdFusion it wouldn't have helped anyway.
  - singlow 14y ago
    That is in fact what was done in an application I worked on. We used ColdFusion for both the Front-End and the "Commerce Server" and the commerce server only took credit card information from the front-end and could not give it back. It was behind its own hardware firewall and any exploit in ColdFusion would have been hidden from the front end unless it could be exploited through the API methods we exposed through the gateway. To gain access to any other url endpoints you would need to access the server from within the firewall by connecting to a VPN gateway on a separate physical network that was only connected by a dedicated T1 to our office. We did this before there was PCI, because it made sense. The whole system has now been rewritten in dotNET. It would be interesting to see if I can find that old CF app. It ran solid for almost ten years on the same 3 NT4 servers handling tens of thousands of users per day. And it could have probably run on one server except that we isolated the commerce and MSSQL functions from the front-end app.
- 14y ago

prawn 14y ago

To be honest, I'd rather have my credit card number in the wild than be worrying about my sites/servers being compromised. Obviously neither is ideal, however.

egeozcan 14y ago

It's about time we start seeing some OAuth bank account logins for payments.

citricsquid 14y ago

Must have been a coincidence my card was stolen then. At least I don't need to move away from Linode now, hurray.

espennilsen 14y ago

It seems seclists was affected http://seclists.org/nmap-dev/2013/q2/3 http://seclists.org/nmap-dev/2013/q2/3

scottjad 14y ago

What isn't in this post: - Any mention of the claim that they had made a deal with the hackers. No attempt to deny the claim. - Any mention of when the systems were compromised or when they discovered that they were compromised. The post starts with "Yesterday," mentions working "around the clock" and mentions Friday, giving the impression that this is all very recent, when according to the hacker Linode has known for two weeks.

ScottWhigham 14y ago
If it was your company and you had decided to make a deal with the hackers, would you publicize or publicly address that fact? My thinking is that doing so would incentivize others to do similar acts because you are an easy mark.
cjh_ 14y ago
I find it somewhat worrying that they haven't directly address the points of the hacker's claim. I am particularly interested to find out whether linode did enter into a deal with the hackers.

nfm 14y ago

"Despite the uselessness of these hashes"... Hrm.

neya 14y ago

Ahh, they still don't mention if even their ex-customers are affected by this. Now, I'm really paranoid! Let's just hope for the best...but when I read about 'storing passwords in plain text' I did get a little turned off. I know the Linode team is very technically strong, infact, they've aided me so many times. It hurts to see them make such a silly mistake :(

zhoutong 14y ago

EDIT 2: It seems that I have confused the "encryption" with normal hashing. So this comment is invalid if they are doing encryption properly. I wonder how hard it is to brute-force the credit card numbers. Given the last 4 digits, the Luhn algorithm and the starting digit of common cards (e.g. 4 for Visa), there are essentially only 10 digits left to crack. That's just 10 billion combinations. As part of the source code is exposed, most likely the encryption algorithm/sequence is already known. If the passphrase is shared among all credit cards and the encryption algorithm is a fast one, there's a fair chance that eventually these credit cards are going to be exposed all at once. Or am I missing anything? EDIT: Even if the encryption algorithm is as slow as 10,000 tries per second, exhaustively attempting every single credit card number possible is going to take 277 CPU-hours, or about a day in three quad-core machines.

dchest 14y ago
If they used encryption properly (random or unique IV depending on cipher mode of operation), you can't bruteforce credit card numbers (encrypted content), you can only try to bruteforce the key used for encryption. See http://en.wikipedia.org/wiki/Semantic_security http://en.wikipedia.org/wiki/Semantic_security and http://en.wikipedia.org/wiki/Ciphertext_indistinguishability http://en.wikipedia.org/wiki/Ciphertext_indistinguishability
- zhoutong 14y ago
  Right. I guess I had confused it with hashing. If the credit cards were hashed then it would be relatively easy to crack. Edited my comment. Thanks!

alinspired 14y ago

very reassuring update for someone who didn't read the IRC logs with hacker and i think it will do the trick for most of linode users

antihero 14y ago

Who on earth uses ColdFusion any more? Surely when they re-built it a while back they could have used something more secure like Django?

mekoka 14y ago
I agree with questioning their choice of ColdFusion, but I have some reserves about your proposed alternative to use Django. I don't know how secure Django is, but I know it's big. And my perception is generally that anything that big is bound to have a few loose ends, if looked under the magnifier.

chris_wot 14y ago

"Along with the encrypted credit card, the last four digits are stored in clear text to assist in lookups and for display on things like your Account tab and payment receipt emails." I really don't see the need for that at all. What sort of credit card "lookup" are they doing exactly?

lucaspiller 14y ago
I guess it's mainly to display stuff. When you have multiple cards on Amazon you can choose which one to use, however without seeing the last four digits its pretty hard to tell which is which.
mekoka 14y ago
It's not for them, it's for users. It helps to know which of your CCs is going to be charged.
UnoriginalGuy 14y ago
It is fairly standard. Go look at your Amazon account, you'll see the last four digits.

abalone 14y ago

If the hackers got to the ColdFusion source code then it is quite possible they decrypted credit card numbers. The passphrase may not be stored in the database but presumably the decryption key is kept in memory by the app server while running. Otherwise it wouldn't be able to charge anything. And if the hackers could modify the ColdFusion source code, they can get to anything the app server can, including the decryption key. ColdFusion uses an interpreted language so injecting code is possible with just filesystem access. Linode says there's "no evidence" of decrypted card numbers but I think it's more a matter of no evidence.. yet. They also call salted hashes "useless" to hackers which betrays a rather unenlightened perspective on infosec.

jaxbot 14y ago
From what HTP said on IRC, the credit card numbers were decrypted within the application context, which would be consistent with what you said. But during the IRC conversation, HTP willingly gave out the last four digits of people's CC#s, but ignored any request for digits before that, which would be more consistent with what Linode is stating. It's all a very confusing world where nobody has a definite answer. Just 2 cents, don't act like I'm sharing a fact or anything
- nenolod 14y ago
  Last 4 is all you need to wreak some serious havoc though. Service cancellations and whatnot...
- abalone 14y ago
  There's no inconsistency. Linode said "Credit card numbers in our database are stored in encrypted format... Along with the encrypted credit card, the last four digits are stored in clear text..."

thaumaturgy 14y ago

They reset the passwords on Friday. Even assuming that they've been working for the last four days on discovering the extent of the intrusion, there's a pretty big gap between Friday's announcement: "This activity appears to have been a coordinated attempt to access the account of one of our customers. This customer is aware of this activity and we have determined its extent and impact. We have found no evidence that any Linode data of any other customer was accessed. In addition, we have found no evidence that payment information of any customer was accessed. ... Out of an abundance of caution, however, we have decided to implement a Linode Manager password reset," and their disclosure today. I find that I'm uncertain whether they would've released today's disclosure if it weren't for the IRC chat log getting passed around online. I don't get why they thought it would be prudent to reset everyone's manager passwords "out of an abundance of caution", but didn't bother to mention the possible leak of payment information. We don't have any information on the strength of the passphrase used (nor can they provide any, of course); we have no idea how feasible it might be to brute-force it. Even if the full credit card numbers are never broken, the last four digits are still available in the clear. That has been enough to give unauthorized persons access to accounts on other services in the past (see the Matt Honan thing from last year: http://www.wired.com/gadgetlab/2012/08/apple-amazon-mat-honan-hacking/all/ http://www.wired.com/gadgetlab/2012/08/apple-amazon-mat-hona...). Depending on what other information was stored in the database, customers may be at risk of having accounts compromised elsewhere using the billing information from Linode. Linode really should have included that information in Friday's announcement. Also, ColdFusion has had 103 different CVEs going back to 1999 (http://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=coldfusion http://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=coldfusion). Adobe doesn't really have a sterling record when it comes to this stuff. A nice example is the recent CVE-2013-0632 (http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2013-0632 http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2013-0632), a remote code execution exploit that was discovered already in use back on January 4th but didn't have a fix available until January 15th. Linode -- and their customers -- are a really big, juicy target for future CF vulnerabilities.

corford 14y ago
This is what Caker (founder of Linode IIRC?) has to say about the strength of the passphrase: "our private key is stored only in encrypted format. The passphrase is not guessable, sufficiently long and complex, not based on dictionary words, and not stored anywhere but in our heads." [1] [1] http://blog.linode.com/2013/04/16/security-incident-update/comment-page-1/#comment-419022 http://blog.linode.com/2013/04/16/security-incident-update/c...

codeulike 14y ago

Its not a "zero-day exploit" if there's been a patch out for it for N days, where N>0, right?

acqq 14y ago
Exactly. Zero-day were only if the attackers gained access before these N days. Did they?

mekoka 14y ago

The one thing that puzzles me above anything else about this whole thing is: "Linode uses ColdFusion". I mean, you have this Linux company that's pretty much an open source champion to us Linux guys and then... Adobe ColdFusion. Really? I know nothing about ColdFusion, don't get me wrong, I just find it an incredibly odd choice for a Linux company. That having been said, I think Linode just learnt a really unpleasant lesson and will become an even more secure provider for it. I believe that moving my sites away from them at this moment would be like selling stocks when they're about to go up.

rjknight 14y ago
Linode is a Linux company, but it's also a company founded in 2002-3. Back then, there weren't many great options for web programming and ColdFusion would have made a degree of sense (PHP 4.0 was released in 2000, and Ruby on Rails 1.0 didn't appear until 2004; Django was released in 2005). A modern, reasonably-secure web stack in those days really did mean "PHP with register_globals=Off". (Facebook and YouTube were both created later, and in PHP). It's a bit surprising that they haven't moved away from CF, but this is a good example of how hard it can be to transition away from legacy technologies. It's also not clear that security would have been improved by adopting, say, Rails, given that there has been no shortage in security issues there either.
- hellweaver666 14y ago
  Also, if it works for them, it may not be financially viable to rewrite all of their code. For example, I work for a large company that uses Perl for all the backend stuff on our website. If we were to change to a more "modern" stack, we would have to stop and focus on that for a period of time instead of pushing the company forward and improving the customer experience etc. In the meantime, Perl works for us. It's not cool or sexy but it gets the job done. Probably a similar situation at Linode.
  - arc_of_descent 14y ago
    > In the meantime, Perl works for us. It's not cool or sexy > but it gets the job done. What is not cool about Perl? You don't need to transition to a new framework outside of Perl. You can upgrade your backend by investing your time in rewriting the Perl code to newer standards.

ig1 14y ago

I've canceled the card I used with Linode, the hacker may or may not have got access to the card number, but given the level of uncertainty I'd just rather cancel my card and get a new one reissued. I imagine my bank feels the same way. Remember that depending on where you're based / your agreement with your bank that you might lose any standard "protection" for any fraudulent transactions that occur after you learn your cards been compromised.

qompiler 14y ago
This should be the top comment and the reason why companies should disclose this information on time and more often. Just like you would block your card if you get robbed, you should block your card if someone steals your cc data.
trebor 14y ago
And for exactly that reason I've had my credit card reissued as well, even though no fraudulent charges have been made. The risk is just too great for me to feel safe with the old credit card number. Edit: Fixed spelling mistake.
hvs 14y ago
Agreed. As soon as I heard about this, I called my bank and canceled the card (no suspicious activity on it) and then regenerated 32 character passwords for all of my logins associated with Linode. An ounce of prevention and all that.
kaiserama 14y ago
I did the same thing. I had to read HN to even have a clue about this as their email correspondence/site gave no indication that this was even a possibility. I would assume a large majority of their clients don't read HN and would have no line of sight to do the same. It's unfortunate that this happened, but I'll be looking for another host in the near future.

mark_lee 14y ago

I don't know why so many guys here who obviously are not customers of linode care about this case so much, I've been a user of linode for 5 years, it's a company that you know you can trust, that's simply what I can tell from my experience.

threeseed 14y ago
I am a former customer who was burned the last time they were hacked and were less than truthful. Now it has happened again and nothing has changed. Clearly non existent security audits (cleartext passwords ?) and still lots of unanswered questions. It should be crystal clear by now that Linode is fundamentally untrustworthy.
- VLM 14y ago
  Typical hackernews binary thinking. Trust is a spectrum not a binary. I'd trust them more than the 16 yr old waitress at the local restaurant who is legally judgment proof and as a juvenile frankly has no punishment related reason not to rip me off. Every time I pay with a CC at a restaurant and don't get my number stolen it pretty much restores my faith in humanity. Or I'd trust them more than the seedy psuedo-organized crime operator of my local gas station. Or the supermarket checker, again, juvenile and judgement proof. Would I date/marry/have kids with linode? Well probably not that much trust. Luckily rather than sharing precious bodily fluids with them, all I've been sharing is a 16 digit number which if lost can be replaced at no cost to myself other than some time. Where it gets fuzzy is something in the thousands of dollar range. Would I trust linode with the keys to my car? Well... it is insured, but the deductible is about a kilobuck because for me that's (more or less) pocket change...
  - pizzeys 14y ago
    > I'd trust them more than the 16 yr old waitress at the local restaurant who is legally judgment proof and as a juvenile frankly has no punishment related reason not to rip me off. Every time I pay with a CC at a restaurant and don't get my number stolen it pretty much restores my faith in humanity. Know what we did in the UK to solve this problem? Identified it as a weakness and switched to a system that removes this weakness. Linode aren't too good at that bit.

facorreia 14y ago

As a computer user, I'm thoroughly tired of Adobe's products constant stream of security vulnerabilities. Flash, Adobe Reader, ColdFusion. I avoid their products mostly for this reason. Flash, which I can't avoid completely, I have set up to only run when I say so, which I only do for a couple websites I trust. And why should a static document reader be able to run arbitrary code completely eludes me. Many companies should learn the lesson from Microsoft's huge shift years ago when they implemented the Trustworthy Computing Initiative. They have much larger targets (an operating system, a hypervisor, a database server, a full office suite, etc.) compared to a browser plugin or a document reader, and in my experience they're doing a far better job at keeping security at an acceptable level. Comparing IIS+ASP.NET security versus some frameworks such as Rails (another huge source of vulnerable web applications) is also enlightening.

kijin 14y ago
> why should a static document reader be able to run arbitrary code Because PDF was never meant to be a static format, and Adobe Reader was never meant to be a static document reader. However, since most PDFs are static anyway, I think it would be a very good idea for PDF reader softwares to treat every PDF as a static document by default, and only allow scripting after the user clicks Yes on a scary-looking warning.
- facorreia 14y ago
  Correct. I wasn't clear, so let me clarify: I regard every PDF I read as a static document. That's how I see them and that's how I read them, with readers that treat them as static. I have no use for a PDF document that wants to acts as software and I wouldn't use a bug-nest of a piece of software to run it.
  - purephase 14y ago
    You're not the target market for those types of PDF's then. In large corporate environments, PDF's are roughly the equivalent of small web-apps that handle a lot of the data entry and sharing across the organization. I'm not a proponent of this approach, but it is what it is and it is the reason that Reader is such a bloated security-ridden nightmare.

spyder 14y ago

I think the only thing that could somewhat restore the trust in Linode would be a third-party security audit from a reputable company.

DigitalSea 14y ago

Hey Linode how about addressing the claims that the private and public keys were both stored in the same location on the compromised manager server. Is that true? And if so, why wasn't it revealed? Did they get access to any credit card information or was it truly just the last 4 digits used for lookup purposes? In my opinion these are the real questions us Linode customers want to know the answers to, sooner rather than later.

FiloSottile 14y ago
> Credit card numbers in our database are stored in encrypted format, using public and private key encryption. The private key is itself encrypted with passphrase encryption and the complex passphrase is not stored electronically.
cmelbye 14y ago
They did address that.

pc86 14y ago

I was going to post a TL;DR but the article is pretty short. Suffice it to say there are things in the database that should not be there. Encrypted credit card numbers along with the public and private keys? Plain text Lish passwords? I am curious why the fix for this specific vulnerability[0] wasn't implemented even though it's been out for a week now (obviously slightly less by the time they were affected). [0] http://www.adobe.com/support/security/bulletins/apsb13-10.html http://www.adobe.com/support/security/bulletins/apsb13-10.ht...

gr3yh47 14y ago

How can it be a 0-day vulnerability if it was addressed in a patch that came out more than a day ago? I understand patch cycles and I'm not hating on them for not having the latest update when it's been less than a week but still...

singlow 14y ago
It is a 0-day because the vulnerability was disclosed publicly before the patch was made available.

kbtr 14y ago

latest claims on #linode by HTP http://pastebin.com/7WXRDyAg http://pastebin.com/7WXRDyAg

tlrobinson 14y ago

"Despite the uselessness of these hashes, as you know we expired Linode Manager passwords on Friday." The uselessness of hashed passwords depends entirely on the hashing algorithm and the strength of the password.

moxie 14y ago

"There were occurrences of Lish passwords in clear text in our database." I think that this is what's most criminal about the whole event. It's probably safe to assume that "there were occurances of Lish passwords in clear text..." is a euphemism for "we stored all Lish passwords in cleartext." Simply invalidating these is probably not enough. It's very likely that people reused their root login passwords as their Lish passwords, meaning that Linodes running without firewalled sshd are potentially vulnerable. I hope Linode communicated who had "the occurrences" of compromised cleartext Lish passwords, so that those users could take appropriate action.

sabat 14y ago
The immediate, apparent, utter dishonesty of Linode in this incident had me closing my account yesterday—and watching my credit card's recent activity. It's bad enough that their security was awful; it's so much worse that they're trying to spin the incident instead of being 100% honest about it, and perhaps offering to compensate those who may have been wronged.
peterwwillis 14y ago
The lish passwords are the easiest to abuse immediately, but the larger impact is on all the user accounts with weak reused passwords. While all of the manager passwords were reset, each VM's root password was not. Once a hashed password is cracked their root password is vulnerable (if it was reused). And that's probably a lot more people than had exposed lish passwords. Everyone should go reset their virtual machine's root password if they reused them.
yogo 14y ago
Exactly! I'm pretty sure it was all Lish passwords. Yesterday the news was all focused on the credit card information but now it makes sense why Lish passwords were not working (when I was 100% sure I had it correct--Friday's email said nothing about lish iirc). Interestingly I even mentioned this as part of a ticket and they brushed it off as nothing... hmm.
powertower 14y ago
I've been a Linode customer since the start of 2007 and my biggest worry right now is this... > Credit card numbers in our database are stored in encrypted format, using public and private key encryption. The private key is itself encrypted with passphrase encryption and the complex passphrase is not stored electronically. By not providing further clarification that the private key was stored outside that compromised host in that update, you can read between the lines that they are trying hard not to address that particular issue. I really don't want to assume that the attacker can now brute force the private key's passphrase, and it might not take long if that "complex" passphrase was a word out of /usr/share/dict/words. Or worse, that the billing process was monitored via the compromised network, RAM, or a key logger - to get the unencrypted private key or it's passphrase.

jlongster 14y ago

I can't believe they don't address the claim that the public and private keys were stored in the same place. It makes me wonder if that's actually true, which knocks my trust down just a little bit more.

deleted 14y ago
[deleted]

hemancuso 14y ago

I wish PCI compliance required companies that take credit card info for any sort of recurring billing declare if they store your credit card information or if it is stored for them by their gateway/merchant account provider. There are a thousands of very small SAAS services out there that bill monthly which I assume would take advantage of gateway stored credit card info, then you find out that a comparatively larger more trustworthy player like Linode doesn't. Similarly it would be nice to also have them declare if they use something like Braintree's transparent redirect such that they never even see the card numbers.

darkarmani 14y ago

I'm assuming they mean these hashes are useless for the attacker, not for Linode users? "Linode Manager user passwords are not stored in our database, but their salted and cryptographically hashed representations are. Despite the uselessness of these hashes, as you know we expired Linode Manager passwords on Friday."

brandon272 14y ago

Linode exceeds technically in providing reliable VPS instances. Unfortunately, that's overshadowed by their lax approach toward security and their inability to be transparent when problems come up. This is not the first incidence where their lack of transparency has come under fire. This latest update dances around the issue and doesn't go far enough to explain what happened, whether or not your data is at risk, and what is being done to rectify it. If you are a Linode customer reading this, there is no indication as to whether or not you should cancel your credit card number. You, the customer, are left to decide on your own without all the information. Does Linode have an obligation to advise their customers to request new card numbers? It doesn't look as if they card information was stored in compliance with PCI-DSS and their systems were breached and private information was compromised. Is "the private key was also compromised but we don't think they can guess it, so let's just move on" good enough? Customers shouldn't have to read between the lines on security updates like this. I expect the companies I deal with to be up front and transparent, even if it makes them look bad. The reality here is that they messed up, not because they didn't patch a week-old exploit soon enough, but because of the poor security practices they had in place internally that allowed the hacker to get the information they did, and by not being up front with their customers to begin with. Any company can tell you that they take security seriously and that their customers best interests are their highest priority, but what ultimately matters are your actions. What are your security policies? What is your company culture toward security? How quickly do you announce problems and how transparent are you in those announcements?

dangrossman 14y ago
> It doesn't look as if they card information was stored in compliance with PCI-DSS Why do you say that? The requirements are basically to use strong encryption, only store those encrypted numbers on a system not accessible from the public internet, have auditable access controls and logs, and have a procedure for keeping systems up-to-date. The last 4 digits of the card are not subject to the same requirements as the full account number. All of that doesn't prevent the scenario that happened playing out -- someone gets into an internet-connected server, then can connect from there to the database server with the encrypted credit card numbers.

itsgettingcold 14y ago

This is deeply concerning for those of us who have business dependencies on linode. It certainly shakes confidence, at the very least.

andyhmltn 14y ago

This is really annoying for a number of reasons: 1) Well the obvious one is my card info is out there now 2) I really love Linode. Always respond to my tickets within 10 minutes, great prices and an awesome interface. It's such a shame that this has happened.