6 ms·

Good to see some discussion and acknowledgment from Linode, but their treatment of the credit card encryption is distressing. They don't address the claim that

by cheald 14y ago

Good to see some discussion and acknowledgment from Linode, but their treatment of the credit card encryption is distressing. They don't address the claim that the private keys were on the servers' filesystems, and instead focus on them being password protected. If the password is brute-forced, then it's game over.

tsahyt 14y ago
My thoughts exactly. Then again, I'd expect them to use a rather strong password for sensitive data. Considering 10^12 tries per second (which is probably very far fetched) and a 64 character alphanumeric password, my credit card -- probably the whole galaxy -- will have long expired until the password can be expected to be cracked. At that point, it really depends on how strong their passphrase is. If it's a dictionary word, we're pretty much fucked.
derefr 14y ago
I think a lot of people are failing to understand how credit card processing works. The private key needs to live on some machine, if you actually want to have any CC requests processed at some point. If the private key itself isn't on the frontend/user-facing server, then instead, when the frontend node is hacked, the hacker will find a shared secret or signing key to auth the frontend node against the backend node that does process the CC requests. The hacker won't be able to take your CC number for themselves in this case, but they'll sure be able to put spurious/malicious charges on your account.
- JonnieCache 14y ago
  I suppose it could have been kept only in memory, rather than just chilling on the filsystem in a file called private_key which is basically what happened AFAIK. It's surely within the boundaries of sense to load the key manually each time the app boots. Thinking about it though, that's basically the role of the key passphrase.
  - derefr 14y ago
    Right; this is a common experience with SSL/X.509 signing keys--they're encrypted by default, so any server you configure with your ssl.pem file will prompt for a passphrase on startup, and only keep the decrypted key in memory. That's why this line, that most SSL providers give in their instructions, is so dangerous: openssl rsa -in server.key.org -out server.key It strips this (rather useful) encryption, giving any successful attacker to your frontend server your private key, in exchange for letting the node come up non-interactively.
  - gingerlime 14y ago
    This is what ssh-agent does, and I could only hope Linode was using something similar. I think the statement suggests they did. > The private key is itself encrypted with passphrase encryption and the passphrase is not stored electronically So even if the hacker bragged about obtaining both the public and private keys, the private key was protected and therefore unusable without the passphrase. At this point, we can all hope linode used a very complex passphrase... My gut feeling is that they did.
    - joeyh 14y ago
      Storing a passphrase (or unlocked private key) in memory is storing it electronically. Once the attacker has root, they can dump the memory and access this information. The attacker claims to have gigabytes of data, which could well include a memory image (or swap partition image, if the program that stored this in memory didn't lock the memory to prevent it being swapped out).
      - joeyh 14y ago
        Attacker claims to have done so in this irc log: http://pastebin.com/7WXRDyAg http://pastebin.com/7WXRDyAg > 07:52 < HTP> the CCrypter class of the linode application context was accessable from outside the wwwroot using undocumented ColdFusion methods. i was fully able to decrypt the ccs using the in-memory privkey that they supplied the password for. > 07:53 < HTP> coldfusion runs as a single process, and its memory can be accessed using the ColdFusion wrapper
- rainforest 14y ago
  From their description it appears encrypted CC numbers were in the database amongst the other customer information. Sure, the data has to live somewhere, but the apparent situation of CC, customer, private key all accessible to the frontend looks sub-optimal.
  - gingerlime 14y ago
    there's a difference between obtaining the private key and the private key file, which according to Linode was protected with a (hopefully strong) passphrase...
    - rainforest 14y ago
      It might have been encrypted, but surely it must be exposed to their system somewhere to enable them to make charges? Is it feasible, if that was the case, that the attackers could use that to exfiltrate decrypted CC info? I suppose this wild speculation isn't helpful or conducive and waiting on more information might be a better idea. I think the reputation damage has already been done, judging by the comments on the previous post though.
      - gingerlime 14y ago
        It depends on how they implemented it. Maybe there's an operator that logs on once a day and enters the passphrase, and then they use it to charge credit cards. So the private key is exposed only for a brief period. Another possibility is that the passphrase is entered when the machine is booted, and then the private key (not the passphrase) is loaded into memory. This is how e.g. ssh-agent[1] works. When this mechanism is used, you don't get access to the private key itself without direct memory access (which is possible, but not that easy to carry out, and would typically require root access and debug tools etc). However, the private key can be then used to carry out crypto operations. From the IRC chat logs, it looks like ryan got the private key file, but he didn't get the private key itself. At least he didn't seem to provide any solid proof to suggest he did. [1]https://en.wikipedia.org/wiki/Ssh-agent https://en.wikipedia.org/wiki/Ssh-agent
- InclinedPlane 14y ago
  CC data however does NOT have to live on any machine that anyone except your merchant bank owns. If you are writing against billing APIs in the 21st century then you only need to store a token which you use for recurring billing. Not everyone does this, but it's a reasonably good practice, especially if you have had intrusions in the past.
  - shadowfox 14y ago
    Out of curiosity, is this token generation sufficiently general or is it tied to a specific bank/credit card provider? (Basically is it easy to switch providers without having all your customers enter their cc information again?)
    - nenolod 14y ago
      Unfortunately it is tied to the merchant. :(
- deleted 14y ago
  [deleted]
reidrac 14y ago
I agree with you. Besides, how do they use the private key? Are they using a GPG agent o anything like that? Do they enter the password every time they need to use the private key? Having the private key in the web frontend sounds like a bad idea (as this incident has proved), they should use a backend system that is not accessible from the Internet. I don't know how they system works but the web frontend needs to encrypt CC numbers but I'm not sure it is required to decrypt them as that could be performed in a more restricted environment. Security incidents suck, but the good news is that they now have a chance to improve their security.
zcam 14y ago
Even then, attackers with half a brain (and malicious intent) would also have backdoored the payment page to harvest "fresh" credit cards numbers during the period they went unnoticed (possibly days/weeks). It's another good reason to leave the credit card handling to a reliable 3rd party imo.
entropy_ 14y ago
That doesn't make any sense. It's exactly like saying: "Oh sure, the credit card numbers were encrypted but if the encryption key is brute-forced then it's game over". The CC numbers were encrypted using a public key. The corresponding private key was itself encrypted using "[a] complex passphrase [that] is not stored electronically". They could be lying, but if they are not then this is perfectly valid security. Using this scheme, you can store new CC numbers any time you want using the public key. You'd have a regular billing cycle(say a month) with billings happening in a short time window(All my linode accounts get billed within minutes of each other so I think this is the case for linode). When you want to bill your customers you'd input the passphrase which wouldn't be stored anywhere, would be used to temporarily decrypt the CC numbers and bill everyone. Unless you are attacked in that window(or an attacker gains access before that window and can remain unnoticed until it happens, and even then they'd need root access to read the memory and not just access to the filesystem) then the CC numbers are pretty safe against any attack that leaks files/databases/etc...
- thaumaturgy 14y ago
  > When you want to bill your customers you'd input the passphrase which wouldn't be stored anywhere, would be used to temporarily decrypt the CC numbers and bill everyone. I've had account changes billed to my card within minutes, even (and especially) in the middle of the night (when I do most of that stuff). I'm having a little bit of trouble imagining that someone's sitting in front of a terminal and typing a passphrase into a prompt every three to five minutes, all day long.
  - entropy_ 14y ago
    You make a good point
  - corford 14y ago
    Interesting. Whenever I make changes that affect how much I'll be billed, the amounts get added or removed and take effect when the card next gets charged (which happens once a month). At least that's how I remember it... going to dig through some old statements now to verify this as its been a while since I last changed anything that would alter what they charge me.

by cheald 14y ago

tsahyt 14y ago

My thoughts exactly. Then again, I'd expect them to use a rather strong password for sensitive data. Considering 10^12 tries per second (which is probably very far fetched) and a 64 character alphanumeric password, my credit card -- probably the whole galaxy -- will have long expired until the password can be expected to be cracked. At that point, it really depends on how strong their passphrase is. If it's a dictionary word, we're pretty much fucked.

derefr 14y ago

I think a lot of people are failing to understand how credit card processing works. The private key needs to live on some machine, if you actually want to have any CC requests processed at some point. If the private key itself isn't on the frontend/user-facing server, then instead, when the frontend node is hacked, the hacker will find a shared secret or signing key to auth the frontend node against the backend node that does process the CC requests. The hacker won't be able to take your CC number for themselves in this case, but they'll sure be able to put spurious/malicious charges on your account.

JonnieCache 14y ago
I suppose it could have been kept only in memory, rather than just chilling on the filsystem in a file called private_key which is basically what happened AFAIK. It's surely within the boundaries of sense to load the key manually each time the app boots. Thinking about it though, that's basically the role of the key passphrase.
- derefr 14y ago
  Right; this is a common experience with SSL/X.509 signing keys--they're encrypted by default, so any server you configure with your ssl.pem file will prompt for a passphrase on startup, and only keep the decrypted key in memory. That's why this line, that most SSL providers give in their instructions, is so dangerous: openssl rsa -in server.key.org -out server.key It strips this (rather useful) encryption, giving any successful attacker to your frontend server your private key, in exchange for letting the node come up non-interactively.
- gingerlime 14y ago
  This is what ssh-agent does, and I could only hope Linode was using something similar. I think the statement suggests they did. > The private key is itself encrypted with passphrase encryption and the passphrase is not stored electronically So even if the hacker bragged about obtaining both the public and private keys, the private key was protected and therefore unusable without the passphrase. At this point, we can all hope linode used a very complex passphrase... My gut feeling is that they did.
  - joeyh 14y ago
    Storing a passphrase (or unlocked private key) in memory is storing it electronically. Once the attacker has root, they can dump the memory and access this information. The attacker claims to have gigabytes of data, which could well include a memory image (or swap partition image, if the program that stored this in memory didn't lock the memory to prevent it being swapped out).
    - joeyh 14y ago
      Attacker claims to have done so in this irc log: http://pastebin.com/7WXRDyAg http://pastebin.com/7WXRDyAg > 07:52 < HTP> the CCrypter class of the linode application context was accessable from outside the wwwroot using undocumented ColdFusion methods. i was fully able to decrypt the ccs using the in-memory privkey that they supplied the password for. > 07:53 < HTP> coldfusion runs as a single process, and its memory can be accessed using the ColdFusion wrapper
rainforest 14y ago
From their description it appears encrypted CC numbers were in the database amongst the other customer information. Sure, the data has to live somewhere, but the apparent situation of CC, customer, private key all accessible to the frontend looks sub-optimal.
- gingerlime 14y ago
  there's a difference between obtaining the private key and the private key file, which according to Linode was protected with a (hopefully strong) passphrase...
  - rainforest 14y ago
    It might have been encrypted, but surely it must be exposed to their system somewhere to enable them to make charges? Is it feasible, if that was the case, that the attackers could use that to exfiltrate decrypted CC info? I suppose this wild speculation isn't helpful or conducive and waiting on more information might be a better idea. I think the reputation damage has already been done, judging by the comments on the previous post though.
    - gingerlime 14y ago
      It depends on how they implemented it. Maybe there's an operator that logs on once a day and enters the passphrase, and then they use it to charge credit cards. So the private key is exposed only for a brief period. Another possibility is that the passphrase is entered when the machine is booted, and then the private key (not the passphrase) is loaded into memory. This is how e.g. ssh-agent[1] works. When this mechanism is used, you don't get access to the private key itself without direct memory access (which is possible, but not that easy to carry out, and would typically require root access and debug tools etc). However, the private key can be then used to carry out crypto operations. From the IRC chat logs, it looks like ryan got the private key file, but he didn't get the private key itself. At least he didn't seem to provide any solid proof to suggest he did. [1]https://en.wikipedia.org/wiki/Ssh-agent https://en.wikipedia.org/wiki/Ssh-agent
InclinedPlane 14y ago
CC data however does NOT have to live on any machine that anyone except your merchant bank owns. If you are writing against billing APIs in the 21st century then you only need to store a token which you use for recurring billing. Not everyone does this, but it's a reasonably good practice, especially if you have had intrusions in the past.
- shadowfox 14y ago
  Out of curiosity, is this token generation sufficiently general or is it tied to a specific bank/credit card provider? (Basically is it easy to switch providers without having all your customers enter their cc information again?)
  - nenolod 14y ago
    Unfortunately it is tied to the merchant. :(
deleted 14y ago
[deleted]

reidrac 14y ago

I agree with you. Besides, how do they use the private key? Are they using a GPG agent o anything like that? Do they enter the password every time they need to use the private key? Having the private key in the web frontend sounds like a bad idea (as this incident has proved), they should use a backend system that is not accessible from the Internet. I don't know how they system works but the web frontend needs to encrypt CC numbers but I'm not sure it is required to decrypt them as that could be performed in a more restricted environment. Security incidents suck, but the good news is that they now have a chance to improve their security.

zcam 14y ago

Even then, attackers with half a brain (and malicious intent) would also have backdoored the payment page to harvest "fresh" credit cards numbers during the period they went unnoticed (possibly days/weeks). It's another good reason to leave the credit card handling to a reliable 3rd party imo.

entropy_ 14y ago

That doesn't make any sense. It's exactly like saying: "Oh sure, the credit card numbers were encrypted but if the encryption key is brute-forced then it's game over". The CC numbers were encrypted using a public key. The corresponding private key was itself encrypted using "[a] complex passphrase [that] is not stored electronically". They could be lying, but if they are not then this is perfectly valid security. Using this scheme, you can store new CC numbers any time you want using the public key. You'd have a regular billing cycle(say a month) with billings happening in a short time window(All my linode accounts get billed within minutes of each other so I think this is the case for linode). When you want to bill your customers you'd input the passphrase which wouldn't be stored anywhere, would be used to temporarily decrypt the CC numbers and bill everyone. Unless you are attacked in that window(or an attacker gains access before that window and can remain unnoticed until it happens, and even then they'd need root access to read the memory and not just access to the filesystem) then the CC numbers are pretty safe against any attack that leaks files/databases/etc...

thaumaturgy 14y ago
> When you want to bill your customers you'd input the passphrase which wouldn't be stored anywhere, would be used to temporarily decrypt the CC numbers and bill everyone. I've had account changes billed to my card within minutes, even (and especially) in the middle of the night (when I do most of that stuff). I'm having a little bit of trouble imagining that someone's sitting in front of a terminal and typing a passphrase into a prompt every three to five minutes, all day long.
- entropy_ 14y ago
  You make a good point
- corford 14y ago
  Interesting. Whenever I make changes that affect how much I'll be billed, the amounts get added or removed and take effect when the card next gets charged (which happens once a month). At least that's how I remember it... going to dig through some old statements now to verify this as its been a while since I last changed anything that would alter what they charge me.