6 ms·

I'm fairly confident that Linode has been compromised since July, if not earlier. PagerDuty moved off of Linode after an incident in July. We've been under stri

I'm fairly confident that Linode has been compromised since July, if not earlier. PagerDuty moved off of Linode after an incident in July. We've been under strict gag orders from legal about that incident until today when Linode finally announced their compromise. Really, the only way I can see that this attacker could have gotten in the way they did (they logged into our Linode Manager account on the first try using a username that wasn't used anywhere but in Linode Manager, using valid 2fa and valid password) was if they had access to the Linode Manager database. I'm pretty sure that the recent WP Engine compromise was achieved through the same attack vector, too

masterleep 11y ago
If the Linode Manager database only stores the hash, how would they know the password? How did you find out about the illicit login?
- ceejayoz 11y ago
  If they've been more compromised than something like a simple SQL injection vulnerability, it could be something like added code to log passwords or post them off somewhere else.
- stcredzero 11y ago
  If the Linode Manager database only stores the hash, how would they know the password? Weak password? Weak hashing algorithm?
- Someone1234 11y ago
  > If the Linode Manager database only stores the hash, how would they know the password? Hashes can be turned back into plain text, it is just computationally expensive to do so. Hashing only slows down an attack (and or increases the cost), it doesn't not mitigate one. In particular if the hashes aren't salted then a rainbow table is an extremely effective way of breaking all of the hashes concurrently. The main method of doing so: Generate the hash for every combination of typable characters up to a given length (e.g. MD5() A-Za-z0-9 & specials up to 8x characters). This can be mitigated using a more computationally expensive hashing routine (or increasing the work factor on a less computationally expensive one) and salts. But given enough time OR computing power, all hashes will be broken. AWS makes breaking hashes a lot cheaper as you can bid on spare capacity and perform the operations relatively cheaply.
  - alextgordon 11y ago
    This only works if the input password has low entropy. You would think that people using Linode are savvy enough to be using long, randomly generated passwords.
    - Someone1234 11y ago
      > This only works if the input password has low entropy. If you're generating every single possible password up to e.g. 8 characters the password's quality doesn't matter, only the length does.
      - TheOtherHobbes 11y ago
        12 character random strings are an absolute minimum for a secure password, because brute forcing and tabling start to become impractical. Longer strings are even better. I wouldn't consider an 8 char password secure, no matter what the entropy is.
      - Buge 11y ago
        Yes length matters. That's why pretty much all "randomly generated" passwords are long. Mine are 20 characters.
      - yeukhon 11y ago
        A good secure password hash should be generated with a salt to slow down the process, in addition to using strong (slow) KDF function like scrypt. State actor like NSA could have store all possible 8 character combination today in a massive storage facility. God knows. But the size of such rainbow table is only effective if there is no salt, as with salt the hash is now different despite the underlying password is the same, thus there will never be enough storage if salt is present. But the most effective "rainbow table"-like table is a look-up table with the followings: * leaked password in plaintext, associate with email and any ID (forum username??) * hash all of those passwords without salt * hashes (with salt) of known leaked passwords (you try pas$w0rd and found a match for some hash with salt) - this only works if your attack succeed. If you do a quick count you won't be surprise most passwords are fairly short and simple. If two complex passwords appear to be very similar, you can assume with a good probability they are used by the same person. You can learn some private data from just looking at password (e.g. birthday, pet's name, door number, company they worked for, sport team they root for, which many turn out to be the crucial hint or actual answer to security questions.) I have never opened or downloaded any leaked data and don't know if it legal for use at all, but the black market probably has over petabyte volume of such data available. It would be very interesting to see the whole world attack couple hashes per day. Imagine you go to a website, it gives you some plaintext, and you run a couple quick scrypt with random salt, and return the response. Now with a billion online users, run this every day once, you may end up finding one successful match of "this password == this hash with this salt" once in a while. But hey, that's what botnet can do...and then bitcoin!
- ryanlol 11y ago
  They don't need to know the password with db write access. In fact, depending on how the sessions are managed the attacker might just need read access to log in without a password.
ryanlol 11y ago
Interestingly, while I know for a fact that you're correct. When someone asked about this on #linode the ops immediate reaction was to deny it (as it was every other time they got hacked). < naqod> alexf: any legitimacy to this https://news.ycombinator.com/item?id=10845619 https://news.ycombinator.com/item?id=10845619 ? <@alexf> naqod: again I'm not in ops so I don't have the deets, but my gut reaction is to say No effing Way You'd imagine that by now they wouldn't be so quick to deny this stuff.
- mcone 11y ago
  As a former Linode employee, I find all of your comments very interesting. But if it's true that you're being prosecuted, why are you incriminating yourself here?
  - Perihelion 11y ago
    All of the people involved have been very open about their identities and what they've been up to. It's sort of their MO. Pure speculation on my part, but I imagine it's basically a giant middle finger to everyone else.
  - ryanlol 11y ago
    Double jeopardy, I was previously convicted of 50700 counts of unauthorized access to various coldfusion sites. Among those was linode.
    - Someone1234 11y ago
      This one? http://www.upi.com/Top_News/World-News/2015/07/08/Finnish-teen-convicted-of-hacking-50700-computers/5981436373963/ http://www.upi.com/Top_News/World-News/2015/07/08/Finnish-te...
      - ryanlol 11y ago
        Yeah.
Thalagyrt 11y ago
This was my post from SomethingAwful, it was copied/pasted here. No idea who copied/pasted it. Rather than writing up a huge reply with more info, I'll link to another reply with more details on this same thread since someone else already did the writing: https://news.ycombinator.com/item?id=10845985 https://news.ycombinator.com/item?id=10845985
dfoxinator 11y ago
I find this very interesting because I think it brings up a specific question. If this is actually from July, and they knew about it, why would Linode choose this specific time to announce the breach publicly? It seems very curious being that they are already getting punished from the DDoS attack and if they waited 6 months they could have waited until the dust settles a little bit. Unless, of course, the group involved in the breach are also the ones unleashing the DDoS attacks. Which would also make me think there has been communication between the group and Linode, in contrary to what Linode stated.
- Thalagyrt 11y ago
  I think since multiple customers were hit by this and are presumably all putting pressure on them about it, their hand may have been forced.
  - dfoxinator 11y ago
    Could be, but the timing still seems really weird to me. If they did an investigation like they said they did, I just don't see why you would come out with it right now. Because from Pager Duty post it seems that no one could do anything to make them disclose it sooner.
    - Thalagyrt 11y ago
      I can't speak for the other folks that were compromised this way, but we decided to just cut our losses and move on at PagerDuty and spent the 30 days after the compromise migrating everything that was running there over to Azure. No point in putting pressure on a company that stonewalls you.
      - dfoxinator 11y ago
        That's a good point. Not worth your time for a company like Linode that doesn't really care about its customers. I think people mistake the quick support responses to basic questions as them caring, but when it really comes down to the important things like security and communication during a crisis, it's clear that there is a huge lapse from the leadership level down. Someone in this post wrote about how they stopped $10k worth of Linode service a month and no one tried to get them back or retain them. That was very odd to read, especially for a business of Linode's size ($22 million in revenue isn't that big where you can shrug off $120k/year.) Par for the course it seems, and everything is starting to make sense. Thanks for sharing your story and sorry that happened to your team.
linodemaster 11y ago
Hello, my name is Julius Kivimaki. I am the one who hacked Linode. You guys are all stupid if you think you can catch me. I say, CATCH ME IF YOU CAN, or GTFO! LULZ!

masterleep 11y ago

If the Linode Manager database only stores the hash, how would they know the password? How did you find out about the illicit login?

ceejayoz 11y ago
If they've been more compromised than something like a simple SQL injection vulnerability, it could be something like added code to log passwords or post them off somewhere else.
stcredzero 11y ago
If the Linode Manager database only stores the hash, how would they know the password? Weak password? Weak hashing algorithm?
Someone1234 11y ago
> If the Linode Manager database only stores the hash, how would they know the password? Hashes can be turned back into plain text, it is just computationally expensive to do so. Hashing only slows down an attack (and or increases the cost), it doesn't not mitigate one. In particular if the hashes aren't salted then a rainbow table is an extremely effective way of breaking all of the hashes concurrently. The main method of doing so: Generate the hash for every combination of typable characters up to a given length (e.g. MD5() A-Za-z0-9 & specials up to 8x characters). This can be mitigated using a more computationally expensive hashing routine (or increasing the work factor on a less computationally expensive one) and salts. But given enough time OR computing power, all hashes will be broken. AWS makes breaking hashes a lot cheaper as you can bid on spare capacity and perform the operations relatively cheaply.
- alextgordon 11y ago
  This only works if the input password has low entropy. You would think that people using Linode are savvy enough to be using long, randomly generated passwords.
  - Someone1234 11y ago
    > This only works if the input password has low entropy. If you're generating every single possible password up to e.g. 8 characters the password's quality doesn't matter, only the length does.
    - TheOtherHobbes 11y ago
      12 character random strings are an absolute minimum for a secure password, because brute forcing and tabling start to become impractical. Longer strings are even better. I wouldn't consider an 8 char password secure, no matter what the entropy is.
    - Buge 11y ago
      Yes length matters. That's why pretty much all "randomly generated" passwords are long. Mine are 20 characters.
    - yeukhon 11y ago
      A good secure password hash should be generated with a salt to slow down the process, in addition to using strong (slow) KDF function like scrypt. State actor like NSA could have store all possible 8 character combination today in a massive storage facility. God knows. But the size of such rainbow table is only effective if there is no salt, as with salt the hash is now different despite the underlying password is the same, thus there will never be enough storage if salt is present. But the most effective "rainbow table"-like table is a look-up table with the followings: * leaked password in plaintext, associate with email and any ID (forum username??) * hash all of those passwords without salt * hashes (with salt) of known leaked passwords (you try pas$w0rd and found a match for some hash with salt) - this only works if your attack succeed. If you do a quick count you won't be surprise most passwords are fairly short and simple. If two complex passwords appear to be very similar, you can assume with a good probability they are used by the same person. You can learn some private data from just looking at password (e.g. birthday, pet's name, door number, company they worked for, sport team they root for, which many turn out to be the crucial hint or actual answer to security questions.) I have never opened or downloaded any leaked data and don't know if it legal for use at all, but the black market probably has over petabyte volume of such data available. It would be very interesting to see the whole world attack couple hashes per day. Imagine you go to a website, it gives you some plaintext, and you run a couple quick scrypt with random salt, and return the response. Now with a billion online users, run this every day once, you may end up finding one successful match of "this password == this hash with this salt" once in a while. But hey, that's what botnet can do...and then bitcoin!
ryanlol 11y ago
They don't need to know the password with db write access. In fact, depending on how the sessions are managed the attacker might just need read access to log in without a password.

ryanlol 11y ago

Interestingly, while I know for a fact that you're correct. When someone asked about this on #linode the ops immediate reaction was to deny it (as it was every other time they got hacked). < naqod> alexf: any legitimacy to this https://news.ycombinator.com/item?id=10845619 https://news.ycombinator.com/item?id=10845619 ? <@alexf> naqod: again I'm not in ops so I don't have the deets, but my gut reaction is to say No effing Way You'd imagine that by now they wouldn't be so quick to deny this stuff.

mcone 11y ago
As a former Linode employee, I find all of your comments very interesting. But if it's true that you're being prosecuted, why are you incriminating yourself here?
- Perihelion 11y ago
  All of the people involved have been very open about their identities and what they've been up to. It's sort of their MO. Pure speculation on my part, but I imagine it's basically a giant middle finger to everyone else.
- ryanlol 11y ago
  Double jeopardy, I was previously convicted of 50700 counts of unauthorized access to various coldfusion sites. Among those was linode.
  - Someone1234 11y ago
    This one? http://www.upi.com/Top_News/World-News/2015/07/08/Finnish-teen-convicted-of-hacking-50700-computers/5981436373963/ http://www.upi.com/Top_News/World-News/2015/07/08/Finnish-te...
    - ryanlol 11y ago
      Yeah.

Thalagyrt 11y ago

This was my post from SomethingAwful, it was copied/pasted here. No idea who copied/pasted it. Rather than writing up a huge reply with more info, I'll link to another reply with more details on this same thread since someone else already did the writing: https://news.ycombinator.com/item?id=10845985 https://news.ycombinator.com/item?id=10845985

dfoxinator 11y ago

I find this very interesting because I think it brings up a specific question. If this is actually from July, and they knew about it, why would Linode choose this specific time to announce the breach publicly? It seems very curious being that they are already getting punished from the DDoS attack and if they waited 6 months they could have waited until the dust settles a little bit. Unless, of course, the group involved in the breach are also the ones unleashing the DDoS attacks. Which would also make me think there has been communication between the group and Linode, in contrary to what Linode stated.

Thalagyrt 11y ago
I think since multiple customers were hit by this and are presumably all putting pressure on them about it, their hand may have been forced.
- dfoxinator 11y ago
  Could be, but the timing still seems really weird to me. If they did an investigation like they said they did, I just don't see why you would come out with it right now. Because from Pager Duty post it seems that no one could do anything to make them disclose it sooner.
  - Thalagyrt 11y ago
    I can't speak for the other folks that were compromised this way, but we decided to just cut our losses and move on at PagerDuty and spent the 30 days after the compromise migrating everything that was running there over to Azure. No point in putting pressure on a company that stonewalls you.
    - dfoxinator 11y ago
      That's a good point. Not worth your time for a company like Linode that doesn't really care about its customers. I think people mistake the quick support responses to basic questions as them caring, but when it really comes down to the important things like security and communication during a crisis, it's clear that there is a huge lapse from the leadership level down. Someone in this post wrote about how they stopped $10k worth of Linode service a month and no one tried to get them back or retain them. That was very odd to read, especially for a business of Linode's size ($22 million in revenue isn't that big where you can shrug off $120k/year.) Par for the course it seems, and everything is starting to make sense. Thanks for sharing your story and sorry that happened to your team.

linodemaster 11y ago

Hello, my name is Julius Kivimaki. I am the one who hacked Linode. You guys are all stupid if you think you can catch me. I say, CATCH ME IF YOU CAN, or GTFO! LULZ!