16 ms·

I had a sour taste from what I remember being misleading communication around very serious control plane hacks of linode. A lot of bitcoin theft in 2012 (maybe

I had a sour taste from what I remember being misleading communication around very serious control plane hacks of linode. A lot of bitcoin theft in 2012 (maybe by their own staff?) 2013 some kind of cold fusion / HTP hack Another CF / HTP hack here. 2014 brought the MySQL server no password stuff. 2015 ish some kind of total root compromise? You can get a feel for all this here including the denials / lack of notification. https://news.ycombinator.com/item?id=10845985 https://news.ycombinator.com/item?id=10845985 Maybe 2016 same issue? https://www.zdnet.com/article/cloud-firm-linode-resets-user-passwords-after-suspected-hack/ https://www.zdnet.com/article/cloud-firm-linode-resets-user-... Not a company I'd put much actual production onto. Imagine if AWS had a hacker running around with total root access, able to reset MFA tokens to their own etc with no notice to customers. I'm not even sure such root access exists on AWS.

INTPenis 5y ago
I hate to say it but I recently moved away from Linode after their /64 block in Frankfurt was banned by all Google services. And even though all their kubernetes nodes have a public ipv4 address they were somehow unable to fallback on this when their ipv6 didn't work. And when I suggested this to their support they acted like I was crazy and said there is no way to switch between ipv4 and ipv6. Well I don't work in networking but I do work for a major telco and I know our networking guys could have done that routing change, easy.
- unilynx 5y ago
  Are you referring to the geoip issue ? https://gitlab.com/gitlab-org/gitlab-runner/-/issues/28769 https://gitlab.com/gitlab-org/gitlab-runner/-/issues/28769 This was resolved and it didn’t appear something nefarious or any kind of ban was going on, so if you have some references I’d love to see them
  - INTPenis 5y ago
    I never said it was "nefarious", I'm saying it was badly handled and it brought down all services in an entire location for days, with no fallback to ipv4, no status communication from Linode. It's not malice, it's just incompetence. Why the block was banned I have no idea and I don't really care, it can happen and they need to be able to handle it. I literally had a support case asking what happened, no response, eventually I had to say "look, I'm moving now because all services are down, please tell me what is going on", no answer, I had to move to get my services back up.
    - maccolgan 5y ago
      It seems to be a fault on Google's side, and not Linode's. It's like blaming the car when the road is closed down.
      - INTPenis 5y ago
        Yeah but again, no one is blaming anyone for the root cause here. Only for the handling.
- benatkin 5y ago
  Yep, confirmation from their staff here: https://www.linode.com/community/questions/22243/google-and-linode-server-issues https://www.linode.com/community/questions/22243/google-and-...
- hughrr 5y ago
  We blocked the entire of Linode AS63949 ranges because we were getting attacked from random owned nodes and it was tripping our IDS constantly. Just got fed up with it in the end and decided to hose them. To note, we have had problems with AWS blocking random addresses as well where we've had staff abroad.
  - mattl 5y ago
    Same, but DO as well.
    - archon810 5y ago
      Same but also about a hundred others in my block list, except it's not enough because they keep on coming and new ones popping up all the time.
      - mattl 5y ago
        Would you mind sharing? Email address in profile
- slenk 5y ago
  I am trying to jump through hoops to move to hetzner as I can't get a clean /64 either.
Scoundreller 5y ago
A 2012 Bitcoin hack victim was none other than a lead developer of Bitcoin. Back then, they ran a Bitcoin faucet on it that gave out a paltry 0.25 Bitcoin at a time. I never bothered to jump through those hoops for like a dollar (now about US$10k): http://gavintech.blogspot.com/2012/03/bitcoin-faucet-hacked.html?m=1 http://gavintech.blogspot.com/2012/03/bitcoin-faucet-hacked.... He only lost 5 bitcoin (like $20 then or $200k today), but another lost 3100, or around… $124 million today: https://bitcointalk.org/index.php?topic=66916.0 https://bitcointalk.org/index.php?topic=66916.0 They ran a Bitcoin mining pool and this hack motivated them to create a hardware wallet: https://blog.trezor.io/how-trezor-was-born-from-a-hacking-attack-that-affected-slush-pool-7a538f03fd8f https://blog.trezor.io/how-trezor-was-born-from-a-hacking-at...
- glofish 5y ago
  somebody please explain to me how is it possible that the owner reports losing $124 million, then they casually mention in the reply that: no problem, I'll just cover it with my own money ... (another recent story on ether hack had the same "resolution" the organization just chose to replaced the losses) ... where is that money coming from? does not seem real
  - dsl 5y ago
    Because back then the price of bitcoin was much lower. It is $124 million in today dollars, like $1,000 in back then dollars.
  - Scoundreller 5y ago
    That’s what it would be worth now. Was worth like 1/10000th of that back then. They probably covered it with their previously earned holdings that could have been another fraction of that.
  - saghm 5y ago
    I imagine they repaid the value of the bitcoins at the time, which would have been a lot less
  - deleted 5y ago
imachine1980_ 5y ago
this worry me, any similar compromise in digital ocean for example?
- btgeekboy 5y ago
  Not really a compromise but there was a point where they weren’t wiping block devices by default: https://news.ycombinator.com/item?id=6983097 https://news.ycombinator.com/item?id=6983097 Not sure if that’s changed. (Hopefully it has!)
  - riffic 5y ago
    jeez, a lot has changed since 2013. DO's the only one out of the major VPS players to go public. It's not amateur hour over there anymore.
    - btgeekboy 5y ago
      This was the same era that Linode had the compromises listed. It's a pretty apples-to-apples comparison, though I agree it has been almost a decade since.
frakkingcylons 5y ago
You can’t just imply that the linode staff assisted or were involved in stealing crypto currency from their customers without actually providing any evidence.
- tempnow987 5y ago
  Fair point, I can't edit my comment. My metric though is this: Someone has admin level access to full root on control plane and does things to customer accounts they don't want. A customer complains that the control plane has been used to reset things / asks for logs / etc. That's your opportunity to identify a root level control plane hack, disclose it, do the password reset things, remediate and move forward. OR - You don't disclose things till much later, you deny things, you provide scrubbed logs. Add in the bitcoin angle where this type of story is pretty common (trusted entity runs away with coins because they are easy to steal) - and couple that with how they handled the reports -> as I said in my comment, I'd be careful putting sensitive info onto that platform.
Twirrim 5y ago
I ended up dropping Linode as a result of the hacks, after having been a customer for years, both because of the nature of the hacks, and how they communicated.
lbotos 5y ago
Look, we are random internet people, and it's a "me vs. you" scenario, but as someone who worked at Linode in 2012, we were a small company, that all worked out of one office, with like 3 admins at the time. Yes, there were various hacks. Yes, there were silly vulns, but positing that one of the employees at that time stole bitcoin is something that I won't stand for. Could it have happened, sure. Do I think that it was an inside job? No, not at all. 99% of the people there at that time thought Bitcoin was insanely dumb, and I suspect most of us still do.
- lucb1e 5y ago
  I think in 2012 the tech community's sentiment was actually that Bitcoin was really cool and definitely useful, since it was new and did things in a practical application that we hadn't seen before. It's interesting that hindsight clouds that, it has been viewed negatively for years now but it's not that old yet. (Either way, your main point stands of course.)
  - hnxs 5y ago
    nope, I remember starting my first tech job in 2013, and the only people in tech who cared about it were libertarians, which was a very small subgroup of tech
    - eyelidlessness 5y ago
      It's still heavily dominated by Libertarians. They want an anarchist revolution, without the effort of actually organizing and sustaining it.
  - Beltalowda 5y ago
    > I think in 2012 the tech community's sentiment was actually that Bitcoin was really cool and definitely useful I don't think there was ever really a consensus on this. Lots of people (myself included, but also quite a few friends) always thought Bitcoin was just kinda useless. It's just that in 2012 there were comparatively low stakes (i.e. no massive energy use, not yet massive amounts of people pouring money in it, no massive amounts of "crypto snakeoil") that it just wasn't worth commenting on.
TheSwordsman 5y ago
The January 2016 thing was them finally acknowledging the attack that had happened many months prior, after WPEngine gave Linode the opportunity to announce they were hacked after they were also compromised using the same vector that hit PagerDuty. If Linode had declined, WPEngine was going to do it on their behalf. I couldn't convince the powers that be to make the same demand months prior, even though I was confident, so if WPEngine hadn't pushed the issue I don't know that Linode would have ever disclosed. But that doesn't matter anymore. This was nearly a generation ago in tech companies, and they are now part of a bigger one.
- tempnow987 5y ago
  Yeah, this key issue. https://wptavern.com/wp-engine-identifies-cloud-infrastructure-providers-as-entry-point-for-recent-security-breach https://wptavern.com/wp-engine-identifies-cloud-infrastructu... How do you sit on breaches like this for months and cause all this downline pain to your customers is crazy.
deleted 5y ago
[deleted]

INTPenis 5y ago

I hate to say it but I recently moved away from Linode after their /64 block in Frankfurt was banned by all Google services. And even though all their kubernetes nodes have a public ipv4 address they were somehow unable to fallback on this when their ipv6 didn't work. And when I suggested this to their support they acted like I was crazy and said there is no way to switch between ipv4 and ipv6. Well I don't work in networking but I do work for a major telco and I know our networking guys could have done that routing change, easy.

unilynx 5y ago
Are you referring to the geoip issue ? https://gitlab.com/gitlab-org/gitlab-runner/-/issues/28769 https://gitlab.com/gitlab-org/gitlab-runner/-/issues/28769 This was resolved and it didn’t appear something nefarious or any kind of ban was going on, so if you have some references I’d love to see them
- INTPenis 5y ago
  I never said it was "nefarious", I'm saying it was badly handled and it brought down all services in an entire location for days, with no fallback to ipv4, no status communication from Linode. It's not malice, it's just incompetence. Why the block was banned I have no idea and I don't really care, it can happen and they need to be able to handle it. I literally had a support case asking what happened, no response, eventually I had to say "look, I'm moving now because all services are down, please tell me what is going on", no answer, I had to move to get my services back up.
  - maccolgan 5y ago
    It seems to be a fault on Google's side, and not Linode's. It's like blaming the car when the road is closed down.
    - INTPenis 5y ago
      Yeah but again, no one is blaming anyone for the root cause here. Only for the handling.
benatkin 5y ago
Yep, confirmation from their staff here: https://www.linode.com/community/questions/22243/google-and-linode-server-issues https://www.linode.com/community/questions/22243/google-and-...
hughrr 5y ago
We blocked the entire of Linode AS63949 ranges because we were getting attacked from random owned nodes and it was tripping our IDS constantly. Just got fed up with it in the end and decided to hose them. To note, we have had problems with AWS blocking random addresses as well where we've had staff abroad.
- mattl 5y ago
  Same, but DO as well.
  - archon810 5y ago
    Same but also about a hundred others in my block list, except it's not enough because they keep on coming and new ones popping up all the time.
    - mattl 5y ago
      Would you mind sharing? Email address in profile
slenk 5y ago
I am trying to jump through hoops to move to hetzner as I can't get a clean /64 either.

Scoundreller 5y ago

A 2012 Bitcoin hack victim was none other than a lead developer of Bitcoin. Back then, they ran a Bitcoin faucet on it that gave out a paltry 0.25 Bitcoin at a time. I never bothered to jump through those hoops for like a dollar (now about US$10k): http://gavintech.blogspot.com/2012/03/bitcoin-faucet-hacked.html?m=1 http://gavintech.blogspot.com/2012/03/bitcoin-faucet-hacked.... He only lost 5 bitcoin (like $20 then or $200k today), but another lost 3100, or around… $124 million today: https://bitcointalk.org/index.php?topic=66916.0 https://bitcointalk.org/index.php?topic=66916.0 They ran a Bitcoin mining pool and this hack motivated them to create a hardware wallet: https://blog.trezor.io/how-trezor-was-born-from-a-hacking-attack-that-affected-slush-pool-7a538f03fd8f https://blog.trezor.io/how-trezor-was-born-from-a-hacking-at...

glofish 5y ago
somebody please explain to me how is it possible that the owner reports losing $124 million, then they casually mention in the reply that: no problem, I'll just cover it with my own money ... (another recent story on ether hack had the same "resolution" the organization just chose to replaced the losses) ... where is that money coming from? does not seem real
- dsl 5y ago
  Because back then the price of bitcoin was much lower. It is $124 million in today dollars, like $1,000 in back then dollars.
- Scoundreller 5y ago
  That’s what it would be worth now. Was worth like 1/10000th of that back then. They probably covered it with their previously earned holdings that could have been another fraction of that.
- saghm 5y ago
  I imagine they repaid the value of the bitcoins at the time, which would have been a lot less
- deleted 5y ago

imachine1980_ 5y ago

this worry me, any similar compromise in digital ocean for example?

btgeekboy 5y ago
Not really a compromise but there was a point where they weren’t wiping block devices by default: https://news.ycombinator.com/item?id=6983097 https://news.ycombinator.com/item?id=6983097 Not sure if that’s changed. (Hopefully it has!)
- riffic 5y ago
  jeez, a lot has changed since 2013. DO's the only one out of the major VPS players to go public. It's not amateur hour over there anymore.
  - btgeekboy 5y ago
    This was the same era that Linode had the compromises listed. It's a pretty apples-to-apples comparison, though I agree it has been almost a decade since.

frakkingcylons 5y ago

You can’t just imply that the linode staff assisted or were involved in stealing crypto currency from their customers without actually providing any evidence.

tempnow987 5y ago
Fair point, I can't edit my comment. My metric though is this: Someone has admin level access to full root on control plane and does things to customer accounts they don't want. A customer complains that the control plane has been used to reset things / asks for logs / etc. That's your opportunity to identify a root level control plane hack, disclose it, do the password reset things, remediate and move forward. OR - You don't disclose things till much later, you deny things, you provide scrubbed logs. Add in the bitcoin angle where this type of story is pretty common (trusted entity runs away with coins because they are easy to steal) - and couple that with how they handled the reports -> as I said in my comment, I'd be careful putting sensitive info onto that platform.

Twirrim 5y ago

I ended up dropping Linode as a result of the hacks, after having been a customer for years, both because of the nature of the hacks, and how they communicated.

lbotos 5y ago

Look, we are random internet people, and it's a "me vs. you" scenario, but as someone who worked at Linode in 2012, we were a small company, that all worked out of one office, with like 3 admins at the time. Yes, there were various hacks. Yes, there were silly vulns, but positing that one of the employees at that time stole bitcoin is something that I won't stand for. Could it have happened, sure. Do I think that it was an inside job? No, not at all. 99% of the people there at that time thought Bitcoin was insanely dumb, and I suspect most of us still do.

lucb1e 5y ago
I think in 2012 the tech community's sentiment was actually that Bitcoin was really cool and definitely useful, since it was new and did things in a practical application that we hadn't seen before. It's interesting that hindsight clouds that, it has been viewed negatively for years now but it's not that old yet. (Either way, your main point stands of course.)
- hnxs 5y ago
  nope, I remember starting my first tech job in 2013, and the only people in tech who cared about it were libertarians, which was a very small subgroup of tech
  - eyelidlessness 5y ago
    It's still heavily dominated by Libertarians. They want an anarchist revolution, without the effort of actually organizing and sustaining it.
- Beltalowda 5y ago
  > I think in 2012 the tech community's sentiment was actually that Bitcoin was really cool and definitely useful I don't think there was ever really a consensus on this. Lots of people (myself included, but also quite a few friends) always thought Bitcoin was just kinda useless. It's just that in 2012 there were comparatively low stakes (i.e. no massive energy use, not yet massive amounts of people pouring money in it, no massive amounts of "crypto snakeoil") that it just wasn't worth commenting on.

TheSwordsman 5y ago

The January 2016 thing was them finally acknowledging the attack that had happened many months prior, after WPEngine gave Linode the opportunity to announce they were hacked after they were also compromised using the same vector that hit PagerDuty. If Linode had declined, WPEngine was going to do it on their behalf. I couldn't convince the powers that be to make the same demand months prior, even though I was confident, so if WPEngine hadn't pushed the issue I don't know that Linode would have ever disclosed. But that doesn't matter anymore. This was nearly a generation ago in tech companies, and they are now part of a bigger one.

tempnow987 5y ago
Yeah, this key issue. https://wptavern.com/wp-engine-identifies-cloud-infrastructure-providers-as-entry-point-for-recent-security-breach https://wptavern.com/wp-engine-identifies-cloud-infrastructu... How do you sit on breaches like this for months and cause all this downline pain to your customers is crazy.

deleted 5y ago

[deleted]