15 ms·

Security Notification and Linode Manager Password Reset

233 points by qmr 11y ago

ryanlol 11y ago
This is at least the fifth time I can count that linode has been hacked, really? Maybe it's time to ditch that coldfusion stack? Edit: 1.The bitcoin hacks, March 2012 2.HTP hack, April 15, 2013 (CF exploit) 3.Second HTP hack April 16, 2013 (Another CF exploit) 4.MySQL server that allowed anonymous logins (?!?!) January 19, 2014 5.This hack I'm not counting their domain name and various other parts of their infrastructure provided by third parties being compromised, but if I was the list would be significantly longer.
- ddevault 11y ago
  >Maybe it's time to ditch that coldfusion stack? [Linode developer here] Been working on (re)writing things in Python since I started. It takes a while, but I think everyone recognizes that the CF codebase is difficult to maintain. The good news is that significant progress is being made and we're still doing routine audits of the existing stuff.
  - ryanlol 11y ago
    >we're still doing routine audits of the existing stuff. This doesn't work, auditing coldfusion code is impossible without auditing the entire platform. The whole platform is so full of bugs and strange behaviour that it's actually impossible to produce secure coldfusion code.
    - Sir_Cmpwn 11y ago
      I can't really deny that CF is that bad, but it'd be irresponsible to just let the codebase rust as we rewrite it - and we are rewriting it.
      - ryanlol 11y ago
        When did you start rewriting it? It doesn't take years to replace this stuff.
        ddevault 11y ago
        I was hired in July and have been driving most of this effort and we're shipping rewritten versions of some parts of our infra soon™
        StavrosK 11y ago
        That sounds good, at least, thanks for letting us know.
        deleted 11y ago
        [deleted]
- mapgrep 11y ago
  I was just thinking the same thing. I've been a customer for >10 years but this is getting ridiculous. First 2013 attack was apparently exacerbated by cleartext password storage for LISH (their management shell) and API tokens https://marco.org/2013/04/16/linode-hacked https://marco.org/2013/04/16/linode-hacked The 2012 Bitcoin attack involved a breach of Linode's customer service portal http://arstechnica.com/business/2012/03/bitcoins-worth-228000-stolen-from-customers-of-hacked-webhost/ http://arstechnica.com/business/2012/03/bitcoins-worth-22800... Today's attack is some kind of unspecified or unknown breach involving Linode manager. I guess the obvious commonality here is that all the attacks target the "soft" Linode layers AROUND managing deploys of Linux and Xen/KVM/UML rather than the "hard" targets of those widely used systems. This also happens to be the layer where Linode should be adding value (as opposed to the cheaper VPS providers out there) and I think it's increasingly troublesome that they continue to have such severe security issues. Is this company (CEO - Christopher Aker) not investing in security staff, security training, best practices etc, or are they investing tons and just getting breached because they host so many sites? Unclear. But it's easy to imagine it's the former, from the outside, given all these incidents.
  - ryanlol 11y ago
    Actually the 2013 hack was caused by linode running blatantly misconfigured CF installations, like doing stuff that the manual has big warnings about.
    - sarciszewski 11y ago
      For those not in the know, ryanlol was one of the people on the team involved in the 2013 hacks.
      - throway-not-909 11y ago
        Really? Thomas Asaro told us they were all in jail.
        ryanlol 11y ago
- technion 11y ago
  MySQL server that allowed anonymous logins Has anyone got more information on this? Various Google searches keep pointing me at the other four hacks.
  - ryanlol 11y ago
    https://blog.linode.com/2014/01/19/an-old-system-and-a-swat-team/ https://blog.linode.com/2014/01/19/an-old-system-and-a-swat-... They try to downplay it pretty hard.
    - technion 11y ago
      Thanks for this link. Although it talks about: database accessed using old forum credentials So I'm not sure "anonymous login" would be an entirely accurate description.
      - ryanlol 11y ago
        The server would in fact accept any credentials as it had been started with --skip-grant-tables. Tested it myself.
      - deleted 11y ago
        [deleted]
wiremine 11y ago
The page was slashdotted for me, here's a copy: "Effective immediately, Linode Manager passwords have been expired. You will be prompted to set a new password on your next login. We regret this inconvenience, however this is a necessary precaution. "A security investigation into the unauthorized login of three accounts has led us to the discovery of two Linode.com user credentials on an external machine. This implies user credentials could have been read from our database, either offline or on, at some point. The user table contains usernames, email addresses, securely hashed passwords and encrypted two-factor seeds. The resetting of your password will invalidate the old credentials. "This may have contributed to the unauthorized access of the three Linode customer accounts mentioned above, which were logged into via manager.linode.com. The affected customers were notified immediately. We have found no other evidence of access to Linode infrastructure, including host machines and virtual machine data. "The entire Linode team has been working around the clock to address both this issue and the ongoing DDoS attacks. We’ve retained a well-known third-party security firm to aid in our investigation. Multiple Federal law enforcement authorities are also investigating and have cases open for both issues. When the thorough investigation is complete, we will share an update on the findings. "You may be wondering if the same person or group is behind these malicious acts. We are wondering the same thing. At this point we have no information about who is behind either issue. We have not been contacted by anyone taking accountability or making demands. The acts may be related and they may not be. "The security of your data, the functionality of your servers, and your confidence in Linode are extremely important to all of us. While we feel victimized ourselves, we understand it is our responsibility, and our privilege as your host, to provide the best possible security and service. You can help further enhance the security of your account by always using strong passwords, enabling two-factor authentication, and never using the same password at multiple services. "We sincerely apologize for the recent disruptions in your Linode service. Thank you for your patience, understanding and ongoing trust in Linode."
- madaxe_again 11y ago
  Sadly I trust them as far as I can throw them, and we moved everything important from them to AWS a few years back. We had left a few static sites there but after the shenanigans over the holidays, we're moving our remaining stuff. Sad. We spent >$10k/month with them for a while, before their shit started falling apart. They didn't appear to care at all when we left, so I suppose they have an awful lot of large customers who just don't mind their stack disintegrating beneath them every sixty days at most.
broknbottle 11y ago
Wow, these guys can't get a break.
- JshWright 11y ago
  It's not as if this is just bad luck...
- threeseed 11y ago
  You make your own luck. If they can't be bothered to invest in their tools and processes then this is the sort of thing that happens.
  - workitout 11y ago
    Yeah it's not like ColdFusion powered web sites enabling billions of dollars in commerce and what with all other web development tools being free of security vulnerabilities and all.
    - brandon272 11y ago
      The merits of ColdFusion are irrelevant. The fact of the matter is that their software has continuously been breached.
- fweespeech 11y ago
  They've been using a CF stack that is fundamentally broken from the foundation up for years and was aware of it. Its part of why they hired someone in July to rebuild it in python.
  - workitout 11y ago
    You signed an NDA and have done a full analysis of their code? Cool.
    - fweespeech 11y ago
secstate 11y ago
Sure wish they had sent out an email notification to users instead of a slashdotted blog post. Now the question is how long can Linode stand in the face of these sorts of hacks and network attacks in the face of stiff VPS competition.
- threeseed 11y ago
  Be grateful they posted something to Slashdot at all. Back in 2013 they didn't do that for DAYS after the event happened and someone else posted what happened.
- chibosempai 11y ago
  (Linode Employee) Already got it covered, we are sending out an email to everyone in batches, but pushed out the blog first since it can be seen by everyone right away.
  - dnm 11y ago
    I'm still waiting for mine (6 hours since this was posted to HN).
    - Keverw 11y ago
      I'm still waiting on an email too.
  - dfoxinator 11y ago
    How come, still, no one I know, including myself, has received this email?
qmr 11y ago
Blog is down. Maybe they should host it on WPEngine. (: This sort of thing almost became background noise when I worked there. I can say that things improved somewhat after the HTP compromise, at least for awhile.
throwaway909009 11y ago
I'm fairly confident that Linode has been compromised since July, if not earlier. PagerDuty moved off of Linode after an incident in July. We've been under strict gag orders from legal about that incident until today when Linode finally announced their compromise. Really, the only way I can see that this attacker could have gotten in the way they did (they logged into our Linode Manager account on the first try using a username that wasn't used anywhere but in Linode Manager, using valid 2fa and valid password) was if they had access to the Linode Manager database. I'm pretty sure that the recent WP Engine compromise was achieved through the same attack vector, too
- masterleep 11y ago
  If the Linode Manager database only stores the hash, how would they know the password? How did you find out about the illicit login?
  - ceejayoz 11y ago
    If they've been more compromised than something like a simple SQL injection vulnerability, it could be something like added code to log passwords or post them off somewhere else.
  - stcredzero 11y ago
    If the Linode Manager database only stores the hash, how would they know the password? Weak password? Weak hashing algorithm?
  - Someone1234 11y ago
    > If the Linode Manager database only stores the hash, how would they know the password? Hashes can be turned back into plain text, it is just computationally expensive to do so. Hashing only slows down an attack (and or increases the cost), it doesn't not mitigate one. In particular if the hashes aren't salted then a rainbow table is an extremely effective way of breaking all of the hashes concurrently. The main method of doing so: Generate the hash for every combination of typable characters up to a given length (e.g. MD5() A-Za-z0-9 & specials up to 8x characters). This can be mitigated using a more computationally expensive hashing routine (or increasing the work factor on a less computationally expensive one) and salts. But given enough time OR computing power, all hashes will be broken. AWS makes breaking hashes a lot cheaper as you can bid on spare capacity and perform the operations relatively cheaply.
dmuth 11y ago
Looks like the reason their blog is down is because... it's now being targeted by a DoS: http://status.linode.com/incidents/kldhjpjnfnkj http://status.linode.com/incidents/kldhjpjnfnkj Attacking a blog talking about the hack? It sure seems that someone has a grudge against Linode. :-/
- Mahn 11y ago
  At this point I'm starting to wonder whether this isn't a competitor putting their investors money to work. It's otherwise utterly bizzare that someone would be so obsessive in damaging Linode. I really hope they make the details of the investigation public...
  - Someone1234 11y ago
    It makes more sense than you might imagine... Linode, Github, Stackoverflow, Imgur, they've all been targeted. But what do they have in common? In a word: popularity. The core reason these sites are targeted is because it is impressive to others. The source of this is typically two fold: - For the lolz. Someone with a botnet just wants to show off, taking down something known gives them more notoriety. - For a sales pitch. Someone has botnet capacity that they want to sell, and "I took down Linode" is a great way of demoing that to potential buyers. The first one is more common when someone finds a new method of traffic multiplication and just wants to show it off (i.e. they trick a third party into DoS/DDoSing a target). The second one is legitimate criminal enterprises.
    - Buge 11y ago
      Github was also targeted by China to try to get them to delete certain repos that were unfriendly to China. Some people also saw it as showing off their power.
  - vertis 11y ago
    Based on the number of negative reviews on glassdoor, it could also be a former employee.
TheSwordsman 11y ago
I'm glad to see that this information has now been publicly disclosed. In July 2015, we suffered a compromise at PagerDuty via the Linode Manager. I hope that we can provide a bit more of an official in-depth post-mortem of our compromise, but I'd be happy to disclose some of the details here. Using the access gained within the Linode Manager, the attacker reset the root password on a few systems, and used Lish to gain root access. We were alerted to this activity and fully revoked the attacker's access within 60 minutes of the first node being compromised. Working with Linode support, we discovered which user account was being used and completely deactivated the user. We also isolated the VMs, and performed forensics on read-only copies of their disk images. In our situation the attacker knew one of our user's passwords and MFA secret. This allowed them to provide valid authentication credentials for an account in the Linode Manager. It's worth noting that all of our active user accounts had two-factor authentication enabled. An interesting data point was that the user who had their account compromised was no longer in possession of the MFA secret themselves. Their cell phone had been reset (thus deleting all data) 8 months prior. The user could not log in to the Linode Manager if they wanted, so it was our determination that the key could not have been obtained from the user and was more likely on Linode's side. We also have evidence from access logs provided by Linode that the attackers tried to authenticate as an ex-employee, whose username ONLY existed in the Linode database. It was absolutely unique and was not used elsewhere by the employee making the username an accidental honeypot. This was another piece of data supporting that Linode was the source of our compromise. We immediately reached out to them not only to inform them of their compromise, but to assist them in investigating it. We were confident that the Linode database had been breached, and that the secret key used to encrypt information in the database had been compromised as well. In addition to reaching out to Linode, we also worked with a third-party security firm to audit our work done during the incident. Likewise, around the same time we reached out to law enforcement to assist in investigating the attack. I believe our public disclosure includes this information[1]. This was in the middle of July 2015. We did not get confirmation in July that there was a breach of the Linode Manager or any associated credentials. In the end, we migrated away from Linode because of this breach (even before it was publicly disclosed) in Aug 2015. We also never were able to confidently disclose that Linode was the vector due to lack of confirmation from their end. While all of us who responded to the incident were confident they were the source, we now thankfully have the data to confirm it. [1] https://www.pagerduty.com/blog/july-2015-security-announcement/ https://www.pagerduty.com/blog/july-2015-security-announceme...
deleted 11y ago
[deleted]
msh 11y ago
Why the hell have they not emailed their customers about this! This is not the kind of thing I want to learn from HN.
- dangrossman 11y ago
  They're doing so as we speak. It takes a while to send 400,000 mails if you actually want them to be delivered to inboxes.
  - chucky_z 11y ago
    400k emails isn't that many though. :(
    - dwightgunning 11y ago
      I haven't received any email. It's over 12 hours since the notice went up on status.linode.com and the Twitter account.
- cakes 11y ago
  Agreed - especially having read this earlier (not being able to fix it at the time), still not getting the email (like 6 hours have gone by) and I'm resetting because I should and still no communication from Linode. Concerning since I had to reset my password and regen my 2fa using only my old password...
dnm 11y ago
And of course there's the ongoing ddos attack that is preventing me from reaching manager.linode.com...
wtbob 11y ago
This is Yet Another Reminder to use unique, unguessable, unmemorable passwords for all online services. It's a question of when, not if, any particular password database will be compromised. While 'password1' and 'this is my long unguessable password' and 'm4r1g0ld' and 'false lemur capacitor paperclip' will all eventually be guessed, 'sF0PSQMwK85fe9xanqJRm9nty9cJGHJsVmti' will never, ever be.
- wtbob 11y ago
  Why was this downvoted? It's true. The example password is 36 characters; a mixed-case-and-numbers alphabet has just over 6 bit per character; the example password is thus 217 bits. It's not going to be guessed.
- ryanlol 11y ago
  Except passwords aren't even a relevant part of this breach. They're worth absolutely nothing compared to the ridiculous amounts of customer data linode controls.
tptacek 11y ago
Don't use Linode.
- workitout 11y ago
  Don't lecture me.
reconbot 11y ago
Not to throw fuel on the fire but grumbling ex-employees aren't surprised. I've heard a few stories about not investing in infrastructure or even bug fixes. "They wont even look at bug fixes with major support costs." I know it's hard to triage what to do when running a company but when employees are proud of their product a breach like this isn't surprising.
sarciszewski 11y ago
The actual answer is much more sinister than that. Which is kind of hilarious.
- lawnchair_larry 11y ago
  What is the actual answer?
  - sarciszewski 11y ago
    Last I heard, compromising Bitcoin exchanges for lulz and mad profit. Allegedly. So if the rumor is true, he wasn't technically lying.
    - jacquesm 11y ago
      I don't think that if all you have is 'allegedly' it is up to you to make such accusations here in public.
      - sarciszewski 11y ago
        This is stuff he shares openly on IRC. In public channels. Whether any of it is true or not remains to be seen. He's not ever given me any reason to believe he's lying, however.
- dang 11y ago
  That crosses into personal attack, which is not ok, even if it's just insinuated. We detached this subthread from https://news.ycombinator.com/item?id=10847715 https://news.ycombinator.com/item?id=10847715 and marked it off-topic.
  - sarciszewski 11y ago
    It's not a personal attack, it's something that ryanlol publicly claims in ##security on freenode FFS. If it wasn't, I wouldn't have referenced it From what I gather, he thinks law enforcement is hilarious.
relaxatorium 11y ago
and now getting server errors & timeouts when I try to get to the manager to change my passwords. this has been a hellaciously thorough attack.
tyingq 11y ago
For those not keeping track, that's the third major breach I've heard of where user data was exposed. Previous Incidents: April, 2013: https://blog.linode.com/2013/04/16/security-incident-update/ https://blog.linode.com/2013/04/16/security-incident-update/ January 2014: https://blog.linode.com/2014/01/19/an-old-system-and-a-swat-team/ https://blog.linode.com/2014/01/19/an-old-system-and-a-swat-... (they mention the breach almost in passing...the swatting is unrelated) I really want Linode to do well, but this may be it for me.
codezero 11y ago
Glad they posted it. I just logged into one of my linode boxes that I don't use for very much but keep around, and it was rooted... doh.
- Ronsenshi 11y ago
  Could be just a coincidence if you keep SSH open on standard port. SSH bots don't sleep.
  - codezero 11y ago
    The host was only accessible via private key or Linode's LISH shell. That's what seems most suspicious. There is some minor evidence remaining in the .bash_history that is curious.
    - maaarghk 11y ago
      I'd be interested in a write-up about anything you find in that .bash_history or logs, would you consider writing one?
      - codezero 11y ago
        Here's the main bulk of the obviously bad things: 3 ls 4 ls -al 5 chown syslog auth.log 6 ls -al 7 chown syslog kern.log 8 ls -al 9 chown syslog syslog 10 ls -al 11 echo -n '' > /media/xvda/root/.bash_history 12 echo -n '' > /root/.bash_history 13 echo -n '' > /root/.viminfo 14 L=$(find /var/log -type f); for F in $L; do echo -n '' > $F; done 15 rm -rf /etc/ssh/*_key* #remove host keys 16 rm -rf /var/lib/dhcp/* # dhcp leases 17 echo "echo 'options rotate' >> /etc/resolv.conf" > /etc/dhcp/dhclient-exit-hooks.d/rotate 18 ls 19 ls -al /var/log 20 ls 21 ls -al /var/log 22 exit 23 ls 24 ls -al /var/log 27 adduser in 28 su - in 29 vi /etc/sudoers 30 vi /etc/gro 31 vi /etc/group 32 groupadd --help 33 groups 34 groupmod 35 groupadd --help 36 vi /etc/group 37 su - in
superuser2 11y ago
With Linode's extended DDoS I have not been able to get into the Manager for a couple weeks. I'd really like to cancel my account with them (and they do keep billing) but I don't appear to have any tools short of a chargeback. Anyone else in the same boat?
- mapunk 11y ago
  You don't mention contacting support. Shouldn't that be your second avenue after trying to access the account dashboard?
  - superuser2 11y ago
    I have always contacted support by opening a ticket in the Manager. The main website is also down. Is there an out-of-band way to reach them?
    - IKnowComputer 11y ago
      Phone? 855-454-6633 (+1-609-380-7100)
- stephen223 11y ago
  Why not call them on the phone? They have a toll-free number for support.
- dwightgunning 11y ago
  The main website is down for me and has been flakey but http://manage.linode.com http://manage.linode.com hasn't been a problem for me over the last few weeks.
aenglander 11y ago
This is a perfect poster child for decentralized authentication. Centralized stores of passwords and secrets are just asking for trouble. When will we learn?
- lsc 11y ago
  so... I completely agree that shared secret authentication is a bad idea, and I use public key authentication wherever I can (password auth is disabled for ssh on every server I control; I do everything with ssh public keys.) However, I've yet to set a public key authentication scheme that users would find acceptable for web applications. Do you really expect all users to setup x.509 auth in the browser? What is your public key solution to authenticate the web-applications that customers demand?
cylo 11y ago
So, outside of the major cloud providers, what are the good alternatives?
- peteretep 11y ago
  I like Bytemark.
- tyingq 11y ago
  VPS providers that are typically pitched as Linode Alternatives. Not 1:1 equivalents, but may work depending on your needs. Vultr, Ramnode, Wable, iWStack Or, depending on the number of VPS's you have, you might like: - Aliyun, the cloud service from Alibaba - A dedicated server from OVH, or their mid-tier brand, SoYouStart. This is my personal favorite. They have real DDOS protection, Data centers in North America and Europe, reasonable web interfaces, lots of available IPV4 space, and DIRT CHEAP prices. Run proxmox as the distro, and you get a decent interface to create and manage VPS instances. The most important piece would be to try and split instances across at least two of these providers so that you have some fast recourse if something goes wrong. For the things I'm running, doing a nightly rsync of the data from one provider to another suffices as reasonable insurance.
SuperKlaus 11y ago
Not the first time they poorly handled a security issue: https://blog.linode.com/2014/01/19/an-old-system-and-a-swat-team/ https://blog.linode.com/2014/01/19/an-old-system-and-a-swat-...
- dfoxinator 11y ago
  There are also a number of reviews on Glassdoor where supposed Linode employees say they were asked by management to lie to customers about security breaches.
  - throway-not-909 11y ago
    We were.
dantiberian 11y ago
I enabled two factor on my Linode account recently and saw this worrying copy: > If you lose your token and get locked out of the Linode Manager, email support@linode.com to regain access to your account. > Should you need us to disable your Two-Factor Authentication, the following information is required: > An image of the front and back of the payment card on file, which clearly shows both the last 6 digits and owner of the card. > An image of the front and back of the matching government-issued photo ID. There doesn't seem to be a way to say "I have my big boy pants on, don't let anyone in under any circumstances". This is the first 2FA setup I've seen that still allows bypass by contacting support. EDIT: I also find it odd that you have to manually generate a scratch code, and they don't automatically generate it for you. Again, all of the other 2FA setups I've gone through have done this.
- pfg 11y ago
  > There doesn't seem to be a way to say "I have my big boy pants on, don't let anyone in under any circumstances". This is the first 2FA setup I've seen that still allows bypass by contacting support. Sadly, this is quite common, especially with organizations that provide phone support[1]. [1]: http://krebsonsecurity.com/2015/12/2016-reality-lazy-authentication-still-the-norm/ http://krebsonsecurity.com/2015/12/2016-reality-lazy-authent...
tlrobinson 11y ago
Comments from Linode employees, Linode customers who were hacked, and a guy who hacked Linode (different incident) I love HN.
jafingi 11y ago
That's it. I'm done with Linode. I will be keeping one server on Linode, but my new startup where I need several load balancers, Web servers, DB replication will be hosted elsewhere. Do you guys know any good providers? Not AWS.
- laseeey 11y ago
  Can I ask why not AWS?
  - jafingi 11y ago
    I just don't think the performance is good enough for pure web hosting. Much slower/$ than Linode and DO. Their uptime and infrastructure is great, but you're totally on your own (e.g. if you're getting DDoS'ed). Also their traffic cost is not as easy to predict than DO/Linode/Vultr.
- pfg 11y ago
  OVH is hard to beat when it comes to raw performance/$. Hardware and network has been quite reliable for me (using SoYouStart and OVH), and their DDoS filter is excellent. I'm not really familiar with their cloud/VPS offering. That being said, they only have one data center in NA at the moment, and they recently suffered a rather nasty network outage in that data center. Would definitely look into them if your target market is primarily EU, but if you're mostly NA-based, YMMV.
linodemaster 11y ago
Hello, my name is Julius Kivimaki. I am the one who hacked Linode. You guys are all stupid if you think you can catch me. I say, CATCH ME IF YOU CAN, or GTFO! LULZ!
linodemaster 11y ago
Hello, my name is Julius Kivimaki. I am the one who hacked Linode. You guys are all stupid if you think you can catch me. I say, CATCH ME IF YOU CAN, or GTFO! LULZ!
linodemaster 11y ago
Hello, my name is Julius Kivimaki. I am the one who hacked Linode. You guys are all stupid if you think you can catch me. I say, CATCH ME IF YOU CAN, or GTFO! LULZ!

233 points by qmr 11y ago

This is at least the fifth time I can count that linode has been hacked, really? Maybe it's time to ditch that coldfusion stack? Edit: 1.The bitcoin hacks, March 2012 2.HTP hack, April 15, 2013 (CF exploit) 3.Second HTP hack April 16, 2013 (Another CF exploit) 4.MySQL server that allowed anonymous logins (?!?!) January 19, 2014 5.This hack I'm not counting their domain name and various other parts of their infrastructure provided by third parties being compromised, but if I was the list would be significantly longer.

ddevault 11y ago
>Maybe it's time to ditch that coldfusion stack? [Linode developer here] Been working on (re)writing things in Python since I started. It takes a while, but I think everyone recognizes that the CF codebase is difficult to maintain. The good news is that significant progress is being made and we're still doing routine audits of the existing stuff.
- ryanlol 11y ago
  >we're still doing routine audits of the existing stuff. This doesn't work, auditing coldfusion code is impossible without auditing the entire platform. The whole platform is so full of bugs and strange behaviour that it's actually impossible to produce secure coldfusion code.
  - Sir_Cmpwn 11y ago
    I can't really deny that CF is that bad, but it'd be irresponsible to just let the codebase rust as we rewrite it - and we are rewriting it.
    - ryanlol 11y ago
      When did you start rewriting it? It doesn't take years to replace this stuff.
      - ddevault 11y ago
        I was hired in July and have been driving most of this effort and we're shipping rewritten versions of some parts of our infra soon™
        StavrosK 11y ago
        That sounds good, at least, thanks for letting us know.
      - deleted 11y ago
        [deleted]
mapgrep 11y ago
I was just thinking the same thing. I've been a customer for >10 years but this is getting ridiculous. First 2013 attack was apparently exacerbated by cleartext password storage for LISH (their management shell) and API tokens https://marco.org/2013/04/16/linode-hacked https://marco.org/2013/04/16/linode-hacked The 2012 Bitcoin attack involved a breach of Linode's customer service portal http://arstechnica.com/business/2012/03/bitcoins-worth-228000-stolen-from-customers-of-hacked-webhost/ http://arstechnica.com/business/2012/03/bitcoins-worth-22800... Today's attack is some kind of unspecified or unknown breach involving Linode manager. I guess the obvious commonality here is that all the attacks target the "soft" Linode layers AROUND managing deploys of Linux and Xen/KVM/UML rather than the "hard" targets of those widely used systems. This also happens to be the layer where Linode should be adding value (as opposed to the cheaper VPS providers out there) and I think it's increasingly troublesome that they continue to have such severe security issues. Is this company (CEO - Christopher Aker) not investing in security staff, security training, best practices etc, or are they investing tons and just getting breached because they host so many sites? Unclear. But it's easy to imagine it's the former, from the outside, given all these incidents.
- ryanlol 11y ago
  Actually the 2013 hack was caused by linode running blatantly misconfigured CF installations, like doing stuff that the manual has big warnings about.
  - sarciszewski 11y ago
    For those not in the know, ryanlol was one of the people on the team involved in the 2013 hacks.
    - throway-not-909 11y ago
      Really? Thomas Asaro told us they were all in jail.
      - ryanlol 11y ago
technion 11y ago
MySQL server that allowed anonymous logins Has anyone got more information on this? Various Google searches keep pointing me at the other four hacks.
- ryanlol 11y ago
  https://blog.linode.com/2014/01/19/an-old-system-and-a-swat-team/ https://blog.linode.com/2014/01/19/an-old-system-and-a-swat-... They try to downplay it pretty hard.
  - technion 11y ago
    Thanks for this link. Although it talks about: database accessed using old forum credentials So I'm not sure "anonymous login" would be an entirely accurate description.
    - ryanlol 11y ago
      The server would in fact accept any credentials as it had been started with --skip-grant-tables. Tested it myself.
    - deleted 11y ago
      [deleted]

wiremine 11y ago

The page was slashdotted for me, here's a copy: "Effective immediately, Linode Manager passwords have been expired. You will be prompted to set a new password on your next login. We regret this inconvenience, however this is a necessary precaution. "A security investigation into the unauthorized login of three accounts has led us to the discovery of two Linode.com user credentials on an external machine. This implies user credentials could have been read from our database, either offline or on, at some point. The user table contains usernames, email addresses, securely hashed passwords and encrypted two-factor seeds. The resetting of your password will invalidate the old credentials. "This may have contributed to the unauthorized access of the three Linode customer accounts mentioned above, which were logged into via manager.linode.com. The affected customers were notified immediately. We have found no other evidence of access to Linode infrastructure, including host machines and virtual machine data. "The entire Linode team has been working around the clock to address both this issue and the ongoing DDoS attacks. We’ve retained a well-known third-party security firm to aid in our investigation. Multiple Federal law enforcement authorities are also investigating and have cases open for both issues. When the thorough investigation is complete, we will share an update on the findings. "You may be wondering if the same person or group is behind these malicious acts. We are wondering the same thing. At this point we have no information about who is behind either issue. We have not been contacted by anyone taking accountability or making demands. The acts may be related and they may not be. "The security of your data, the functionality of your servers, and your confidence in Linode are extremely important to all of us. While we feel victimized ourselves, we understand it is our responsibility, and our privilege as your host, to provide the best possible security and service. You can help further enhance the security of your account by always using strong passwords, enabling two-factor authentication, and never using the same password at multiple services. "We sincerely apologize for the recent disruptions in your Linode service. Thank you for your patience, understanding and ongoing trust in Linode."

madaxe_again 11y ago
Sadly I trust them as far as I can throw them, and we moved everything important from them to AWS a few years back. We had left a few static sites there but after the shenanigans over the holidays, we're moving our remaining stuff. Sad. We spent >$10k/month with them for a while, before their shit started falling apart. They didn't appear to care at all when we left, so I suppose they have an awful lot of large customers who just don't mind their stack disintegrating beneath them every sixty days at most.

broknbottle 11y ago

Wow, these guys can't get a break.

JshWright 11y ago
It's not as if this is just bad luck...
threeseed 11y ago
You make your own luck. If they can't be bothered to invest in their tools and processes then this is the sort of thing that happens.
- workitout 11y ago
  Yeah it's not like ColdFusion powered web sites enabling billions of dollars in commerce and what with all other web development tools being free of security vulnerabilities and all.
  - brandon272 11y ago
    The merits of ColdFusion are irrelevant. The fact of the matter is that their software has continuously been breached.
fweespeech 11y ago
They've been using a CF stack that is fundamentally broken from the foundation up for years and was aware of it. Its part of why they hired someone in July to rebuild it in python.
- workitout 11y ago
  You signed an NDA and have done a full analysis of their code? Cool.
  - fweespeech 11y ago

secstate 11y ago

Sure wish they had sent out an email notification to users instead of a slashdotted blog post. Now the question is how long can Linode stand in the face of these sorts of hacks and network attacks in the face of stiff VPS competition.

threeseed 11y ago
Be grateful they posted something to Slashdot at all. Back in 2013 they didn't do that for DAYS after the event happened and someone else posted what happened.
chibosempai 11y ago
(Linode Employee) Already got it covered, we are sending out an email to everyone in batches, but pushed out the blog first since it can be seen by everyone right away.
- dnm 11y ago
  I'm still waiting for mine (6 hours since this was posted to HN).
  - Keverw 11y ago
    I'm still waiting on an email too.
- dfoxinator 11y ago
  How come, still, no one I know, including myself, has received this email?

qmr 11y ago

Blog is down. Maybe they should host it on WPEngine. (: This sort of thing almost became background noise when I worked there. I can say that things improved somewhat after the HTP compromise, at least for awhile.

throwaway909009 11y ago

I'm fairly confident that Linode has been compromised since July, if not earlier. PagerDuty moved off of Linode after an incident in July. We've been under strict gag orders from legal about that incident until today when Linode finally announced their compromise. Really, the only way I can see that this attacker could have gotten in the way they did (they logged into our Linode Manager account on the first try using a username that wasn't used anywhere but in Linode Manager, using valid 2fa and valid password) was if they had access to the Linode Manager database. I'm pretty sure that the recent WP Engine compromise was achieved through the same attack vector, too

masterleep 11y ago
If the Linode Manager database only stores the hash, how would they know the password? How did you find out about the illicit login?
- ceejayoz 11y ago
  If they've been more compromised than something like a simple SQL injection vulnerability, it could be something like added code to log passwords or post them off somewhere else.
- stcredzero 11y ago
  If the Linode Manager database only stores the hash, how would they know the password? Weak password? Weak hashing algorithm?
- Someone1234 11y ago
  > If the Linode Manager database only stores the hash, how would they know the password? Hashes can be turned back into plain text, it is just computationally expensive to do so. Hashing only slows down an attack (and or increases the cost), it doesn't not mitigate one. In particular if the hashes aren't salted then a rainbow table is an extremely effective way of breaking all of the hashes concurrently. The main method of doing so: Generate the hash for every combination of typable characters up to a given length (e.g. MD5() A-Za-z0-9 & specials up to 8x characters). This can be mitigated using a more computationally expensive hashing routine (or increasing the work factor on a less computationally expensive one) and salts. But given enough time OR computing power, all hashes will be broken. AWS makes breaking hashes a lot cheaper as you can bid on spare capacity and perform the operations relatively cheaply.

dmuth 11y ago

Looks like the reason their blog is down is because... it's now being targeted by a DoS: http://status.linode.com/incidents/kldhjpjnfnkj http://status.linode.com/incidents/kldhjpjnfnkj Attacking a blog talking about the hack? It sure seems that someone has a grudge against Linode. :-/

Mahn 11y ago
At this point I'm starting to wonder whether this isn't a competitor putting their investors money to work. It's otherwise utterly bizzare that someone would be so obsessive in damaging Linode. I really hope they make the details of the investigation public...
- Someone1234 11y ago
  It makes more sense than you might imagine... Linode, Github, Stackoverflow, Imgur, they've all been targeted. But what do they have in common? In a word: popularity. The core reason these sites are targeted is because it is impressive to others. The source of this is typically two fold: - For the lolz. Someone with a botnet just wants to show off, taking down something known gives them more notoriety. - For a sales pitch. Someone has botnet capacity that they want to sell, and "I took down Linode" is a great way of demoing that to potential buyers. The first one is more common when someone finds a new method of traffic multiplication and just wants to show it off (i.e. they trick a third party into DoS/DDoSing a target). The second one is legitimate criminal enterprises.
  - Buge 11y ago
    Github was also targeted by China to try to get them to delete certain repos that were unfriendly to China. Some people also saw it as showing off their power.
- vertis 11y ago
  Based on the number of negative reviews on glassdoor, it could also be a former employee.

TheSwordsman 11y ago

I'm glad to see that this information has now been publicly disclosed. In July 2015, we suffered a compromise at PagerDuty via the Linode Manager. I hope that we can provide a bit more of an official in-depth post-mortem of our compromise, but I'd be happy to disclose some of the details here. Using the access gained within the Linode Manager, the attacker reset the root password on a few systems, and used Lish to gain root access. We were alerted to this activity and fully revoked the attacker's access within 60 minutes of the first node being compromised. Working with Linode support, we discovered which user account was being used and completely deactivated the user. We also isolated the VMs, and performed forensics on read-only copies of their disk images. In our situation the attacker knew one of our user's passwords and MFA secret. This allowed them to provide valid authentication credentials for an account in the Linode Manager. It's worth noting that all of our active user accounts had two-factor authentication enabled. An interesting data point was that the user who had their account compromised was no longer in possession of the MFA secret themselves. Their cell phone had been reset (thus deleting all data) 8 months prior. The user could not log in to the Linode Manager if they wanted, so it was our determination that the key could not have been obtained from the user and was more likely on Linode's side. We also have evidence from access logs provided by Linode that the attackers tried to authenticate as an ex-employee, whose username ONLY existed in the Linode database. It was absolutely unique and was not used elsewhere by the employee making the username an accidental honeypot. This was another piece of data supporting that Linode was the source of our compromise. We immediately reached out to them not only to inform them of their compromise, but to assist them in investigating it. We were confident that the Linode database had been breached, and that the secret key used to encrypt information in the database had been compromised as well. In addition to reaching out to Linode, we also worked with a third-party security firm to audit our work done during the incident. Likewise, around the same time we reached out to law enforcement to assist in investigating the attack. I believe our public disclosure includes this information[1]. This was in the middle of July 2015. We did not get confirmation in July that there was a breach of the Linode Manager or any associated credentials. In the end, we migrated away from Linode because of this breach (even before it was publicly disclosed) in Aug 2015. We also never were able to confidently disclose that Linode was the vector due to lack of confirmation from their end. While all of us who responded to the incident were confident they were the source, we now thankfully have the data to confirm it. [1] https://www.pagerduty.com/blog/july-2015-security-announcement/ https://www.pagerduty.com/blog/july-2015-security-announceme...

deleted 11y ago

[deleted]

msh 11y ago

Why the hell have they not emailed their customers about this! This is not the kind of thing I want to learn from HN.

dangrossman 11y ago
They're doing so as we speak. It takes a while to send 400,000 mails if you actually want them to be delivered to inboxes.
- chucky_z 11y ago
  400k emails isn't that many though. :(
  - dwightgunning 11y ago
    I haven't received any email. It's over 12 hours since the notice went up on status.linode.com and the Twitter account.
cakes 11y ago
Agreed - especially having read this earlier (not being able to fix it at the time), still not getting the email (like 6 hours have gone by) and I'm resetting because I should and still no communication from Linode. Concerning since I had to reset my password and regen my 2fa using only my old password...

dnm 11y ago

And of course there's the ongoing ddos attack that is preventing me from reaching manager.linode.com...

wtbob 11y ago

This is Yet Another Reminder to use unique, unguessable, unmemorable passwords for all online services. It's a question of when, not if, any particular password database will be compromised. While 'password1' and 'this is my long unguessable password' and 'm4r1g0ld' and 'false lemur capacitor paperclip' will all eventually be guessed, 'sF0PSQMwK85fe9xanqJRm9nty9cJGHJsVmti' will never, ever be.

wtbob 11y ago
Why was this downvoted? It's true. The example password is 36 characters; a mixed-case-and-numbers alphabet has just over 6 bit per character; the example password is thus 217 bits. It's not going to be guessed.
ryanlol 11y ago
Except passwords aren't even a relevant part of this breach. They're worth absolutely nothing compared to the ridiculous amounts of customer data linode controls.

tptacek 11y ago

Don't use Linode.

workitout 11y ago
Don't lecture me.

reconbot 11y ago

Not to throw fuel on the fire but grumbling ex-employees aren't surprised. I've heard a few stories about not investing in infrastructure or even bug fixes. "They wont even look at bug fixes with major support costs." I know it's hard to triage what to do when running a company but when employees are proud of their product a breach like this isn't surprising.

sarciszewski 11y ago

The actual answer is much more sinister than that. Which is kind of hilarious.

lawnchair_larry 11y ago
What is the actual answer?
- sarciszewski 11y ago
  Last I heard, compromising Bitcoin exchanges for lulz and mad profit. Allegedly. So if the rumor is true, he wasn't technically lying.
  - jacquesm 11y ago
    I don't think that if all you have is 'allegedly' it is up to you to make such accusations here in public.
    - sarciszewski 11y ago
      This is stuff he shares openly on IRC. In public channels. Whether any of it is true or not remains to be seen. He's not ever given me any reason to believe he's lying, however.
dang 11y ago
That crosses into personal attack, which is not ok, even if it's just insinuated. We detached this subthread from https://news.ycombinator.com/item?id=10847715 https://news.ycombinator.com/item?id=10847715 and marked it off-topic.
- sarciszewski 11y ago
  It's not a personal attack, it's something that ryanlol publicly claims in ##security on freenode FFS. If it wasn't, I wouldn't have referenced it From what I gather, he thinks law enforcement is hilarious.

relaxatorium 11y ago

and now getting server errors & timeouts when I try to get to the manager to change my passwords. this has been a hellaciously thorough attack.

tyingq 11y ago

For those not keeping track, that's the third major breach I've heard of where user data was exposed. Previous Incidents: April, 2013: https://blog.linode.com/2013/04/16/security-incident-update/ https://blog.linode.com/2013/04/16/security-incident-update/ January 2014: https://blog.linode.com/2014/01/19/an-old-system-and-a-swat-team/ https://blog.linode.com/2014/01/19/an-old-system-and-a-swat-... (they mention the breach almost in passing...the swatting is unrelated) I really want Linode to do well, but this may be it for me.

codezero 11y ago

Glad they posted it. I just logged into one of my linode boxes that I don't use for very much but keep around, and it was rooted... doh.

Ronsenshi 11y ago
Could be just a coincidence if you keep SSH open on standard port. SSH bots don't sleep.
- codezero 11y ago
  The host was only accessible via private key or Linode's LISH shell. That's what seems most suspicious. There is some minor evidence remaining in the .bash_history that is curious.
  - maaarghk 11y ago
    I'd be interested in a write-up about anything you find in that .bash_history or logs, would you consider writing one?
    - codezero 11y ago
      Here's the main bulk of the obviously bad things: 3 ls 4 ls -al 5 chown syslog auth.log 6 ls -al 7 chown syslog kern.log 8 ls -al 9 chown syslog syslog 10 ls -al 11 echo -n '' > /media/xvda/root/.bash_history 12 echo -n '' > /root/.bash_history 13 echo -n '' > /root/.viminfo 14 L=$(find /var/log -type f); for F in $L; do echo -n '' > $F; done 15 rm -rf /etc/ssh/*_key* #remove host keys 16 rm -rf /var/lib/dhcp/* # dhcp leases 17 echo "echo 'options rotate' >> /etc/resolv.conf" > /etc/dhcp/dhclient-exit-hooks.d/rotate 18 ls 19 ls -al /var/log 20 ls 21 ls -al /var/log 22 exit 23 ls 24 ls -al /var/log 27 adduser in 28 su - in 29 vi /etc/sudoers 30 vi /etc/gro 31 vi /etc/group 32 groupadd --help 33 groups 34 groupmod 35 groupadd --help 36 vi /etc/group 37 su - in

superuser2 11y ago

With Linode's extended DDoS I have not been able to get into the Manager for a couple weeks. I'd really like to cancel my account with them (and they do keep billing) but I don't appear to have any tools short of a chargeback. Anyone else in the same boat?

mapunk 11y ago
You don't mention contacting support. Shouldn't that be your second avenue after trying to access the account dashboard?
- superuser2 11y ago
  I have always contacted support by opening a ticket in the Manager. The main website is also down. Is there an out-of-band way to reach them?
  - IKnowComputer 11y ago
    Phone? 855-454-6633 (+1-609-380-7100)
stephen223 11y ago
Why not call them on the phone? They have a toll-free number for support.
dwightgunning 11y ago
The main website is down for me and has been flakey but http://manage.linode.com http://manage.linode.com hasn't been a problem for me over the last few weeks.

aenglander 11y ago

This is a perfect poster child for decentralized authentication. Centralized stores of passwords and secrets are just asking for trouble. When will we learn?

lsc 11y ago
so... I completely agree that shared secret authentication is a bad idea, and I use public key authentication wherever I can (password auth is disabled for ssh on every server I control; I do everything with ssh public keys.) However, I've yet to set a public key authentication scheme that users would find acceptable for web applications. Do you really expect all users to setup x.509 auth in the browser? What is your public key solution to authenticate the web-applications that customers demand?

cylo 11y ago

So, outside of the major cloud providers, what are the good alternatives?

peteretep 11y ago
I like Bytemark.
tyingq 11y ago
VPS providers that are typically pitched as Linode Alternatives. Not 1:1 equivalents, but may work depending on your needs. Vultr, Ramnode, Wable, iWStack Or, depending on the number of VPS's you have, you might like: - Aliyun, the cloud service from Alibaba - A dedicated server from OVH, or their mid-tier brand, SoYouStart. This is my personal favorite. They have real DDOS protection, Data centers in North America and Europe, reasonable web interfaces, lots of available IPV4 space, and DIRT CHEAP prices. Run proxmox as the distro, and you get a decent interface to create and manage VPS instances. The most important piece would be to try and split instances across at least two of these providers so that you have some fast recourse if something goes wrong. For the things I'm running, doing a nightly rsync of the data from one provider to another suffices as reasonable insurance.

SuperKlaus 11y ago

Not the first time they poorly handled a security issue: https://blog.linode.com/2014/01/19/an-old-system-and-a-swat-team/ https://blog.linode.com/2014/01/19/an-old-system-and-a-swat-...

dfoxinator 11y ago
There are also a number of reviews on Glassdoor where supposed Linode employees say they were asked by management to lie to customers about security breaches.
- throway-not-909 11y ago
  We were.

dantiberian 11y ago

I enabled two factor on my Linode account recently and saw this worrying copy: > If you lose your token and get locked out of the Linode Manager, email support@linode.com to regain access to your account. > Should you need us to disable your Two-Factor Authentication, the following information is required: > An image of the front and back of the payment card on file, which clearly shows both the last 6 digits and owner of the card. > An image of the front and back of the matching government-issued photo ID. There doesn't seem to be a way to say "I have my big boy pants on, don't let anyone in under any circumstances". This is the first 2FA setup I've seen that still allows bypass by contacting support. EDIT: I also find it odd that you have to manually generate a scratch code, and they don't automatically generate it for you. Again, all of the other 2FA setups I've gone through have done this.

pfg 11y ago
> There doesn't seem to be a way to say "I have my big boy pants on, don't let anyone in under any circumstances". This is the first 2FA setup I've seen that still allows bypass by contacting support. Sadly, this is quite common, especially with organizations that provide phone support[1]. [1]: http://krebsonsecurity.com/2015/12/2016-reality-lazy-authentication-still-the-norm/ http://krebsonsecurity.com/2015/12/2016-reality-lazy-authent...

tlrobinson 11y ago

Comments from Linode employees, Linode customers who were hacked, and a guy who hacked Linode (different incident) I love HN.

jafingi 11y ago

That's it. I'm done with Linode. I will be keeping one server on Linode, but my new startup where I need several load balancers, Web servers, DB replication will be hosted elsewhere. Do you guys know any good providers? Not AWS.

laseeey 11y ago
Can I ask why not AWS?
- jafingi 11y ago
  I just don't think the performance is good enough for pure web hosting. Much slower/$ than Linode and DO. Their uptime and infrastructure is great, but you're totally on your own (e.g. if you're getting DDoS'ed). Also their traffic cost is not as easy to predict than DO/Linode/Vultr.
pfg 11y ago
OVH is hard to beat when it comes to raw performance/$. Hardware and network has been quite reliable for me (using SoYouStart and OVH), and their DDoS filter is excellent. I'm not really familiar with their cloud/VPS offering. That being said, they only have one data center in NA at the moment, and they recently suffered a rather nasty network outage in that data center. Would definitely look into them if your target market is primarily EU, but if you're mostly NA-based, YMMV.

linodemaster 11y ago

Hello, my name is Julius Kivimaki. I am the one who hacked Linode. You guys are all stupid if you think you can catch me. I say, CATCH ME IF YOU CAN, or GTFO! LULZ!

linodemaster 11y ago

Hello, my name is Julius Kivimaki. I am the one who hacked Linode. You guys are all stupid if you think you can catch me. I say, CATCH ME IF YOU CAN, or GTFO! LULZ!

linodemaster 11y ago

Hello, my name is Julius Kivimaki. I am the one who hacked Linode. You guys are all stupid if you think you can catch me. I say, CATCH ME IF YOU CAN, or GTFO! LULZ!