20 ms·

Linode Manager Security Incident

186 points by ErneX 15y ago

glfomfn 15y ago
I am failing to understand what exactly happened. The user who was affected by the incident quoted an email from linode that stated "Our investigation has revealed a customer support interface was used to access your account.", based on that and all the information of that post you get the impression that through the 'interface' the attacker was able to change the vps root password. Now a reply from linode comes and says "The portal does not have access to credit card information or Linode Manager user passwords". So if the portal doesn't have access to Linode Manager how the attacker gained ability to change the root passwords ? Thy should give more details on the incident, i do have a certain trust in the ability of linode to have a secure environment & i can understand that things like that will happen at some point to everyone. However its one thing for someone to get access in your system because you had your roots password to 'password' and another if there was a bug that got exploited.(yea this is an extreme example)
- citricsquid 15y ago
  > So if the portal doesn't have access to Linode Manager They didn't say that, they said it doesn't have access to the passwords. They have an interface to change details, they just can't read them. So they can reset your password to "hunter2" but they can't see if it's "hunter2".
  - glfomfn 15y ago
    You are right, my bad on that. Still this looks like a Public relations post by them than giving out facts. They should be explaining what the attacker could do by gaining access on that interface, the ability of the attacker to change the password has the same consequences. The point is that exploited interface had a backdoor access to the virtual machines (to be able to change passwords or w/e)
    - nknight 15y ago
      I understand how this might be confusing to a third party, but Linode's response thus far makes perfect sense to those of us who have been customers for a while. We're pretty aware of the general parameters of Linode's internal systems.
- dissident 15y ago
  It sounds like Linode's user system is different from their server management system. They probably have some administration tools for resetting VPS passwords, but don't have access to sensitive user details.
- keeran 15y ago
  Likewise. Suspicious events prompted an immediate investigation and the compromised credentials used by this intruder were then restricted Does that mean the credentials were gained outside of Linode and used to change the root passwords of the accounts for the purposes of the theft, or were those credentials used as part of an exploit in Linodes systems?
- dangrossman 15y ago
  I run a web app. I built an administrative interface for managing it. This interface includes the ability to log me in to any individual user's account (by appropriately initializing a session with that user's ID, the same as logging in normally would do), and to reset any individual user's password. This interface does not let me view any user's passwords; it's both technically impossible, as I don't store them in plain text or encrypted form, and unnecessary. Unless you believe they're lying, then Linode has the same thing. Some interface where they can access their users' Linode Manager accounts, but that interface does not show Linode Manager passwords or credit card information.
frsandstone 15y ago
Well handled.
- mrschwabe 15y ago
  Agreed. This is in sharp contrast to the Media Temple / Plesk debacle. Which affected how many thousands of sites? How long did it take them to admit it? Meanwhile, Linode reports full details of a hack affecting no more than 8 accounts the SAME day.
  - ErneX 15y ago
    It's good the fast acknowledgement, but I find it lacks in detail. I'd love to know exactly what kind of measures are they going to take so the chances of this happening again are truly reduced. I share a Linode with a friend and at work we deal with 4 so this is a concern.
    - Legion 15y ago
      Further comment is indeed warranted, but that part can come in a few days, when there's more clarity on the scope of the incident and a fully-hashed plan of attack for closing the vulnerability.
      - blakeperdue 15y ago
        I hope so. They said they will be "reviewing our policies and procedures to prevent this from ever recurring." That doesn't imply they will be posting a public update. As a Linode customer, I would like to know details of how this breach occurred and what actions will be taken to prevent it in the future.
  - unseen 15y ago
    Full details? It's as few details as they can possibly report. I guess thats the actual underlying problem here. They don't want to give all the details because then it would quickly become apparent that their security system and protocols for managing hundreds of servers was extremely lackluster.
Lazare 15y ago
There's already quite a lot of discussion of this incident here: http://news.ycombinator.com/item?id=3654110 http://news.ycombinator.com/item?id=3654110
deleted 15y ago
[deleted]
orofino 15y ago
I'd like to understand the what actions will be taken to prevent similar attacks in the future. Also, what can I as a linode customer to prevent my host from being compromised in a similar fashion. Implementation of two factor authentication for your customers and requiring it for a root password reset would go a ways to preventing similar attacks.
- eli 15y ago
  Two-factor authentication for their support personnel you mean?
  - brown9-2 15y ago
    Also, is this customer service portal available via a public URL? Shouldn't some sort of VPN access be required to even get on the network hosting these things?
    - ErneX 15y ago
      It is public yes, this is it: http://manager.linode.com/ http://manager.linode.com/
      - eli 15y ago
        I think there is some confusion here. That's the link to the customer portal, which needs to be pretty public. I have no idea how Linode's support team access accounts, but I would hope it is less public.
        ErneX 15y ago
        The title of their blog post is: "Linode Manager Security Incident" and that's exactly the name of the customer website where you can manage your instance, billing, etc. I think someone found a way to gain access to any Linode customer account through the customer website and from there shut down the instance, changed root password and rebooted (you can do that from there).
strags 15y ago
While the incident is unfortunate, I do give them credit for being up-front, honest, and relatively speedy in their response. Sad to say, I'm not sure a lot of other hosting providers would be as quick to admit culpability. That said, there's no way that resetting the root password should be something a customer service rep can do. Particularly when Linode are very explicit about not being a service for newbs - and are in general unwilling to provide help setting up a Linux system.
amalag 15y ago
No mention of liability. So they are fixing it, but someone is out of $12k of bitcoins. Linode says, tough luck for trusting us with your valuables.
- techiferous 15y ago
  Best thing to do would be for Linode to generously reimburse these people, but they have no obligation to do so. As a policy, it would neither be good business nor appropriate for Linode to assume all of the risk in a situation like this. Also, if Linode did put a policy in place to assume some of the risk (some sort of insurance policy) they open themselves up to scams (just get your friend to rob your bitcoins and cash in on Linode's good will insurance policy).
  - iamd 15y ago
    I disagree. I think the best thing for them to do is to let the Bitcoin user eat the loss. Whenever you're in this sort of shared environment, be it Linode or EC2 or anything else, you open yourself up to risks, and it's your responsibility to address them. Bitcoin is adopted only by fairly sophisticated geeks, who really should know better. If Linode uses my monthly fees to reimburse this fellow, I'll snap-cancel my account, because that's not what I pay for. I want them to spend on staff and equipment, not reimbursing some incompetent speculators who suffered bad results because of their lousy judgment.
  - larrys 15y ago
    "Best thing to do would be for Linode to generously reimburse these people" Creates a precedent and greater future liability.
    - techiferous 15y ago
      Good point.
  - plox 15y ago
    The scam you mention would be difficult to pull off as there would have to be evidence that the account was compromised on Linode's end for the customer to make a claim.
look_lookatme 15y ago
I'm surprised that a tool with this level of access isn't walled off behind a VPN.
rdl 15y ago
I really hope the various bitcoin related incidents don't poison the well for online currencies in general. Yes, you need a greater level of security when dealing with transferrable, relatively anonymous or pseudonymous (and rapidly extractable) online assets that you don't need for book-entry accounting with an audit trail and reversible transactions (credit cards, ACH, etc.). Yes, this is beyond what even most banks currently use. No, it's not beyond current technological state of the art. Gaming (i.e. casinos), at least some of them, do a reasonable job with some very similar security problems.
- wmf 15y ago
  Wasn't the well pretty thoroughly poisoned by e-Gold and such? (We realize they're not the same as Bitcoin, but most don't.)
  - rdl 15y ago
    Yeah, kinda. Although E-Gold's problem was force majeure, not technical (did they ever get hacked? people stole passwords all the time, but since it was double entry, it was pretty easy to reverse). osGold, etc. were probably better parallels. But yes, this whole field has had a long series of technical, business model, governance, and government related failures.
    - StavrosK 15y ago
      Can you explain what happened with eGold, osGold, etc? I haven't heard anything about it.
      - rdl 15y ago
        e-gold was shut down by the government for running an unlicensed money service business. They were officially Nevis based, but actually run out of Melbourne, FL (where they started originally); criminal charges were involved. The main reason was a particularly motivated federal prosecutor and the specific uses e-Gold users found -- child porn, paying for stolen cc's, etc. It wasn't a general thing about PATRIOT/MSB laws. osGold was just an internal fraud; digital currency, give me your (gold) and issue tokens -- then one day it disappeared, taking the original value with it. There were hack attacks by third parties against some other gold currency systems at the time, but I don't remember the details.
shazow 15y ago
Perhaps people affected by the many Bitcoin Exchange password leaks were reusing their passwords on Linode?
- frankydp 15y ago
  The breach originated on the Lionode customer service system. Which the attacker used to reset user passwords. Unlikely the incidents are directly related.
nwmcsween 15y ago
Can someone with a legal background comment on the viability of the victims taking legal action?
- keypusher 15y ago
  I've got to wonder too. If you rent out an apartment and someone breaks in with a key they stole from the landlord and takes your TV, can you sue the landlord? On the other hand, if you rent a security deposit box from a bank and it gets broken into, is the bank liable?
  - mindstab 15y ago
    It all comes down to insurance. No one generally pays out of their pocket, they pay with insurance money. The reason you pay so much at a bank for a tiny box, where as, space wise, you pay a fraction for your appartment, or essentially change for a storage locker, is that the bank expects you to put valualbles in that space and so pays lots of that money to inssurance so if they do get robbed, they are covered and can pay you. The other places... dont. And usually have clauses in their contract saying as much: "We aren't insured for valueables, so don't store them here, it may not be safe and we won't (and can't) pay you back if they get stolen. Linode falls into the second category, cheap, uninsurred storage space. You want storage space insurred against digital robbery, a) good luck b) expect to pay a lot more than $20 or $30 a month
  - mbreese 15y ago
    I think the threshold would be, did Linode take "reasonable" precautions in protecting the servers in question. Just like the landlord or bank. So long as they take reasonable precautions, they can't be held liable. You can expect for your landlord to keep their copy of the key in some sort of lockbox, but you aren't going to expect them to keep it in a pressure-sensitive safe, guarded by movie-style lasers and a German Shepard. You expect a little more security out of your bank. The real question is: why were services that act like bitcoin banks storing their coins on Linode in the first place.
    - 15y ago
jostmey 15y ago
Hey, I'm one of those guys! But I wasn't stupid enough to leave a stockpile of bitcoins sitting around to steal.
- olefoo 15y ago
  I wonder about that. It seems to me that the bitcoin wallet shouldn't have been accessible after reboot, at least until someone came by to give the agent managing it the passphrase that would allow it to decrypt it's state. From the sound of things the wallet was just laying around unencrypted?
  - ceol 15y ago
    https://bitcointalk.org/index.php?topic=66979.msg778666#msg778666 https://bitcointalk.org/index.php?topic=66979.msg778666#msg7... According to the admin of bitcoinica, he didn't have the wallet encrypted because the ruby gem he was using to process withdrawals didn't support it.
    - ashconnor 15y ago
      Doesn't even matter if the wallet was encrypted, he'd still have to have the password somewhere on the server.
      - lmm 15y ago
        Not if he entered it manually on every reboot. There'd still be attacks around snooping the contents of the wallet from memory, but the ability to do a root password reset that involves rebooting the box wouldn't be enough (and it seems like that may have been all the access that the attackers here had).
        deleted 15y ago
        [deleted]
    - AdamGibbins 15y ago
meow 15y ago
May be its time for some cloud insurance (covering outages/security etc) ? I don't mind paying a little extra if it mitigates risk.
- tptacek 15y ago
  Near unlimited liability and virtually zero ability to model the likelihood of events. A hard product to design.
  - meow 15y ago
    Liability might be unlimited, but most of the insurance products for other categories cap it based on premium. So that may not be much of a problem. I agree with the difficulty of modelling the events though... they need more data for any kind of modelling.. I'm guessing amazon already has some kind of insurance to cover its risks under AWS service agreement..
videoappeal 15y ago
All this talk about banks being safe yada yada and cloud hosting not safe for US50k. Real banking companies (with billions of dollars on hand) do use commodity cloud hosting including Linode, for even sensitive parts. Take for example Natwest online banking login. On initial login page they load a cookie via an image from www.advanced-web-analytics.com and then once you enter a customer number the next page loads a ...drum roll... javascript file from www.omni-traffic.com. Now who can tell me what one can do when you have control over the Javascript on a banking login page? Ah crap. It looks they have been moved to Amazon EC2, ~8 months ago they were hosted on conventional Linode VPSs. Points still stands though.
- tomg 15y ago
  In my experience working on a US financial website, a bank would never consider using a VPS like Linode to store actual banking and customer data. It's not even close to Level 1 PCI compliant.
  - videoappeal 15y ago
    Well tomg, when I researched this ~8 months ago there were at least 2 US financial websites that were using the same specialized analytic company that injected JS into banking login pages that were hosted on Linode VPSes.
    - tomg 15y ago
      Oh I don't doubt you. I'm not an expert on this, heck, I wasn't even allowed on to the actual servers (because of said compliance). I don't know the guidelines for login pages or what kind of security third party JS libs are supposed to have (also PCI is not a law, afaik). What I'm asserting is that the servers that store the actual banking and customer data have very high security standards. It's one thing to store front end website code on a VPS, it's a totally other thing to store your database with customer & bank data on Linode. The bitcoin breach seems analogous to Bank of America storing your account information on Linode and trusting it as the Real Data. Does that make sense?
    - 15y ago
VonLipwig 15y ago
The message seems to be... Only 8 accounts were affected. Do not worry. Minor breach. Not much harm done. It seems to me the truth is the attacker looked for bitcoin wallets and emptied them. The fact he could identify 8 accounts and access them suggests the attacker could have accessed far more accounts if they wished. I think this is the most worrying thing about the breach. I don't really understand how bitcoin works but it seems that people with wallets need to set up multiple wallets on multiple providers and limit the amount of bit coins in each wallet to limit any losses from breaches like this. If I was a linode customer I would be thinking about moving. This message, while fairly open, doesn't give me much confidence there aren't other security issues with the platform.
- mdda 15y ago
  It wasn't clear from the memo, but how did the attacker know which 8 specific accounts had 'references to bitcoin' if they didn't access other accounts too?
  - calloc 15y ago
    Are you able to name your VM's on Linode?
    - ukd1 15y ago
      I doubt it's that: I would have worked out the target sites hosts (looking up the IP), made a list, worked out the most popular by value - probably worked out it was linode and gone from there...once in, I'd bet it's a simple task of searching for the IP in their interface.
      - mdda 15y ago
        Ok, asking the other way around, did Linode check out the non-attacked VPSs to determine whether they didn't have bitcoin wallets on them? I'm not disagreeing with anything else said here - it's just that it seems that the memo was very conclusive in stating something that couldn't be known unless there was more VPS introspection than they claim...
desade 15y ago
In Linode's FAQ, they mention that their virtualization management software was developed in-house. They seem quite proud of this, quipping: "The Linode Manager is custom software, written in house, and is not for sale (although others have tried to mimic it)." It seems to me that this is a classic example of security failures following inevitably from a lack of peer review. Maybe Linode didn't consider its LM software to be peer-reviewable, but I bet the victims of the bitcoin thefts wish that someone else had tested the code (and human systems surrounding it) for vulnerabilities. Is this not exactly what Bruce Schneier frequently points out? Anything that must withstand attacks to protect the valuables within should be tested by attacking it. A lot. My hunch is that the vulnerability exploited by this attacker would have been found and fixed already if the LM software were more open.
- jarito 15y ago
  I think you may be a little off here. The statements in the thread seem to indicate that the compromise was not based on a vulnerability in custom software, but compromised credentials. You can certainly argue that the management console should be protected by two-factor (and it should be), but their software doesn't seem to be at fault here. I would be willing to bet that they have had the system tested by external security contractors and scanned with automated scanning tools. This seems to be a people problem and features problem not a vulnerability problem. I guess we just don't know at this point. You may very well be correct. I guess if you want to use an open source provider, just make sure they are running OpenStack (openstack.org).

186 points by ErneX 15y ago

I am failing to understand what exactly happened. The user who was affected by the incident quoted an email from linode that stated "Our investigation has revealed a customer support interface was used to access your account.", based on that and all the information of that post you get the impression that through the 'interface' the attacker was able to change the vps root password. Now a reply from linode comes and says "The portal does not have access to credit card information or Linode Manager user passwords". So if the portal doesn't have access to Linode Manager how the attacker gained ability to change the root passwords ? Thy should give more details on the incident, i do have a certain trust in the ability of linode to have a secure environment & i can understand that things like that will happen at some point to everyone. However its one thing for someone to get access in your system because you had your roots password to 'password' and another if there was a bug that got exploited.(yea this is an extreme example)

citricsquid 15y ago
> So if the portal doesn't have access to Linode Manager They didn't say that, they said it doesn't have access to the passwords. They have an interface to change details, they just can't read them. So they can reset your password to "hunter2" but they can't see if it's "hunter2".
- glfomfn 15y ago
  You are right, my bad on that. Still this looks like a Public relations post by them than giving out facts. They should be explaining what the attacker could do by gaining access on that interface, the ability of the attacker to change the password has the same consequences. The point is that exploited interface had a backdoor access to the virtual machines (to be able to change passwords or w/e)
  - nknight 15y ago
    I understand how this might be confusing to a third party, but Linode's response thus far makes perfect sense to those of us who have been customers for a while. We're pretty aware of the general parameters of Linode's internal systems.
dissident 15y ago
It sounds like Linode's user system is different from their server management system. They probably have some administration tools for resetting VPS passwords, but don't have access to sensitive user details.
keeran 15y ago
Likewise. Suspicious events prompted an immediate investigation and the compromised credentials used by this intruder were then restricted Does that mean the credentials were gained outside of Linode and used to change the root passwords of the accounts for the purposes of the theft, or were those credentials used as part of an exploit in Linodes systems?
dangrossman 15y ago
I run a web app. I built an administrative interface for managing it. This interface includes the ability to log me in to any individual user's account (by appropriately initializing a session with that user's ID, the same as logging in normally would do), and to reset any individual user's password. This interface does not let me view any user's passwords; it's both technically impossible, as I don't store them in plain text or encrypted form, and unnecessary. Unless you believe they're lying, then Linode has the same thing. Some interface where they can access their users' Linode Manager accounts, but that interface does not show Linode Manager passwords or credit card information.

frsandstone 15y ago

Well handled.

mrschwabe 15y ago
Agreed. This is in sharp contrast to the Media Temple / Plesk debacle. Which affected how many thousands of sites? How long did it take them to admit it? Meanwhile, Linode reports full details of a hack affecting no more than 8 accounts the SAME day.
- ErneX 15y ago
  It's good the fast acknowledgement, but I find it lacks in detail. I'd love to know exactly what kind of measures are they going to take so the chances of this happening again are truly reduced. I share a Linode with a friend and at work we deal with 4 so this is a concern.
  - Legion 15y ago
    Further comment is indeed warranted, but that part can come in a few days, when there's more clarity on the scope of the incident and a fully-hashed plan of attack for closing the vulnerability.
    - blakeperdue 15y ago
      I hope so. They said they will be "reviewing our policies and procedures to prevent this from ever recurring." That doesn't imply they will be posting a public update. As a Linode customer, I would like to know details of how this breach occurred and what actions will be taken to prevent it in the future.
- unseen 15y ago
  Full details? It's as few details as they can possibly report. I guess thats the actual underlying problem here. They don't want to give all the details because then it would quickly become apparent that their security system and protocols for managing hundreds of servers was extremely lackluster.

Lazare 15y ago

There's already quite a lot of discussion of this incident here: http://news.ycombinator.com/item?id=3654110 http://news.ycombinator.com/item?id=3654110

deleted 15y ago

[deleted]

orofino 15y ago

I'd like to understand the what actions will be taken to prevent similar attacks in the future. Also, what can I as a linode customer to prevent my host from being compromised in a similar fashion. Implementation of two factor authentication for your customers and requiring it for a root password reset would go a ways to preventing similar attacks.

eli 15y ago
Two-factor authentication for their support personnel you mean?
- brown9-2 15y ago
  Also, is this customer service portal available via a public URL? Shouldn't some sort of VPN access be required to even get on the network hosting these things?
  - ErneX 15y ago
    It is public yes, this is it: http://manager.linode.com/ http://manager.linode.com/
    - eli 15y ago
      I think there is some confusion here. That's the link to the customer portal, which needs to be pretty public. I have no idea how Linode's support team access accounts, but I would hope it is less public.
      - ErneX 15y ago
        The title of their blog post is: "Linode Manager Security Incident" and that's exactly the name of the customer website where you can manage your instance, billing, etc. I think someone found a way to gain access to any Linode customer account through the customer website and from there shut down the instance, changed root password and rebooted (you can do that from there).

strags 15y ago

While the incident is unfortunate, I do give them credit for being up-front, honest, and relatively speedy in their response. Sad to say, I'm not sure a lot of other hosting providers would be as quick to admit culpability. That said, there's no way that resetting the root password should be something a customer service rep can do. Particularly when Linode are very explicit about not being a service for newbs - and are in general unwilling to provide help setting up a Linux system.

amalag 15y ago

No mention of liability. So they are fixing it, but someone is out of $12k of bitcoins. Linode says, tough luck for trusting us with your valuables.

techiferous 15y ago
Best thing to do would be for Linode to generously reimburse these people, but they have no obligation to do so. As a policy, it would neither be good business nor appropriate for Linode to assume all of the risk in a situation like this. Also, if Linode did put a policy in place to assume some of the risk (some sort of insurance policy) they open themselves up to scams (just get your friend to rob your bitcoins and cash in on Linode's good will insurance policy).
- iamd 15y ago
  I disagree. I think the best thing for them to do is to let the Bitcoin user eat the loss. Whenever you're in this sort of shared environment, be it Linode or EC2 or anything else, you open yourself up to risks, and it's your responsibility to address them. Bitcoin is adopted only by fairly sophisticated geeks, who really should know better. If Linode uses my monthly fees to reimburse this fellow, I'll snap-cancel my account, because that's not what I pay for. I want them to spend on staff and equipment, not reimbursing some incompetent speculators who suffered bad results because of their lousy judgment.
- larrys 15y ago
  "Best thing to do would be for Linode to generously reimburse these people" Creates a precedent and greater future liability.
  - techiferous 15y ago
    Good point.
- plox 15y ago
  The scam you mention would be difficult to pull off as there would have to be evidence that the account was compromised on Linode's end for the customer to make a claim.

look_lookatme 15y ago

I'm surprised that a tool with this level of access isn't walled off behind a VPN.

rdl 15y ago

I really hope the various bitcoin related incidents don't poison the well for online currencies in general. Yes, you need a greater level of security when dealing with transferrable, relatively anonymous or pseudonymous (and rapidly extractable) online assets that you don't need for book-entry accounting with an audit trail and reversible transactions (credit cards, ACH, etc.). Yes, this is beyond what even most banks currently use. No, it's not beyond current technological state of the art. Gaming (i.e. casinos), at least some of them, do a reasonable job with some very similar security problems.

wmf 15y ago
Wasn't the well pretty thoroughly poisoned by e-Gold and such? (We realize they're not the same as Bitcoin, but most don't.)
- rdl 15y ago
  Yeah, kinda. Although E-Gold's problem was force majeure, not technical (did they ever get hacked? people stole passwords all the time, but since it was double entry, it was pretty easy to reverse). osGold, etc. were probably better parallels. But yes, this whole field has had a long series of technical, business model, governance, and government related failures.
  - StavrosK 15y ago
    Can you explain what happened with eGold, osGold, etc? I haven't heard anything about it.
    - rdl 15y ago
      e-gold was shut down by the government for running an unlicensed money service business. They were officially Nevis based, but actually run out of Melbourne, FL (where they started originally); criminal charges were involved. The main reason was a particularly motivated federal prosecutor and the specific uses e-Gold users found -- child porn, paying for stolen cc's, etc. It wasn't a general thing about PATRIOT/MSB laws. osGold was just an internal fraud; digital currency, give me your (gold) and issue tokens -- then one day it disappeared, taking the original value with it. There were hack attacks by third parties against some other gold currency systems at the time, but I don't remember the details.

shazow 15y ago

Perhaps people affected by the many Bitcoin Exchange password leaks were reusing their passwords on Linode?

frankydp 15y ago
The breach originated on the Lionode customer service system. Which the attacker used to reset user passwords. Unlikely the incidents are directly related.

nwmcsween 15y ago

Can someone with a legal background comment on the viability of the victims taking legal action?

keypusher 15y ago
I've got to wonder too. If you rent out an apartment and someone breaks in with a key they stole from the landlord and takes your TV, can you sue the landlord? On the other hand, if you rent a security deposit box from a bank and it gets broken into, is the bank liable?
- mindstab 15y ago
  It all comes down to insurance. No one generally pays out of their pocket, they pay with insurance money. The reason you pay so much at a bank for a tiny box, where as, space wise, you pay a fraction for your appartment, or essentially change for a storage locker, is that the bank expects you to put valualbles in that space and so pays lots of that money to inssurance so if they do get robbed, they are covered and can pay you. The other places... dont. And usually have clauses in their contract saying as much: "We aren't insured for valueables, so don't store them here, it may not be safe and we won't (and can't) pay you back if they get stolen. Linode falls into the second category, cheap, uninsurred storage space. You want storage space insurred against digital robbery, a) good luck b) expect to pay a lot more than $20 or $30 a month
- mbreese 15y ago
  I think the threshold would be, did Linode take "reasonable" precautions in protecting the servers in question. Just like the landlord or bank. So long as they take reasonable precautions, they can't be held liable. You can expect for your landlord to keep their copy of the key in some sort of lockbox, but you aren't going to expect them to keep it in a pressure-sensitive safe, guarded by movie-style lasers and a German Shepard. You expect a little more security out of your bank. The real question is: why were services that act like bitcoin banks storing their coins on Linode in the first place.
  - 15y ago

jostmey 15y ago

Hey, I'm one of those guys! But I wasn't stupid enough to leave a stockpile of bitcoins sitting around to steal.

olefoo 15y ago
I wonder about that. It seems to me that the bitcoin wallet shouldn't have been accessible after reboot, at least until someone came by to give the agent managing it the passphrase that would allow it to decrypt it's state. From the sound of things the wallet was just laying around unencrypted?
- ceol 15y ago
  https://bitcointalk.org/index.php?topic=66979.msg778666#msg778666 https://bitcointalk.org/index.php?topic=66979.msg778666#msg7... According to the admin of bitcoinica, he didn't have the wallet encrypted because the ruby gem he was using to process withdrawals didn't support it.
  - ashconnor 15y ago
    Doesn't even matter if the wallet was encrypted, he'd still have to have the password somewhere on the server.
    - lmm 15y ago
      Not if he entered it manually on every reboot. There'd still be attacks around snooping the contents of the wallet from memory, but the ability to do a root password reset that involves rebooting the box wouldn't be enough (and it seems like that may have been all the access that the attackers here had).
      - deleted 15y ago
        [deleted]
  - AdamGibbins 15y ago

meow 15y ago

May be its time for some cloud insurance (covering outages/security etc) ? I don't mind paying a little extra if it mitigates risk.

tptacek 15y ago
Near unlimited liability and virtually zero ability to model the likelihood of events. A hard product to design.
- meow 15y ago
  Liability might be unlimited, but most of the insurance products for other categories cap it based on premium. So that may not be much of a problem. I agree with the difficulty of modelling the events though... they need more data for any kind of modelling.. I'm guessing amazon already has some kind of insurance to cover its risks under AWS service agreement..

videoappeal 15y ago

All this talk about banks being safe yada yada and cloud hosting not safe for US50k. Real banking companies (with billions of dollars on hand) do use commodity cloud hosting including Linode, for even sensitive parts. Take for example Natwest online banking login. On initial login page they load a cookie via an image from www.advanced-web-analytics.com and then once you enter a customer number the next page loads a ...drum roll... javascript file from www.omni-traffic.com. Now who can tell me what one can do when you have control over the Javascript on a banking login page? Ah crap. It looks they have been moved to Amazon EC2, ~8 months ago they were hosted on conventional Linode VPSs. Points still stands though.

tomg 15y ago
In my experience working on a US financial website, a bank would never consider using a VPS like Linode to store actual banking and customer data. It's not even close to Level 1 PCI compliant.
- videoappeal 15y ago
  Well tomg, when I researched this ~8 months ago there were at least 2 US financial websites that were using the same specialized analytic company that injected JS into banking login pages that were hosted on Linode VPSes.
  - tomg 15y ago
    Oh I don't doubt you. I'm not an expert on this, heck, I wasn't even allowed on to the actual servers (because of said compliance). I don't know the guidelines for login pages or what kind of security third party JS libs are supposed to have (also PCI is not a law, afaik). What I'm asserting is that the servers that store the actual banking and customer data have very high security standards. It's one thing to store front end website code on a VPS, it's a totally other thing to store your database with customer & bank data on Linode. The bitcoin breach seems analogous to Bank of America storing your account information on Linode and trusting it as the Real Data. Does that make sense?
  - 15y ago

VonLipwig 15y ago

The message seems to be... Only 8 accounts were affected. Do not worry. Minor breach. Not much harm done. It seems to me the truth is the attacker looked for bitcoin wallets and emptied them. The fact he could identify 8 accounts and access them suggests the attacker could have accessed far more accounts if they wished. I think this is the most worrying thing about the breach. I don't really understand how bitcoin works but it seems that people with wallets need to set up multiple wallets on multiple providers and limit the amount of bit coins in each wallet to limit any losses from breaches like this. If I was a linode customer I would be thinking about moving. This message, while fairly open, doesn't give me much confidence there aren't other security issues with the platform.

mdda 15y ago
It wasn't clear from the memo, but how did the attacker know which 8 specific accounts had 'references to bitcoin' if they didn't access other accounts too?
- calloc 15y ago
  Are you able to name your VM's on Linode?
  - ukd1 15y ago
    I doubt it's that: I would have worked out the target sites hosts (looking up the IP), made a list, worked out the most popular by value - probably worked out it was linode and gone from there...once in, I'd bet it's a simple task of searching for the IP in their interface.
    - mdda 15y ago
      Ok, asking the other way around, did Linode check out the non-attacked VPSs to determine whether they didn't have bitcoin wallets on them? I'm not disagreeing with anything else said here - it's just that it seems that the memo was very conclusive in stating something that couldn't be known unless there was more VPS introspection than they claim...

desade 15y ago

In Linode's FAQ, they mention that their virtualization management software was developed in-house. They seem quite proud of this, quipping: "The Linode Manager is custom software, written in house, and is not for sale (although others have tried to mimic it)." It seems to me that this is a classic example of security failures following inevitably from a lack of peer review. Maybe Linode didn't consider its LM software to be peer-reviewable, but I bet the victims of the bitcoin thefts wish that someone else had tested the code (and human systems surrounding it) for vulnerabilities. Is this not exactly what Bruce Schneier frequently points out? Anything that must withstand attacks to protect the valuables within should be tested by attacking it. A lot. My hunch is that the vulnerability exploited by this attacker would have been found and fixed already if the LM software were more open.

jarito 15y ago
I think you may be a little off here. The statements in the thread seem to indicate that the compromise was not based on a vulnerability in custom software, but compromised credentials. You can certainly argue that the management console should be protected by two-factor (and it should be), but their software doesn't seem to be at fault here. I would be willing to bet that they have had the system tested by external security contractors and scanned with automated scanning tools. This seems to be a people problem and features problem not a vulnerability problem. I guess we just don't know at this point. You may very well be correct. I guess if you want to use an open source provider, just make sure they are running OpenStack (openstack.org).