10 ms·

Linode DDoS continues – Atlanta down for 16+ hours

106 points by gingerlime 11y ago

gingerlime 11y ago
I honestly love Linode and am sure they'll come out better as a result of this. But our customers aren't as understanding. Currently we're playing a bit of cat and mouse and with each data centre going down - we're switching our recovery process into gear and restoring to a different VPS (outside Linode). We have linodes on pretty much all locations, but if this continues at this rate, we simply won't have any linodes left there. It would be very hard to justify going back to Linode afterwards, even with the best intentions to do so. "... So you seriously want us to go back to this hosting provider that caused us all this mess over Christmas / New Year's??"
- jafingi 11y ago
  That's exactly what the reaction the attackers want. However, what are the alternatives? Linode have been dead stable for me the past many years, and delivers what they promise in a transparent way. No overselling of servers. No sudden extra bills. Linode will come out stronger after this, so it won't be able to happen on this scale again. The big question is: Who, with a lot of money, would want to hurt Linode's business in this way? This isn't just a "script kiddie" having fun. It's a very well planned and powerful attack requiring buying large botnet capacity for an extensive amount of time.
  - agildehaus 11y ago
    Genuine question: What does anyone have to gain from Linode's misfortune? What sort of group would do this?
    - mwill 11y ago
      A competitor seems like a pretty obvious answer. But the vps market is fairly crowded, I can't imagine what attacking a single competitor would actually accomplish.
      - rdtsc 11y ago
        > A competitor seems like a pretty obvious answer. I don't know... unless Linode has a large presense in a country where this level of play is expected. I have hard time imagining a large US cloud / hosting company comparible in size or customer base with Linode trying it. There is too high a risk, someone will talkt and reveal it.
      - meshko 11y ago
        Why would a competitor hate Atlanta in particular? Must be going after something hosted there.
    - danieltillett 11y ago
      I have been surprised that this sort of approach has not been more common than it has been. It really is a credit to us as a community that this very low hanging fruit has not been exploited more often.
      - hellbanner 11y ago
        You're crediting the tech community for not exploiting their abilities? BF Skinner, famous behavioral psychologist, wrote about this in "Beyond Freedom and Dignity" -- that much of positive emotions coming from choosing to do the morally "righteous" action exist because of the capacity for evil.
        danieltillett 11y ago
        >You're crediting the tech community for not exploiting their abilities? Yes I am :) It would be easy to engage in morally bad behaviour, yet for the most part we don’t. This is a very good.
    - rdtsc 11y ago
      Maybe the some state actor / mafia drug lord... didn't like one of the customers and put enough effort to shut down the whole service. Github was attacked by China before I believe for hosting some firewall by-passing software. Maybe a competitor, but who would bother to go that low? I can't imagine any of the popular ones trying it -- too much risk it will be exposed.
    - sandaru1 11y ago
      It could be a competitor of someone who is using linode for their hosting. Rather than attacking a single server which would be easier to track down, it's probably easier to attack the whole network.
    - bdcravens 11y ago
      May not be the case here, but Bitcoin ransoms are become more common.
    - jacquesm 11y ago
      Usually these are extortion schemes. Pay x bitcoins to this address or else.
  - onewaystreet 11y ago
    > This isn't just a "script kiddie" having fun. It's a very well planned and powerful attack requiring buying large botnet capacity for an extensive amount of time. These days such things aren't that hard to come by. See the recent Lizard Squad attacks for example.
  - houseofmore 11y ago
    I don't know about dead stable. Fremont?
  - danieltillett 11y ago
    I agree, but I am unsure of the come back stronger sentiment. This looks to be a serious attack driven by a knowledgeably adversary. I really feel for linode at this point.
  - DCoder 11y ago
    > Who, with a lot of money, would want to hurt Linode's business in this way? This isn't just a "script kiddie" having fun. It's a very well planned and powerful attack requiring buying large botnet capacity for an extensive amount of time. Serious question: how much is "a lot of money' in this case? How much does an attack like this cost?
    - AdamJacobMuller 11y ago
      Unfortunately linode hasn't specified enough to give specifics about that. From what they have said however, it seems to me like this is generally a very "smart" attack. Unfortunately, a smart attack can be done much more cheaply in terms of botnet capacity than a dumb volumetric attack (this still combines a volumetric attack, but, in targeted ways). I have delt with these kinds of attacks (defending them) and they are, unfortunately, much too cheap to pull off. Botnet capacity is dirt cheap, if you're willing to spend the time to find/compromise your own botnet hosts, its effectively free.
  - gingerlime 11y ago
    I'm definitely curious to find out, but feels more like gossip to be honest... Given the length of time since the first attacks started and the downtime of the most recent attack, the big questions I have are: 1. What are they doing right now? 2. What mistakes were made / Why didn't they realise the risk beforehand? / (how) could this have been avoided? 3. What can they do in future to prevent this?
houseofmore 11y ago
Linode has been great to us, but we can't risk further outages. We've switched over 20 nodes to Google Cloud for the time being -- thankfully before that 16+ hour outage today in Atlanta. Happy check them out again once the dust settles.
- click170 11y ago
  From an armchair, it seems like it's a good idea to distribute a virtual server farm on multiple providers (Linode, AWS, et al). There's even libraries available to abstract away the provider layer, like libcloud. However, IME it's typical to invest in just one provider. Is anyone currently using libcloud or equiv. and able to share details?
empressplay 11y ago
Let's hope the authorities can identify the bad actor(s) in this case -- if they haven't made extortion demands it's hard not to imagine they're a competitor, and it would be really frustrating if they were able to get away with it.
- zhte415 11y ago
  If a competitor, their funders would likely be non-amused at their actions if publicised (even if already aware).
  - deleted 11y ago
    [deleted]
dantiberian 11y ago
I love Linode but they've done a poor job communicating with their customers. Compare the Linode Twitter feed, their mentions, and the lack of replies to how Slack handled it: https://medium.com/swlh/slackdown-a-lesson-in-brand-interaction-17f8c82b9fce#.4vsz21c0u https://medium.com/swlh/slackdown-a-lesson-in-brand-interact.... The Main Twitter account still hasn't announced the Atlanta DoS. https://twitter.com/linode?lang=en https://twitter.com/linode?lang=en I understand the severity of the issue is different to Slacks', but there should be a bunch of people on the Twitter account replying to people and saving customers. A lot of their customers are angry about lack of communication as much as they are about the downtime.
- empressplay 11y ago
  Although I'm not sure how important it is to announce outages on social media, I'm a customer and I haven't seen an e-mail either warning of an outage or explaining the situation. Generally I think linode is great but taking the tack of "Oh, if they don't notice, we shouldn't point it out to them" is just a bit cowardly.
  - pan69 11y ago
    I love Linode and I agree with you here. It does depend on the situation and I don't think it's necessary to start sending emails for each bit of down time, I mean, if you're savvy enough to run a VPS you should be able to monitor what your running, but since the severity of what's happening at the moment I would have at least expected an email by now explaining what's going on. Did receive my invoice though.. :) It's basically the same with the status updates, I have to "poll" for status updates rather than to be notified. They have this mobile app (which is pretty useless as it currently sits) which would be a great platform to receive push notifications on, even if it was to let you know that there is a change on the status page. There's certainly some missed opportunities here, but then again, I assume they're pretty busy at the moment...
    - tsturzl 11y ago
      I agree with this. They're service clearly doesn't include monitoring as a mention commodity, and the monitoring that they provide pro-bono isn't network monitoring. If you want to know that kind of thing, you should have the monitoring in place to do so, or simply subscribe to their RSS. No VPS provider ever has sent out emails. The only hosting company I ever left was DigitalOcean, because they kept having really silly problems and at the time NY1 was the only datacenter with private networking, which is ridiculously silly to think that your infrastructure communicates over the internet. And the private networking was "experimental" for the longest time, and very limited. I don't think Linode did anything wrong. And moving to another host, I garuntee you that they will not do this either. Chances are if you move hosts you may feed the shitty business who performed this DDoS.
dantiberian 11y ago
The brutal thing about a DDoS on a web hosting company is that it affects their business in very long lasting ways. If an ecommerce site is down they may lose sales for that day, and a small amount of customers. If a hosting provider is down then they can lose many of their customers for life.
- jacquesm 11y ago
  If you look at how interconnected stuff is these days it is really not that far away (if we haven't crossed that point already) were lives will be lost due to crimes like these. The analogy for me is one of roads. If you block a road on purpose then an ambulance might not be able to reach an accident victim in time. The internet is infrastructure, just like roads and purposefully obstructing it wholesale is doing damage to a large number of parties. What is sad is that these people get away with this stuff over and over again, it is very rare for DDoS organizers to be caught, rarer still (if it even ever happened) for them to be sentenced.
  - jops 11y ago
    Agreed. Attacks like this should be turned into a national issue and not left to Linode to deal with alone. I hope the NSA or FBI or whoever are investigating.
    - jacquesm 11y ago
      Likely you'll find they are an international issue.
      - SQLite 11y ago
        Isn't that why we have aircraft carriers?
        jacquesm 11y ago
        Who is we?
cpqq 11y ago
Work in IT. Server was stable & online for 189 days before the 25th, knew them for stability. No notification from them, just a handful of downtime alerts during time with the family. They were completely gone from BGP tables in Newark. Used backups and moved sites to OVH. Don't know who they pissed off, I suspect another NJ competitor, who is known for taking cheap shots at other VPS companies. It's a pain in the ass, but at the same time, how is their network so fragile? You would think at least some of the fragile systems being attacked would be firewalled or at least ACL'd off from the public net. This is what happens when you don't run your own network and rely on other ASN's and uplinks to do the work for you. When it comes to other customers being affected, they will simply null you. Unlike your network ops who would be trying anything they could from OOB to rectify such.
- StanAngeloff 11y ago
  What you are saying has been somewhat confirmed in Linode's latest update on the Atlanta outages [1]. I can't help it but to wonder if Linode were prepared or had a plan in place in case of a DDoS? It appears their upstream provider cuts them off completely once an attack starts/resumes and gradually puts them back on. The cycle then repeats. We are also duplicating in OVH, read good things about their built-in DDoS protection on HN. [1] http://status.linode.com/incidents/cbbcjnhhpkgm http://status.linode.com/incidents/cbbcjnhhpkgm
  - gingerlime 11y ago
    What kind of VPSs are you guys using with OVH? I had a look at OVH, but to be honest got really confused with too many options to choose from... (not to mention I wasn't sure which site I should sign up to, the .com / .co.uk - is this based on the VPS location in any way?). Linode clearly wins on simplicity and clarity. I guess under the current circumstances, I'd be willing to compromise simplicity for better availability though.
    - StanAngeloff 11y ago
      (I work in IT as well. Not affiliated with Linode / OVH) Linode wins on simplicity, agreed. We are in the same boat, OVH has too many offerings. We are looking at their VPS SSD plans [1]. Last thing we want is to be offline again. As such we are also looking for anti-DDoS which is included in the plans. I intent on spinning up a few nodes to try them out first. [1] https://www.ovh.ie/vps/vps-ssd.xml https://www.ovh.ie/vps/vps-ssd.xml
deleted 11y ago
[deleted]
frik 11y ago
This story fall from #2 on HN to page #102 in seconds. Is it because of only 38 upvotes vs 43 comments? The story is just 3 hours old. @HN / dang: What's going on with HN sorting algorithm? The #3 story on HN is 1 hour old and has just 9 upvotes and 2 comments: "Churchill and His Money, or Lack of It": https://news.ycombinator.com/item?id=10825575 https://news.ycombinator.com/item?id=10825575 Screenshot: http://s3.postimg.org/6dn7h6w5v/hn_linode_fall.png http://s3.postimg.org/6dn7h6w5v/hn_linode_fall.png
- cpqq 11y ago
  So I'm not the only one who saw this quickly disappear. Couldn't find it on the 2nd or 3rd page, very odd.
- dang 11y ago
  It set off the flamewar detector. Looks like an edge case in the algorithm. Btw you should email hn@ycombinator.com instead of asking us questions here. It's pretty random whether or not we see the latter.
- stefantalpalaru 11y ago
  What's happening is we're all watched over by machines of loving grace. There's no way to communicate with them to report errors, so it's better to stop talking about it in public. Just relax and trust the system. It knows what's best for you.
rrcap 11y ago
In this moment I can access to my linode and my website is operational. News about status of attack?
DarthSith 11y ago
I'm a Linode customer. I have been calling on a regular basis because I have a hosted service for my customers (thousands of them) that has been all but dead for 24 hours now. If I hear 'we are working on it' one more time, I may lose my mind. My customers are suffering severely, and I'm losing thousands of dollars as we speak. They keep telling me they are working with their upstream provider and that it's out of their hands. I'm not paying their upstream provider. I don't have a service agreement with their upstream provider. I'm paying them based on the service agreement I have with them. This is either a completely new level of DDoS or they are just completely incompetent in their way of handling it. In any case, I believe that Linode is going to suffer greatly for this in terms of lost customers. They're going to lose me by the end of next week, that's for certain.
- deleted 11y ago
  [deleted]
- dwightgunning 11y ago
  >> This is either a completely new level of DDoS or they are just completely incompetent in their way of handling it. My take is that it's somewhere in the middle. This type of outage has in the past, and will potentially in future, hit providers at all price points regardless of their SLAs and guarantees. In my experience (and other comments indicate) Linode are genearlly very reliable. This isn't a mickey mouse, dirt-cheap VPS operation. The communication hasn't been awesome but by keeping an eye on status.linode.com I personally have felt reasonably well informed. If you trust they're working on it and doing the best they can. Other commenters suggest this is a fairly sophisticated, deliberate and sustained attack. They're putting significant resources toward it. It also seems whoever is behind it has potentially gained inside knowledge of their network topology. Based that understand, I'm not wasting my time calling them or sitting around hitting F5. I'm working to improve my systems' architecture for resiliency in this type of situation. That involves geo-distributed, multi-site redundancy and fail-over. My advice: be optimistic and proactive.
- brandon272 11y ago
  Why do you not have a disaster recovery plan that allows you to restore services to your customers in case of outage or other emergency? I'm asking sincerely. I wouldn't be able to sleep at night if I had a SaaS product with thousands of customers and no way to restore service if my primary provider went down for an extended period of time.
gwright 11y ago
Every business endeavor has associated risks, which can be mitigated in a variety of ways for a variety of costs. Offloading the responsibility for continuation of your business to Linode (or any other data center provider) is unfair. A history of uptime, verbal promises, or fancy SLA terms should never be interpreted to mean that disasters won't happen. A ten day long DDOS is a disaster and in this case a man-made disaster. Using Linode (or another provider) instead of building your own data center is more cost effective, but it means you are no longer in direct control of your infrastructure (decreased costs, increased risk). Designing your application to span multiple availability zones (data centers) can mitigate single points of failure within a single vendor but is more expensive than operating in a single zone. Designing your application to span multiple vendors can mitigate single vendor failures (or changes in offerings from a single vendor) but is even more expensive. And still there are ways to mitigate these costs, business interruption insurance can help cover the costs for moving to a new data center or vendor in case of a disaster (such as hiring staff, overtime, etc.). Lost profits can be covered by business interruption insurance also. Of course it is expensive to operate any business in a hostile environment. A seaside restaurant better be prepared to weather a hurricane. I wonder how much money has been spent on security cameras, guards, metal detectors, and so on since 9/11? The increasing occurrence of targeted DDOS (and other types) of attacks is the physical equivalent of an increasingly hostile environment and is going to be associated with higher costs. In the longer term, I think we need to find ways to get law enforcement better suited to deal with these problems but ultimately I think we'll need to radically change the way we handle network operations and the technical foundations of the network such as content centric networking (https://en.wikipedia.org/wiki/Content_centric_networking https://en.wikipedia.org/wiki/Content_centric_networking)
mjrpes 11y ago
Je suis Linode! This is nothing less than cyber-terrorism, and I hope the FBI is involved. A problem I see with people throwing Linode under the bus is that only the mega hosting providers (AWS/Google) have the resources to mitigate such attacks. I wound hope that the industry can find a solution out there that allows smaller players (Linode, Digital Ocean, etc) to run a hosting business without the threat of DDoS. If everyone moves to the big players, it is a loss for us all: feature-wise, quality-wise, security-wise, cost-wise.
- larrymcp 11y ago
  "Je suis Linode" what an awesome sentiment... I just signed up for a Linode server. Going to use it for part of our offsite backup storage and to monitor some nodes on our primary network. Huzzah...
- tsturzl 11y ago
  Still sticking with Linode. This is definitely not their fault. If anything I blame ISPs for letting it happen. ISP's could just simply filter traffic by filtering forged packets, then dropping traffic at a backbone would be trivial, making most botnets obsolete. AWS and Google have resources in place for their own benefit, and hosting with them means you get these benefits to some degree. But the issue is that this complicated infrastructure is only reachable by giants(amazon, google, rackspace, etc). AWS probably spends enough money, collectively, on DDoS mitigation and fallback that it could acquire another hosting company for that sum of money. The to consider the shady practices of who ever is trying to sabotage Linode's business, the likely goal is to direct business elsewhere. So you may end up moving to a hosting company who is shady enough to have caused you a loss of profits for their own profits. Somethings up here, and I'm really curious to find out who did this. I'm not leaving a business because they were a victim of espionage.
ugexe 11y ago
their status page is update after update of the same message, apparently being ad libbed by someone with a thesaurus
lambdud 11y ago
I've used Linode increasingly since 2008, now consuming 10 times what I did at the start. I'm preparing to move to AWS today but genuinely hope that Linode comes through with a reasonable explanation of why I can expect this to not be repeated.
leeforkenbrock 11y ago
Well, in good news, it seems Linode's billing system was unharmed during this outage. ;)
meshko 11y ago
I am a (tiny) Linode customer with just one node but for a good number of years. Probably close to 10 now. This is the first outage I have with them. All in all it's a good thing as it made me finally learn how to use EC2 and I now have a backup there. I already shut down the EC2 backup instance and switched back to Linode as they seem to be up now.

106 points by gingerlime 11y ago

I honestly love Linode and am sure they'll come out better as a result of this. But our customers aren't as understanding. Currently we're playing a bit of cat and mouse and with each data centre going down - we're switching our recovery process into gear and restoring to a different VPS (outside Linode). We have linodes on pretty much all locations, but if this continues at this rate, we simply won't have any linodes left there. It would be very hard to justify going back to Linode afterwards, even with the best intentions to do so. "... So you seriously want us to go back to this hosting provider that caused us all this mess over Christmas / New Year's??"

jafingi 11y ago
That's exactly what the reaction the attackers want. However, what are the alternatives? Linode have been dead stable for me the past many years, and delivers what they promise in a transparent way. No overselling of servers. No sudden extra bills. Linode will come out stronger after this, so it won't be able to happen on this scale again. The big question is: Who, with a lot of money, would want to hurt Linode's business in this way? This isn't just a "script kiddie" having fun. It's a very well planned and powerful attack requiring buying large botnet capacity for an extensive amount of time.
- agildehaus 11y ago
  Genuine question: What does anyone have to gain from Linode's misfortune? What sort of group would do this?
  - mwill 11y ago
    A competitor seems like a pretty obvious answer. But the vps market is fairly crowded, I can't imagine what attacking a single competitor would actually accomplish.
    - rdtsc 11y ago
      > A competitor seems like a pretty obvious answer. I don't know... unless Linode has a large presense in a country where this level of play is expected. I have hard time imagining a large US cloud / hosting company comparible in size or customer base with Linode trying it. There is too high a risk, someone will talkt and reveal it.
    - meshko 11y ago
      Why would a competitor hate Atlanta in particular? Must be going after something hosted there.
  - danieltillett 11y ago
    I have been surprised that this sort of approach has not been more common than it has been. It really is a credit to us as a community that this very low hanging fruit has not been exploited more often.
    - hellbanner 11y ago
      You're crediting the tech community for not exploiting their abilities? BF Skinner, famous behavioral psychologist, wrote about this in "Beyond Freedom and Dignity" -- that much of positive emotions coming from choosing to do the morally "righteous" action exist because of the capacity for evil.
      - danieltillett 11y ago
        >You're crediting the tech community for not exploiting their abilities? Yes I am :) It would be easy to engage in morally bad behaviour, yet for the most part we don’t. This is a very good.
  - rdtsc 11y ago
    Maybe the some state actor / mafia drug lord... didn't like one of the customers and put enough effort to shut down the whole service. Github was attacked by China before I believe for hosting some firewall by-passing software. Maybe a competitor, but who would bother to go that low? I can't imagine any of the popular ones trying it -- too much risk it will be exposed.
  - sandaru1 11y ago
    It could be a competitor of someone who is using linode for their hosting. Rather than attacking a single server which would be easier to track down, it's probably easier to attack the whole network.
  - bdcravens 11y ago
    May not be the case here, but Bitcoin ransoms are become more common.
  - jacquesm 11y ago
    Usually these are extortion schemes. Pay x bitcoins to this address or else.
- onewaystreet 11y ago
  > This isn't just a "script kiddie" having fun. It's a very well planned and powerful attack requiring buying large botnet capacity for an extensive amount of time. These days such things aren't that hard to come by. See the recent Lizard Squad attacks for example.
- houseofmore 11y ago
  I don't know about dead stable. Fremont?
- danieltillett 11y ago
  I agree, but I am unsure of the come back stronger sentiment. This looks to be a serious attack driven by a knowledgeably adversary. I really feel for linode at this point.
- DCoder 11y ago
  > Who, with a lot of money, would want to hurt Linode's business in this way? This isn't just a "script kiddie" having fun. It's a very well planned and powerful attack requiring buying large botnet capacity for an extensive amount of time. Serious question: how much is "a lot of money' in this case? How much does an attack like this cost?
  - AdamJacobMuller 11y ago
    Unfortunately linode hasn't specified enough to give specifics about that. From what they have said however, it seems to me like this is generally a very "smart" attack. Unfortunately, a smart attack can be done much more cheaply in terms of botnet capacity than a dumb volumetric attack (this still combines a volumetric attack, but, in targeted ways). I have delt with these kinds of attacks (defending them) and they are, unfortunately, much too cheap to pull off. Botnet capacity is dirt cheap, if you're willing to spend the time to find/compromise your own botnet hosts, its effectively free.
- gingerlime 11y ago
  I'm definitely curious to find out, but feels more like gossip to be honest... Given the length of time since the first attacks started and the downtime of the most recent attack, the big questions I have are: 1. What are they doing right now? 2. What mistakes were made / Why didn't they realise the risk beforehand? / (how) could this have been avoided? 3. What can they do in future to prevent this?

houseofmore 11y ago

Linode has been great to us, but we can't risk further outages. We've switched over 20 nodes to Google Cloud for the time being -- thankfully before that 16+ hour outage today in Atlanta. Happy check them out again once the dust settles.

click170 11y ago
From an armchair, it seems like it's a good idea to distribute a virtual server farm on multiple providers (Linode, AWS, et al). There's even libraries available to abstract away the provider layer, like libcloud. However, IME it's typical to invest in just one provider. Is anyone currently using libcloud or equiv. and able to share details?

empressplay 11y ago

Let's hope the authorities can identify the bad actor(s) in this case -- if they haven't made extortion demands it's hard not to imagine they're a competitor, and it would be really frustrating if they were able to get away with it.

zhte415 11y ago
If a competitor, their funders would likely be non-amused at their actions if publicised (even if already aware).
- deleted 11y ago
  [deleted]

dantiberian 11y ago

I love Linode but they've done a poor job communicating with their customers. Compare the Linode Twitter feed, their mentions, and the lack of replies to how Slack handled it: https://medium.com/swlh/slackdown-a-lesson-in-brand-interaction-17f8c82b9fce#.4vsz21c0u https://medium.com/swlh/slackdown-a-lesson-in-brand-interact.... The Main Twitter account still hasn't announced the Atlanta DoS. https://twitter.com/linode?lang=en https://twitter.com/linode?lang=en I understand the severity of the issue is different to Slacks', but there should be a bunch of people on the Twitter account replying to people and saving customers. A lot of their customers are angry about lack of communication as much as they are about the downtime.

empressplay 11y ago
Although I'm not sure how important it is to announce outages on social media, I'm a customer and I haven't seen an e-mail either warning of an outage or explaining the situation. Generally I think linode is great but taking the tack of "Oh, if they don't notice, we shouldn't point it out to them" is just a bit cowardly.
- pan69 11y ago
  I love Linode and I agree with you here. It does depend on the situation and I don't think it's necessary to start sending emails for each bit of down time, I mean, if you're savvy enough to run a VPS you should be able to monitor what your running, but since the severity of what's happening at the moment I would have at least expected an email by now explaining what's going on. Did receive my invoice though.. :) It's basically the same with the status updates, I have to "poll" for status updates rather than to be notified. They have this mobile app (which is pretty useless as it currently sits) which would be a great platform to receive push notifications on, even if it was to let you know that there is a change on the status page. There's certainly some missed opportunities here, but then again, I assume they're pretty busy at the moment...
  - tsturzl 11y ago
    I agree with this. They're service clearly doesn't include monitoring as a mention commodity, and the monitoring that they provide pro-bono isn't network monitoring. If you want to know that kind of thing, you should have the monitoring in place to do so, or simply subscribe to their RSS. No VPS provider ever has sent out emails. The only hosting company I ever left was DigitalOcean, because they kept having really silly problems and at the time NY1 was the only datacenter with private networking, which is ridiculously silly to think that your infrastructure communicates over the internet. And the private networking was "experimental" for the longest time, and very limited. I don't think Linode did anything wrong. And moving to another host, I garuntee you that they will not do this either. Chances are if you move hosts you may feed the shitty business who performed this DDoS.

dantiberian 11y ago

The brutal thing about a DDoS on a web hosting company is that it affects their business in very long lasting ways. If an ecommerce site is down they may lose sales for that day, and a small amount of customers. If a hosting provider is down then they can lose many of their customers for life.

jacquesm 11y ago
If you look at how interconnected stuff is these days it is really not that far away (if we haven't crossed that point already) were lives will be lost due to crimes like these. The analogy for me is one of roads. If you block a road on purpose then an ambulance might not be able to reach an accident victim in time. The internet is infrastructure, just like roads and purposefully obstructing it wholesale is doing damage to a large number of parties. What is sad is that these people get away with this stuff over and over again, it is very rare for DDoS organizers to be caught, rarer still (if it even ever happened) for them to be sentenced.
- jops 11y ago
  Agreed. Attacks like this should be turned into a national issue and not left to Linode to deal with alone. I hope the NSA or FBI or whoever are investigating.
  - jacquesm 11y ago
    Likely you'll find they are an international issue.
    - SQLite 11y ago
      Isn't that why we have aircraft carriers?
      - jacquesm 11y ago
        Who is we?

cpqq 11y ago

Work in IT. Server was stable & online for 189 days before the 25th, knew them for stability. No notification from them, just a handful of downtime alerts during time with the family. They were completely gone from BGP tables in Newark. Used backups and moved sites to OVH. Don't know who they pissed off, I suspect another NJ competitor, who is known for taking cheap shots at other VPS companies. It's a pain in the ass, but at the same time, how is their network so fragile? You would think at least some of the fragile systems being attacked would be firewalled or at least ACL'd off from the public net. This is what happens when you don't run your own network and rely on other ASN's and uplinks to do the work for you. When it comes to other customers being affected, they will simply null you. Unlike your network ops who would be trying anything they could from OOB to rectify such.

StanAngeloff 11y ago
What you are saying has been somewhat confirmed in Linode's latest update on the Atlanta outages [1]. I can't help it but to wonder if Linode were prepared or had a plan in place in case of a DDoS? It appears their upstream provider cuts them off completely once an attack starts/resumes and gradually puts them back on. The cycle then repeats. We are also duplicating in OVH, read good things about their built-in DDoS protection on HN. [1] http://status.linode.com/incidents/cbbcjnhhpkgm http://status.linode.com/incidents/cbbcjnhhpkgm
- gingerlime 11y ago
  What kind of VPSs are you guys using with OVH? I had a look at OVH, but to be honest got really confused with too many options to choose from... (not to mention I wasn't sure which site I should sign up to, the .com / .co.uk - is this based on the VPS location in any way?). Linode clearly wins on simplicity and clarity. I guess under the current circumstances, I'd be willing to compromise simplicity for better availability though.
  - StanAngeloff 11y ago
    (I work in IT as well. Not affiliated with Linode / OVH) Linode wins on simplicity, agreed. We are in the same boat, OVH has too many offerings. We are looking at their VPS SSD plans [1]. Last thing we want is to be offline again. As such we are also looking for anti-DDoS which is included in the plans. I intent on spinning up a few nodes to try them out first. [1] https://www.ovh.ie/vps/vps-ssd.xml https://www.ovh.ie/vps/vps-ssd.xml

deleted 11y ago

[deleted]

frik 11y ago

This story fall from #2 on HN to page #102 in seconds. Is it because of only 38 upvotes vs 43 comments? The story is just 3 hours old. @HN / dang: What's going on with HN sorting algorithm? The #3 story on HN is 1 hour old and has just 9 upvotes and 2 comments: "Churchill and His Money, or Lack of It": https://news.ycombinator.com/item?id=10825575 https://news.ycombinator.com/item?id=10825575 Screenshot: http://s3.postimg.org/6dn7h6w5v/hn_linode_fall.png http://s3.postimg.org/6dn7h6w5v/hn_linode_fall.png

cpqq 11y ago
So I'm not the only one who saw this quickly disappear. Couldn't find it on the 2nd or 3rd page, very odd.
dang 11y ago
It set off the flamewar detector. Looks like an edge case in the algorithm. Btw you should email hn@ycombinator.com instead of asking us questions here. It's pretty random whether or not we see the latter.
stefantalpalaru 11y ago
What's happening is we're all watched over by machines of loving grace. There's no way to communicate with them to report errors, so it's better to stop talking about it in public. Just relax and trust the system. It knows what's best for you.

rrcap 11y ago

In this moment I can access to my linode and my website is operational. News about status of attack?

DarthSith 11y ago

I'm a Linode customer. I have been calling on a regular basis because I have a hosted service for my customers (thousands of them) that has been all but dead for 24 hours now. If I hear 'we are working on it' one more time, I may lose my mind. My customers are suffering severely, and I'm losing thousands of dollars as we speak. They keep telling me they are working with their upstream provider and that it's out of their hands. I'm not paying their upstream provider. I don't have a service agreement with their upstream provider. I'm paying them based on the service agreement I have with them. This is either a completely new level of DDoS or they are just completely incompetent in their way of handling it. In any case, I believe that Linode is going to suffer greatly for this in terms of lost customers. They're going to lose me by the end of next week, that's for certain.

deleted 11y ago
[deleted]
dwightgunning 11y ago
>> This is either a completely new level of DDoS or they are just completely incompetent in their way of handling it. My take is that it's somewhere in the middle. This type of outage has in the past, and will potentially in future, hit providers at all price points regardless of their SLAs and guarantees. In my experience (and other comments indicate) Linode are genearlly very reliable. This isn't a mickey mouse, dirt-cheap VPS operation. The communication hasn't been awesome but by keeping an eye on status.linode.com I personally have felt reasonably well informed. If you trust they're working on it and doing the best they can. Other commenters suggest this is a fairly sophisticated, deliberate and sustained attack. They're putting significant resources toward it. It also seems whoever is behind it has potentially gained inside knowledge of their network topology. Based that understand, I'm not wasting my time calling them or sitting around hitting F5. I'm working to improve my systems' architecture for resiliency in this type of situation. That involves geo-distributed, multi-site redundancy and fail-over. My advice: be optimistic and proactive.
brandon272 11y ago
Why do you not have a disaster recovery plan that allows you to restore services to your customers in case of outage or other emergency? I'm asking sincerely. I wouldn't be able to sleep at night if I had a SaaS product with thousands of customers and no way to restore service if my primary provider went down for an extended period of time.

gwright 11y ago

Every business endeavor has associated risks, which can be mitigated in a variety of ways for a variety of costs. Offloading the responsibility for continuation of your business to Linode (or any other data center provider) is unfair. A history of uptime, verbal promises, or fancy SLA terms should never be interpreted to mean that disasters won't happen. A ten day long DDOS is a disaster and in this case a man-made disaster. Using Linode (or another provider) instead of building your own data center is more cost effective, but it means you are no longer in direct control of your infrastructure (decreased costs, increased risk). Designing your application to span multiple availability zones (data centers) can mitigate single points of failure within a single vendor but is more expensive than operating in a single zone. Designing your application to span multiple vendors can mitigate single vendor failures (or changes in offerings from a single vendor) but is even more expensive. And still there are ways to mitigate these costs, business interruption insurance can help cover the costs for moving to a new data center or vendor in case of a disaster (such as hiring staff, overtime, etc.). Lost profits can be covered by business interruption insurance also. Of course it is expensive to operate any business in a hostile environment. A seaside restaurant better be prepared to weather a hurricane. I wonder how much money has been spent on security cameras, guards, metal detectors, and so on since 9/11? The increasing occurrence of targeted DDOS (and other types) of attacks is the physical equivalent of an increasingly hostile environment and is going to be associated with higher costs. In the longer term, I think we need to find ways to get law enforcement better suited to deal with these problems but ultimately I think we'll need to radically change the way we handle network operations and the technical foundations of the network such as content centric networking (https://en.wikipedia.org/wiki/Content_centric_networking https://en.wikipedia.org/wiki/Content_centric_networking)

mjrpes 11y ago

Je suis Linode! This is nothing less than cyber-terrorism, and I hope the FBI is involved. A problem I see with people throwing Linode under the bus is that only the mega hosting providers (AWS/Google) have the resources to mitigate such attacks. I wound hope that the industry can find a solution out there that allows smaller players (Linode, Digital Ocean, etc) to run a hosting business without the threat of DDoS. If everyone moves to the big players, it is a loss for us all: feature-wise, quality-wise, security-wise, cost-wise.

larrymcp 11y ago
"Je suis Linode" what an awesome sentiment... I just signed up for a Linode server. Going to use it for part of our offsite backup storage and to monitor some nodes on our primary network. Huzzah...
tsturzl 11y ago
Still sticking with Linode. This is definitely not their fault. If anything I blame ISPs for letting it happen. ISP's could just simply filter traffic by filtering forged packets, then dropping traffic at a backbone would be trivial, making most botnets obsolete. AWS and Google have resources in place for their own benefit, and hosting with them means you get these benefits to some degree. But the issue is that this complicated infrastructure is only reachable by giants(amazon, google, rackspace, etc). AWS probably spends enough money, collectively, on DDoS mitigation and fallback that it could acquire another hosting company for that sum of money. The to consider the shady practices of who ever is trying to sabotage Linode's business, the likely goal is to direct business elsewhere. So you may end up moving to a hosting company who is shady enough to have caused you a loss of profits for their own profits. Somethings up here, and I'm really curious to find out who did this. I'm not leaving a business because they were a victim of espionage.

ugexe 11y ago

their status page is update after update of the same message, apparently being ad libbed by someone with a thesaurus

lambdud 11y ago

I've used Linode increasingly since 2008, now consuming 10 times what I did at the start. I'm preparing to move to AWS today but genuinely hope that Linode comes through with a reasonable explanation of why I can expect this to not be repeated.

leeforkenbrock 11y ago

Well, in good news, it seems Linode's billing system was unharmed during this outage. ;)

meshko 11y ago

I am a (tiny) Linode customer with just one node but for a good number of years. Probably close to 10 now. This is the first outage I have with them. All in all it's a good thing as it made me finally learn how to use EC2 and I now have a backup there. I already shut down the EC2 backup instance and switched back to Linode as they seem to be up now.