5 ms·

DDOS attacks are a fact of life, nice to know that Rails Machine will throw you under the bus when one happens. Doesn't seem to fit their homepage description:

DDOS attacks are a fact of life, nice to know that Rails Machine will throw you under the bus when one happens. Doesn't seem to fit their homepage description: "You write Rails apps. We deploy, manage, support, monitor, and scale them. Done."

jsprinkles 14y ago
So if someone throws multiple tens of gigabits at your customer, and your upstream threatens to turn off your entire hosting company, you would respond "no way, we're going the extra mile for our customer"? Rails Machine was, in all likelihood, compelled to act to either (a) preserve its relationship with is upstream or (b) preserve its relationship with its other paying customers that do not attract DoS attacks. Your idealism will fall down quickly when one annoying customer threatens service for every last one of your clients: it becomes a "do I continue to get paid or do I fight for this customer" equation. Price out mitigation equipment and the multiple high-level engineers you will need to administer it before you respond telling me how wrong I am, by the way. The gear alone is an engineer's salary just to get started. Edit: Fine, I dropped the Amazon example.
- jayzee 14y ago
  Amazon did not drop Wikileaks fearing DDOS attacks. They dropped Wikileaks fearing Lieberman.
  - deleted 14y ago
    [deleted]
    - seanp2k2 14y ago
      >"Noteworthy: Getting attacked is against ToS at many hosts." Yes, and the action we take is largely dependent on the magnitude of the attack and how it affects /other/ clients.
- saurik 14y ago
  I don't remember it being about either DDoS (if anything, I thought they went to Amazon to avoid the DDoS, seemingly somewhat successfully) or "ToS" (which to me has an implication that Amazon decided they didn't like the service, as opposed to caving under the pressure of other people not liking the service). http://arstechnica.com/security/news/2010/12/wikileaks-kicked-out-of-amazons-cloud.ars http://arstechnica.com/security/news/2010/12/wikileaks-kicke... > The Wikileaks website migrated to Amazon's cloud hosting service yesterday after being hit by a distributed denial of service (DDoS) attack. Amazon decided to discontinue serving the controversial website this morning in response to pressure from critics, including prominent members of Congress. ... Senator Joe Lieberman (I-CT), chairman of the Homeland Security and Governmental Affairs Committee, was among the congressmen who pressured Amazon to stop hosting Wikileaks. He told AFP this morning that he plans to question Amazon about its relationship with Wikileaks.
  - deleted 14y ago
    [deleted]
    - saurik 14y ago
      Well, if you want to believe that public statement (which I see maybe you don't in the second paragraph you edited in), then you also have to retract the DDoS argument, as Amazon expressly and clearly states that that is an incorrect assessment. > There have also been reports that it was prompted by massive DDOS attacks. That too is inaccurate. There were indeed large-scale DDOS attacks, but they were successfully defended against.
- postfuturist 14y ago
  Any given site might not experience a DDoS attack, but if you run a hosting company, it will happen. The frequency depends on how many customers you have, how popular they are, etc. If a DDoS attack catches you by surprise, then you are very ill-prepared. One possible strategy is to throw the targeted customer under the bus and call it a day. For hosts of a certain size, that may be reasonable, as long as you clearly communicate that to customers. I've seen these attacks mitigated single-handedly by an experienced fellow with no access to fancy equipment. I believe it went something like this, change the DNS to point at some EC2 instances to do front-end load balancing with some scripts that detect and drop connections from attacking IP addresses and do severe rate limiting. You can proxy legitimate requests back to the original servers. Granted, not everyone knows how to do that in a pinch, but it shouldn't be a big deal for a hosting company. You only have to figure out how to do it once and when you detect a DDoS attack, flip the switch for that customer.
  - jsprinkles 14y ago
    > I've seen these attacks mitigated single-handedly by an experienced fellow with no access to fancy equipment. I believe it went something like this, change the DNS to point at some EC2 instances to do front-end load balancing with some scripts that detect and drop connections from attacking IP addresses and do severe rate limiting. You can proxy legitimate requests back to the original servers. That isn't how it works, but I appreciate you trying to explain it to me (I have personal experience as a former employee of a hosting company, in this exact position). A typical DoS attack is designed to flood the pipe, usually with UDP. UDP spam attacks are far simpler than layer 7 attacks that you describe, and layer 7 attacks (if small enough) are easier to handle. As a point of reference, I don't even have access to a botnet these days but I could knock your home cable connection offline in about five minutes of my time. In your scenario, if you were to update DNS to point the victim at some EC2 instances, you'd accomplish DoS attacking EC2 instead as they only offer you gigabit connectivity on most EC2 instances. If the DoS attack is multiple gigabit, that isn't going to help you unless Amazon steps in and mitigates about as best they can. A DoS mitigation strategy generally has nothing to do with your application. If an engineer's first answer to a huge DoS attack -- as in, larger than the uplink -- is iptables or "scripts to detect and drop connections", that engineer is uninformed. I'm sorry. I have personally witnessed a 40 gigabit/sec attack. I'm sure the smart engineers at CloudFlare have seen bigger. Upstreams don't give a shit what's flowing across the wire, and a small TCP SYN flood directed at your server won't gain any attention from your host or their upstream. A multiple-gigabit attack that takes down the entire uplink will.
- kijin 14y ago
  > paying customers that do not attract DoS attacks A little off topic, but I've always felt a slightly uneasy about the concept of "attracting" DDoS attacks. Sure, if you knowingly piss off a bunch of script kiddies, you're attracting attacks. But it seems that nowadays, any site that hosts user-generated content is at risk of being attacked for any random reason. And yet, a lot of people talk about "customers who attract attacks" as if those customers are to blame. It almost sounds like blaming women who wear certain types of clothes for attracting sex crime. Of course, the fact that you didn't do anything to provoke an attack might be irrelevant when your upstream faces a choice between cutting you loose and eating hundreds of thousands of dollars. People need to do what they need to do to protect their networks. Nonetheless, I'm curious what you guys think about the concept of "attracting attacks". If you blame the victim even a little bit, does that affect your judgment about what should be done in the case of an attack?
  - jsprinkles 14y ago
    > It almost sounds like blaming women who wear certain types of clothes for attracting sex crime. I completely stopped reading this comment here, when you wrote this, because that was a completely off-base comparison and has absolutely nothing to do with the topic at hand. Worse, you probably know it; I wouldn't assume you to be stupid. And that was a mountainously stupid comment. In hosting, there are customers that attract DoS attacks. Period. Ask anybody who does hosting. IRC servers are a canonical example and are DoS magnets. Torrent trackers are another. Pastebin sites, like Pastie, are becoming another (look who uses pastebin.com a lot: Anonymous). Hell, Facebook and Google probably takes several DoS attacks a day just by nature of being well-known. Don't even conflate my argument with an undertone of sexism.
    - kijin 14y ago
      Sorry if my comment came across as suggesting that your argument had an "undertone of sexism". I definitely wasn't trying to say anything of the sort. It was just an analogy that popped into my mind, and I don't think it was a particularly bad analogy. But I don't think your unwarranted indignation adds anything to the question that I was trying to ask to other HNers. Unfortunately, that question was at the end of my comment, past the point where you stopped reading.
- deleted 14y ago
  [deleted]

jsprinkles 14y ago

So if someone throws multiple tens of gigabits at your customer, and your upstream threatens to turn off your entire hosting company, you would respond "no way, we're going the extra mile for our customer"? Rails Machine was, in all likelihood, compelled to act to either (a) preserve its relationship with is upstream or (b) preserve its relationship with its other paying customers that do not attract DoS attacks. Your idealism will fall down quickly when one annoying customer threatens service for every last one of your clients: it becomes a "do I continue to get paid or do I fight for this customer" equation. Price out mitigation equipment and the multiple high-level engineers you will need to administer it before you respond telling me how wrong I am, by the way. The gear alone is an engineer's salary just to get started. Edit: Fine, I dropped the Amazon example.

jayzee 14y ago
Amazon did not drop Wikileaks fearing DDOS attacks. They dropped Wikileaks fearing Lieberman.
- deleted 14y ago
  [deleted]
  - seanp2k2 14y ago
    >"Noteworthy: Getting attacked is against ToS at many hosts." Yes, and the action we take is largely dependent on the magnitude of the attack and how it affects /other/ clients.
saurik 14y ago
I don't remember it being about either DDoS (if anything, I thought they went to Amazon to avoid the DDoS, seemingly somewhat successfully) or "ToS" (which to me has an implication that Amazon decided they didn't like the service, as opposed to caving under the pressure of other people not liking the service). http://arstechnica.com/security/news/2010/12/wikileaks-kicked-out-of-amazons-cloud.ars http://arstechnica.com/security/news/2010/12/wikileaks-kicke... > The Wikileaks website migrated to Amazon's cloud hosting service yesterday after being hit by a distributed denial of service (DDoS) attack. Amazon decided to discontinue serving the controversial website this morning in response to pressure from critics, including prominent members of Congress. ... Senator Joe Lieberman (I-CT), chairman of the Homeland Security and Governmental Affairs Committee, was among the congressmen who pressured Amazon to stop hosting Wikileaks. He told AFP this morning that he plans to question Amazon about its relationship with Wikileaks.
- deleted 14y ago
  [deleted]
  - saurik 14y ago
    Well, if you want to believe that public statement (which I see maybe you don't in the second paragraph you edited in), then you also have to retract the DDoS argument, as Amazon expressly and clearly states that that is an incorrect assessment. > There have also been reports that it was prompted by massive DDOS attacks. That too is inaccurate. There were indeed large-scale DDOS attacks, but they were successfully defended against.
postfuturist 14y ago
Any given site might not experience a DDoS attack, but if you run a hosting company, it will happen. The frequency depends on how many customers you have, how popular they are, etc. If a DDoS attack catches you by surprise, then you are very ill-prepared. One possible strategy is to throw the targeted customer under the bus and call it a day. For hosts of a certain size, that may be reasonable, as long as you clearly communicate that to customers. I've seen these attacks mitigated single-handedly by an experienced fellow with no access to fancy equipment. I believe it went something like this, change the DNS to point at some EC2 instances to do front-end load balancing with some scripts that detect and drop connections from attacking IP addresses and do severe rate limiting. You can proxy legitimate requests back to the original servers. Granted, not everyone knows how to do that in a pinch, but it shouldn't be a big deal for a hosting company. You only have to figure out how to do it once and when you detect a DDoS attack, flip the switch for that customer.
- jsprinkles 14y ago
  > I've seen these attacks mitigated single-handedly by an experienced fellow with no access to fancy equipment. I believe it went something like this, change the DNS to point at some EC2 instances to do front-end load balancing with some scripts that detect and drop connections from attacking IP addresses and do severe rate limiting. You can proxy legitimate requests back to the original servers. That isn't how it works, but I appreciate you trying to explain it to me (I have personal experience as a former employee of a hosting company, in this exact position). A typical DoS attack is designed to flood the pipe, usually with UDP. UDP spam attacks are far simpler than layer 7 attacks that you describe, and layer 7 attacks (if small enough) are easier to handle. As a point of reference, I don't even have access to a botnet these days but I could knock your home cable connection offline in about five minutes of my time. In your scenario, if you were to update DNS to point the victim at some EC2 instances, you'd accomplish DoS attacking EC2 instead as they only offer you gigabit connectivity on most EC2 instances. If the DoS attack is multiple gigabit, that isn't going to help you unless Amazon steps in and mitigates about as best they can. A DoS mitigation strategy generally has nothing to do with your application. If an engineer's first answer to a huge DoS attack -- as in, larger than the uplink -- is iptables or "scripts to detect and drop connections", that engineer is uninformed. I'm sorry. I have personally witnessed a 40 gigabit/sec attack. I'm sure the smart engineers at CloudFlare have seen bigger. Upstreams don't give a shit what's flowing across the wire, and a small TCP SYN flood directed at your server won't gain any attention from your host or their upstream. A multiple-gigabit attack that takes down the entire uplink will.
kijin 14y ago
> paying customers that do not attract DoS attacks A little off topic, but I've always felt a slightly uneasy about the concept of "attracting" DDoS attacks. Sure, if you knowingly piss off a bunch of script kiddies, you're attracting attacks. But it seems that nowadays, any site that hosts user-generated content is at risk of being attacked for any random reason. And yet, a lot of people talk about "customers who attract attacks" as if those customers are to blame. It almost sounds like blaming women who wear certain types of clothes for attracting sex crime. Of course, the fact that you didn't do anything to provoke an attack might be irrelevant when your upstream faces a choice between cutting you loose and eating hundreds of thousands of dollars. People need to do what they need to do to protect their networks. Nonetheless, I'm curious what you guys think about the concept of "attracting attacks". If you blame the victim even a little bit, does that affect your judgment about what should be done in the case of an attack?
- jsprinkles 14y ago
  > It almost sounds like blaming women who wear certain types of clothes for attracting sex crime. I completely stopped reading this comment here, when you wrote this, because that was a completely off-base comparison and has absolutely nothing to do with the topic at hand. Worse, you probably know it; I wouldn't assume you to be stupid. And that was a mountainously stupid comment. In hosting, there are customers that attract DoS attacks. Period. Ask anybody who does hosting. IRC servers are a canonical example and are DoS magnets. Torrent trackers are another. Pastebin sites, like Pastie, are becoming another (look who uses pastebin.com a lot: Anonymous). Hell, Facebook and Google probably takes several DoS attacks a day just by nature of being well-known. Don't even conflate my argument with an undertone of sexism.
  - kijin 14y ago
    Sorry if my comment came across as suggesting that your argument had an "undertone of sexism". I definitely wasn't trying to say anything of the sort. It was just an analogy that popped into my mind, and I don't think it was a particularly bad analogy. But I don't think your unwarranted indignation adds anything to the question that I was trying to ask to other HNers. Unfortunately, that question was at the end of my comment, past the point where you stopped reading.
deleted 14y ago
[deleted]