12 ms·

Still not a good move for Akamai, though. I get him speaking out for them about the hosting having been free, but Akamai is now the CDN that got bullied into k

Still not a good move for Akamai, though. I get him speaking out for them about the hosting having been free, but Akamai is now the CDN that got bullied into kicking someone of their service against their own will. Terrible PR, and that mud will stick in tech circles. Akamai folds under pressure. I know it's a crude comparison, but we don't negotiate with terrorists for a reason.

dangero 10y ago
I agree and I could see them reversing their stance now with the press around it. More than likely a middle manager approved the pro bono usage and once it got expensive that manager found him/herself in hot water over the resources being consumed. A few apologetic emails to Krebs from that middle manager, but saying he had to go, and boom we find ourselves here.
lsc 10y ago
I would bet money that the attack was truly epic... to the point where it was impacting (or was about to impact) other Akamai customers. Because you are right, this is terrible PR, and Akamai knows it. DDoS mitigation is fundamentally a problem of "who has more bandwidth?" - if the attacker has more bandwidth than you (and how much bandwidth you have depends on "to where" ) - it's over. The problem is that the economics, right now, are heavily tipped in favor of the attacker.
- masklinn 10y ago
  > I would bet money that the attack was truly epic... Seems like it not only was one of the most expensive attacks so far (and by far the biggest one to have ever hit Prolexic, according to them), it also made little use of reflection of amplification making it much harder to mitigate. > to the point where it was impacting (or was about to impact) other Akamai customers. That's exactly the scale it had reached, and Akamai provided free service to Krebs, which was nice of them but only to the extent that it wasn't significantly impacting customers.
  - lsc 10y ago
    >That's exactly the scale it had reached, and Akamai provided free service to Krebs, which was nice of them but only to the extent that it wasn't significantly impacting customers. I... wonder if anything different would have been done for a paying customer. I mean, if the attack was big enough to take down other customers, and if Akamai had the choice between kicking one customer and all customers being down?
    - masklinn 10y ago
      I'd expect them to have gone further and probably invested/committed more resources to it.
      - wpietri 10y ago
        Why's that? The theory is that it was impacting other customers. If the notion is that they kick off the low-revenue person to preserve the larger revenue stream, then it would seem that the only way to get real protection from Akamai is to be their largest customer.
        masklinn 10y ago
        > Why's that? The theory is that it was impacting other customers. If the notion is that they kick off the low-revenue person Krebs was not a "low-revenue person" he was a "no-revenue person" they provided the service pro-bono. With customers they'd have a contract, and while I don't know Akamai's contracts I assume they either have specific service clauses and/or "use clauses" where protection costs get charged to the customer.
        thaeli 10y ago
        Nope. Contract basically says "here are the service levels Akamai commits to meet" and "Customer's sole remedy is to cancel service if Akamai does not meet these levels." Looking at their contract language does not make me feel any better about whether they'd drop a paying customer in similar circumstances. (Quoting from a version available online at http://contracts.onecle.com/akamai/msa.shtml http://contracts.onecle.com/akamai/msa.shtml) 8.1.3 Akamai shall meet or exceed the network availability, capacity and operations levels as set forth in Section 2 above; provided that Customer's sole remedy for the breach of this provision by Akamai shall be the termination rights set forth in Section 10.2 below. 10.2 TERMINATION UPON DEFAULT. Either party may terminate this Agreement in the event that the other party materially defaults in performing any obligation under this Agreement and such default continues unremedied for a period of thirty (30) days following, written notice of default; provided, however, that in the event this Agreement is terminated by Customer due to Akamai's breach of its representations under Section 8.1.3 above and failure to cure, Customer's sole remedy shall be its election to terminate the Agreement without further liability to either party (except for Customer's obligation to pay all accrued and unpaid fees outstanding at the date of termination).
- manarth 10y ago
  I would bet money that the attack was truly epic From the article: "The assault has flooded Krebs' site with more than 620 Gbps per second of traffic — nearly double what Akamai has seen in the past." Sounds pretty epic.
  - julianwachholz 10y ago
    It's accelerating!
  - lightedman 10y ago
    If it takes a mere 620 G-bit to screw Akamai, then they're obviously not much of a content distribution platform. I only need a few thousand compromised machines in the right countries to run 5+ T-bit scale attacks. This isn't the 90s any more, and Akamai has no reason to not have improved bandwidth capabilities, unless they're doing like the Telecoms companies by saving that money and never using it to improve their own infrastructure.
    - daddykotex 10y ago
      I was impressed by that number too, but after reading your comment it seems awfully small. In 2016, what kind of attack do you expect CDN like Akamai to handle?
      - lightedman 10y ago
        Chattanooga has symmetrical G-bit fiber to the home. 5,000 Chat-town residents got uppity, think of what would happen to Akamai if they couldn't deal with 600-ish G-bit.
        Anasufovic 10y ago
        Yeah if that network was designed to handle continuous 1gbit per home.
        10y ago
- Pyxl101 10y ago
  > If the attacker has more bandwidth than you (and how much bandwidth you have depends on "to where" ) - it's over. Is there no way to stop such attacks by coordinating with your upstream ISP, or with the sources of the traffic? Why do backbones allow it to be carried? Is the problem that these attacks are too many different individual streams to identify and filter? It seems like there ought to be some way to hierarchically punt the problem to network operators. "Your network is contributing 10 gigabytes per second to this DDOS attack. Identify the sources and shut them down." - times each identifiable traffic stream. Is there no way to capture a list of all IPs involved in the traffic and quickly distribute a "shut off this device" request to the origin network? Maybe a good-faith collaboration of different ISPs could result in quick shut-downs. Or if networks don't cooperate, then the next-nearest border does it for them (and if they don't like the policies under which their neighbor suspends the traffic, they can sign up to do it themselves). Imagine something like a mini automated DMCA type request. "This IP is DDOSing me", signed by the operator of a reputable network, having the effect of suppressing origin traffic from the IP when received by a reputable network. (Any abuse of the mechanism causes the network to lose its privilege to participate, and DMCA requests related to that network fall upon its neighbors.) Perhaps the suppression would be destination-limited so as not to be vulnerable to too much abuse of the mechanism.
  - vidarh 10y ago
    It is possible to mitigate, but it takes time. > Is there no way to capture a list of all IPs involved in the traffic and quickly distribute a "shut off this device" request to the origin network? Now hackers have a new attack vector.
    - pbhjpbhj 10y ago
      Without net neutrality economics would likely fix this. "We'll limit our 60GBps traffic to this network block because our peering arrangement makes that expensive." Couldn't an ISP just throttle traffic to a particular network block on receipt of an authorised request. And that seemingly provides a general solution. Record output to particular network blocks and throttle traffic when it peaks beyond statistically normal bounds? Basically, applying damping.
dahdum 10y ago
What would have been a bunch of positive earned media is now a flood of negative reaction and lingering questions that they buckled under the pressure. The biggest DDOS ever, and Akamai dumps the client rather than defend it. However they spin it, doesn't look good.
- headmelted 10y ago
  Exactly this. Frank Leighton needs to make a statement about this, immediately, and he better pull the Rabbit of Caerbannog out of his hat.
wpietri 10y ago
> Terrible PR, and that mud will stick in tech circles. Akamai folds under pressure. Definitely. The lesson I'd take from this is that Akamai isn't serious about DDOS protection. For me, buying DDOS protection is something like buying insurance. I don't expect to need it, but if the worst happens, I expect them to stick with me. The way I measure insurance providers is by asking friends how it was when they had a claim. It strikes me as especially bad that they're doing it in the moment. It'd be bad enough if they said, "Sorry, Brian, this is too big a distraction; you've got 90 days to find a new home." But that they're dropping him in the middle of an attack? That means I can't trust Akamai.
- voidz 10y ago
  They gave Krebs the service free of charge.
  - headmelted 10y ago
    I mentioned that above, but I think this misses my point, which is: If Akamai can't provide their service for free, then they shouldn't provide their service for free. A CDN's whole business is resilience, which in this case makes them the bodyguard, not a bystander. Whatever your opinion of Cloudflare, it seems clear to me that Matthew Prince keenly understands this, hence him reaching out and offering to step in and get Krebs back online. tldr; If Akamai can't do the one job they exist to do in the face of an (albeit well armed) assailant, then they're the problem, not Krebs.
    - acdha 10y ago
      They gave him service for free and covered many, many smaller attacks, including the 20-100Gbps attacks he reported earlier this month. The fact that they decided not to cover one of the largest attacks ever documented pro bono isn't great but I don't think any of their clients would fail to understand the difference between a favor and a signed contract.
    - pizzapill 10y ago
      "This was flagged to my attention and I've reviewed all the interactions between the author and our team [cloudflare]. The site in question was using the free version of CloudFlare's service. On February 2, 2013, the site came under a substantial Layer 7 DDoS attack. While we provide basic DDoS mitigation for all customers (even those on the Free CloudFlare plan), for the mitigation of large attacks a site needs at least the Business tier of CloudFlare's service. In an effort to keep the site online, our ops team enabled I'm Under Attack Mode, which is available for Free customers and enhances DDoS protection. The attack continued and began to affect the performance of other CloudFlare customers, at which point we routed traffic to the site away from our network." https://news.ycombinator.com/item?id=5214480 https://news.ycombinator.com/item?id=5214480
elmin 10y ago
I can almost guarantee CloudFlare will offer to step into the breach.
bisRepetita 10y ago
In an ideal world, they would not cave... but someone has to pay the giant bill. Who would chip in? The site must be constantly under attack, and it must cost Akamai a fortune in real $$ all year long. And, when their service is performing well, nobody is talking about it, so there must be very little positive PR. If someone is ready to foot a 620 Gbps bandwith bill all year long, I am pretty sure Akamai will be more than happy and able to scale up further. Too bad there is no always a cleaner/smarter solution than pure bandwidth and $$ to fight those attacks.
snowwrestler 10y ago
As someone who actually pays for DDOS protection, this doesn't alter my opinion of Akamai. I already know that Akamai is expensive and not particularly good at it. They are a CDN who is trying to make some money on the side with unused bandwidth, and will protect their CDN business if it comes down to it. They don't invest in active defense. In fact, I know folks who actually had to block malware being served from Akamai-owned IPs! I suspect that is why they gave service to Krebs for free in the first place--they need the marketing.

dangero 10y ago

I agree and I could see them reversing their stance now with the press around it. More than likely a middle manager approved the pro bono usage and once it got expensive that manager found him/herself in hot water over the resources being consumed. A few apologetic emails to Krebs from that middle manager, but saying he had to go, and boom we find ourselves here.

lsc 10y ago

I would bet money that the attack was truly epic... to the point where it was impacting (or was about to impact) other Akamai customers. Because you are right, this is terrible PR, and Akamai knows it. DDoS mitigation is fundamentally a problem of "who has more bandwidth?" - if the attacker has more bandwidth than you (and how much bandwidth you have depends on "to where" ) - it's over. The problem is that the economics, right now, are heavily tipped in favor of the attacker.

masklinn 10y ago
> I would bet money that the attack was truly epic... Seems like it not only was one of the most expensive attacks so far (and by far the biggest one to have ever hit Prolexic, according to them), it also made little use of reflection of amplification making it much harder to mitigate. > to the point where it was impacting (or was about to impact) other Akamai customers. That's exactly the scale it had reached, and Akamai provided free service to Krebs, which was nice of them but only to the extent that it wasn't significantly impacting customers.
- lsc 10y ago
  >That's exactly the scale it had reached, and Akamai provided free service to Krebs, which was nice of them but only to the extent that it wasn't significantly impacting customers. I... wonder if anything different would have been done for a paying customer. I mean, if the attack was big enough to take down other customers, and if Akamai had the choice between kicking one customer and all customers being down?
  - masklinn 10y ago
    I'd expect them to have gone further and probably invested/committed more resources to it.
    - wpietri 10y ago
      Why's that? The theory is that it was impacting other customers. If the notion is that they kick off the low-revenue person to preserve the larger revenue stream, then it would seem that the only way to get real protection from Akamai is to be their largest customer.
      - masklinn 10y ago
        > Why's that? The theory is that it was impacting other customers. If the notion is that they kick off the low-revenue person Krebs was not a "low-revenue person" he was a "no-revenue person" they provided the service pro-bono. With customers they'd have a contract, and while I don't know Akamai's contracts I assume they either have specific service clauses and/or "use clauses" where protection costs get charged to the customer.
        thaeli 10y ago
        Nope. Contract basically says "here are the service levels Akamai commits to meet" and "Customer's sole remedy is to cancel service if Akamai does not meet these levels." Looking at their contract language does not make me feel any better about whether they'd drop a paying customer in similar circumstances. (Quoting from a version available online at http://contracts.onecle.com/akamai/msa.shtml http://contracts.onecle.com/akamai/msa.shtml) 8.1.3 Akamai shall meet or exceed the network availability, capacity and operations levels as set forth in Section 2 above; provided that Customer's sole remedy for the breach of this provision by Akamai shall be the termination rights set forth in Section 10.2 below. 10.2 TERMINATION UPON DEFAULT. Either party may terminate this Agreement in the event that the other party materially defaults in performing any obligation under this Agreement and such default continues unremedied for a period of thirty (30) days following, written notice of default; provided, however, that in the event this Agreement is terminated by Customer due to Akamai's breach of its representations under Section 8.1.3 above and failure to cure, Customer's sole remedy shall be its election to terminate the Agreement without further liability to either party (except for Customer's obligation to pay all accrued and unpaid fees outstanding at the date of termination).
manarth 10y ago
I would bet money that the attack was truly epic From the article: "The assault has flooded Krebs' site with more than 620 Gbps per second of traffic — nearly double what Akamai has seen in the past." Sounds pretty epic.
- julianwachholz 10y ago
  It's accelerating!
- lightedman 10y ago
  If it takes a mere 620 G-bit to screw Akamai, then they're obviously not much of a content distribution platform. I only need a few thousand compromised machines in the right countries to run 5+ T-bit scale attacks. This isn't the 90s any more, and Akamai has no reason to not have improved bandwidth capabilities, unless they're doing like the Telecoms companies by saving that money and never using it to improve their own infrastructure.
  - daddykotex 10y ago
    I was impressed by that number too, but after reading your comment it seems awfully small. In 2016, what kind of attack do you expect CDN like Akamai to handle?
    - lightedman 10y ago
      Chattanooga has symmetrical G-bit fiber to the home. 5,000 Chat-town residents got uppity, think of what would happen to Akamai if they couldn't deal with 600-ish G-bit.
      - Anasufovic 10y ago
        Yeah if that network was designed to handle continuous 1gbit per home.
        10y ago
Pyxl101 10y ago
> If the attacker has more bandwidth than you (and how much bandwidth you have depends on "to where" ) - it's over. Is there no way to stop such attacks by coordinating with your upstream ISP, or with the sources of the traffic? Why do backbones allow it to be carried? Is the problem that these attacks are too many different individual streams to identify and filter? It seems like there ought to be some way to hierarchically punt the problem to network operators. "Your network is contributing 10 gigabytes per second to this DDOS attack. Identify the sources and shut them down." - times each identifiable traffic stream. Is there no way to capture a list of all IPs involved in the traffic and quickly distribute a "shut off this device" request to the origin network? Maybe a good-faith collaboration of different ISPs could result in quick shut-downs. Or if networks don't cooperate, then the next-nearest border does it for them (and if they don't like the policies under which their neighbor suspends the traffic, they can sign up to do it themselves). Imagine something like a mini automated DMCA type request. "This IP is DDOSing me", signed by the operator of a reputable network, having the effect of suppressing origin traffic from the IP when received by a reputable network. (Any abuse of the mechanism causes the network to lose its privilege to participate, and DMCA requests related to that network fall upon its neighbors.) Perhaps the suppression would be destination-limited so as not to be vulnerable to too much abuse of the mechanism.
- vidarh 10y ago
  It is possible to mitigate, but it takes time. > Is there no way to capture a list of all IPs involved in the traffic and quickly distribute a "shut off this device" request to the origin network? Now hackers have a new attack vector.
  - pbhjpbhj 10y ago
    Without net neutrality economics would likely fix this. "We'll limit our 60GBps traffic to this network block because our peering arrangement makes that expensive." Couldn't an ISP just throttle traffic to a particular network block on receipt of an authorised request. And that seemingly provides a general solution. Record output to particular network blocks and throttle traffic when it peaks beyond statistically normal bounds? Basically, applying damping.

dahdum 10y ago

What would have been a bunch of positive earned media is now a flood of negative reaction and lingering questions that they buckled under the pressure. The biggest DDOS ever, and Akamai dumps the client rather than defend it. However they spin it, doesn't look good.

headmelted 10y ago
Exactly this. Frank Leighton needs to make a statement about this, immediately, and he better pull the Rabbit of Caerbannog out of his hat.

wpietri 10y ago

> Terrible PR, and that mud will stick in tech circles. Akamai folds under pressure. Definitely. The lesson I'd take from this is that Akamai isn't serious about DDOS protection. For me, buying DDOS protection is something like buying insurance. I don't expect to need it, but if the worst happens, I expect them to stick with me. The way I measure insurance providers is by asking friends how it was when they had a claim. It strikes me as especially bad that they're doing it in the moment. It'd be bad enough if they said, "Sorry, Brian, this is too big a distraction; you've got 90 days to find a new home." But that they're dropping him in the middle of an attack? That means I can't trust Akamai.

voidz 10y ago
They gave Krebs the service free of charge.
- headmelted 10y ago
  I mentioned that above, but I think this misses my point, which is: If Akamai can't provide their service for free, then they shouldn't provide their service for free. A CDN's whole business is resilience, which in this case makes them the bodyguard, not a bystander. Whatever your opinion of Cloudflare, it seems clear to me that Matthew Prince keenly understands this, hence him reaching out and offering to step in and get Krebs back online. tldr; If Akamai can't do the one job they exist to do in the face of an (albeit well armed) assailant, then they're the problem, not Krebs.
  - acdha 10y ago
    They gave him service for free and covered many, many smaller attacks, including the 20-100Gbps attacks he reported earlier this month. The fact that they decided not to cover one of the largest attacks ever documented pro bono isn't great but I don't think any of their clients would fail to understand the difference between a favor and a signed contract.
  - pizzapill 10y ago
    "This was flagged to my attention and I've reviewed all the interactions between the author and our team [cloudflare]. The site in question was using the free version of CloudFlare's service. On February 2, 2013, the site came under a substantial Layer 7 DDoS attack. While we provide basic DDoS mitigation for all customers (even those on the Free CloudFlare plan), for the mitigation of large attacks a site needs at least the Business tier of CloudFlare's service. In an effort to keep the site online, our ops team enabled I'm Under Attack Mode, which is available for Free customers and enhances DDoS protection. The attack continued and began to affect the performance of other CloudFlare customers, at which point we routed traffic to the site away from our network." https://news.ycombinator.com/item?id=5214480 https://news.ycombinator.com/item?id=5214480

elmin 10y ago

I can almost guarantee CloudFlare will offer to step into the breach.

bisRepetita 10y ago

In an ideal world, they would not cave... but someone has to pay the giant bill. Who would chip in? The site must be constantly under attack, and it must cost Akamai a fortune in real $$ all year long. And, when their service is performing well, nobody is talking about it, so there must be very little positive PR. If someone is ready to foot a 620 Gbps bandwith bill all year long, I am pretty sure Akamai will be more than happy and able to scale up further. Too bad there is no always a cleaner/smarter solution than pure bandwidth and $$ to fight those attacks.

snowwrestler 10y ago

As someone who actually pays for DDOS protection, this doesn't alter my opinion of Akamai. I already know that Akamai is expensive and not particularly good at it. They are a CDN who is trying to make some money on the side with unused bandwidth, and will protect their CDN business if it comes down to it. They don't invest in active defense. In fact, I know folks who actually had to block malware being served from Akamai-owned IPs! I suspect that is why they gave service to Krebs for free in the first place--they need the marketing.