6 ms·

I've always wondered how Google would deal with a client on GCP being DDoSed. Mainly as were in online advertising and DDoS extortion isn't uncommon. Guess wit

I've always wondered how Google would deal with a client on GCP being DDoSed. Mainly as were in online advertising and DDoS extortion isn't uncommon. Guess with this I'll now find out, as crapping on Krebs' site is practically a right of passage when you've got a botnet now.

MaysonL 10y ago
Trying to access 130.211.45.45, which is listed in OP as the IP adress, keep getting 502 server errors.
- poooogles 10y ago
  Sure you're sending the right host header?
- neitsab 10y ago
  You can try hardcoding the IP address in your hosts file (see my comment below: https://news.ycombinator.com/item?id=12574428#12575021 https://news.ycombinator.com/item?id=12574428#12575021). This will get you to the site through a regular request.
hrrsn 10y ago
I'd wager this is Project Shield rather than regular GCP. https://projectshield.withgoogle.com/public/ https://projectshield.withgoogle.com/public/
- eric_bullington 10y ago
  This is the first I've heard of Google's Project Shield. Very cool project, is it relatively new? Does anyone know if this was the result of someone's 20% time or a top-down Google initiative?
  - kerneis 10y ago
    Project Shield is part of Jigsaw (formerly "Google Ideas"): see https://jigsaw.google.com/ https://jigsaw.google.com/ and https://medium.com/jigsaw https://medium.com/jigsaw for more related projects and context. Google Ideas is 5 years old now. I'm not aware of it having started as a 20% project. [disclaimer: I work at Google, but not on Jigsaw.]
    - r3bl 10y ago
      I'm working for a journalistic organization that cooperated with Google to create one of the graduated Jigsaw projects (Investigative Dashboard). We were considering using Project Shield relatively recently (as in, some time this year), but at that time, the info about how it actually worked was... well, non-existing on their website. IIRC, their SSL support is also fairly new, which made it pretty unusable until very recently.
      - eli 10y ago
        Well by design it's putting Google in a MITM position for your site (as is Cloudflare for the matter). You're either OK with that or you're not. I don't see how they'd be able to effectively stop Layer 7 attacks without being able to see unencrypted requests.
      - boulos 10y ago
        Interestingly, I've never looked deeply at what Project Shield does under the covers but I know what all the underlying technology "must" be. The main difference between GCLB and Shield, is that Shield is a free service (operated by a different group, as already mentioned) explicitly for those at risk of censorship. As another poster indicated, if you want someone to terminate SSL for you, you're going to need to hand them a key. We encrypt ours at rest, provide the same security and care to your secret material as we do Google properties, and as you can see with our Customer Supplied Encryption Key support for Persistent Disk and GCS, we care a lot about letting you control access to your data. If you don't mind me asking, to whom are you comfortable uploading your keys to? Disclosure: I work on Google Cloud, so I'm actively trying to take your money in exchange for our services.
- robin_reala 10y ago
  Interestingly their FAQ says: Project Shield welcomes applications from websites serving news, human rights, or elections monitoring content. We do not provide service to other types of content, including gaming, businesses, or individual blogs. I guess Brian’s a reasonable exception.
  - jpgvm 10y ago
    Further down they explicitly mention journalists, which I believe is a reasonable category for Krebs.
- pfg 10y ago
  There's a "server: shield" response header for krebsonsecurity.com, so this is probably a good guess.
  - Springtime 10y ago
    Krebs confirmed this in his latest blog post [1]. Today, I am happy to report that the site is back up — this time under Project Shield, a free program run by Google to help protect journalists from online censorship. [1] https://krebsonsecurity.com/2016/09/the-democratization-of-censorship/ https://krebsonsecurity.com/2016/09/the-democratization-of-c...
pktgen 10y ago
I realize Krebs seems to be using Shield, not vanilla GCP, but I'd like to know the answer to this as well. Krebs' article describes two attack methods, fake HTTP requests and a simple volumetric flood of GRE packets. L7 mitigation clearly isn't part of GCP's service. The GRE packets should be a different story -- if the customer isn't using GRE, they should be able to drop this traffic easily using the GCE firewall or GCE load balancer. The vulnerability report at http://www.securityfocus.com/archive/1/532239 http://www.securityfocus.com/archive/1/532239 suggests the GCE firewall is, or at least was as of May 2014, ineffective against such an attack (even if your firewall rule set will drop the problem traffic). But it also says the GCE load balancers do not have the same problem. Krebs didn't mention how much of the 600+ Gbps was GRE, but let's say a GCE load balancer was configured to pass through only TCP/80 and received 100 Gbps of non-TCP traffic. Would the load balancer function as intended and drop all of this with no interruption to legitimate TCP traffic, would Google null route you, or something else?
- poooogles 10y ago
  I fear it would be null routing. Would be good to know someone with some experience of this that's not from Google. We can get all the reassurance from an account manager that they'd never stop our service. Real life doesn't always play out that way though...
  - boulos 10y ago
    I don't think that's true for GCLB. We might rate limit packets arriving or some such, but I'm not a networking expect (as with other posts, I'm sure someone else at Google will send me an excellent email explaining a subtlety here). We generally don't talk about DoS attacks at Google (sorry). But I can say we're familiar with them, and that Cloud uses the same front end and networking infrastructure as the rest of Google here. We do understand (per project, the unit of billing) if there's tons of ingress/egress from a single project, and your intuition is right: it would be wise to respond to "Hey, is this 1 Tbps of traffic legitimate?" quickly ;). Disclosure: I'm an engineer in Cloud (mostly GCE), but since I'm not an account manager I refuse to reassure you ;).
roel_v 10y ago
*rite (sorry)
- poooogles 10y ago
  TIL. Never knew that thanks.
qoqiwd8o1i2j 10y ago
rite of passage, rite of spring ...
boulos 10y ago
As I said down thread, we (Google) and some of our customers are routinely the target of attacks (DDoS and otherwise). We don't talk about it (sorry? yet?), but we have mechanisms to protect you as well as ourselves and other customers. My hope is we'll be talking about a particularly high profile one soon, but if you've got an account and want to talk to us privately, we're here to help. Disclosure: I work on Cloud, but I'm not a networking expert.
jaypaulynice 10y ago
The underlying technology isn't that difficult to understand...there is a web application firewall that sits in front of the web server serving the real content. There is also a load balancer to distribute the requests. You can block the attack based on IP, request headers, rate control, geo, etc. The problem with some ddos attacks is the size of the request body. That's what usually overwhelms a server. Ddos attacks do knock out the servers, but since the requests are distributed, then you don't see that. Because so many servers are getting knocked out, then real paying customers can't serve their content. I'm guessing that's why they pulled the plug. Google's infrastructure is so large that knocking out a few servers won't impact much, they just spin them back up.

MaysonL 10y ago

Trying to access 130.211.45.45, which is listed in OP as the IP adress, keep getting 502 server errors.

poooogles 10y ago
Sure you're sending the right host header?
neitsab 10y ago
You can try hardcoding the IP address in your hosts file (see my comment below: https://news.ycombinator.com/item?id=12574428#12575021 https://news.ycombinator.com/item?id=12574428#12575021). This will get you to the site through a regular request.

hrrsn 10y ago

I'd wager this is Project Shield rather than regular GCP. https://projectshield.withgoogle.com/public/ https://projectshield.withgoogle.com/public/

eric_bullington 10y ago
This is the first I've heard of Google's Project Shield. Very cool project, is it relatively new? Does anyone know if this was the result of someone's 20% time or a top-down Google initiative?
- kerneis 10y ago
  Project Shield is part of Jigsaw (formerly "Google Ideas"): see https://jigsaw.google.com/ https://jigsaw.google.com/ and https://medium.com/jigsaw https://medium.com/jigsaw for more related projects and context. Google Ideas is 5 years old now. I'm not aware of it having started as a 20% project. [disclaimer: I work at Google, but not on Jigsaw.]
  - r3bl 10y ago
    I'm working for a journalistic organization that cooperated with Google to create one of the graduated Jigsaw projects (Investigative Dashboard). We were considering using Project Shield relatively recently (as in, some time this year), but at that time, the info about how it actually worked was... well, non-existing on their website. IIRC, their SSL support is also fairly new, which made it pretty unusable until very recently.
    - eli 10y ago
      Well by design it's putting Google in a MITM position for your site (as is Cloudflare for the matter). You're either OK with that or you're not. I don't see how they'd be able to effectively stop Layer 7 attacks without being able to see unencrypted requests.
    - boulos 10y ago
      Interestingly, I've never looked deeply at what Project Shield does under the covers but I know what all the underlying technology "must" be. The main difference between GCLB and Shield, is that Shield is a free service (operated by a different group, as already mentioned) explicitly for those at risk of censorship. As another poster indicated, if you want someone to terminate SSL for you, you're going to need to hand them a key. We encrypt ours at rest, provide the same security and care to your secret material as we do Google properties, and as you can see with our Customer Supplied Encryption Key support for Persistent Disk and GCS, we care a lot about letting you control access to your data. If you don't mind me asking, to whom are you comfortable uploading your keys to? Disclosure: I work on Google Cloud, so I'm actively trying to take your money in exchange for our services.
robin_reala 10y ago
Interestingly their FAQ says: Project Shield welcomes applications from websites serving news, human rights, or elections monitoring content. We do not provide service to other types of content, including gaming, businesses, or individual blogs. I guess Brian’s a reasonable exception.
- jpgvm 10y ago
  Further down they explicitly mention journalists, which I believe is a reasonable category for Krebs.
pfg 10y ago
There's a "server: shield" response header for krebsonsecurity.com, so this is probably a good guess.
- Springtime 10y ago
  Krebs confirmed this in his latest blog post [1]. Today, I am happy to report that the site is back up — this time under Project Shield, a free program run by Google to help protect journalists from online censorship. [1] https://krebsonsecurity.com/2016/09/the-democratization-of-censorship/ https://krebsonsecurity.com/2016/09/the-democratization-of-c...

pktgen 10y ago

I realize Krebs seems to be using Shield, not vanilla GCP, but I'd like to know the answer to this as well. Krebs' article describes two attack methods, fake HTTP requests and a simple volumetric flood of GRE packets. L7 mitigation clearly isn't part of GCP's service. The GRE packets should be a different story -- if the customer isn't using GRE, they should be able to drop this traffic easily using the GCE firewall or GCE load balancer. The vulnerability report at http://www.securityfocus.com/archive/1/532239 http://www.securityfocus.com/archive/1/532239 suggests the GCE firewall is, or at least was as of May 2014, ineffective against such an attack (even if your firewall rule set will drop the problem traffic). But it also says the GCE load balancers do not have the same problem. Krebs didn't mention how much of the 600+ Gbps was GRE, but let's say a GCE load balancer was configured to pass through only TCP/80 and received 100 Gbps of non-TCP traffic. Would the load balancer function as intended and drop all of this with no interruption to legitimate TCP traffic, would Google null route you, or something else?

poooogles 10y ago
I fear it would be null routing. Would be good to know someone with some experience of this that's not from Google. We can get all the reassurance from an account manager that they'd never stop our service. Real life doesn't always play out that way though...
- boulos 10y ago
  I don't think that's true for GCLB. We might rate limit packets arriving or some such, but I'm not a networking expect (as with other posts, I'm sure someone else at Google will send me an excellent email explaining a subtlety here). We generally don't talk about DoS attacks at Google (sorry). But I can say we're familiar with them, and that Cloud uses the same front end and networking infrastructure as the rest of Google here. We do understand (per project, the unit of billing) if there's tons of ingress/egress from a single project, and your intuition is right: it would be wise to respond to "Hey, is this 1 Tbps of traffic legitimate?" quickly ;). Disclosure: I'm an engineer in Cloud (mostly GCE), but since I'm not an account manager I refuse to reassure you ;).

roel_v 10y ago

*rite (sorry)

poooogles 10y ago
TIL. Never knew that thanks.

qoqiwd8o1i2j 10y ago

rite of passage, rite of spring ...

boulos 10y ago

As I said down thread, we (Google) and some of our customers are routinely the target of attacks (DDoS and otherwise). We don't talk about it (sorry? yet?), but we have mechanisms to protect you as well as ourselves and other customers. My hope is we'll be talking about a particularly high profile one soon, but if you've got an account and want to talk to us privately, we're here to help. Disclosure: I work on Cloud, but I'm not a networking expert.

jaypaulynice 10y ago

The underlying technology isn't that difficult to understand...there is a web application firewall that sits in front of the web server serving the real content. There is also a load balancer to distribute the requests. You can block the attack based on IP, request headers, rate control, geo, etc. The problem with some ddos attacks is the size of the request body. That's what usually overwhelms a server. Ddos attacks do knock out the servers, but since the requests are distributed, then you don't see that. Because so many servers are getting knocked out, then real paying customers can't serve their content. I'm guessing that's why they pulled the plug. Google's infrastructure is so large that knocking out a few servers won't impact much, they just spin them back up.