5 ms·

What are the privacy implications of this? Will Cloudflare be able to see GitLab traffic? Was/is Fastly able to before? And is there any reason to trust one ove

What are the privacy implications of this? Will Cloudflare be able to see GitLab traffic? Was/is Fastly able to before? And is there any reason to trust one over the other?

eganist 7y ago
> As a result of the complexity and requirements, we realized we would like to have a solution for CDN, WAF, and DDOS protection with one vendor. [emphasis mine] Cloudflare will see All The Things™ as TLS termination at Cloudflare is required for Cloudflare to be able to analyze web traffic and block common attacks. Just based on that one sentence, it seems like a value play more than anything. That they care enough for the defense-in-depth is probably a good sign, but that also shows a bias in favor of security over privacy. Depending on your personal or corporate privacy preferences/politics, that may or may not be desirable, but it's also for you to decide if any resulting risks are acceptable. My hope is that Gitlab pressed Cloudflare on the process changes that took place after the WAF outage last year. --- (this comment has been heavily edited since it was originally published)
- hartator 7y ago
  I don’t think that’s true. You can serve your own SSL certificate through Cloudflare and they won’t be able to inspect it?
  - eganist 7y ago
    > I don’t think that’s true. You can serve your own SSL certificate through Cloudflare and they won’t be able to inspect it? Unless there's been a serious homomorphic encryption breakthrough, it's impossible to impose layer 7 intrusion prevention if layer 6 encryption is passthrough. They subscribed to consolidate the listed offerings (CDN, WAF, DDoS protection). That WAF is mentioned at all is a tip that Cloudflare must decrypt and analyze Gitlab's traffic before re-encrypting it for delivery to Gitlab's application servers.
  - judge2020 7y ago
    There are 2 different "use your own SSL" solutions at the moment (for enterprise): A. Uploading your own certificate - this just uploads it and CF still does HTTPS interception. B. Spectrum - Cloudflare will proxy the actual port 443 TCP connections (which might be the plan for SSH connections to GitLab that are not altssh.gitlab.com). The issue is that this only does layer 4 IP reputation DDOS protection; no WAF, etc is done so this most likely isn't what they will do for the main gitlab.com site.
    - T4cC0re 7y ago
      In the case of Cloudflare, there also is option C, "Keyless-SSL", where you control the key. (https://www.cloudflare.com/ssl/keyless-ssl/ https://www.cloudflare.com/ssl/keyless-ssl/) But that is irrelevant in our case. We opted to use Cloudflare Spectrum to facilitate our change. There is no way to mix Spectrum and non-Spectrum on one hostname, but Spectrum allows us to choose between TCP, HTTP, and HTTPS pipelines. While we intend to TCP proxy port 22 on gitlab.com and 443 on altssh.gitlab.com, this also allows us to use Cloudflare's HTTP(s) content processing pipeline on ports 80 and 443 on gitlab.com. In the latter case Cloudflare will be terminating TLS, performing WAF, CDN and Workers duty, then re-encrypt and send the request to our origin. You can look at some visualization of this here: https://gitlab.com/gitlab-com/gl-infra/readiness/tree/master/cloudflare#architecture https://gitlab.com/gitlab-com/gl-infra/readiness/tree/master...
IfOnlyYouKnew 7y ago
I trust Cloudflare a whole lot more than Gitlab. CF makes all the right noises around privacy for their 1.1.1.1 service, for example. Gitlab is still the team that didn't notice that three of their three independent backup mechanisms had not been working for weeks to months. And while wholly cloning Github is legal, I started having serious doubts about their ethical instincts when they repeatedly tried to incite people against Github for cloning minor features. Beyond that: it depends. I tend to believe large vendors are worse for privacy of generic consumers, because they have the scale where it pays off to sell data or do "big data" analysis. They are better for individual high-value targets because they have more. to lose, and less to win, relative to whatever they could expect by screwing you over. Protection against rogue employees is also something that happens in larger teams, if at all.
- eganist 7y ago
  > Gitlab is still the team that didn't notice that three of their three independent backup mechanisms had not been working for weeks to months. So, this is a really good point. Someone made the smart decision within Gitlab to outsource some of the harder problems to an outside party with serious experience on the topics rather than trying to script their way out of the challenge themselves. > Protection against rogue employees is also something that happens in larger teams, if at all. ref: amazon w/r/t capital one for a good example of this exact failure mode. (Can't tell if we're in agreement; your statement is a bit ambiguous, but given "if at all," I'm assuming we're on the same page)
  - seanhunter 7y ago
    My understanding is that the Capital One breach didn't involve any secret information known only to Amazon employees. CapOne had a misconfigured firewall and allowed public internet traffic to servers which were not hardened and got compromised. The person behind the breach happens to have previously worked at Amazon, but it wasn't a rogue Amazon employee breaching an AWS customer.
    - eganist 7y ago
T4cC0re 7y ago
Here are my answers to those questions: (Note: I am a GitLab employee, so they are probably a bit biased) - What are the privacy implications of this? Theoretically, any 3rd party in line of traffic (in this case it is Cloudflare) has the ability to monitor and alter traffic. This is especially true when said 3rd party does inspections into TLS traffic. The privacy implications are, that they can see and alter any traffic between the client and the origin. But in the case of Cloudflare and GitLab, that is desired, as we want to take advantage of Cloudflare's Web Application Firewall to further protect us from 0-days and other threats, where mitigation using traditional methods might cause a delay, that cloud lead to exploitation of those, causing bigger harm. My personal opinion on this (it does not necessarily reflect the view of GitLab) is, that Cloudflare is a vendor I personally trust in terms of adhering to what they say in regards to privacy. While they, as any other vendor, could be breached, I see the benefit of their Security solution bigger, than the breach-risk. - Will Cloudflare be able to see GitLab traffic? Yes, partially. SSH is not inspected by CF, despite being proxied by them. HTTP can be inspected by anybody on the net and yes, thus CF, too. HTTPS will be terminated by Cloudflare (and re-encrypted when talking to our origin) for some reasons: - CDN: They need to know which resource you request in order to serve it from a cache - WAF: Once enabled (not from the start) it will scan requests for malicious content and either block or challenge the request - Workers: A highly integrated FaaS platform, we can use to dynamically authenticate requests to cached resources and other logic we may want to execute on the edge of Cloudflare's network. Despite all that Cloudflare does not log raw requests. On our end, we only see the information about where the request came from, user agents, etc. You can find out more about what Cloudflare logs over here: https://blog.cloudflare.com/what-cloudflare-logs/ https://blog.cloudflare.com/what-cloudflare-logs/ - Was/is Fastly able to before? Fastly is in the same position as Cloudflare will be in the future. Right now https://about.gitlab.com https://about.gitlab.com is served via fastly pointing to an origin, which we control. If fastly would be breached, the attacker could control the contents of about.gitlab.com. Static assets are also hosted via fastly and are also subject to be possibly altered by an attacker. - And is there any reason to trust one over the other? The decision within GitLab was made purely on technical grounds. Cloudflare was the only solution meeting our criteria to keep serving traffic via SSH on the same host as we do right now. Speaking for myself again, I trust Cloudflare and have personally been a customer of theirs for a long time 2012-ish). While I do not have that kind of relationship with fastly, I believe both vendors to be trustworthy in doing their best to protect the privacy of their users. You can find more information about how we are going to deploy Cloudflare here: https://gitlab.com/gitlab-com/gl-infra/readiness/tree/master/cloudflare#architecture https://gitlab.com/gitlab-com/gl-infra/readiness/tree/master...

What are the privacy implications of this? Will Cloudflare be able to see GitLab traffic? Was/is Fastly able to before? And is there any reason to trust one over the other?

eganist 7y ago

> As a result of the complexity and requirements, we realized we would like to have a solution for CDN, WAF, and DDOS protection with one vendor. [emphasis mine] Cloudflare will see All The Things™ as TLS termination at Cloudflare is required for Cloudflare to be able to analyze web traffic and block common attacks. Just based on that one sentence, it seems like a value play more than anything. That they care enough for the defense-in-depth is probably a good sign, but that also shows a bias in favor of security over privacy. Depending on your personal or corporate privacy preferences/politics, that may or may not be desirable, but it's also for you to decide if any resulting risks are acceptable. My hope is that Gitlab pressed Cloudflare on the process changes that took place after the WAF outage last year. --- (this comment has been heavily edited since it was originally published)

hartator 7y ago
I don’t think that’s true. You can serve your own SSL certificate through Cloudflare and they won’t be able to inspect it?
- eganist 7y ago
  > I don’t think that’s true. You can serve your own SSL certificate through Cloudflare and they won’t be able to inspect it? Unless there's been a serious homomorphic encryption breakthrough, it's impossible to impose layer 7 intrusion prevention if layer 6 encryption is passthrough. They subscribed to consolidate the listed offerings (CDN, WAF, DDoS protection). That WAF is mentioned at all is a tip that Cloudflare must decrypt and analyze Gitlab's traffic before re-encrypting it for delivery to Gitlab's application servers.
- judge2020 7y ago
  There are 2 different "use your own SSL" solutions at the moment (for enterprise): A. Uploading your own certificate - this just uploads it and CF still does HTTPS interception. B. Spectrum - Cloudflare will proxy the actual port 443 TCP connections (which might be the plan for SSH connections to GitLab that are not altssh.gitlab.com). The issue is that this only does layer 4 IP reputation DDOS protection; no WAF, etc is done so this most likely isn't what they will do for the main gitlab.com site.
  - T4cC0re 7y ago
    In the case of Cloudflare, there also is option C, "Keyless-SSL", where you control the key. (https://www.cloudflare.com/ssl/keyless-ssl/ https://www.cloudflare.com/ssl/keyless-ssl/) But that is irrelevant in our case. We opted to use Cloudflare Spectrum to facilitate our change. There is no way to mix Spectrum and non-Spectrum on one hostname, but Spectrum allows us to choose between TCP, HTTP, and HTTPS pipelines. While we intend to TCP proxy port 22 on gitlab.com and 443 on altssh.gitlab.com, this also allows us to use Cloudflare's HTTP(s) content processing pipeline on ports 80 and 443 on gitlab.com. In the latter case Cloudflare will be terminating TLS, performing WAF, CDN and Workers duty, then re-encrypt and send the request to our origin. You can look at some visualization of this here: https://gitlab.com/gitlab-com/gl-infra/readiness/tree/master/cloudflare#architecture https://gitlab.com/gitlab-com/gl-infra/readiness/tree/master...

IfOnlyYouKnew 7y ago

I trust Cloudflare a whole lot more than Gitlab. CF makes all the right noises around privacy for their 1.1.1.1 service, for example. Gitlab is still the team that didn't notice that three of their three independent backup mechanisms had not been working for weeks to months. And while wholly cloning Github is legal, I started having serious doubts about their ethical instincts when they repeatedly tried to incite people against Github for cloning minor features. Beyond that: it depends. I tend to believe large vendors are worse for privacy of generic consumers, because they have the scale where it pays off to sell data or do "big data" analysis. They are better for individual high-value targets because they have more. to lose, and less to win, relative to whatever they could expect by screwing you over. Protection against rogue employees is also something that happens in larger teams, if at all.

eganist 7y ago
> Gitlab is still the team that didn't notice that three of their three independent backup mechanisms had not been working for weeks to months. So, this is a really good point. Someone made the smart decision within Gitlab to outsource some of the harder problems to an outside party with serious experience on the topics rather than trying to script their way out of the challenge themselves. > Protection against rogue employees is also something that happens in larger teams, if at all. ref: amazon w/r/t capital one for a good example of this exact failure mode. (Can't tell if we're in agreement; your statement is a bit ambiguous, but given "if at all," I'm assuming we're on the same page)
- seanhunter 7y ago
  My understanding is that the Capital One breach didn't involve any secret information known only to Amazon employees. CapOne had a misconfigured firewall and allowed public internet traffic to servers which were not hardened and got compromised. The person behind the breach happens to have previously worked at Amazon, but it wasn't a rogue Amazon employee breaching an AWS customer.
  - eganist 7y ago

T4cC0re 7y ago

Here are my answers to those questions: (Note: I am a GitLab employee, so they are probably a bit biased) - What are the privacy implications of this? Theoretically, any 3rd party in line of traffic (in this case it is Cloudflare) has the ability to monitor and alter traffic. This is especially true when said 3rd party does inspections into TLS traffic. The privacy implications are, that they can see and alter any traffic between the client and the origin. But in the case of Cloudflare and GitLab, that is desired, as we want to take advantage of Cloudflare's Web Application Firewall to further protect us from 0-days and other threats, where mitigation using traditional methods might cause a delay, that cloud lead to exploitation of those, causing bigger harm. My personal opinion on this (it does not necessarily reflect the view of GitLab) is, that Cloudflare is a vendor I personally trust in terms of adhering to what they say in regards to privacy. While they, as any other vendor, could be breached, I see the benefit of their Security solution bigger, than the breach-risk. - Will Cloudflare be able to see GitLab traffic? Yes, partially. SSH is not inspected by CF, despite being proxied by them. HTTP can be inspected by anybody on the net and yes, thus CF, too. HTTPS will be terminated by Cloudflare (and re-encrypted when talking to our origin) for some reasons: - CDN: They need to know which resource you request in order to serve it from a cache - WAF: Once enabled (not from the start) it will scan requests for malicious content and either block or challenge the request - Workers: A highly integrated FaaS platform, we can use to dynamically authenticate requests to cached resources and other logic we may want to execute on the edge of Cloudflare's network. Despite all that Cloudflare does not log raw requests. On our end, we only see the information about where the request came from, user agents, etc. You can find out more about what Cloudflare logs over here: https://blog.cloudflare.com/what-cloudflare-logs/ https://blog.cloudflare.com/what-cloudflare-logs/ - Was/is Fastly able to before? Fastly is in the same position as Cloudflare will be in the future. Right now https://about.gitlab.com https://about.gitlab.com is served via fastly pointing to an origin, which we control. If fastly would be breached, the attacker could control the contents of about.gitlab.com. Static assets are also hosted via fastly and are also subject to be possibly altered by an attacker. - And is there any reason to trust one over the other? The decision within GitLab was made purely on technical grounds. Cloudflare was the only solution meeting our criteria to keep serving traffic via SSH on the same host as we do right now. Speaking for myself again, I trust Cloudflare and have personally been a customer of theirs for a long time 2012-ish). While I do not have that kind of relationship with fastly, I believe both vendors to be trustworthy in doing their best to protect the privacy of their users. You can find more information about how we are going to deploy Cloudflare here: https://gitlab.com/gitlab-com/gl-infra/readiness/tree/master/cloudflare#architecture https://gitlab.com/gitlab-com/gl-infra/readiness/tree/master...