7 ms·
Upcoming changes to our CDN for GitLab.com
- eat_veggies 7y agoWhat are the privacy implications of this? Will Cloudflare be able to see GitLab traffic? Was/is Fastly able to before? And is there any reason to trust one over the other?
- eganist 7y ago> As a result of the complexity and requirements, we realized we would like to have a solution for CDN, WAF, and DDOS protection with one vendor. [emphasis mine] Cloudflare will see All The Things™ as TLS termination at Cloudflare is required for Cloudflare to be able to analyze web traffic and block common attacks. Just based on that one sentence, it seems like a value play more than anything. That they care enough for the defense-in-depth is probably a good sign, but that also shows a bias in favor of security over privacy. Depending on your personal or corporate privacy preferences/politics, that may or may not be desirable, but it's also for you to decide if any resulting risks are acceptable. My hope is that Gitlab pressed Cloudflare on the process changes that took place after the WAF outage last year. --- (this comment has been heavily edited since it was originally published)
- hartator 7y agoI don’t think that’s true. You can serve your own SSL certificate through Cloudflare and they won’t be able to inspect it?
- eganist 7y ago> I don’t think that’s true. You can serve your own SSL certificate through Cloudflare and they won’t be able to inspect it? Unless there's been a serious homomorphic encryption breakthrough, it's impossible to impose layer 7 intrusion prevention if layer 6 encryption is passthrough. They subscribed to consolidate the listed offerings (CDN, WAF, DDoS protection). That WAF is mentioned at all is a tip that Cloudflare must decrypt and analyze Gitlab's traffic before re-encrypting it for delivery to Gitlab's application servers.
- judge2020 7y agoThere are 2 different "use your own SSL" solutions at the moment (for enterprise): A. Uploading your own certificate - this just uploads it and CF still does HTTPS interception. B. Spectrum - Cloudflare will proxy the actual port 443 TCP connections (which might be the plan for SSH connections to GitLab that are not altssh.gitlab.com). The issue is that this only does layer 4 IP reputation DDOS protection; no WAF, etc is done so this most likely isn't what they will do for the main gitlab.com site.
- T4cC0re 7y agoIn the case of Cloudflare, there also is option C, "Keyless-SSL", where you control the key. (https://www.cloudflare.com/ssl/keyless-ssl/ https://www.cloudflare.com/ssl/keyless-ssl/) But that is irrelevant in our case. We opted to use Cloudflare Spectrum to facilitate our change. There is no way to mix Spectrum and non-Spectrum on one hostname, but Spectrum allows us to choose between TCP, HTTP, and HTTPS pipelines. While we intend to TCP proxy port 22 on gitlab.com and 443 on altssh.gitlab.com, this also allows us to use Cloudflare's HTTP(s) content processing pipeline on ports 80 and 443 on gitlab.com. In the latter case Cloudflare will be terminating TLS, performing WAF, CDN and Workers duty, then re-encrypt and send the request to our origin. You can look at some visualization of this here: https://gitlab.com/gitlab-com/gl-infra/readiness/tree/master/cloudflare#architecture https://gitlab.com/gitlab-com/gl-infra/readiness/tree/master...
- IfOnlyYouKnew 7y agoI trust Cloudflare a whole lot more than Gitlab. CF makes all the right noises around privacy for their 1.1.1.1 service, for example. Gitlab is still the team that didn't notice that three of their three independent backup mechanisms had not been working for weeks to months. And while wholly cloning Github is legal, I started having serious doubts about their ethical instincts when they repeatedly tried to incite people against Github for cloning minor features. Beyond that: it depends. I tend to believe large vendors are worse for privacy of generic consumers, because they have the scale where it pays off to sell data or do "big data" analysis. They are better for individual high-value targets because they have more. to lose, and less to win, relative to whatever they could expect by screwing you over. Protection against rogue employees is also something that happens in larger teams, if at all.
- eganist 7y ago> Gitlab is still the team that didn't notice that three of their three independent backup mechanisms had not been working for weeks to months. So, this is a really good point. Someone made the smart decision within Gitlab to outsource some of the harder problems to an outside party with serious experience on the topics rather than trying to script their way out of the challenge themselves. > Protection against rogue employees is also something that happens in larger teams, if at all. ref: amazon w/r/t capital one for a good example of this exact failure mode. (Can't tell if we're in agreement; your statement is a bit ambiguous, but given "if at all," I'm assuming we're on the same page)
- seanhunter 7y agoMy understanding is that the Capital One breach didn't involve any secret information known only to Amazon employees. CapOne had a misconfigured firewall and allowed public internet traffic to servers which were not hardened and got compromised. The person behind the breach happens to have previously worked at Amazon, but it wasn't a rogue Amazon employee breaching an AWS customer.
- eganist 7y ago
- T4cC0re 7y agoHere are my answers to those questions: (Note: I am a GitLab employee, so they are probably a bit biased) - What are the privacy implications of this? Theoretically, any 3rd party in line of traffic (in this case it is Cloudflare) has the ability to monitor and alter traffic. This is especially true when said 3rd party does inspections into TLS traffic. The privacy implications are, that they can see and alter any traffic between the client and the origin. But in the case of Cloudflare and GitLab, that is desired, as we want to take advantage of Cloudflare's Web Application Firewall to further protect us from 0-days and other threats, where mitigation using traditional methods might cause a delay, that cloud lead to exploitation of those, causing bigger harm. My personal opinion on this (it does not necessarily reflect the view of GitLab) is, that Cloudflare is a vendor I personally trust in terms of adhering to what they say in regards to privacy. While they, as any other vendor, could be breached, I see the benefit of their Security solution bigger, than the breach-risk. - Will Cloudflare be able to see GitLab traffic? Yes, partially. SSH is not inspected by CF, despite being proxied by them. HTTP can be inspected by anybody on the net and yes, thus CF, too. HTTPS will be terminated by Cloudflare (and re-encrypted when talking to our origin) for some reasons: - CDN: They need to know which resource you request in order to serve it from a cache - WAF: Once enabled (not from the start) it will scan requests for malicious content and either block or challenge the request - Workers: A highly integrated FaaS platform, we can use to dynamically authenticate requests to cached resources and other logic we may want to execute on the edge of Cloudflare's network. Despite all that Cloudflare does not log raw requests. On our end, we only see the information about where the request came from, user agents, etc. You can find out more about what Cloudflare logs over here: https://blog.cloudflare.com/what-cloudflare-logs/ https://blog.cloudflare.com/what-cloudflare-logs/ - Was/is Fastly able to before? Fastly is in the same position as Cloudflare will be in the future. Right now https://about.gitlab.com https://about.gitlab.com is served via fastly pointing to an origin, which we control. If fastly would be breached, the attacker could control the contents of about.gitlab.com. Static assets are also hosted via fastly and are also subject to be possibly altered by an attacker. - And is there any reason to trust one over the other? The decision within GitLab was made purely on technical grounds. Cloudflare was the only solution meeting our criteria to keep serving traffic via SSH on the same host as we do right now. Speaking for myself again, I trust Cloudflare and have personally been a customer of theirs for a long time 2012-ish). While I do not have that kind of relationship with fastly, I believe both vendors to be trustworthy in doing their best to protect the privacy of their users. You can find more information about how we are going to deploy Cloudflare here: https://gitlab.com/gitlab-com/gl-infra/readiness/tree/master/cloudflare#architecture https://gitlab.com/gitlab-com/gl-infra/readiness/tree/master...
- secondo 7y agoAs a result of the complexity and requirements, we realized we would like to have a solution for CDN, WAF, and DDOS protection with one vendor. Depending on your definition of ddos protection Fastly has those things along with granular and semi scriptable configurability via vcl. Granted it’s been 2-3 years since I last did a comparison between Fastly and Cloudflare but I can’t understand why a company with presumably a need for configurability at the edge would make this choice. Cloudflare’s entry tiers are unbeatable in price but beyond that it’s extremely bulky.
- eganist 7y agoIs there anything you wish Cloudflare offered with Workers (https://developers.cloudflare.com/workers/ https://developers.cloudflare.com/workers/) or VCL->Workers conversion (https://developers.cloudflare.com/workers/archive/recipes/vcl-conversion/ https://developers.cloudflare.com/workers/archive/recipes/vc...) specifically?
- T4cC0re 7y agoOne of the key reasons we decided to utilize Cloudflare as our vendor for WAF and CDN was, that they uniquely offer to run SSH (or any TCP application, really) and HTTP(s) on the same hostname.
- secondo 7y agoYou’re setting yourself up for some future and potentially hard to introspect problems for yourself and your users with this setup. There are good reasons to keep those endpoints separate. As an example, the anycast solution you’ll rely on on that host, while great for short lived tcp sessions (https) will be problematic for your long lived sessions (ssh) which will be disconnected during routing changes in cloudflares network - ddos or operational changes. That is unless you establish sub flows for those sessions using a unicast IP address but now you’re accumulating additional complexity for the sake of wanting it all on a single hostname.
- T4cC0re 7y ago
- deeblering4 7y agoIs anyone else getting worried about the amount of the internet that is proxied through cloudflare?
- hoistbypetard 7y agoThat worries me the same way the consolidation of email services does, even though I don't find CloudFlare themselves as scary as I find some of the email providers. We've seen monocultures before. They can be really unhealthy for the ecosystem.
- jesseb 7y agoIt's going to be interesting to see how much of the net is unavailable when they inevitably have a major outage.
- eganist 7y agohttps://blog.cloudflare.com/details-of-the-cloudflare-outage-on-july-2-2019/ https://blog.cloudflare.com/details-of-the-cloudflare-outage...
- filmgirlcw 7y agoAs opposed to the amount of the internet hosted by Amazon? Like, I understand the argument, I’m just not sure why we get angsty about one type of market dominance and not another.
- im3w1l 7y agoHow much of the internet is hosted by Amazon?
- BlueTemplar 7y agoFrom that "I tried to boycott the GAFAM" story, IIRC it was "about half" ?
- stephenr 7y ago
- cagenut 7y agowow you rarely see "tech" companies migrate in this direction (vs say enterprise or smb).
- porker 7y agoI wonder how much Fastly's pricing impacted this? OT but my clients are small players (10s of GB traffic a month); but they value availability and low latency for their dynamic content. Fastly (and varnish) are ideal for this, but every time I talk to Fastly about services beyond the PAYG-CDN it's "min $2k/month", "$5k/month + fees". I'd rather use them than Cloudflare, I've been accused of being a walking advert for Fastly - but they are pushing me towards other providers and proving actively hostile to small clients. Good for them if they are raking it in from enterprise customers. But AWS manages to cater for both ends of the market, and it's a shame to see good services price themselves into "enterprise-only" territory.
- jgrahamc 7y agoThere's a comment below from a Gitlab employee that mentions the breadth of the Cloudflare feature set as being key: https://news.ycombinator.com/item?id=22073753 https://news.ycombinator.com/item?id=22073753
- deleted 7y ago[deleted]
- m4lvin 7y agoDoes this mean GitLab.com will become unusable via Tor? https://support.cloudflare.com/hc/en-us/articles/203306930-Understanding-Cloudflare-Tor-support-and-Onion-Routing https://support.cloudflare.com/hc/en-us/articles/203306930-U...
- T4cC0re 7y agoThat is a very good question. In general, what you can expect to see when visiting a Cloudflare site via Tor is a CAPTCHA challenge to authenticate your browser to show you're not a bot. As to whether we can enable the `Onion Routing` setting, which would cause your browser to connect to Cloudflare via Tor and not via an exit-node, I don't know if we may enable it. There might be some compliance issues with that, which may prevent us from enabling that. I have however created an issue on this for further investigation whether that might be possible: https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/8954 https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues...
- jgrahamc 7y agoIf you're using TBB then you should NOT expect to see a CAPTCHA from Cloudflare at all. Unless the customer specifically wants to challenge Tor users we do not challenge them just because they use TBB.
- mattbee 7y agoOh - we're back to talking about trust & Cloudflare again. Cloudflare have always hosted [] booter / stresser / DDoS-for-hire sites. Right now I can find 8 sites on their network when I search for "booter" (it's been higher). So Cloudflare are responsible for keeping criminal activity online, and defending them from visibility to their downstream hosts, who might otherwise cut them off. Cloudflare also sell DDoS protection from the same criminals. Because they host them, they can see where attacks will happen next - an advantage other big CDNs won't tolerate. That adds up to a protection racket - in the 90s this type of shady business wouldn't have been able to source connectivity. But now they're the good guys :( I don't get it. [] Cloudflare have tended to cry "we don't host anything!". But if they stopped providing service, these sites probably couldn't exist. I call that mission-critical hosting.
- jakejarvis 7y agovia https://gitlab.com/gitlab-com/gl-infra/readiness/tree/master/cloudflare#execution https://gitlab.com/gitlab-com/gl-infra/readiness/tree/master...: > 1. Once traffic is ensured to flow though Cloudflare, we initiate decommission of Route53. > We would disable the transfer lock and generate an auth code. > immediately after, we move the domain over to the Cloudflare registry I love the overall plan but this part would worry me... The most obvious contingency plan when you're putting so many eggs into one basket is to keep a kill-switch somewhere like your domain registrar, where you can abandon ship entirely by switching nameservers in the worst of scenarios, right? Correct me if I'm wrong but when the Cloudflare dashboard went down a few weeks (months?) ago, no sort of DNS-level changes would have been possible. (Either way, I don't think you can even set external nameservers for domains on Cloudflare Registrar yet?) Just curious about the thinking behind this particular move and why the pros outweigh the cons of leaving the domain where it is.