7 ms·
Container-Native Multi-Cluster Global Load Balancing with Cloud Armor on GCP
- pmlnr 7y agoIs the title peak buzzword-era, or is there more to come?
- dna_polymerase 7y agoThe page greets you with: > Kubernetes Accelerated Your Path to Enterprise Cloud Native I have no idea what they want from me.
- djohnston 7y agoThey want money!
- LeonM 7y agoNah, it doesn't contain blockchain or serverless, so there is still some runway left.
- minxomat 7y agoThis might be an elaborate demo for the upcoming thisblogpostdoesnotexist ML text synthesis project. Just for fun, I used Grover to generate an article based on the headline: > What is the cheapest way to optimize the cloud? Going globally? Using open-source cloud computing platforms? Choosing AWS in the U.S. and for the remaining countries? Choosing SaaS? Or, using AWS and establishing a global presence on the EC2 Container Manager platform? > I know because I did all four. And I won’t do them again. > I am not arguing that globalizing your compute resources through a community cloud is any less important than globalizing your compute resources through your enterprise cloud (though I am somewhat of a skeptic about the value of community clouds). The important point is that migrating to a global cloud requires a state-of-the-art approach to container-native load balancing, container-native load balancing on AWS, and container-native load balancing on other clouds. And you need to be prepared to pay for it.
- politelemon 7y agoCan you tell me a little more about Grover, is it a CLI you just pointed at a URL or a little bit more to generate that article?
- minxomat 7y agoI have a local version of Grover and the model checkpoints, so I used a python script. But I think there's an online demo somewhere.
- DangitBobby 7y agoOkay, but unless Grover can accurately describe how to deploy services and load balancers on GCP, generate valid kubernetes yaml that matches the contents of the article, and generate an infrastructure diagram that matches the infrastructure described in the article, then this was certainly not generated by Grover. It in no way resembles something generated by some ML text synthesis. I'm not sure how anyone who read it could come to the conclusion that it was.
- deleted 7y ago[deleted]
- ljm 7y agoI remember when load balancing was a case of setting up apache or nginx on a server and letting it handle the upstream connections.
- zzzcpan 7y agoIt still is. Proper "global load balancing" is essentially a CDN and nginx with say bind9 running on a bunch of edge nodes from different hosting providers without all those container-native buzzwords is the easiest way to do it.
- manigandham 7y agoIt still is... if you're just running a few apps on single server, or 1 load balancer VM in front of a few other servers. This article is talking about routing traffic from a single IP globally to multiple backend Kubernetes clusters that can be running in any region. Traffic will automatically go to the closest region (with available capacity) or otherwise fallback to other endpoints, while also bypassing much of the Kubernetes network stack to go straight to the running pods via GCP's software-defined network. The complexity here is warranted if you need the flexibility and features. If you don't then you can just stick with nginx.
- ec109685 7y agoI think that is unfair. Each of those buzz words were concepts in the article or actual products that GCP offers. However, this is embarrassing to GCP that this isn’t easier. These were very basic requirements from this customer, yet so much rigmarole was needed to get it set up.
- manigandham 7y agoMulti-cluster global load balancing isn't "very basic" and no other cloud is any easier. AWS requires stacking the Global Accelerator on top of their ALB/ELBs but doesn't have smart routing across clusters and Azure only has Frontdoor which requires completely manual setup and has no backend integration.
- ec109685 7y agoDeploying an app to more than one geographic region on earth should be very basic. For mostly read only sites, it is mean to make your users suffer extra latency and your reliability goes down. The authors had to go read source code to figure out obscure annotations at one point to get container aware networking to work, which then wasn’t compatible with Google’s tools to setup global ingress.
- manigandham 7y agoThat isn't very basic and never has been. If it's just a read-only site then all you need is a CDN, not load balancing with multiple regions running Kubernetes clusters. What other cloud vendor or PaaS offers this? The only one that has seamless global deployment is Cloudflare's Workers but that's limited to Javascript code in a lightweight functions environment.
- ec109685 7y agoThey are making one part of this easier at least: https://twitter.com/vicnastea/status/1232751949117702145?s=20 https://twitter.com/vicnastea/status/1232751949117702145?s=2...
- manigandham 7y agoThey are product and feature names used by the GCP platform (and this article). It's perfectly accurate and descriptive.
- rossmohax 7y ago> This means the Service must be created first, then when the corresponding NEG is created the name can be queried and added to the Terraform project. These kind of reverse dependencies, where app level changes have to be reflected in infrastructure are source of endless bugs and headaches. To do it right, you'd want to encode app and infra changes in the same ubiquitous tool, but "infrastructure as a code" tools such as terraform suck big times at app level deploys. Pulumi takes steps in right direction, where it is actually not that painfull to manage everything, that is what I ended up using for very similar problem configuring GCPs load balancers for my k8s apps.
- cbushko 7y agoWe use terraform for our kubernetes app configuration and it works well. Being able to pass values from one resource to another is extremely powerful. For example, the database instance credentials down into cloudsql proxy. The only and most frustrating missing piece of the terraform kubernetes provider is CRDs. I hacked around it by using some yaml provider bit I would like a cleaner solution.
- rossmohax 7y agoTF kubernetes provider works for simplest cases only. How would you wait for PV to be created, when your TF resources specify PVC? That is to help users to diagnose problems with deployments.
- cppforlife 7y agosomeone asked me to wrap get-kapp.io (one of my projects) with terraform so that they can pass down infra settings to their k8s config. i ended up writing https://githubc.com/k14s/terraform-provider-k14s https://githubc.com/k14s/terraform-provider-k14s (example: https://github.com/k14s/terraform-provider-k14s/tree/master/examples/guestbook https://github.com/k14s/terraform-provider-k14s/tree/master/...). i havent been asked about case where app info is needed by terraform, but may be thats something to add...
- solatic 7y ago> To do it right, you'd want to encode app and infra changes in the same ubiquitous tool Well, that's exactly what we do with Dhall, which lets us put static types on infrastructure tooling that prints out unpredictable output but in a predictable format and transform that output into the format that other dependent tools expect. Everything is then glued together with simple shell scripts and run in a CI environment. Once you understand the pattern - use an uber-config plus typed tool outputs to generate downstream config and apply all config in an idempotent way - you wonder why you ever did things any other way.
- znpy 7y agoCLOUD NATIVE DEVOPS RAFT-BASED BLOCKCHAIN-ENABLED MULTI CLOUD RESILIENT CONTAINERIZED MULTI-REGION MULTI-AZ static blog generator.
- choward 7y agoI'm sold. Link please.
- bryanlarsen 7y agoI downvoted this because even though I like it, it belongs at the bottom of the conversation below any real discussion of the actual article.
- Aperocky 7y agoSarcasm is in the realm of discussion though, see 'Will Any Crap We Put into Graphene Increase Its Electrocatalytic Effect?' https://pubs.acs.org/doi/10.1021/acsnano.9b00184 https://pubs.acs.org/doi/10.1021/acsnano.9b00184
- deleted 7y ago[deleted]
- tyfon 7y agoPersonally I think it belongs at the top. When the article looks like it is generated by the buzzword generator, that should be the real discussion.
- bryanlarsen 7y agoIt's quite a good article with a solution for a real problem. That it looks like a buzzword generator to HN says more about HN than it does about the article, IMO.
- deleted 7y ago[deleted]
- 7y ago
- jonstewart 7y agoIs it web scale?
- cbushko 7y agoThis is a decent article on how the load balancing between GCP loadbalancers and kubernetes works. We use terraform end to end and tried the http/https L7 loadbalancer first in our setup but I had a heck of a time with: - there is no API for the annotation mentioned in the article so if you miss it while setting up a backend, nothing works - gRPC was hard to get going. Most gRPC examples out there use some port such as 50051 so you assume that is needed. gRPC does work over 443. We currently use istios ingressgateway as a loadbalancer that you set in your kubernetes setup. It works but I don't know if it is better or not. We had to run an ingressgateway pod as a daemonset on every node so that we could get the real IP addresses from requests for security logging. That was a pain.
- fulafel 7y agoWhy would GCP restrict the ports like this?
- tn890 7y agoGCP has a lot of ports restrictions. A place I used to work at migrated from GCP to AWS because they don't allow port 25 incoming; if you're doing e-mail, you 100% need this.
- jkaplowitz 7y agoThey block it outbound, not inbound - email receiving should work fine in GCP. I believe that's the only mandatory firewall restriction, and even that one can have exceptions made for use cases such as your former employer, as the other reply to you said. The point is to make sure that anything sending email from GCP has someone properly attending to it, so that GCP does not become a source of spam, rather than to prevent email service providers or the like from using GCP. For many companies that aren't an email service provider, it's often a better use of IT funds to send via one anyway, given how much work is involved in maintaining security and deliverability of an outbound email server in 2020. Most of those support sending through ports that aren't 25, and Google has some nice free tier deals for GCP customers.
- 0xEFF 7y agoGCP will allow port 25 and smtp, just not by default. Talk to your TAM to make it happen.
- whatsmyusername 7y agoI've yet to see a compelling argument for the complication of kubernetes or the 'its not AWS' nature of GCP over a bog standard AWS ALB with ECS cluster tasks attached to it.
- minxomat 7y agoBecause k8s mostly prevents lock-in. A much better idea is to run EKS on Fargate. You still won't have to manage a cluster, but at least you can now use standard k8s manifests that work in any other cluster instead of the ECS homebrew stuff. You can keep ALB, because ALB can run as an Ingress Controller.
- whatsmyusername 7y agoEKS billing is hilariously bad (Managed SFTP is the same way). It's obvious they built it as a feature parity thing and they don't want you to actually use it. I can run all of my container infrastructure for a property for the cost of the control plane for EKS (before you attach hosts and before you spend how much time figuring out why hosts won't attach to it). It's just bad.
- whatsmyusername 7y agoLock-in on the cloud platform with >60% market share? I'm indifferent. AWS is the VMWare of our age, you'll never get fired for suggesting it. There are workloads that are inappropriate for it (see: Dropbox) but those are few and far between.
- auspex 7y agoThis seems to be a common rhetoric from people who have not invested the time in k8s. It is not as complicated as people think.
- whatsmyusername 7y agoI've used k8s at various times since it's inception. I've yet to see a compelling argument for running your control plane on the same layer as your workloads. I have always found this to be a recipe of disaster when 'something' in your service dependency chain breaks. Also not having load balancer support built in kills it for me. Yeah you can do metallb or nginx ingress but it means they punted one of the major components needed to make it, 'cloud agnostic.' Meanwhile in AWS we have ECS which works out the box on fargate, their EC2 hosts, or your own EC2 hosts and on hardware I can easily get (and dev) on docker swarm.
- vemacs 7y agoThis tool makes what the OP is trying to do much easier: https://github.com/GoogleCloudPlatform/gke-autoneg-controller https://github.com/GoogleCloudPlatform/gke-autoneg-controlle... you can have GKE configure the NEG on the BS directly, and not have to interrogate K8s for the name of the NEG to add to the BS via terraform.
- je42 7y agowhen I experimented with GCP, I found it weird that cloud-run would not work together with the GLB. That was really a surprise.