4 ms·
Hey GKE team, can you give any details about how this works? What does it assume/require from the underlying infrastructure?
by wmf 8y ago
Hey GKE team, can you give any details about how this works? What does it assume/require from the underlying infrastructure?
- puzzle 8y agoExactly. As described, it sounds a bit like magic. Of course, it's not. Thus, one wonders about what happens in the lower layers of the stack, namely who gets to run them (GKE or the HW owners): * DNS (not the Kubernetes in-cluster one like kube-dns) * DHCP * LDAP or equivalent * SSH and its keys * on-prem security of the cloud identities (does it require TPM? SGX?) * bootloaders * base OS image * drivers for attached storage, GPUs, etc. * firewall rules * routing
- ninkendo 8y agoNot to mention probably the hardest bit, which is how does it do persistent storage? Running k8s on the various cloud providers tends to use storage engines for those providers (ebs, etc)... Does it ship with ceph out of the box? Some in-house block store? What happens when it breaks? Persistent volumes are IMO the very hardest thing to get right, and for me it's the big reason why I'd rather put my trust in a hosted solution in the first place. If they solve this, and make it as seamless and easy as using cloud storage offerings, they've completely changed the game. Somehow I think they have a ways to go.
- puzzle 8y agoI assumed they would punt persistent storage for the first release, kinda like they launched Cloud Filestore with only NFSv3 support and without snapshots. As you say, it's a tough problem to tackle. If they to ship with something, I'd expect them to go for Ceph. As far as I can tell, if you squint hard enough, its lowest layers are the ones that resemble the most those at Google (Colossus/D).
- merb 8y ago> Not to mention probably the hardest bit, which is how does it do persistent storage? on bare-metal this is solved with rook.io. load balancing (not the api servers) is also solved with metallb.
- ninkendo 8y ago> on bare-metal this is solved with rook.io Do you have a source for this? Is there documentation anywhere that says GKE On-Prem is using rook? Or are you just saying "people who use kubernetes on-prem often use rook.io"?
- cube2222 8y agoRook is creating software for solving this problem. This does not at all mean that the problem really is solved. I have yet to hear about any big production deployments of rook, care to provide any to support your claims?
- ferrantim 8y agoI can attest that persistent storage is the hard part! Full disclosure, I work for a company[1] who makes a persistent storage solution for containers/Kubernetes. We are absolutely seeing that our large customers (folks like GE, Verizon, Dreamworks, Comcast, etc) are running "cloud native" applications on-prem as well as in the public cloud so this is a really smart move for Google. [1] https://portworx.com https://portworx.com
- jessebro 8y agoJesse here. I'm the eng manager for GKE and GKE On-Prem Storage lifecycle over at Google. For basic persistent volumes, we have working vSphere (block) support, which allows us to do a lot (including persistent/stateful services). We also have great storage abstraction layers built into K8S - CSI, FlexVolumes, and a large suite of in-tree plugins - so adding additional ones is pretty easy. I think you're talking about something else however - specifically scaled/distributed storage services. We're investigating options here, though keep in mind you should have no problems running containerized storage services in GKE On-prem. They would be on top of the existing block support I mentioned above. I saw a couple comments in other threads from some vendors that sell solutions that do just that. For storage systems that don't containerize, that is a different discussion. Happy to talk more.
- westonh 8y agoWe're going to give a breakout session that goes into more depth on Wednesday @ 4:35pm. IO244. Some quick details: It's a bit of a split between what GKE runs and what the customer runs. Alpha runs on vSphere 6.5 and we're packing up a Google-hardened OS in much the same way we package GKE for GCP. A lot of the integrations for things like networking and storage will be coming from partners. We'll also have remote mgmt capabilities so we can manage the cluster's control plane in much the same way our SREs do for GKE.
- ryanSrich 8y agoWill this be something like COS or even CoreOS? Also, I'm curious to hear more about this part: > GKE On-Prem has a fully integrated stack of hardened components, including OS, container runtime, Kubernetes, and the cloud to which it connects. Which runtime are you shipping? CRI-O? What type of outgoing cloud connection is that? I have so many questions. I'm actually at the conference this week if you're willing to grab coffee.
- westonh 8y agoHappy to chat more. DM @westonhutchins and we can setup a time.
- alxbog 8y agoIs there a video available for session IO244?
- karangoeluw 8y agoKaran from the GKE On-Prem eng team here. In the Alpha, we are supporting vSphere 6.5. Which part of infra are you most curious about knowing?
- ninkendo 8y agoOuch. Way to bury the lede! Is this ever going to be a bare-metal thing? Like probably many others, I'm not really interested in doing on-prem virtualization... kubernetes is interesting to me because containers are a better abstraction than virtual machines in the first place. Why add a virtualization layer if you don't have to? (I get that it makes your life easier as the developer of this product, but having to run a virtualization IaaS between your metal and your orchestration makes the whole thing rather uninteresting IMO.)
- karangoeluw 8y agoThere are definitely enterprises looking to move away from virtualization. We have that in mind. We are exploring additional options, such as bare metal support, based on customer demand. Send me an email (karangoel [at] google) if you'd be interested in talking more about bare metal.
- alxbog 8y ago+1 for Bare Metal support.
- rushins 8y agoi definately want to see bare metal with GKE on-prem as VSPHER E is additional cost
- rushins 8y agohello karan so what is the model for bare metal with GKE on-prem. The reason is for VSPHERE 6.5 is additional cost (license per cores to VMWARE and VCENTER license) which we want to avoid to use bare metal only.