13 ms·

Kubernetes 1.18

160 points by onlydole 7y ago

taywrobel 7y ago
Reminder to everyone that unless you have a truly massive or complex system, you probably don’t need to run K8s, and will save yourself a ton of headaches avoiding it in favor of a more simple system or using a managed option.
- bithavoc 7y ago
  I think the definition of "massive or complex system" varies between developers with different backgrounds, in your opinion, what's considered truly massive and complex system that may require Kubernetes?
- jjtheblunt 7y ago
  What are recommendable simpler system examples, or managed options you allude to?
  - alpb 7y ago
    Google Cloud Run, AWS Fargate, Google App Engine, Heroku etc. are comparable experiences to Kubernetes if you have the flexibility of (1) running on cloud (2) not having to configure host OS or rely on host GPUs etc. Disclaimer: I work at Google Cloud Run.
    - gravypod 7y ago
      Does cloud run or fargate have an equivalent to a statefulset? Persistent storage and the ability to talk to a specific node/instance?
    - jjtheblunt 7y ago
      Thanks; Fargate was what i was anticipating, or Heroku, hadn't looked at Cloud Run but definitely will.
    - yahyaheee 7y ago
      You have vendor lock-in with all of these though
      - alpb 7y ago
        Not really, no. Cloud Run implements the Knative API, so you can actually take it away and run it on any Kubernetes cluster anywhere on a cloud or in your datacenter.
        gtaylor 7y ago
        Doesn't that require Istio and a few other things as well? I wouldn't wish Istio configuration and maintenance on anyone but the larger shops/teams. You might not be technically locked in, but not everyone can afford to dedicate the attention to tuning and maintaining Istio. Source: at mid-sized company who has Istio in the stack.
        alpb 7y ago
        Knative just needs a gateway (LB); not a full mesh. Istio is the default option. Alternatively you can use Gloo, Ambassador or something specifically built for Knative such as Kourier.
      - bg24 7y ago
        Vendor lock-in does not come from using GKE/EKS/AKS or derivatives. What ends up happening is that your application consumes services (storage, analytics, etc.). You start using those services from the cloud provider, which makes sense as long as it is the right thing for your business (aligns w/ your cloud-native blueprint). Kubernetes, by itself, is cloud-provider agnostic.
    - mstg 7y ago
      Hello! Cloud Run would be better if it supported Cloud IAP and wildcard managed certificates. Overall though it is an amazing service and it is really fast and easy to use.
    - threeseed 7y ago
      This comment makes no sense. There is AWS Fargate for Kubernetes, AWS Elastic Kubernetes Service, DigitalOcean Kubernetes, Google Kubernetes Engine etc. All of which are on the cloud and all of which don't require you to configure host OS etc. Some offer full control over node configuration e.g. EKS whilst others manage that for you e.g. Fargate.
      - alpb 7y ago
        Thanks for repeating what I already said. I responded to the question asked. OP asked if there are simpler alternatives. These are simpler alternatives to Kubernetes. I don't know which part you're not getting, but it appears that this person's intention is not to learn Kubernetes or deal with nodes in the first place.
    - psankar 7y ago
      Since you mentioned about CloudRun, I had one query. I run docker compose locally for development. For prod, I just use a different docker compose file (with some values changed, for example the postgres database url etc.). I do this from a 5 USD per month droplet/vm. I can launch multiple services like this for my microservices platform. I can use a hosted database solution for another 15 USD per month, to get backups etc. Also I get a generous 1 TB bandwidth and predictable performance for my system as a whole. In the past I have used appengine and been bitten by their update times (took more than 20 mins for a single code update, things could have improved now). Also I need to write deployment artifacts for each service. Now is there any benefit that cloud run (or any paas) could offer compared to this ? Would it be not easier to just stay with docker-compose and defer upgrading to kubernetes until you turn profitable or become unmanageable with a single VM ?
  - aphexairlines 7y ago
    AWS SAM has been great for running rust in AWS Lambda + API Gateway and for managing tables in DynamoDB.
    - jjtheblunt 7y ago
      thanks
  - kawsper 7y ago
    Nomad and Consul from Hashicorp.
    - merb 7y ago
      which is way more complex to setup than k3s
      - mm89 7y ago
        Perhaps installation and config are not as streamlined as one of the many available k8s setup tools, but the tools themselves are much easier to understand and less broad than the k8s system. Used nomad/consul/fabio at a previous job for running containers, it was very easy to adopt. Way less new concepts as well. It's worth mentioning I later chose GCP managed kubernetes for a small cluster to run at my startup. I had to learn a few new things, but I'm not familiar of any "nomad as a service" offerings, so I went with k8s on Google cloud.
      - Legogris 7y ago
        When was the last time you used it? I found it straightforward and not so many knobs to twist. ACLs and cert management can be a bit of a PITA the first time (especially the former, and it’s something you really want to do before you’re replying on Consul for mission-critical stuff), but that’s about it. Still, those two things are mainly confusing due to poor documentation and not that complicated once you get up to speed, so it’s mainly the first time you do a new setup.
        Legogris 7y ago
        Downvoters: Care to elaborate what you think is wrong with my comment?
- empath75 7y ago
  People say this every time anything related to k8s gets posted and I always wonder who it’s addressed to. The system doesn’t actually have to be that complex for kubernetes to be useful and kubernetes isn’t that hard to run. We’re in the process of switching from ecs to kubernetes and while it’s not an easy thing to make ready for production, it enables so much that wouldn’t even be possible with ecs. To me this advice is only useful for tiny startups running a handful of web servers.
  - outworlder 7y ago
    There is one more advantage vs ECS: there is no longer any lock-in. You can have more capabilities, using standard solutions that work anywhere you want.
  - snypox 7y ago
    What does it enable compared to ECS?
- outworlder 7y ago
  Not sure why this disclaimer has to be posted every time there's a discussion on K8s. It is a tool, if you need to use it, do use it. If not, don't. Although I would argue that you need to know what trade offs you are making if you have the right use-case (multiple containers you need to orchestrate, preferably across multiple machines) and you are not using it or a similar tool. There are lots of best-practices and features you get out of the box, that you would have to implement yourself. You get: * Deployments and updates (rolling if you so wish) * Secret management * Configuration management * Health Checks * Load balancing * Resource limits * Logging And so on(not even going into stateful here), but you get the picture. Whatever you don't get out of the box, you can easily add. Want Prometheus? That's an easy helm install away. Almost every system starts out by being 'simple'. The question is, it going to _stay_ simple? If so, sure, you can docker run your container and forget about it.
  - rumanator 7y ago
    I would stress the out-of-the-box support for blue-green deployments in a system that is fully versioned, thus supports rollbacks like a champ.
    - predakanga 7y ago
      This is news to me, do you have any links to k8s's support for blue-green deploys? I've been holding off setting up a system like Spinnaker because I'd read it was coming (in the form of custom deployment strategies), but can't find anything current on the subject.
      - rumanator 7y ago
        At the service level Kubernetes offers deployments https://github.com/kubernetes/kubernetes/tree/master/pkg/controller/deployment https://github.com/kubernetes/kubernetes/tree/master/pkg/con... At an application level then the strategy consists of having two applications deployed and update the application's ingress after the deployment controller finishes updating the deployments.
  - 7y ago
- RexM 7y ago
  I've enjoyed learning K8s for my not massive nor complex personal workload. It's running and is more hands off than without it. I'm using a managed digital ocean cluster. I no longer have to worry about patching the OS as it's all handled for me. I also don't have to worry about having a server with a bunch of specialized packages installed, although I suppose only using containers could have gotten me that far. I haven't had a ton of headaches. So, I guess people's experiences may differ. It's interesting to me that K8s always draws out the "you probably don't need it" comments.
- haolez 7y ago
  When I think about k8s complexity, I can only understand this argument if I'm the one dealing with the infrastructure required. If I have to install k8s in my servers, then I'll probably need to think hard about security, certificates, hardware failures, monitoring, alerting, etc. It's a lot of work. However, if I use a managed k8s service, I probably don't have to think about any of that. I can focus on the metrics of my application, and not the cluster itself. At least, that's how I think it should work. I haven't used k8s in a while.
  - empath75 7y ago
    Honestly getting a cluster running is the easiest part. It’s all the add one like istio that make it complicated.
    - haolez 7y ago
      You don't need Istio if your application is simple. I think Istio makes more sense when your application makes heavy use of peer-to-peer pod connections. If you can get away with a simple queue as a bus, it should remain simple. I think!
    - rumanator 7y ago
      > Honestly getting a cluster running is the easiest part. It’s all the add one like istio that make it complicated. That's like saying running windows is the easy part, it's all the add-ons like Microsoft office that make it complicated.
- ojhughes 7y ago
  This false information really needs to die. k8s is a sane choice in many cases not just hyper scale. Regular business apps benefit from rolling upgrades and load balancing between replicas. Managed k8s platforms like GKE make cluster management a breeze. Developer tooling such as Skaffold makes developing with k8s seamless. I expect k8s to continue growing and will soon take over much of the existing Cloud Foundry estate in F500 companies .
  - jasdfajdsfk 7y ago
    false? the only thing false here is k8s marketing - k8s is never a sane choice, it is insecure, performance sucks and costs companies unbelievable amounts of dough k8s deserves to die in a fire
  - dx034 7y ago
    Running k8s is much harder and takes more time than just having a few VMs with docker on it. Many applications never need to scale. I really like k8s from a user perspective but it's no easy task to set up. And managed solutions don't always work for everyone (and aren't always cost efficient).
    - ojhughes 7y ago
      Most companies will have an Ops team to manage the prod clusters
    - ithkuil 7y ago
      If you run the whole thing (app + k8s) then I do agree with you that it's more complex and you're likely be better off without it. But, k8s offers a very good way to split the responsibility of managing the infrastructure and managing the applications that run on top. Many people work in medium to big corporations that have a bunch of people that are in charge of managing the compute infrastructure. I certainly prefer k8s as an API between me and the infrastructure, as opposed to filing tickets and/or using some ad-hoc and often in-house automation that lets me deploy my stuff in a way that is acceptable for the "infrastructure guys".
- te_chris 7y ago
  Yeah, why would anyone with a simple system want an elegant way to declaratively describe the state of their runtime?
threatofrain 7y ago
Anyone have a recommended guide for Kubernetes?
- bryogenic 7y ago
  https://github.com/alta3/kubernetes-the-alta3-way https://github.com/alta3/kubernetes-the-alta3-way ansible playbooks based on "the hard way" but without being dependent on Google cloud
- geuis 7y ago
  I’ve been getting up to speed over the last few months. Don’t do what I did: googling and going through random top hits. Most of these are 1-off blog articles that revolve around “install Helm and then do these 6 things” or “just kubectl apply random.link.com/some-script”. Doing that just leads to tons of confusion and anger. My recommendation: Suck it up and read through the official kubernetes docs. Their docs aren’t written in a way to easily explain core concepts unfortunately. However, slogging through it will give you some initial exposure to concepts and will let you know where to go back to later when your making mental connections. Next, look for k8s tutorials from Digital Ocean and Linode. In my opinion, they’re the best written guides for demonstrating how to get from A to B. You’ll start running through those guides and be referencing back to the official docs. Gradually bridges will form on your head and you’ll get an intermediate, functional level of competence.
- deleted 7y ago
  [deleted]
- thedance 7y ago
  There aren't any. Someone should write one. My beef with them all is they start from the top down, taking giant leaps of logic and assuming that everyone has the same use cases and motivations. In my personal opinion, humble though it may not be, the way to understand Kubernetes is from the bottom up. First of all, figure out how Linux starts your process. I've noticed that a lot of people who are afraid of k8s are afraid of it partly because they have no idea how a Linux machine works anyway. So start there. After you've got that figured out, learn about control groups. What kinds of resources can be controlled? How do you create and destroy control groups? How do you put processes in them? Same thing for namespaces. How do they work? What can a process see from its perspective inside a process namespace? If you grok all that, then you are ready to grok the K8s Container Runtime Interface. And, if you really grokked it, you understand that Docker is besides the point, isolation and namespaces are optional, and so forth. All k8s demands from the "container runtime" is that a thing has a name and a defined lifecycle. Once you understand how k8s works at the pod and node level then it should be perfectly obvious how it works at higher levels. That's why I really want to see someone write the bottom-up guide!
fosk 7y ago
Although I understand the fear that many have of being left behind the technology curve by not having the time - or the chance - to run Kubernetes in their day to day work, we really must appreciate that Kubernetes really is the future of infrastructure. For those that are looking at Kubernetes with suspicion, it is a natural instinct to think of K8s as a threat to all the knowledge we have built in the past few years, but it doesn't have to be that way. So many things that we would have built ourselves (deployments, upgrades, monitoring, etc) can now be streamlined with K8s, and existing knowledge around those topics will just make our transition to Kubernetes faster (besides still being able to use most of our expertise within K8s anyways). Managed solutions make K8s easy to use, while we can still benefit from being able to run out workloads left and right on any cloud vendor. In one word: portability. Which in the day and age of cloud vendor lock-in it is to be protected at all cost. I know that some organizations allow to allocate time to explore new technologies and learn new practices. If your organization has no policy around this, it is worth try asking. Ultimately it will benefit the organization and the business as a whole, as they will be able to build a solid foundation to more rapidly transition and execute on their digital products. Kubernetes is good for business.
- drdaeman 7y ago
  > K8s as a threat to all the knowledge we have built in the past few years, but it doesn't have to be that way All this knowledge is still relevant. Kubernetes is an extra layer on top of all the classical technologies out there. It is not a replacement - besides some fancy experiments, it doesn't replace GNU/Linux. If you're going to run a Kubernetes cluster, you still have to know all the "classical" networking, GNU/Linux system administration and architecture, etc. Having a fancy task scheduler for containers, and a tool that sets up clever network bridges (or whatever your preferred CNI flavor does), and a bunch of other nice extras does not remove the requirement to know what's underneath. And in any non-standard situation, there are chances that you'll have to build things from scratch. For example, I do want to run a K8s cluster over a cjdns overlay network (which doesn't have IPv4, so Flannel or Weave won't work). Haven't figured this out yet. And if you're not setting up and managing the infrastructure - it's about the same as it had always been. Just a different user interface to manage your deployments/storage/networking. > Managed solutions make K8s easy to use I would disagree. "Using" is the same (that's the whole point of K8s). "Install and maintain" is easier, but only in a sense that you don't do this. ;)
crb 7y ago
Podcast interview with the release team lead: https://kubernetespodcast.com/episode/096-kubernetes-1.18/ https://kubernetespodcast.com/episode/096-kubernetes-1.18/
robbiet480 7y ago
Really annoying that AWS EKS only got K8S 1.15 last week.
QUFB 7y ago
I've seen the cloud agnostic nature of Kuberbetes mentioned in many posts here. This is only true on the surface for a significant number of use cases and deployment models. Once petabytes of data are out there in your GCP or AWS environment, "portability" will be costly due to extortionistic pricing of egress bandwidth.
- bcheung 7y ago
  At that scale it is probably a better option to look into something like AWS Snowball and ship it on physical drives. At the petabyte scale there will probably be huge cost savings to using your own private cloud on your own hardware.
- hodgesrm 7y ago
  A lot of people think of multi-cloud as some kind of arbitrage where you jump quickly between markets. Running applications in cloud environments is a lot more like leasing property. Once you set up there are costs to moving. The portability argument boils down to saying you are not boxed in. If things get bad enough you can move. This is a big long-term advantage for businesses because it means you can correct mistakes or adjust to changing business conditions. That's what most people who run companies are really looking for.
  - MrBuddyCasino 7y ago
    It also leads to nullifying the advantages of the cloud. The whole point are the proprietary services they offer, and use those instead of building them yourself. Trying to be „cloud agnostic“ is one of the biggest mistakes one could make.
    - closeparen 7y ago
      You're deploying to clusters of x86 machines in the first place because people in the 90s realized they were a little too into the proprietary APIs of their big iron vendors.
    - growse 7y ago
EdwardDiego 7y ago
I'm pleased to see the changes in the HPA, having pod scale-up/down periods be tied to a systemwide setting was a bit painful.
- hiroshi3110 7y ago
  This one? https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.18.md#new-api-fields https://github.com/kubernetes/kubernetes/blob/master/CHANGEL... > autoscaling/v2beta2 HorizontalPodAutoscaler added a spec.behavior field that allows scale behavior to be configured. Behaviors are specified separately for scaling up and down. In each direction a stabilization window can be specified as well as a list of policies and how to select amongst them. Policies can limit the absolute number of pods added or removed, or the percentage of pods added or removed. (#74525, @gliush) [SIG API Machinery, Apps, Autoscaling and CLI]
  - EdwardDiego 7y ago
    Yep, in the version I'm on (1.15) there's only global flags and config[1] which apply to all HPAs, but not all apps should scale the same way - our net facing glorified REST apps can easily scale up with, say, a 1-2m window, but our pipeline apps sharing a Kafka consumer group should be scaled more cautiously (as consumer group rebalancing is a stop-the-world event for group members) 1: https://v1-15.docs.kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/#algorithm-details https://v1-15.docs.kubernetes.io/docs/tasks/run-application/...
joseph 7y ago
Shamelesss plug: Keights, my Kubernetes installer[1] for AWS supports 1.18 (the latest version available on EKS is 1.15). Keights also supports running etcd separate from the control plane, lets you choose instance sizes for the control plane, and can run in GovCloud. 1. https://github.com/cloudboss/keights https://github.com/cloudboss/keights
- llarsson 7y ago
  How does it compare to kops or, dare I say it, kubespray?
  - joseph 7y ago
    Hi, I've often worked in corporate environments where it wasn't necessarily allowed to spin up a VPC, or to create an internet gateway. In the time I was creating this, many companies who are in healthcare or are otherwise locked down, could not use kops due to its requirement for an internet gateway. I created keights to fill that space, so that anyone could run Kubernetes in AWS, even in air gapped environments. This is pretty common nowadays, by the way - enterprises have a team to manage all AWS accounts, and they set up VPCs and connectivity ahead of time, before development teams get access to the account; access to the internet is through a proxy only, and no one can modify the network. Not to mention, most of the access to Amazon's services can now be done without an internet gateway, using VPC endpoints. Keights fits well in this world of locked down network access, and it works well even in GovCloud (you would need to build the AMI there as my public AMIs cannot be shared with GovCloud accounts). Keights and Kubespray both use Ansible, however they do it in a very different way. (Disclaimer: I haven't used kubespray, only looked over the documentation). Keights uses Ansible roles to build CloudFormation stacks to produce a cluster. The nodes in the cluster bootstrap themselves using systemd services that are baked into the AMI; Ansible does not run on the nodes in the cluster. Kubespray, as I understand it, uses a traditional Ansible approach of pushing configurations over ssh to nodes in its inventory. To my knowledge, it does not actually build the machines in the cluster, it just configures existing machines. Keights does the full end-to-end automation to bring up a working cluster, including the creation of all required AWS resources (autoscaling groups, a load balancer for the apiserver, security groups, etc - though you do provide certain resources as parameters, for example your VPC ID and subnet IDs, due to the aforementioned requirements to fit into locked-down environments).
eric_khun 7y ago
Anyone solved properly the CPU Throttling issues they are seeing with kubernetes ? Is this release solved it? We are seeing a lot of throttling on every deployments, what impact our latency, even when setting a really high cpu limit. The solutions seems to be: - remove the limit completely. Not a fan of this one since we really don't want a service going over a given limit... - using the static management cpu policy [2] Not a fan because some services doesn't need a "whole" cpu to run... Anyone has any other solutions? Thanks! [1] https://github.com/kubernetes/kubernetes/issues/67577 https://github.com/kubernetes/kubernetes/issues/67577 [2] https://kubernetes.io/docs/tasks/administer-cluster/cpu-management-policies/#static-policy https://kubernetes.io/docs/tasks/administer-cluster/cpu-mana...
- dilyevsky 7y ago
  1. It makes no sense to quota your cpu with the exception of very specific cases (like metered usage). You’re just throwing away compute cycles 2. Same applies to dedicate cores for pretty much same reasons Having said that if you really really want quota but don’t want shit tail latency I suggest setting cfs_quota_period to under 5ms via kubelet flag
  - eric_khun 7y ago
    Thanks! wouldn't that be an issue if a pod "take over" the node , if for some reasons a request use too much CPU?
    - dilyevsky 7y ago
      Not really if you ensure every pod sets cpu request (which sets up cgroups cpu.shares) and your kubelet and system services are running in separate top-level cgroups (—kube-reserved and —system-reserved flags) you have reserved enforcement enabled. On full node contention every container will just consume its proportional share. This is not to say that someone malicious wouldn’t be able to dos a node but untrusted workload is a whole separate topic
  - rumanator 7y ago
dragonsh 7y ago
Containers are not necessary if the systems are build with something like Guix or Nix as they provide transaction level updates to applications and dependency and secure by default as no need to run a daemon with root access to run, monitor and manage containers. They provide same way of managing application deployment, the way application source code is managed with versioned deployments and rollbacks all baked in. But as with any technology Guix and Nix are still decade ahead of the present and may pick up later when technology converge back to running application servers and other dependent software in isolation with user level namespaces. Kubernetes try to solve one problem and create 10 other infrastructure problems to manage and instead of working on application, tie the company to a specific distribution or cloud service provider. So far there is nothing revolutionary in it unless the startup or company adopting k8s is google size operations. From a software developer perspective which is the main audience of HN it will be popular as most of them dream or wants to work for company of size google. Startup founders want to solve the scaling problem like google in the beginning as everyone dreams to be google size from day one. Kubernetes complexity is useful at large scale for majority i.e. over 90% of the deployments simple containers, bare metal or VM with traditional configuration management will be sufficient.
- gigatexal 7y ago
  Kubernetes is here. Better to learn it and use it where appropriate than to fight it.
  - dragonsh 7y ago
    I don't need to fight it, I am not looking for a job and follow the herd mentality. My own startup is pretty happy with Guix and related infrastructure on bare-metal, it works well for us and we can still stand on shoulders of giants who have done much better job in managing large distributed infrastructure.
    - gigatexal 7y ago
      That’s awesome. I’m not evangelizing it or saying you should use it as I did say “where appropriate” and at your startup it doesn’t seem appropriate. There are probably 10x more engineers that have experience with k8s than with guix though so ... perhaps that could be a factor though not a reason to change out the infra completely.
jasdfajdsfk 7y ago
k8s posts == spam
deboflo 7y ago
Kubernetes? No thanks.
mehdix 7y ago
FWIWI a few days back I upgraded my microk8s-based k8s to 1.18 beta channel and it solved at least ImagePullBackOff problem that I had with pulling from my private GitLab registry. Worked like charm.
lprd 7y ago
I've taken a couple k8s courses, I understand all the small parts that make up k8s, but yet it seems that there are still no easy solutions to install on bare metal. The default recommendation is to always just roll with a managed solution. This is slightly irritating considering there are plently of companies out there who own their own infrastructure. There are plenty of great developer distributions out there (k3s, kind, minikube, microk8s), but those are single node only, and aren't meant for production use. I'm still searching for a solid guide on how to get k8s installed on your own hardware. Any suggestions would be very appreciated!
- ojhughes 7y ago
  k3s supports multinode
  - lprd 7y ago
    Ah, I wasn't aware. Does it support HA as well?
    - deleted 7y ago
      [deleted]
    - ojhughes 7y ago
      Yes if you use an external DB for the k8s control plane https://rancher.com/docs/k3s/latest/en/installation/datastore https://rancher.com/docs/k3s/latest/en/installation/datastor...
      - pas 7y ago
        There's also an experimental embedded DQlite (raft + sqlite) thingie too! https://rancher.com/docs/k3s/latest/en/installation/ha-embedded/ https://rancher.com/docs/k3s/latest/en/installation/ha-embed...
- mike1o1 7y ago
  https://www.ansibleforkubernetes.com/ https://www.ansibleforkubernetes.com/ I've found this pretty good starting point. Keep in mind the book is still a work in progress. Note, I haven't actually run k8s in production, but this book has helped me get something up and running in VM's pretty quickly using Ansible on top of kubeadm.

160 points by onlydole 7y ago

Reminder to everyone that unless you have a truly massive or complex system, you probably don’t need to run K8s, and will save yourself a ton of headaches avoiding it in favor of a more simple system or using a managed option.

bithavoc 7y ago
I think the definition of "massive or complex system" varies between developers with different backgrounds, in your opinion, what's considered truly massive and complex system that may require Kubernetes?
jjtheblunt 7y ago
What are recommendable simpler system examples, or managed options you allude to?
- alpb 7y ago
  Google Cloud Run, AWS Fargate, Google App Engine, Heroku etc. are comparable experiences to Kubernetes if you have the flexibility of (1) running on cloud (2) not having to configure host OS or rely on host GPUs etc. Disclaimer: I work at Google Cloud Run.
  - gravypod 7y ago
    Does cloud run or fargate have an equivalent to a statefulset? Persistent storage and the ability to talk to a specific node/instance?
  - jjtheblunt 7y ago
    Thanks; Fargate was what i was anticipating, or Heroku, hadn't looked at Cloud Run but definitely will.
  - yahyaheee 7y ago
    You have vendor lock-in with all of these though
    - alpb 7y ago
      Not really, no. Cloud Run implements the Knative API, so you can actually take it away and run it on any Kubernetes cluster anywhere on a cloud or in your datacenter.
      - gtaylor 7y ago
        Doesn't that require Istio and a few other things as well? I wouldn't wish Istio configuration and maintenance on anyone but the larger shops/teams. You might not be technically locked in, but not everyone can afford to dedicate the attention to tuning and maintaining Istio. Source: at mid-sized company who has Istio in the stack.
        alpb 7y ago
        Knative just needs a gateway (LB); not a full mesh. Istio is the default option. Alternatively you can use Gloo, Ambassador or something specifically built for Knative such as Kourier.
    - bg24 7y ago
      Vendor lock-in does not come from using GKE/EKS/AKS or derivatives. What ends up happening is that your application consumes services (storage, analytics, etc.). You start using those services from the cloud provider, which makes sense as long as it is the right thing for your business (aligns w/ your cloud-native blueprint). Kubernetes, by itself, is cloud-provider agnostic.
  - mstg 7y ago
    Hello! Cloud Run would be better if it supported Cloud IAP and wildcard managed certificates. Overall though it is an amazing service and it is really fast and easy to use.
  - threeseed 7y ago
    This comment makes no sense. There is AWS Fargate for Kubernetes, AWS Elastic Kubernetes Service, DigitalOcean Kubernetes, Google Kubernetes Engine etc. All of which are on the cloud and all of which don't require you to configure host OS etc. Some offer full control over node configuration e.g. EKS whilst others manage that for you e.g. Fargate.
    - alpb 7y ago
      Thanks for repeating what I already said. I responded to the question asked. OP asked if there are simpler alternatives. These are simpler alternatives to Kubernetes. I don't know which part you're not getting, but it appears that this person's intention is not to learn Kubernetes or deal with nodes in the first place.
  - psankar 7y ago
    Since you mentioned about CloudRun, I had one query. I run docker compose locally for development. For prod, I just use a different docker compose file (with some values changed, for example the postgres database url etc.). I do this from a 5 USD per month droplet/vm. I can launch multiple services like this for my microservices platform. I can use a hosted database solution for another 15 USD per month, to get backups etc. Also I get a generous 1 TB bandwidth and predictable performance for my system as a whole. In the past I have used appengine and been bitten by their update times (took more than 20 mins for a single code update, things could have improved now). Also I need to write deployment artifacts for each service. Now is there any benefit that cloud run (or any paas) could offer compared to this ? Would it be not easier to just stay with docker-compose and defer upgrading to kubernetes until you turn profitable or become unmanageable with a single VM ?
- aphexairlines 7y ago
  AWS SAM has been great for running rust in AWS Lambda + API Gateway and for managing tables in DynamoDB.
  - jjtheblunt 7y ago
    thanks
- kawsper 7y ago
  Nomad and Consul from Hashicorp.
  - merb 7y ago
    which is way more complex to setup than k3s
    - mm89 7y ago
      Perhaps installation and config are not as streamlined as one of the many available k8s setup tools, but the tools themselves are much easier to understand and less broad than the k8s system. Used nomad/consul/fabio at a previous job for running containers, it was very easy to adopt. Way less new concepts as well. It's worth mentioning I later chose GCP managed kubernetes for a small cluster to run at my startup. I had to learn a few new things, but I'm not familiar of any "nomad as a service" offerings, so I went with k8s on Google cloud.
    - Legogris 7y ago
      When was the last time you used it? I found it straightforward and not so many knobs to twist. ACLs and cert management can be a bit of a PITA the first time (especially the former, and it’s something you really want to do before you’re replying on Consul for mission-critical stuff), but that’s about it. Still, those two things are mainly confusing due to poor documentation and not that complicated once you get up to speed, so it’s mainly the first time you do a new setup.
      - Legogris 7y ago
        Downvoters: Care to elaborate what you think is wrong with my comment?
empath75 7y ago
People say this every time anything related to k8s gets posted and I always wonder who it’s addressed to. The system doesn’t actually have to be that complex for kubernetes to be useful and kubernetes isn’t that hard to run. We’re in the process of switching from ecs to kubernetes and while it’s not an easy thing to make ready for production, it enables so much that wouldn’t even be possible with ecs. To me this advice is only useful for tiny startups running a handful of web servers.
- outworlder 7y ago
  There is one more advantage vs ECS: there is no longer any lock-in. You can have more capabilities, using standard solutions that work anywhere you want.
- snypox 7y ago
  What does it enable compared to ECS?
outworlder 7y ago
Not sure why this disclaimer has to be posted every time there's a discussion on K8s. It is a tool, if you need to use it, do use it. If not, don't. Although I would argue that you need to know what trade offs you are making if you have the right use-case (multiple containers you need to orchestrate, preferably across multiple machines) and you are not using it or a similar tool. There are lots of best-practices and features you get out of the box, that you would have to implement yourself. You get: * Deployments and updates (rolling if you so wish) * Secret management * Configuration management * Health Checks * Load balancing * Resource limits * Logging And so on(not even going into stateful here), but you get the picture. Whatever you don't get out of the box, you can easily add. Want Prometheus? That's an easy helm install away. Almost every system starts out by being 'simple'. The question is, it going to _stay_ simple? If so, sure, you can docker run your container and forget about it.
- rumanator 7y ago
  I would stress the out-of-the-box support for blue-green deployments in a system that is fully versioned, thus supports rollbacks like a champ.
  - predakanga 7y ago
    This is news to me, do you have any links to k8s's support for blue-green deploys? I've been holding off setting up a system like Spinnaker because I'd read it was coming (in the form of custom deployment strategies), but can't find anything current on the subject.
    - rumanator 7y ago
      At the service level Kubernetes offers deployments https://github.com/kubernetes/kubernetes/tree/master/pkg/controller/deployment https://github.com/kubernetes/kubernetes/tree/master/pkg/con... At an application level then the strategy consists of having two applications deployed and update the application's ingress after the deployment controller finishes updating the deployments.
- 7y ago
RexM 7y ago
I've enjoyed learning K8s for my not massive nor complex personal workload. It's running and is more hands off than without it. I'm using a managed digital ocean cluster. I no longer have to worry about patching the OS as it's all handled for me. I also don't have to worry about having a server with a bunch of specialized packages installed, although I suppose only using containers could have gotten me that far. I haven't had a ton of headaches. So, I guess people's experiences may differ. It's interesting to me that K8s always draws out the "you probably don't need it" comments.
haolez 7y ago
When I think about k8s complexity, I can only understand this argument if I'm the one dealing with the infrastructure required. If I have to install k8s in my servers, then I'll probably need to think hard about security, certificates, hardware failures, monitoring, alerting, etc. It's a lot of work. However, if I use a managed k8s service, I probably don't have to think about any of that. I can focus on the metrics of my application, and not the cluster itself. At least, that's how I think it should work. I haven't used k8s in a while.
- empath75 7y ago
  Honestly getting a cluster running is the easiest part. It’s all the add one like istio that make it complicated.
  - haolez 7y ago
    You don't need Istio if your application is simple. I think Istio makes more sense when your application makes heavy use of peer-to-peer pod connections. If you can get away with a simple queue as a bus, it should remain simple. I think!
  - rumanator 7y ago
    > Honestly getting a cluster running is the easiest part. It’s all the add one like istio that make it complicated. That's like saying running windows is the easy part, it's all the add-ons like Microsoft office that make it complicated.
ojhughes 7y ago
This false information really needs to die. k8s is a sane choice in many cases not just hyper scale. Regular business apps benefit from rolling upgrades and load balancing between replicas. Managed k8s platforms like GKE make cluster management a breeze. Developer tooling such as Skaffold makes developing with k8s seamless. I expect k8s to continue growing and will soon take over much of the existing Cloud Foundry estate in F500 companies .
- jasdfajdsfk 7y ago
  false? the only thing false here is k8s marketing - k8s is never a sane choice, it is insecure, performance sucks and costs companies unbelievable amounts of dough k8s deserves to die in a fire
- dx034 7y ago
  Running k8s is much harder and takes more time than just having a few VMs with docker on it. Many applications never need to scale. I really like k8s from a user perspective but it's no easy task to set up. And managed solutions don't always work for everyone (and aren't always cost efficient).
  - ojhughes 7y ago
    Most companies will have an Ops team to manage the prod clusters
  - ithkuil 7y ago
    If you run the whole thing (app + k8s) then I do agree with you that it's more complex and you're likely be better off without it. But, k8s offers a very good way to split the responsibility of managing the infrastructure and managing the applications that run on top. Many people work in medium to big corporations that have a bunch of people that are in charge of managing the compute infrastructure. I certainly prefer k8s as an API between me and the infrastructure, as opposed to filing tickets and/or using some ad-hoc and often in-house automation that lets me deploy my stuff in a way that is acceptable for the "infrastructure guys".
te_chris 7y ago
Yeah, why would anyone with a simple system want an elegant way to declaratively describe the state of their runtime?

threatofrain 7y ago

Anyone have a recommended guide for Kubernetes?

bryogenic 7y ago
https://github.com/alta3/kubernetes-the-alta3-way https://github.com/alta3/kubernetes-the-alta3-way ansible playbooks based on "the hard way" but without being dependent on Google cloud
geuis 7y ago
I’ve been getting up to speed over the last few months. Don’t do what I did: googling and going through random top hits. Most of these are 1-off blog articles that revolve around “install Helm and then do these 6 things” or “just kubectl apply random.link.com/some-script”. Doing that just leads to tons of confusion and anger. My recommendation: Suck it up and read through the official kubernetes docs. Their docs aren’t written in a way to easily explain core concepts unfortunately. However, slogging through it will give you some initial exposure to concepts and will let you know where to go back to later when your making mental connections. Next, look for k8s tutorials from Digital Ocean and Linode. In my opinion, they’re the best written guides for demonstrating how to get from A to B. You’ll start running through those guides and be referencing back to the official docs. Gradually bridges will form on your head and you’ll get an intermediate, functional level of competence.
deleted 7y ago
[deleted]
thedance 7y ago
There aren't any. Someone should write one. My beef with them all is they start from the top down, taking giant leaps of logic and assuming that everyone has the same use cases and motivations. In my personal opinion, humble though it may not be, the way to understand Kubernetes is from the bottom up. First of all, figure out how Linux starts your process. I've noticed that a lot of people who are afraid of k8s are afraid of it partly because they have no idea how a Linux machine works anyway. So start there. After you've got that figured out, learn about control groups. What kinds of resources can be controlled? How do you create and destroy control groups? How do you put processes in them? Same thing for namespaces. How do they work? What can a process see from its perspective inside a process namespace? If you grok all that, then you are ready to grok the K8s Container Runtime Interface. And, if you really grokked it, you understand that Docker is besides the point, isolation and namespaces are optional, and so forth. All k8s demands from the "container runtime" is that a thing has a name and a defined lifecycle. Once you understand how k8s works at the pod and node level then it should be perfectly obvious how it works at higher levels. That's why I really want to see someone write the bottom-up guide!

fosk 7y ago

Although I understand the fear that many have of being left behind the technology curve by not having the time - or the chance - to run Kubernetes in their day to day work, we really must appreciate that Kubernetes really is the future of infrastructure. For those that are looking at Kubernetes with suspicion, it is a natural instinct to think of K8s as a threat to all the knowledge we have built in the past few years, but it doesn't have to be that way. So many things that we would have built ourselves (deployments, upgrades, monitoring, etc) can now be streamlined with K8s, and existing knowledge around those topics will just make our transition to Kubernetes faster (besides still being able to use most of our expertise within K8s anyways). Managed solutions make K8s easy to use, while we can still benefit from being able to run out workloads left and right on any cloud vendor. In one word: portability. Which in the day and age of cloud vendor lock-in it is to be protected at all cost. I know that some organizations allow to allocate time to explore new technologies and learn new practices. If your organization has no policy around this, it is worth try asking. Ultimately it will benefit the organization and the business as a whole, as they will be able to build a solid foundation to more rapidly transition and execute on their digital products. Kubernetes is good for business.

drdaeman 7y ago
> K8s as a threat to all the knowledge we have built in the past few years, but it doesn't have to be that way All this knowledge is still relevant. Kubernetes is an extra layer on top of all the classical technologies out there. It is not a replacement - besides some fancy experiments, it doesn't replace GNU/Linux. If you're going to run a Kubernetes cluster, you still have to know all the "classical" networking, GNU/Linux system administration and architecture, etc. Having a fancy task scheduler for containers, and a tool that sets up clever network bridges (or whatever your preferred CNI flavor does), and a bunch of other nice extras does not remove the requirement to know what's underneath. And in any non-standard situation, there are chances that you'll have to build things from scratch. For example, I do want to run a K8s cluster over a cjdns overlay network (which doesn't have IPv4, so Flannel or Weave won't work). Haven't figured this out yet. And if you're not setting up and managing the infrastructure - it's about the same as it had always been. Just a different user interface to manage your deployments/storage/networking. > Managed solutions make K8s easy to use I would disagree. "Using" is the same (that's the whole point of K8s). "Install and maintain" is easier, but only in a sense that you don't do this. ;)

crb 7y ago

Podcast interview with the release team lead: https://kubernetespodcast.com/episode/096-kubernetes-1.18/ https://kubernetespodcast.com/episode/096-kubernetes-1.18/

robbiet480 7y ago

Really annoying that AWS EKS only got K8S 1.15 last week.

QUFB 7y ago

I've seen the cloud agnostic nature of Kuberbetes mentioned in many posts here. This is only true on the surface for a significant number of use cases and deployment models. Once petabytes of data are out there in your GCP or AWS environment, "portability" will be costly due to extortionistic pricing of egress bandwidth.

bcheung 7y ago
At that scale it is probably a better option to look into something like AWS Snowball and ship it on physical drives. At the petabyte scale there will probably be huge cost savings to using your own private cloud on your own hardware.
hodgesrm 7y ago
A lot of people think of multi-cloud as some kind of arbitrage where you jump quickly between markets. Running applications in cloud environments is a lot more like leasing property. Once you set up there are costs to moving. The portability argument boils down to saying you are not boxed in. If things get bad enough you can move. This is a big long-term advantage for businesses because it means you can correct mistakes or adjust to changing business conditions. That's what most people who run companies are really looking for.
- MrBuddyCasino 7y ago
  It also leads to nullifying the advantages of the cloud. The whole point are the proprietary services they offer, and use those instead of building them yourself. Trying to be „cloud agnostic“ is one of the biggest mistakes one could make.
  - closeparen 7y ago
    You're deploying to clusters of x86 machines in the first place because people in the 90s realized they were a little too into the proprietary APIs of their big iron vendors.
  - growse 7y ago

EdwardDiego 7y ago

I'm pleased to see the changes in the HPA, having pod scale-up/down periods be tied to a systemwide setting was a bit painful.

hiroshi3110 7y ago
This one? https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.18.md#new-api-fields https://github.com/kubernetes/kubernetes/blob/master/CHANGEL... > autoscaling/v2beta2 HorizontalPodAutoscaler added a spec.behavior field that allows scale behavior to be configured. Behaviors are specified separately for scaling up and down. In each direction a stabilization window can be specified as well as a list of policies and how to select amongst them. Policies can limit the absolute number of pods added or removed, or the percentage of pods added or removed. (#74525, @gliush) [SIG API Machinery, Apps, Autoscaling and CLI]
- EdwardDiego 7y ago
  Yep, in the version I'm on (1.15) there's only global flags and config[1] which apply to all HPAs, but not all apps should scale the same way - our net facing glorified REST apps can easily scale up with, say, a 1-2m window, but our pipeline apps sharing a Kafka consumer group should be scaled more cautiously (as consumer group rebalancing is a stop-the-world event for group members) 1: https://v1-15.docs.kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/#algorithm-details https://v1-15.docs.kubernetes.io/docs/tasks/run-application/...

joseph 7y ago

Shamelesss plug: Keights, my Kubernetes installer[1] for AWS supports 1.18 (the latest version available on EKS is 1.15). Keights also supports running etcd separate from the control plane, lets you choose instance sizes for the control plane, and can run in GovCloud. 1. https://github.com/cloudboss/keights https://github.com/cloudboss/keights

llarsson 7y ago
How does it compare to kops or, dare I say it, kubespray?
- joseph 7y ago
  Hi, I've often worked in corporate environments where it wasn't necessarily allowed to spin up a VPC, or to create an internet gateway. In the time I was creating this, many companies who are in healthcare or are otherwise locked down, could not use kops due to its requirement for an internet gateway. I created keights to fill that space, so that anyone could run Kubernetes in AWS, even in air gapped environments. This is pretty common nowadays, by the way - enterprises have a team to manage all AWS accounts, and they set up VPCs and connectivity ahead of time, before development teams get access to the account; access to the internet is through a proxy only, and no one can modify the network. Not to mention, most of the access to Amazon's services can now be done without an internet gateway, using VPC endpoints. Keights fits well in this world of locked down network access, and it works well even in GovCloud (you would need to build the AMI there as my public AMIs cannot be shared with GovCloud accounts). Keights and Kubespray both use Ansible, however they do it in a very different way. (Disclaimer: I haven't used kubespray, only looked over the documentation). Keights uses Ansible roles to build CloudFormation stacks to produce a cluster. The nodes in the cluster bootstrap themselves using systemd services that are baked into the AMI; Ansible does not run on the nodes in the cluster. Kubespray, as I understand it, uses a traditional Ansible approach of pushing configurations over ssh to nodes in its inventory. To my knowledge, it does not actually build the machines in the cluster, it just configures existing machines. Keights does the full end-to-end automation to bring up a working cluster, including the creation of all required AWS resources (autoscaling groups, a load balancer for the apiserver, security groups, etc - though you do provide certain resources as parameters, for example your VPC ID and subnet IDs, due to the aforementioned requirements to fit into locked-down environments).

eric_khun 7y ago

Anyone solved properly the CPU Throttling issues they are seeing with kubernetes ? Is this release solved it? We are seeing a lot of throttling on every deployments, what impact our latency, even when setting a really high cpu limit. The solutions seems to be: - remove the limit completely. Not a fan of this one since we really don't want a service going over a given limit... - using the static management cpu policy [2] Not a fan because some services doesn't need a "whole" cpu to run... Anyone has any other solutions? Thanks! [1] https://github.com/kubernetes/kubernetes/issues/67577 https://github.com/kubernetes/kubernetes/issues/67577 [2] https://kubernetes.io/docs/tasks/administer-cluster/cpu-management-policies/#static-policy https://kubernetes.io/docs/tasks/administer-cluster/cpu-mana...

dilyevsky 7y ago
1. It makes no sense to quota your cpu with the exception of very specific cases (like metered usage). You’re just throwing away compute cycles 2. Same applies to dedicate cores for pretty much same reasons Having said that if you really really want quota but don’t want shit tail latency I suggest setting cfs_quota_period to under 5ms via kubelet flag
- eric_khun 7y ago
  Thanks! wouldn't that be an issue if a pod "take over" the node , if for some reasons a request use too much CPU?
  - dilyevsky 7y ago
    Not really if you ensure every pod sets cpu request (which sets up cgroups cpu.shares) and your kubelet and system services are running in separate top-level cgroups (—kube-reserved and —system-reserved flags) you have reserved enforcement enabled. On full node contention every container will just consume its proportional share. This is not to say that someone malicious wouldn’t be able to dos a node but untrusted workload is a whole separate topic
- rumanator 7y ago

dragonsh 7y ago

Containers are not necessary if the systems are build with something like Guix or Nix as they provide transaction level updates to applications and dependency and secure by default as no need to run a daemon with root access to run, monitor and manage containers. They provide same way of managing application deployment, the way application source code is managed with versioned deployments and rollbacks all baked in. But as with any technology Guix and Nix are still decade ahead of the present and may pick up later when technology converge back to running application servers and other dependent software in isolation with user level namespaces. Kubernetes try to solve one problem and create 10 other infrastructure problems to manage and instead of working on application, tie the company to a specific distribution or cloud service provider. So far there is nothing revolutionary in it unless the startup or company adopting k8s is google size operations. From a software developer perspective which is the main audience of HN it will be popular as most of them dream or wants to work for company of size google. Startup founders want to solve the scaling problem like google in the beginning as everyone dreams to be google size from day one. Kubernetes complexity is useful at large scale for majority i.e. over 90% of the deployments simple containers, bare metal or VM with traditional configuration management will be sufficient.

gigatexal 7y ago
Kubernetes is here. Better to learn it and use it where appropriate than to fight it.
- dragonsh 7y ago
  I don't need to fight it, I am not looking for a job and follow the herd mentality. My own startup is pretty happy with Guix and related infrastructure on bare-metal, it works well for us and we can still stand on shoulders of giants who have done much better job in managing large distributed infrastructure.
  - gigatexal 7y ago
    That’s awesome. I’m not evangelizing it or saying you should use it as I did say “where appropriate” and at your startup it doesn’t seem appropriate. There are probably 10x more engineers that have experience with k8s than with guix though so ... perhaps that could be a factor though not a reason to change out the infra completely.

jasdfajdsfk 7y ago

k8s posts == spam

deboflo 7y ago

Kubernetes? No thanks.

mehdix 7y ago

FWIWI a few days back I upgraded my microk8s-based k8s to 1.18 beta channel and it solved at least ImagePullBackOff problem that I had with pulling from my private GitLab registry. Worked like charm.

lprd 7y ago

I've taken a couple k8s courses, I understand all the small parts that make up k8s, but yet it seems that there are still no easy solutions to install on bare metal. The default recommendation is to always just roll with a managed solution. This is slightly irritating considering there are plently of companies out there who own their own infrastructure. There are plenty of great developer distributions out there (k3s, kind, minikube, microk8s), but those are single node only, and aren't meant for production use. I'm still searching for a solid guide on how to get k8s installed on your own hardware. Any suggestions would be very appreciated!

ojhughes 7y ago
k3s supports multinode
- lprd 7y ago
  Ah, I wasn't aware. Does it support HA as well?
  - deleted 7y ago
    [deleted]
  - ojhughes 7y ago
    Yes if you use an external DB for the k8s control plane https://rancher.com/docs/k3s/latest/en/installation/datastore https://rancher.com/docs/k3s/latest/en/installation/datastor...
    - pas 7y ago
      There's also an experimental embedded DQlite (raft + sqlite) thingie too! https://rancher.com/docs/k3s/latest/en/installation/ha-embedded/ https://rancher.com/docs/k3s/latest/en/installation/ha-embed...
mike1o1 7y ago
https://www.ansibleforkubernetes.com/ https://www.ansibleforkubernetes.com/ I've found this pretty good starting point. Keep in mind the book is still a work in progress. Note, I haven't actually run k8s in production, but this book has helped me get something up and running in VM's pretty quickly using Ansible on top of kubeadm.