11 ms·

Kubernetes Exposed: One YAML Away from Disaster

182 points by consumer451 3y ago

0xbadcafebee 3y ago
For those unaware: Kubernetes has long had horrible default security. RBAC isn't even enabled by default. Nearly every cluster I've seen allows regular users and applications to breakout into the host nodes by default. Network ACLs, if they are ever used, almost never block either intra-cluster or extra-cluster attacks. Unencrypted API ports are standard. The distinctions between "cluster-level" resources and "everything else" unnecessarily complicates creating fine-grained rules. Default access even with RBAC is too wide-open. I won't even get into ABAC... Basically, the geniuses at Google decided to make clusters insecure by default, because being an admin is, like, hard, man. So guess what? People deployed clusters that were insecure by default. I don't think it has ever been easier to take over cloud computing environments just by exploiting a dinky web app.
- znpy 3y ago
  > Basically, the geniuses at Google decided to make clusters insecure by default, because being an admin is, like, hard, man. So guess what? People deployed clusters that were insecure by default. There is a lot to unpack here, but while I agree with the main point (kubernetes defauls suck) the main problem I’ve seen all across the industry is that people expect to be able to install stuff around and that to be magically secure, functional and performant. It just doesn’t work like that. Systems administration is its own discipline and you can’t expect to follow some random tutorial off the internet and be ready for production. Also, it doesn’t help that the kubernetes documentation is awful. I remember my time working with kubernetes, it’s a patchwork of stuff… it does require time and a fairly high pain threshold to properly master it.
  - orthoxerox 3y ago
    Well, there are two opposing approaches: "up and running" defaults or production defaults. You choose "up-and-running" defaults if you want people to be able to immediately run your software. So this means no encryption, no auth (or "the first user is the admin" kind of auth), storing your state in a bundled SQLite DB, etc. This helps with growth, because random devs can download the container image, run it and check out your features in five minutes. And then run it in production. With production defaults everything is locked down. Go issue some certificates and make sure your OS (or JRE) trusts them, because your nodes won't talk to each other without TLS. No one has any rights to anything by default, can't even log in until you go and edit authN/Z configuration. The container will kill itself if the env var for the database password is still "change_this!". It's a pain to deploy this, but a pleasure to run. Ideally, all products should come with two sets of defaults. Your AIO quickstart image should be easy to run. Your production-grade image or the configs that come with the distribution should be locked down tighter than Fort Knox. K8s is a piece of infrastructure that doesn't really benefit from an "up and running" approach, but it has this kind of defaults for some reason instead of having every default policy resolve to "not allowed".
    - hedora 3y ago
      OpenBSD is an existence proof of a third route: Carefully manage complexity, and write a good manual. OpenBSD is about 22M lines of code; kubernetes is 58M, so the code bases are of comparable complexity (OpenBSD uses many more programming languages than kubernetes does). I'd argue that the base OpenBSD install provides far more utility than Kubernetes, but market share numbers disagree. Sources: https://openhub.net/p/kubernetes https://openhub.net/p/kubernetes https://www.reddit.com/r/openbsd/comments/d9a8c9/how_many_lines_of_code_does_openbsd_65_contain/?rdt=50542 https://www.reddit.com/r/openbsd/comments/d9a8c9/how_many_li... Edit: Using open hub for both says OpenBSD is 75M lines: https://openhub.net/p/openbsd https://openhub.net/p/openbsd
      - lakomen 3y ago
        I don't follow. OpenBSD is an OS, k8s is a set of services running on an OS.
        code_biologist 3y ago
        Both are things deployed to host network visible services.
    - mulmen 3y ago
      > or "the first user is the admin" kind of auth My mind immediately went to Eddie Izzard’s “do you have a flag” bit.
  - deleted 3y ago
    [deleted]
  - ryukafalz 3y ago
    > There is a lot to unpack here, but while I agree with the main point (kubernetes defauls suck) the main problem I’ve seen all across the industry is that people expect to be able to install stuff around and that to be magically secure, functional and performant. > It just doesn’t work like that. Systems administration is its own discipline and you can’t expect to follow some random tutorial off the internet and be ready for production. Disagree. I can install an app to my Sandstorm instance and reasonably expect it to be secure, functional, and performant (inasmuch as the app is performant anyway). The tools we use make things difficult, but I don't think there's as much irreducible complexity here as you're implying. We can and should build better tools.
    - mplewis 3y ago
      Sandstorm is a very different category of system from k8s.
      - znpy 3y ago
        I had the same feeling, but frankly I didn’t bother replying because somebody putting sandstorm and kubernetes on the same level shows their complete lack of any understanding whatsoever. Kinda proves my point: some random amateur sysadmin thinking their experience with sandstorm is enough to put kubernetes in production. But thanks for replying, anyway.
        ryukafalz 3y ago
        I worked with K8s professionally for several years but thanks for assuming I have no idea what I'm talking about. Yes, they're obviously different classes of systems, but not in the ways that matter for this discussion. If the question is whether or not you can design a hosting platform with secure defaults and managed/tightly-scoped communication between applications, the answer is plainly yes. Is Sandstorm as scalable and fault tolerant as Kubernetes? Obviously not.
  - raesene9 3y ago
    Out of curiosity, which bits of the k8s documentation (if there are specific bits) did you find lacking? If it's security related stuff, I can open some issues to see if things can get improved :)
    - znpy 3y ago
      K8s documentation falls essentially in two categories: either reference page for k8s components (like the kubelet) that lists all the options woth a very very very brief description … or conceptual tutorial-like that explain some concepts, but without much details. Nothing in the middle. There is little documentation on how to put the various pieces together. That’s all on various random webpages outside of the main kubernetes.io website. The setup tutorial only describes some basic kubeadm-based setup. There’s a lot to improve, if you ask me.
- benatkin 3y ago
  Horrible scaling to tiny clusters as well. I'm glad they kept the design pure though. Unencrypted means you get to choose the encryption. Redis didn't have built in encrypted tcp servers for a long time either. When they added it there was a good discussion about the overhead: https://github.com/redis/redis/issues/7595 https://github.com/redis/redis/issues/7595
- k8sfellow 3y ago
  RBAC is now enabled by default. But I wonder how many of these clusters that are exploitable are Openshift for example. Openshift had secure by default settings from as long as I can remember. Non-root pods as default, api-server and even metric collection ports are protected by authentication. Openshift also had much strict RBAC settings from very beginning(like even in Openshift 3.x days, when k8s had no RBAC by default and you had to opt-in).
  - vbezhenar 3y ago
    What I don't like about standard kubernetes (kubeadm) is how hard it is to configure pods being secure by default, so developers couldn't just mount host directories into the pods. You would need complicated configs for that and nobody does that.
    - raesene9 3y ago
      For less complex options in modern k8s, you could configure Pod Security Admission on a namespace basis. It's not too complex as you just add some annotations to a namespace to activate it. The baseline policy will disable hostPath volumes which helps a bit, alongside quite a few other restrictions.
      - vbezhenar 3y ago
        I'm aware about that, but the problem is: it's just some kind of good will. First of all: you need to actually add those annotations to the namespace. If someone adds his own namespace, it won't be there. Second: anyone can just remove those annotations (without complicated RBAC rules) and do hostpath and other things. I'm not talking about hackers, but just some less experienced people who might read that advice on forums or chats and decided to just proceed without asking. One can use kyverno to enforce those namespace, but it's so heavyweight solution, I actually deployed it and it was terrible decision, so now I have to plan to decommission it. There's mechanism to actually enforce default baseline policy, so even if it's not set on a namespace, it'll be "baseline". However to configure that mechanism one need to configure admission controller. But good luck configuring it with kubeadm. It requires some cryptic YAMLs, YAML patches and whatnot. You can't just edit some configmap in the cluster and be done with it. https://kubernetes.io/docs/tasks/configure-pod-container/enforce-standards-admission-controller/ https://kubernetes.io/docs/tasks/configure-pod-container/enf... this is what I'm talking about. It's more issue with kubeadm, I guess, which really makes things harder than they should be. I can't just run kubeadm join control-plane, I need to craft configuration, add this AdmissionConfiguration to it, remember to update it on all control nodes if I decide to change it later.
- ransom1538 3y ago
  "Default access even with RBAC is too wide-open." Not following this. Isn't a k8s cluster behind a network firewall? Only exposing 443? To me RBAC would be useful in a specific case where you want k8 service accounts (no one uses these) to have different levels of security. That sounds awful.
  - alephnerd 3y ago
    Jeff in Engineering might have deployed an app spanning vCenter as well as EKS, and to continue to allow the app to function wrote subpar ingress/egress rules.
  - mulmen 3y ago
    443 is just a port. If we assume that implies HTTPS then that’s still only TLS. If I exploit the service (“securely”!) I am inside your network. I can then pivot to your management layer. Game over.
  - vbezhenar 3y ago
    > Isn't a k8s cluster behind a network firewall? Of course not, unless you configure it this way. > RBAC would be useful in a specific case where you want k8 service accounts (no one uses these) to have different levels of security. What? How comes no one uses service accounts? They're used extensively for many things.
  - deleted 3y ago
    [deleted]
- cratermoon 3y ago
  I've seen non-tech corporations adopting Kubernetes without realizing that it's just one part of Google's infrastructure, not the entire operations side. There's an entirely separate chunk of infrastructure that provides the kind of security you mention. K8s is a "do one thing and do it well" that does workload orchestration. Google's BeyondCorp system enforces a zero-trust policy. It's not left to each part of the infrastructure to independently implement.
  - alfalfasprout 3y ago
    Well, Google uses Borg internally which isn't exactly Kubernetes. But yes, the idea is that for it to be useful as infra it has to come without a lot of magic automatically attached.
- raesene9 3y ago
  Whilst Kubernetes defaults are indeed not great, they aren't that relevant unless you're building your clusters from scratch instead of using a k8s distribution, and I don't think I've seen anyone do that in a long time. RBAC is enabled by default on just about every distribution I've seen (with the notable exception of microk8s) Lack of default admission control to prevent someone with pod creation access breakout to the host is indeed pretty common Unencrypted APIs hasn't really been a problem in years, the insecure API port and read-only Kubelet were a thing but they've been disabled for quite a few versions now. I'd definitely agree that early k8s versions had very bad security, but things have been improving gradually over the versions :)
  - 0xbadcafebee 3y ago
    Well first of all, there are still thousands of clusters running with the old insecure defaults, so that's bad. Both GKE and EKS currently have insecure defaults. There's a half dozen ways to take over the entire cluster with the default settings. The unencrypted API port is only this year being deprecated on GKE, I don't know about EKS. Since you can only grant permissions, not deny, most people end up granting too many permissions, defeating the purpose of least privilege. Default permissions for users on GKE/EKS are way too wide open. You can do things you just shouldn't need to do as a regular dev. You shouldn't be able to just deploy a helm chart to a random cluster and then all of a sudden there's an unauthenticated public service on the internet. But that's what every single distribution of K8s gives users by default. Regardless of whether you think that's fine or not, this is how companies get exposed on a regular basis, so it is a real-world persistent security problem that is perpetuated by the defaults.
    - raesene9 3y ago
      People not upgrading is indeed a problem but not one the distributions or k8s can particularly address, unless they force upgrade people which with deprecation/removal of APIs doesn't tend to be popular :) The support lifecycle is pretty short, compared to general enterprise software, so unless orgs are happy running unsupported versions of k8s they're on 1.23 for EKS or 1.24 for GKE by this point. I wouldn't have said that EKS, in particular, had very bad defaults, IME it's the best of the managed distributions for security. Any references for 1/2 dozen ways of taking over a cluster from defaults settings? I'd genuinely be interested as I collect information about Kubernetes security, so I'm always on the look-out for more references. for unencrypted API are you thinking of the Insecure API port or read-only Kubelet? For the insecure API port I don't think that's been a thing for a really long time on managed distros. The only time I remember seeing it was on ACS (the predecessor to AKS) where it was exposed to the container network. read-only Kubelet was more of a thing as that was on worker nodes, but it was less risky.
geek_at 3y ago
I had a similar experience when I tried to use kubernetes in my home lab. Crashed it twice and since the technology and tools evolve so fast almost all tutorials I found were already so out of date that the commands didn't work and some tools were already deprecated. I moved to docker swarm and love it. It's so much easier, straight forward, automatic ingress network and failover were all working out of the box. I'll stay with swarm for now
- GauntletWizard 3y ago
  Do you go to the hot new restaurant in town on their grand opening, and then leave a review that says "the food was great, but it was too crowded and the service was slow", and never go there again? That sounds like what you did there.
  - globuous 3y ago
    Is that not what you’d do? That sounds like an unusually useful review tbh.
    - starttoaster 3y ago
      The point of their comment, I think, is that a restaurant's grand opening is not always indicative of future experiences. The cooks/chefs, and front of house staff are all fairly new to the environment, menus might change as they find out that certain menu items are throwing off their timing or they don't have a great way to procure certain ingredients, and it will take time to all settle into a routine that works. I think they were saying that is akin to the state of kubernetes, where everything is too new to really judge it today. Let me make clear my bias before I get to my opinion: I'm a big fan of what kubernetes has accomplished with managing the deployment of applications. But I think comparing current day kubernetes to a restaurant's grand opening is naïve. Kubernetes is amazing at what it does, and for how highly flexible it is. But it is highly flexible because of how advanced it has become, which in turn requires more knowledgeable administrators. You can take off with Docker Swarm because... it's very simple. It doesn't aim to be flexible, it just aims to be the docker-compose of distributed systems. Which is fine if you don't need the flexibility that kubernetes offers anyway. And while I do appreciate kubernetes for all of its strengths, going back towards the point of this OP, I do believe it's a bit of a security nightmare. For example, the kubernetes operator pattern often just requires this application to run in your cluster with a ClusterRole granting it read and potentially write levels of access to ConfigMaps and Secrets around your cluster. Often without much in the ways of a method of telling the kubernetes API server what namespaces it is allowed to view those objects in. Worth mentioning though that secrets management is something that Docker Swarm also handles fairly poorly, and honestly isn't much of a security regression from the normal "deploy a VM that has all your decrypted secrets sitting on the system" anyway.
ozarker 3y ago
A couple years ago I came into a company that had recently switched almost everything to Kubernetes. The company expected the devs to learn how to own and operate the cluster. Most of the devs there couldn't care less about anything except how to deploy their code to it (probably rightfully so). But I was shocked to find the k8s api was exposed to the internet and unauthenticated. I brought the problem up with the higher-ups and they were pretty lethargic about it. Moral of the story I guess is if you're gonna choose to operate infrastructure like Kubernetes make sure you have experienced people trained to do so and don't expect your devs to wear another hat.
- andhapp 3y ago
  DevOps is the new way, I think. Developers create the code and also know how to run and maintain infra :-)
  - sharts 3y ago
    I don’t think that’s true anymore. Most companies that seriously deploy k8s tend to have dedicated teams for it. There’s no way an app developer is going to be able to do their work while also knowing the ins and outs of cluster mgmt.
    - bonestamp2 3y ago
      > Most companies that seriously deploy k8s tend to have dedicated teams for it. Ya, and everywhere I've worked, that team is called "devops". The application developers are a different team. Maybe that's not the norm, but I thought it was.
      - funcDropShadow 3y ago
        > Ya, and everywhere I've worked, that team is called "devops". But then that team is only doing operations and not development. Unless they are committers to k8s.
        3y ago
debarshri 3y ago
This is super interesting. If you actually look at a Loadbalancer logs without a WAF in place, you will see huge amount of bot activity. What is not clear in the blogs is how did they actually accessed the API server without a valid service token. Secondly, they index alot on IPs, that implied these are k8s cluster that are not managed (EKS, GKE etc). Is my assumption correct?
- abatilo 3y ago
  The GKE "default" is to bind a public IP to your control plane. So these could still be managed
- raesene9 3y ago
  So it is possible to expose Kubernetes without auth (https://raesene.github.io/blog/2023/03/18/lets-talk-about-anonymous-access-to-Kubernetes/ https://raesene.github.io/blog/2023/03/18/lets-talk-about-an... has some details) but it's not a default You can make that mis-configuration with managed distros, which are generally exposed to the Internet by default.
  - debarshri 3y ago
    Ah that's what it is. Opening up API server that's allows anonymous auth is a very bad idea.
koito17 3y ago
I am actually a bit surprised there are any clusters out there exposing their service ports. Do people not run firewalls in their nodes? Sure, it may cause some headaches due to a range of random IPs and random ports that Kubernetes expects to be able to bind, but I assume any sysadmin will know how to deal with this. I know k3s has the awesome advice of disabling firewalld[1], but presumably the clusters mentioned in this article are largely using kubeadm or some other tool rather than k3s. With that said, I agree that secrets management on k8s is pretty bad (unless you use an external service like Hashicorp Vault, a frequent pattern that I run into with k8s). Even with RBAC and secrets in a namespace being inaccessible from other namespaces, the moment someone has permissions to read secrets from a namespace, its essentially game over for you, as noted in this paragraph. > Another potential attack vector within a Kubernetes cluster is accessing internal services. ... In cases where authentication is required, the credentials can typically be found within the Kubernetes secrets space. [1] https://docs.k3s.io/advanced#red-hat-enterprise-linux--centos https://docs.k3s.io/advanced#red-hat-enterprise-linux--cento...
- merb 3y ago
  The firewalld is for the trusted zone… and for the apiserver 6443 https port k3s comes with rbac enabled by default so if you want to access it via kubelet you need to open it?! trusted zone is used by overlay networks or node internal stuff (or if you manually put your interface into it)
- crabbone 3y ago
  Why would you be surprised? In most cases the only thing users need from a Kubernetes cluster is to access it from the outside... like... why wouldn't they expose it to the outside world? -- this is the most useful thing you can do with it. And the range of things you might want to expose... oh, god only knows. Think about such trivial stuff as FTP or iSCSI -- kinda useful, right? -- but Kubernetes rules for port exposure are idiotic, because you cannot expose ranges. And quite a few of popular Internet protocols assume they can allocate whatever port they want and start talking on it. And then you start fighting Kubernetes... and you'll win, but it will be some custom security solution... with the obvious problem of it being a custom security solution. Now, add to this that Kubernetes will typically mess with your iptables and maybe also arptables, and if your firewall is based on eg. iptables, then you might be out of luck using it with Kubernetes, and again, will end up rolling something "special". Obviously, you might also want, say, DNS to work inside your cluster, but that's a separate can of worms.
barakm 3y ago
This is also why managed Kubernetes is a useful thing (EKS, GKE, et al)... but if you still want to do it yourself, maybe look into some Kubernetes distros (like Typhoon (https://typhoon.psdn.io https://typhoon.psdn.io) which I run on my clusters)
- lakomen 3y ago
  But managed is expensive. You have to pay at least double vs unmanaged (in the dedicated world, ofc there is no "unmanaged" k8s offer from those cloud providers, just an analogy). You are then bound to that cloud and its traffic prices and all those you mentioned are filthy expensive when it comes to traffic.
  - alephnerd 3y ago
    If you are dealing with the level of scale that allows you to fully take advantage of K8s, the cost of managed offerings is less than that of deploying and managing Openshift on-prem. Also, the beancounters don't like on-prem as much anymore due to the upfront cost outlay. I've been working with a number of major enterprise accounts and they've all been adopting managed K8s offerings for this very reason.
    - deleted 3y ago
      [deleted]
erulabs 3y ago
I mean, it’s not the best argument to say a system is “one action away from disaster”. If you pull the escape lever in an F16 for no reason you’re also pretty screwed. If you intentionally fly into the ground no one blames General Dynamics for making the plane too complex. It turns out complex tools for complex tasks require some skilled operators. Yes kubernetes is overkill for your home lab, no it’s not overkill for enterprise-grade AWS orchestration with hundreds of apps and millions of users. News at 11.
- ramraj07 3y ago
  I want to host a website not bomb the Middle East. Most enterprise apps are still websites that need no more than elastic beanstalk. If your analogy is to compare k8s to f16 then maybe you are not doing your argument any service.
  - alephnerd 3y ago
    If your dealing with dynamic scaling needs K8s are quite nice. There's a reason why most Telcos have deployed 5G functionality using K8s. If you are in a situation where you want to follow a decoupled/microservices development pattern (ie. split component ownership across dev teams) as well as manage scalability during high load periods, K8s is hard to beat. K8s is a useful tool, but can be overkill in a number of situations. Just like a hammer, you can't use it to solve for every usecase. P.S. F-35s use K8s clusters (or might be K3s) to deploy their EWS.
  - erulabs 3y ago
    Well both hosting websites and bombing the middle east make use of Kubernetes. [1] 1. https://www.theregister.com/2020/06/03/kubernetes_b_21_bomber/ https://www.theregister.com/2020/06/03/kubernetes_b_21_bombe... I've spent my entire career being on-call for web servers and Elastic Beanstalk is not enough. Deployment failures, deployment speed, deployment safety, upgrades, local testing... There is a reason AWS themselves is pushing EKS and not Elastic Beanstalk, and it's not because Kubernetes is trendy, it's because it's incredibly useful.
    - 3y ago
alephnerd 3y ago
Let the Blackhat blogspam begin!
crabbone 3y ago
Here's my Kubernetes security horror story. You probably know about resolv.conf, but you might not know about "search domains". A feature that should've never existed, but was added because "why not?". If you don't know what it does, in first approximation, it creates a domain alias. This is sometimes used to save some typing when using DNS for service discovery in larger clusters in a very hierarchical way (when names start to look like a.b.c.d.e.f.g...com, it's tempting to alias them to something shorter). So, long story short. Ubuntu LTS had a bug (actually a whole bunch of them) in the same script that was responsible for generation of this file (it's a bad idea to have this file to begin with... but hey, who cares, right?). So, one aspect of the bug was that a variable name "DOMAINS" was used instead of its value. And that became a search address in resolv.conf. Now, this search works by appending the search path to the domain name, if DNS server tells you it cannot find it. I had no idea DOMAINS was possible to register as a valid domain name. I mean, it should've been a TLD to be discoverable globally, right? -- Well, it's still a mystery to me, but, for some reason domains of the form a.b.c.DOMAINS are resolved to some AWS EC2 instance... And, the punch line: if you screw up CoreDNS configuration somehow, or something else goes wrong with service discovery inside your Kubernetes cluster on unpatched Ubuntu -- there's a good chance your traffic will go to that EC2 instance. Who's on the other side -- I don't know. I didn't send anything important that way and discovered it because something like Spark operator didn't work... but it could've been much worse. Oh, btw, I believe resolv.conf is used by different CNIs. Maybe Calico and Flanel. I cannot remember atm. why it was used... or maybe it's some kubelet setting. Anyways. Larger point is... large Kubernetes clusters are virtually impossible to systematically audit. Too many things can do too many things. Too much trickery goes on in all sorts of system components that deal with networking. It's also impossible to track who does what and who owns what because there's no user identity. --- I believe that, again, industry adopted an absolute horror show of a program / environment not because it was a great tool, but because "organically" developers started to add "features", creating something that's hard to resist, because, superficially, it creates a lot of value with minimal effort -- but it also lays a trap that is not immediately obvious. And those who find themselves trapped, instead of running away, star adding more "features" to fight the trap. But, in this foolhardy attempt, they only make the trap stronger.
- uberduper 3y ago
superq 3y ago
Servers are so much faster than they were ten years ago that a typical, fully-loaded enterprise app can almost always run on a single mid-sized 10-core box with 256 GB of RAM.. from ten years ago. It's absolutely ludicrous how much power is available now on a single box if you really want to scale up instead of out. (Databases, of course, always like to eat everything because of poorly-optimized statements, but they've never really been good fits for K8s or containers anyway.) Some might say that this isn't webscale, but Stack Overflow runs one of the most popular tech websites in the world.. on only nine web servers: https://www.datacenterdynamics.com/en/news/stack-overflow-still-on-prem-runs-qa-platform-off-just-nine-servers/ https://www.datacenterdynamics.com/en/news/stack-overflow-st... and LiChess is similar: https://lichess.org/costs https://lichess.org/costs
- metadat 3y ago
  > a typical, fully-loaded enterprise app can almost always run on a single mid-sized 10-core box with 256 GB of RAM. If you don't care about high-availability, then yeah. One thing I've learned in my career is that delivering reliable enterprise-grade service is anything but trivial. The level of complexity of any modern stance is high enough that there will inevitably be sharp-edged corner cases which aren't simple or cheap to deal with.
  - spacebanana7 3y ago
    Don't most outages happen because of unforeseen consequences of changes to the application/environment rather than machine failure? Is reducing the exposure to machine failure really worth exposing yourself to the vast complexity of distributed systems?
    - PrayagBhakar 3y ago
      I think the point of such tools is more than just reducing exposure to machine failure. Reducing exposure to machine failure also means that you can run your hardware closer to end of life without having to worry about data outages. The article about stack overflow mentions this little tidbit. > That application handles around 6,000 requests per second, and around 2 billion requests per month. To be able to handle loads of 1M requests per second, your going to need more than one server dispensing results. Especially since only so much data can be stored in memory without running into performance issues. That said, these kinds of tools are an unique use case. Not all programs need them.
gerbilly 3y ago
[flagged]
zaps 3y ago
Super-naive take: why are people such jerks?
raesene9 3y ago
I've done quite a lot of looking at exposed k8s clusters, and one thing this piece doesn't really cover is what percentage of the 350 clusters they identified were in fact honeypots. You can find k8s clusters without auth using Censys as they specifically try to access things like pod lists, and from spinning through lists of those they tend to be obvious honeypots (e.g. lots of ports that wouldn't usually go together being exposed), although I have seen the odd cluster that looks a bit more realistic, unlike Aqua I didn't try accessing it directly as that could be dodgy legally :D If anyone wants to look at the list, Censys handily has a search query for "kubernetes clusters where the pod list is exposed without authentication" https://search.censys.io/search?resource=hosts&q=services.kubernetes.pod_names%3D%22*%22 https://search.censys.io/search?resource=hosts&q=services.ku...
roydivision 3y ago
This article seems to be a very long winded way of saying "Beware of what you are exposing to the internet", using Kubernetes as an example.
jhoelzel 3y ago
While this post has some merit, all of these problems can easily be solved and there are dozens of guides on hit. The entire "CIS Hardening guide" will give you step by step tutorials on how to solve all of this, you just need to bother looking it up. Look, you need a trained sysadmin anyway, no matter where you deploy. See for instance all the ubuntu machines out there that never have seen the admin update it. Or all the terraform scripts that simply get copy pasted by orgs. While I do agree, this is a problem, there are security conscious distributions if kubernetes out there like RKE2 or "rancher government" to make things much more secure. This is actually my job. If you need someone how to secure your cluster and make sure stuff like this is very unlikely to happen, you can find the info in my profile. But what it comes down to is that if your DEVs have direct cluster access and you have no monitoring and alerting setup, you are kind of operating on the same level as copying and pasting stackoverflow answers and pushing them directly to prod without code review.
user6723 3y ago
Linux exposed, one shell command away from disaster rm -rf / Looks like Linux is no good, back to the drawing board, time to redo the while cloud everybody.

182 points by consumer451 3y ago

For those unaware: Kubernetes has long had horrible default security. RBAC isn't even enabled by default. Nearly every cluster I've seen allows regular users and applications to breakout into the host nodes by default. Network ACLs, if they are ever used, almost never block either intra-cluster or extra-cluster attacks. Unencrypted API ports are standard. The distinctions between "cluster-level" resources and "everything else" unnecessarily complicates creating fine-grained rules. Default access even with RBAC is too wide-open. I won't even get into ABAC... Basically, the geniuses at Google decided to make clusters insecure by default, because being an admin is, like, hard, man. So guess what? People deployed clusters that were insecure by default. I don't think it has ever been easier to take over cloud computing environments just by exploiting a dinky web app.

znpy 3y ago
> Basically, the geniuses at Google decided to make clusters insecure by default, because being an admin is, like, hard, man. So guess what? People deployed clusters that were insecure by default. There is a lot to unpack here, but while I agree with the main point (kubernetes defauls suck) the main problem I’ve seen all across the industry is that people expect to be able to install stuff around and that to be magically secure, functional and performant. It just doesn’t work like that. Systems administration is its own discipline and you can’t expect to follow some random tutorial off the internet and be ready for production. Also, it doesn’t help that the kubernetes documentation is awful. I remember my time working with kubernetes, it’s a patchwork of stuff… it does require time and a fairly high pain threshold to properly master it.
- orthoxerox 3y ago
  Well, there are two opposing approaches: "up and running" defaults or production defaults. You choose "up-and-running" defaults if you want people to be able to immediately run your software. So this means no encryption, no auth (or "the first user is the admin" kind of auth), storing your state in a bundled SQLite DB, etc. This helps with growth, because random devs can download the container image, run it and check out your features in five minutes. And then run it in production. With production defaults everything is locked down. Go issue some certificates and make sure your OS (or JRE) trusts them, because your nodes won't talk to each other without TLS. No one has any rights to anything by default, can't even log in until you go and edit authN/Z configuration. The container will kill itself if the env var for the database password is still "change_this!". It's a pain to deploy this, but a pleasure to run. Ideally, all products should come with two sets of defaults. Your AIO quickstart image should be easy to run. Your production-grade image or the configs that come with the distribution should be locked down tighter than Fort Knox. K8s is a piece of infrastructure that doesn't really benefit from an "up and running" approach, but it has this kind of defaults for some reason instead of having every default policy resolve to "not allowed".
  - hedora 3y ago
    OpenBSD is an existence proof of a third route: Carefully manage complexity, and write a good manual. OpenBSD is about 22M lines of code; kubernetes is 58M, so the code bases are of comparable complexity (OpenBSD uses many more programming languages than kubernetes does). I'd argue that the base OpenBSD install provides far more utility than Kubernetes, but market share numbers disagree. Sources: https://openhub.net/p/kubernetes https://openhub.net/p/kubernetes https://www.reddit.com/r/openbsd/comments/d9a8c9/how_many_lines_of_code_does_openbsd_65_contain/?rdt=50542 https://www.reddit.com/r/openbsd/comments/d9a8c9/how_many_li... Edit: Using open hub for both says OpenBSD is 75M lines: https://openhub.net/p/openbsd https://openhub.net/p/openbsd
    - lakomen 3y ago
      I don't follow. OpenBSD is an OS, k8s is a set of services running on an OS.
      - code_biologist 3y ago
        Both are things deployed to host network visible services.
  - mulmen 3y ago
    > or "the first user is the admin" kind of auth My mind immediately went to Eddie Izzard’s “do you have a flag” bit.
- deleted 3y ago
  [deleted]
- ryukafalz 3y ago
  > There is a lot to unpack here, but while I agree with the main point (kubernetes defauls suck) the main problem I’ve seen all across the industry is that people expect to be able to install stuff around and that to be magically secure, functional and performant. > It just doesn’t work like that. Systems administration is its own discipline and you can’t expect to follow some random tutorial off the internet and be ready for production. Disagree. I can install an app to my Sandstorm instance and reasonably expect it to be secure, functional, and performant (inasmuch as the app is performant anyway). The tools we use make things difficult, but I don't think there's as much irreducible complexity here as you're implying. We can and should build better tools.
  - mplewis 3y ago
    Sandstorm is a very different category of system from k8s.
    - znpy 3y ago
      I had the same feeling, but frankly I didn’t bother replying because somebody putting sandstorm and kubernetes on the same level shows their complete lack of any understanding whatsoever. Kinda proves my point: some random amateur sysadmin thinking their experience with sandstorm is enough to put kubernetes in production. But thanks for replying, anyway.
      - ryukafalz 3y ago
        I worked with K8s professionally for several years but thanks for assuming I have no idea what I'm talking about. Yes, they're obviously different classes of systems, but not in the ways that matter for this discussion. If the question is whether or not you can design a hosting platform with secure defaults and managed/tightly-scoped communication between applications, the answer is plainly yes. Is Sandstorm as scalable and fault tolerant as Kubernetes? Obviously not.
- raesene9 3y ago
  Out of curiosity, which bits of the k8s documentation (if there are specific bits) did you find lacking? If it's security related stuff, I can open some issues to see if things can get improved :)
  - znpy 3y ago
    K8s documentation falls essentially in two categories: either reference page for k8s components (like the kubelet) that lists all the options woth a very very very brief description … or conceptual tutorial-like that explain some concepts, but without much details. Nothing in the middle. There is little documentation on how to put the various pieces together. That’s all on various random webpages outside of the main kubernetes.io website. The setup tutorial only describes some basic kubeadm-based setup. There’s a lot to improve, if you ask me.
benatkin 3y ago
Horrible scaling to tiny clusters as well. I'm glad they kept the design pure though. Unencrypted means you get to choose the encryption. Redis didn't have built in encrypted tcp servers for a long time either. When they added it there was a good discussion about the overhead: https://github.com/redis/redis/issues/7595 https://github.com/redis/redis/issues/7595
k8sfellow 3y ago
RBAC is now enabled by default. But I wonder how many of these clusters that are exploitable are Openshift for example. Openshift had secure by default settings from as long as I can remember. Non-root pods as default, api-server and even metric collection ports are protected by authentication. Openshift also had much strict RBAC settings from very beginning(like even in Openshift 3.x days, when k8s had no RBAC by default and you had to opt-in).
- vbezhenar 3y ago
  What I don't like about standard kubernetes (kubeadm) is how hard it is to configure pods being secure by default, so developers couldn't just mount host directories into the pods. You would need complicated configs for that and nobody does that.
  - raesene9 3y ago
    For less complex options in modern k8s, you could configure Pod Security Admission on a namespace basis. It's not too complex as you just add some annotations to a namespace to activate it. The baseline policy will disable hostPath volumes which helps a bit, alongside quite a few other restrictions.
    - vbezhenar 3y ago
      I'm aware about that, but the problem is: it's just some kind of good will. First of all: you need to actually add those annotations to the namespace. If someone adds his own namespace, it won't be there. Second: anyone can just remove those annotations (without complicated RBAC rules) and do hostpath and other things. I'm not talking about hackers, but just some less experienced people who might read that advice on forums or chats and decided to just proceed without asking. One can use kyverno to enforce those namespace, but it's so heavyweight solution, I actually deployed it and it was terrible decision, so now I have to plan to decommission it. There's mechanism to actually enforce default baseline policy, so even if it's not set on a namespace, it'll be "baseline". However to configure that mechanism one need to configure admission controller. But good luck configuring it with kubeadm. It requires some cryptic YAMLs, YAML patches and whatnot. You can't just edit some configmap in the cluster and be done with it. https://kubernetes.io/docs/tasks/configure-pod-container/enforce-standards-admission-controller/ https://kubernetes.io/docs/tasks/configure-pod-container/enf... this is what I'm talking about. It's more issue with kubeadm, I guess, which really makes things harder than they should be. I can't just run kubeadm join control-plane, I need to craft configuration, add this AdmissionConfiguration to it, remember to update it on all control nodes if I decide to change it later.
ransom1538 3y ago
"Default access even with RBAC is too wide-open." Not following this. Isn't a k8s cluster behind a network firewall? Only exposing 443? To me RBAC would be useful in a specific case where you want k8 service accounts (no one uses these) to have different levels of security. That sounds awful.
- alephnerd 3y ago
  Jeff in Engineering might have deployed an app spanning vCenter as well as EKS, and to continue to allow the app to function wrote subpar ingress/egress rules.
- mulmen 3y ago
  443 is just a port. If we assume that implies HTTPS then that’s still only TLS. If I exploit the service (“securely”!) I am inside your network. I can then pivot to your management layer. Game over.
- vbezhenar 3y ago
  > Isn't a k8s cluster behind a network firewall? Of course not, unless you configure it this way. > RBAC would be useful in a specific case where you want k8 service accounts (no one uses these) to have different levels of security. What? How comes no one uses service accounts? They're used extensively for many things.
- deleted 3y ago
  [deleted]
cratermoon 3y ago
I've seen non-tech corporations adopting Kubernetes without realizing that it's just one part of Google's infrastructure, not the entire operations side. There's an entirely separate chunk of infrastructure that provides the kind of security you mention. K8s is a "do one thing and do it well" that does workload orchestration. Google's BeyondCorp system enforces a zero-trust policy. It's not left to each part of the infrastructure to independently implement.
- alfalfasprout 3y ago
  Well, Google uses Borg internally which isn't exactly Kubernetes. But yes, the idea is that for it to be useful as infra it has to come without a lot of magic automatically attached.
raesene9 3y ago
Whilst Kubernetes defaults are indeed not great, they aren't that relevant unless you're building your clusters from scratch instead of using a k8s distribution, and I don't think I've seen anyone do that in a long time. RBAC is enabled by default on just about every distribution I've seen (with the notable exception of microk8s) Lack of default admission control to prevent someone with pod creation access breakout to the host is indeed pretty common Unencrypted APIs hasn't really been a problem in years, the insecure API port and read-only Kubelet were a thing but they've been disabled for quite a few versions now. I'd definitely agree that early k8s versions had very bad security, but things have been improving gradually over the versions :)
- 0xbadcafebee 3y ago
  Well first of all, there are still thousands of clusters running with the old insecure defaults, so that's bad. Both GKE and EKS currently have insecure defaults. There's a half dozen ways to take over the entire cluster with the default settings. The unencrypted API port is only this year being deprecated on GKE, I don't know about EKS. Since you can only grant permissions, not deny, most people end up granting too many permissions, defeating the purpose of least privilege. Default permissions for users on GKE/EKS are way too wide open. You can do things you just shouldn't need to do as a regular dev. You shouldn't be able to just deploy a helm chart to a random cluster and then all of a sudden there's an unauthenticated public service on the internet. But that's what every single distribution of K8s gives users by default. Regardless of whether you think that's fine or not, this is how companies get exposed on a regular basis, so it is a real-world persistent security problem that is perpetuated by the defaults.
  - raesene9 3y ago
    People not upgrading is indeed a problem but not one the distributions or k8s can particularly address, unless they force upgrade people which with deprecation/removal of APIs doesn't tend to be popular :) The support lifecycle is pretty short, compared to general enterprise software, so unless orgs are happy running unsupported versions of k8s they're on 1.23 for EKS or 1.24 for GKE by this point. I wouldn't have said that EKS, in particular, had very bad defaults, IME it's the best of the managed distributions for security. Any references for 1/2 dozen ways of taking over a cluster from defaults settings? I'd genuinely be interested as I collect information about Kubernetes security, so I'm always on the look-out for more references. for unencrypted API are you thinking of the Insecure API port or read-only Kubelet? For the insecure API port I don't think that's been a thing for a really long time on managed distros. The only time I remember seeing it was on ACS (the predecessor to AKS) where it was exposed to the container network. read-only Kubelet was more of a thing as that was on worker nodes, but it was less risky.

geek_at 3y ago

I had a similar experience when I tried to use kubernetes in my home lab. Crashed it twice and since the technology and tools evolve so fast almost all tutorials I found were already so out of date that the commands didn't work and some tools were already deprecated. I moved to docker swarm and love it. It's so much easier, straight forward, automatic ingress network and failover were all working out of the box. I'll stay with swarm for now

GauntletWizard 3y ago
Do you go to the hot new restaurant in town on their grand opening, and then leave a review that says "the food was great, but it was too crowded and the service was slow", and never go there again? That sounds like what you did there.
- globuous 3y ago
  Is that not what you’d do? That sounds like an unusually useful review tbh.
  - starttoaster 3y ago
    The point of their comment, I think, is that a restaurant's grand opening is not always indicative of future experiences. The cooks/chefs, and front of house staff are all fairly new to the environment, menus might change as they find out that certain menu items are throwing off their timing or they don't have a great way to procure certain ingredients, and it will take time to all settle into a routine that works. I think they were saying that is akin to the state of kubernetes, where everything is too new to really judge it today. Let me make clear my bias before I get to my opinion: I'm a big fan of what kubernetes has accomplished with managing the deployment of applications. But I think comparing current day kubernetes to a restaurant's grand opening is naïve. Kubernetes is amazing at what it does, and for how highly flexible it is. But it is highly flexible because of how advanced it has become, which in turn requires more knowledgeable administrators. You can take off with Docker Swarm because... it's very simple. It doesn't aim to be flexible, it just aims to be the docker-compose of distributed systems. Which is fine if you don't need the flexibility that kubernetes offers anyway. And while I do appreciate kubernetes for all of its strengths, going back towards the point of this OP, I do believe it's a bit of a security nightmare. For example, the kubernetes operator pattern often just requires this application to run in your cluster with a ClusterRole granting it read and potentially write levels of access to ConfigMaps and Secrets around your cluster. Often without much in the ways of a method of telling the kubernetes API server what namespaces it is allowed to view those objects in. Worth mentioning though that secrets management is something that Docker Swarm also handles fairly poorly, and honestly isn't much of a security regression from the normal "deploy a VM that has all your decrypted secrets sitting on the system" anyway.

ozarker 3y ago

A couple years ago I came into a company that had recently switched almost everything to Kubernetes. The company expected the devs to learn how to own and operate the cluster. Most of the devs there couldn't care less about anything except how to deploy their code to it (probably rightfully so). But I was shocked to find the k8s api was exposed to the internet and unauthenticated. I brought the problem up with the higher-ups and they were pretty lethargic about it. Moral of the story I guess is if you're gonna choose to operate infrastructure like Kubernetes make sure you have experienced people trained to do so and don't expect your devs to wear another hat.

andhapp 3y ago
DevOps is the new way, I think. Developers create the code and also know how to run and maintain infra :-)
- sharts 3y ago
  I don’t think that’s true anymore. Most companies that seriously deploy k8s tend to have dedicated teams for it. There’s no way an app developer is going to be able to do their work while also knowing the ins and outs of cluster mgmt.
  - bonestamp2 3y ago
    > Most companies that seriously deploy k8s tend to have dedicated teams for it. Ya, and everywhere I've worked, that team is called "devops". The application developers are a different team. Maybe that's not the norm, but I thought it was.
    - funcDropShadow 3y ago
      > Ya, and everywhere I've worked, that team is called "devops". But then that team is only doing operations and not development. Unless they are committers to k8s.
      - 3y ago

debarshri 3y ago

This is super interesting. If you actually look at a Loadbalancer logs without a WAF in place, you will see huge amount of bot activity. What is not clear in the blogs is how did they actually accessed the API server without a valid service token. Secondly, they index alot on IPs, that implied these are k8s cluster that are not managed (EKS, GKE etc). Is my assumption correct?

abatilo 3y ago
The GKE "default" is to bind a public IP to your control plane. So these could still be managed
raesene9 3y ago
So it is possible to expose Kubernetes without auth (https://raesene.github.io/blog/2023/03/18/lets-talk-about-anonymous-access-to-Kubernetes/ https://raesene.github.io/blog/2023/03/18/lets-talk-about-an... has some details) but it's not a default You can make that mis-configuration with managed distros, which are generally exposed to the Internet by default.
- debarshri 3y ago
  Ah that's what it is. Opening up API server that's allows anonymous auth is a very bad idea.

koito17 3y ago

I am actually a bit surprised there are any clusters out there exposing their service ports. Do people not run firewalls in their nodes? Sure, it may cause some headaches due to a range of random IPs and random ports that Kubernetes expects to be able to bind, but I assume any sysadmin will know how to deal with this. I know k3s has the awesome advice of disabling firewalld[1], but presumably the clusters mentioned in this article are largely using kubeadm or some other tool rather than k3s. With that said, I agree that secrets management on k8s is pretty bad (unless you use an external service like Hashicorp Vault, a frequent pattern that I run into with k8s). Even with RBAC and secrets in a namespace being inaccessible from other namespaces, the moment someone has permissions to read secrets from a namespace, its essentially game over for you, as noted in this paragraph. > Another potential attack vector within a Kubernetes cluster is accessing internal services. ... In cases where authentication is required, the credentials can typically be found within the Kubernetes secrets space. [1] https://docs.k3s.io/advanced#red-hat-enterprise-linux--centos https://docs.k3s.io/advanced#red-hat-enterprise-linux--cento...

merb 3y ago
The firewalld is for the trusted zone… and for the apiserver 6443 https port k3s comes with rbac enabled by default so if you want to access it via kubelet you need to open it?! trusted zone is used by overlay networks or node internal stuff (or if you manually put your interface into it)
crabbone 3y ago
Why would you be surprised? In most cases the only thing users need from a Kubernetes cluster is to access it from the outside... like... why wouldn't they expose it to the outside world? -- this is the most useful thing you can do with it. And the range of things you might want to expose... oh, god only knows. Think about such trivial stuff as FTP or iSCSI -- kinda useful, right? -- but Kubernetes rules for port exposure are idiotic, because you cannot expose ranges. And quite a few of popular Internet protocols assume they can allocate whatever port they want and start talking on it. And then you start fighting Kubernetes... and you'll win, but it will be some custom security solution... with the obvious problem of it being a custom security solution. Now, add to this that Kubernetes will typically mess with your iptables and maybe also arptables, and if your firewall is based on eg. iptables, then you might be out of luck using it with Kubernetes, and again, will end up rolling something "special". Obviously, you might also want, say, DNS to work inside your cluster, but that's a separate can of worms.

barakm 3y ago

This is also why managed Kubernetes is a useful thing (EKS, GKE, et al)... but if you still want to do it yourself, maybe look into some Kubernetes distros (like Typhoon (https://typhoon.psdn.io https://typhoon.psdn.io) which I run on my clusters)

lakomen 3y ago
But managed is expensive. You have to pay at least double vs unmanaged (in the dedicated world, ofc there is no "unmanaged" k8s offer from those cloud providers, just an analogy). You are then bound to that cloud and its traffic prices and all those you mentioned are filthy expensive when it comes to traffic.
- alephnerd 3y ago
  If you are dealing with the level of scale that allows you to fully take advantage of K8s, the cost of managed offerings is less than that of deploying and managing Openshift on-prem. Also, the beancounters don't like on-prem as much anymore due to the upfront cost outlay. I've been working with a number of major enterprise accounts and they've all been adopting managed K8s offerings for this very reason.
  - deleted 3y ago
    [deleted]

erulabs 3y ago

I mean, it’s not the best argument to say a system is “one action away from disaster”. If you pull the escape lever in an F16 for no reason you’re also pretty screwed. If you intentionally fly into the ground no one blames General Dynamics for making the plane too complex. It turns out complex tools for complex tasks require some skilled operators. Yes kubernetes is overkill for your home lab, no it’s not overkill for enterprise-grade AWS orchestration with hundreds of apps and millions of users. News at 11.

ramraj07 3y ago
I want to host a website not bomb the Middle East. Most enterprise apps are still websites that need no more than elastic beanstalk. If your analogy is to compare k8s to f16 then maybe you are not doing your argument any service.
- alephnerd 3y ago
  If your dealing with dynamic scaling needs K8s are quite nice. There's a reason why most Telcos have deployed 5G functionality using K8s. If you are in a situation where you want to follow a decoupled/microservices development pattern (ie. split component ownership across dev teams) as well as manage scalability during high load periods, K8s is hard to beat. K8s is a useful tool, but can be overkill in a number of situations. Just like a hammer, you can't use it to solve for every usecase. P.S. F-35s use K8s clusters (or might be K3s) to deploy their EWS.
- erulabs 3y ago
  Well both hosting websites and bombing the middle east make use of Kubernetes. [1] 1. https://www.theregister.com/2020/06/03/kubernetes_b_21_bomber/ https://www.theregister.com/2020/06/03/kubernetes_b_21_bombe... I've spent my entire career being on-call for web servers and Elastic Beanstalk is not enough. Deployment failures, deployment speed, deployment safety, upgrades, local testing... There is a reason AWS themselves is pushing EKS and not Elastic Beanstalk, and it's not because Kubernetes is trendy, it's because it's incredibly useful.
  - 3y ago

alephnerd 3y ago

Let the Blackhat blogspam begin!

crabbone 3y ago

Here's my Kubernetes security horror story. You probably know about resolv.conf, but you might not know about "search domains". A feature that should've never existed, but was added because "why not?". If you don't know what it does, in first approximation, it creates a domain alias. This is sometimes used to save some typing when using DNS for service discovery in larger clusters in a very hierarchical way (when names start to look like a.b.c.d.e.f.g...com, it's tempting to alias them to something shorter). So, long story short. Ubuntu LTS had a bug (actually a whole bunch of them) in the same script that was responsible for generation of this file (it's a bad idea to have this file to begin with... but hey, who cares, right?). So, one aspect of the bug was that a variable name "DOMAINS" was used instead of its value. And that became a search address in resolv.conf. Now, this search works by appending the search path to the domain name, if DNS server tells you it cannot find it. I had no idea DOMAINS was possible to register as a valid domain name. I mean, it should've been a TLD to be discoverable globally, right? -- Well, it's still a mystery to me, but, for some reason domains of the form a.b.c.DOMAINS are resolved to some AWS EC2 instance... And, the punch line: if you screw up CoreDNS configuration somehow, or something else goes wrong with service discovery inside your Kubernetes cluster on unpatched Ubuntu -- there's a good chance your traffic will go to that EC2 instance. Who's on the other side -- I don't know. I didn't send anything important that way and discovered it because something like Spark operator didn't work... but it could've been much worse. Oh, btw, I believe resolv.conf is used by different CNIs. Maybe Calico and Flanel. I cannot remember atm. why it was used... or maybe it's some kubelet setting. Anyways. Larger point is... large Kubernetes clusters are virtually impossible to systematically audit. Too many things can do too many things. Too much trickery goes on in all sorts of system components that deal with networking. It's also impossible to track who does what and who owns what because there's no user identity. --- I believe that, again, industry adopted an absolute horror show of a program / environment not because it was a great tool, but because "organically" developers started to add "features", creating something that's hard to resist, because, superficially, it creates a lot of value with minimal effort -- but it also lays a trap that is not immediately obvious. And those who find themselves trapped, instead of running away, star adding more "features" to fight the trap. But, in this foolhardy attempt, they only make the trap stronger.

uberduper 3y ago

superq 3y ago

Servers are so much faster than they were ten years ago that a typical, fully-loaded enterprise app can almost always run on a single mid-sized 10-core box with 256 GB of RAM.. from ten years ago. It's absolutely ludicrous how much power is available now on a single box if you really want to scale up instead of out. (Databases, of course, always like to eat everything because of poorly-optimized statements, but they've never really been good fits for K8s or containers anyway.) Some might say that this isn't webscale, but Stack Overflow runs one of the most popular tech websites in the world.. on only nine web servers: https://www.datacenterdynamics.com/en/news/stack-overflow-still-on-prem-runs-qa-platform-off-just-nine-servers/ https://www.datacenterdynamics.com/en/news/stack-overflow-st... and LiChess is similar: https://lichess.org/costs https://lichess.org/costs

metadat 3y ago
> a typical, fully-loaded enterprise app can almost always run on a single mid-sized 10-core box with 256 GB of RAM. If you don't care about high-availability, then yeah. One thing I've learned in my career is that delivering reliable enterprise-grade service is anything but trivial. The level of complexity of any modern stance is high enough that there will inevitably be sharp-edged corner cases which aren't simple or cheap to deal with.
- spacebanana7 3y ago
  Don't most outages happen because of unforeseen consequences of changes to the application/environment rather than machine failure? Is reducing the exposure to machine failure really worth exposing yourself to the vast complexity of distributed systems?
  - PrayagBhakar 3y ago
    I think the point of such tools is more than just reducing exposure to machine failure. Reducing exposure to machine failure also means that you can run your hardware closer to end of life without having to worry about data outages. The article about stack overflow mentions this little tidbit. > That application handles around 6,000 requests per second, and around 2 billion requests per month. To be able to handle loads of 1M requests per second, your going to need more than one server dispensing results. Especially since only so much data can be stored in memory without running into performance issues. That said, these kinds of tools are an unique use case. Not all programs need them.

gerbilly 3y ago

[flagged]

zaps 3y ago

Super-naive take: why are people such jerks?

raesene9 3y ago

I've done quite a lot of looking at exposed k8s clusters, and one thing this piece doesn't really cover is what percentage of the 350 clusters they identified were in fact honeypots. You can find k8s clusters without auth using Censys as they specifically try to access things like pod lists, and from spinning through lists of those they tend to be obvious honeypots (e.g. lots of ports that wouldn't usually go together being exposed), although I have seen the odd cluster that looks a bit more realistic, unlike Aqua I didn't try accessing it directly as that could be dodgy legally :D If anyone wants to look at the list, Censys handily has a search query for "kubernetes clusters where the pod list is exposed without authentication" https://search.censys.io/search?resource=hosts&q=services.kubernetes.pod_names%3D%22*%22 https://search.censys.io/search?resource=hosts&q=services.ku...

roydivision 3y ago

This article seems to be a very long winded way of saying "Beware of what you are exposing to the internet", using Kubernetes as an example.

jhoelzel 3y ago

While this post has some merit, all of these problems can easily be solved and there are dozens of guides on hit. The entire "CIS Hardening guide" will give you step by step tutorials on how to solve all of this, you just need to bother looking it up. Look, you need a trained sysadmin anyway, no matter where you deploy. See for instance all the ubuntu machines out there that never have seen the admin update it. Or all the terraform scripts that simply get copy pasted by orgs. While I do agree, this is a problem, there are security conscious distributions if kubernetes out there like RKE2 or "rancher government" to make things much more secure. This is actually my job. If you need someone how to secure your cluster and make sure stuff like this is very unlikely to happen, you can find the info in my profile. But what it comes down to is that if your DEVs have direct cluster access and you have no monitoring and alerting setup, you are kind of operating on the same level as copying and pasting stackoverflow answers and pushing them directly to prod without code review.

user6723 3y ago

Linux exposed, one shell command away from disaster rm -rf / Looks like Linux is no good, back to the drawing board, time to redo the while cloud everybody.