4 ms·

The problem is that the FAANGS run single-process servers at a scale of thousands of VMs, so in their mind 1 VM == 1 App role. They have no concept of sharing a

The problem is that the FAANGS run single-process servers at a scale of thousands of VMs, so in their mind 1 VM == 1 App role. They have no concept of sharing a single VM for dozens of apps.

sofixa 3y ago
> They have no concept of sharing a single VM for dozens of apps They do, it's called a container or a function, which have their own IAM/auth/role.
- zokier 3y ago
  In the context of AWS, both containers (Fargate) and functions (Lambda) run on Firecracker, meaning separate VM for each container/function. No shared VMs there.
  - sofixa 3y ago
    Separate micro-VM which is just there as a security boundary and nothing else, so I'm not sure it's comparable to an EC2.
oblio 3y ago
Well, they're kind of right. Physical servers I get, you can't really do 1:1 without wasting a ton of resources. But VMs can be resized quickly. The problem is that many companies have very rigid IT departments so even though in theory it can be done somewhat easily, in some places it can be a months-long process to actually get it done.
- jiggawatts 3y ago
  You can’t resize a VM below 1 vCPU. Actually in most clouds it’s 2 vCPUs minimum in practice. I have a government customer running 150 web apps in about 20 vCPUs. If you naively multiply 150 apps by the minimum scale for HA, non-prod, and the 2 vCPU minimum for cloud hosts you end up with 450-600 vCPUs! That’s expensive even for governments.
  - vegardx 3y ago
    I don't understand why you'd want to use VMs for something like this when containers exist and you have tools like ECS and Kubernetes. You can serve the same amount of web applications on the same amount of vCPUs and memory footprint, give or take a little bit. As a bonus, if you used AWS ECS, you also get to use things like IMDS (per container/task), Fargate, high availability and autoscaling tools.
    - jiggawatts 3y ago
      Windows. While it's technically possible to run something like an old ASP.NET Web Forms app in a container, it'll be painful.
      - DougBTX 3y ago
        > Windows. You should have put that in your first comment, I've removed my downvote!
      - bob1029 3y ago
        We are really enjoying using Azure Functions to serve web traffic these days. Probably not in exactly the way Microsoft would prefer us to, but it certainly works for us. All our business really requires is simple web forms, some session state and basic get/post request handling. Our users won't cry if they see a page transition. Az functions might not be the most optimal in terms of per request cost, but for the volume of traffic we are pushing it doesn't scale into the equation. Operationally, they're basically magic. TLS, auth, etc. all handled. GitHub deploys with zero drama. Whatever margin we sacrifice at the request billing is more than compensated by how well our engineers are sleeping at night.
    - scarface_74 3y ago
      A. EKS and Docker brings its own complexity B. Enterprises run a ton of commercial off the shelf software hosted on VMs C. Ever heard of Windows? It’s still big in the enterprise
      - 3y ago
baz00 3y ago
That's what EKS is for...
- scarface_74 3y ago
  Yes and that’s a simple solution…
lozenge 3y ago
IMDS is just one option, you can still pass IAM roles to applications in other ways.
threeseed 3y ago
This comment is so odd it's actually doing my head in. Google invented Kubernetes, Apple runs Kubernetes and previously was heavily into Mesos, likewise for Netflix. And I've seen a lot of Engineering tech blogs and worked on AWS for a decade and never have I ever seen anyone spinning up thousands of micro-VMs in that way. Not only are there all sorts of account limits you would immediately hit and have to negotiate but the networking would be problematic to say the least. Meanwhile you can spin up ECS, EKS, Fargate which allow you to run thousands of containerised apps without any headaches at all. Plus they support you having unique IAM roles for each container.
- belter 3y ago
  Kubernetes? You mean the container orchestration system where they forgot to add Multi-tenancy? And no, Namespaces are not Multi-tenancy... https://www.vcluster.com/ https://www.vcluster.com/
  - figmert 3y ago
    What kind of multi-tenancy are you trying to solve that Kubernetes namespaces cannot?
    - deleted 3y ago
      [deleted]
    - chippiewill 3y ago
      Kubernetes has a lot of limitations from a multi tenancy perspective. It's functional, but I think it's not as polished as the rest of Kubernetes which is why Kubernetes has a multi tenancy SIG that spawned the hierarchical namespace controller (https://github.com/kubernetes-sigs/hierarchical-namespaces https://github.com/kubernetes-sigs/hierarchical-namespaces) and virtual clusters (https://github.com/kubernetes-sigs/cluster-api-provider-nested/tree/main/virtualcluster https://github.com/kubernetes-sigs/cluster-api-provider-nest...)
    - belter 3y ago
      The one that the community recognizes never been there in the first place... "Multi-tenancy in Kubernetes: implementation and optimization" - https://www.cncf.io/blog/2022/11/09/multi-tenancy-in-kubernetes-implementation-and-optimization/ https://www.cncf.io/blog/2022/11/09/multi-tenancy-in-kuberne... "...As we can see from the above discussion, the sharing of Kubernetes clusters is not an easy task; multi-tenancy is not a built-in feature of Kubernetes and can only be implemented with other projects’ support in tenant isolation on both control and data planes. This has resulted in considerable learning and adaptation costs for the entire solution. As a result, we are seeing an increasing number of users adopt multi-cluster solutions instead..."

The problem is that the FAANGS run single-process servers at a scale of thousands of VMs, so in their mind 1 VM == 1 App role. They have no concept of sharing a single VM for dozens of apps.

sofixa 3y ago

> They have no concept of sharing a single VM for dozens of apps They do, it's called a container or a function, which have their own IAM/auth/role.

zokier 3y ago
In the context of AWS, both containers (Fargate) and functions (Lambda) run on Firecracker, meaning separate VM for each container/function. No shared VMs there.
- sofixa 3y ago
  Separate micro-VM which is just there as a security boundary and nothing else, so I'm not sure it's comparable to an EC2.

oblio 3y ago

Well, they're kind of right. Physical servers I get, you can't really do 1:1 without wasting a ton of resources. But VMs can be resized quickly. The problem is that many companies have very rigid IT departments so even though in theory it can be done somewhat easily, in some places it can be a months-long process to actually get it done.

jiggawatts 3y ago
You can’t resize a VM below 1 vCPU. Actually in most clouds it’s 2 vCPUs minimum in practice. I have a government customer running 150 web apps in about 20 vCPUs. If you naively multiply 150 apps by the minimum scale for HA, non-prod, and the 2 vCPU minimum for cloud hosts you end up with 450-600 vCPUs! That’s expensive even for governments.
- vegardx 3y ago
  I don't understand why you'd want to use VMs for something like this when containers exist and you have tools like ECS and Kubernetes. You can serve the same amount of web applications on the same amount of vCPUs and memory footprint, give or take a little bit. As a bonus, if you used AWS ECS, you also get to use things like IMDS (per container/task), Fargate, high availability and autoscaling tools.
  - jiggawatts 3y ago
    Windows. While it's technically possible to run something like an old ASP.NET Web Forms app in a container, it'll be painful.
    - DougBTX 3y ago
      > Windows. You should have put that in your first comment, I've removed my downvote!
    - bob1029 3y ago
      We are really enjoying using Azure Functions to serve web traffic these days. Probably not in exactly the way Microsoft would prefer us to, but it certainly works for us. All our business really requires is simple web forms, some session state and basic get/post request handling. Our users won't cry if they see a page transition. Az functions might not be the most optimal in terms of per request cost, but for the volume of traffic we are pushing it doesn't scale into the equation. Operationally, they're basically magic. TLS, auth, etc. all handled. GitHub deploys with zero drama. Whatever margin we sacrifice at the request billing is more than compensated by how well our engineers are sleeping at night.
  - scarface_74 3y ago
    A. EKS and Docker brings its own complexity B. Enterprises run a ton of commercial off the shelf software hosted on VMs C. Ever heard of Windows? It’s still big in the enterprise
    - 3y ago

baz00 3y ago

That's what EKS is for...

scarface_74 3y ago
Yes and that’s a simple solution…

lozenge 3y ago

IMDS is just one option, you can still pass IAM roles to applications in other ways.

threeseed 3y ago

This comment is so odd it's actually doing my head in. Google invented Kubernetes, Apple runs Kubernetes and previously was heavily into Mesos, likewise for Netflix. And I've seen a lot of Engineering tech blogs and worked on AWS for a decade and never have I ever seen anyone spinning up thousands of micro-VMs in that way. Not only are there all sorts of account limits you would immediately hit and have to negotiate but the networking would be problematic to say the least. Meanwhile you can spin up ECS, EKS, Fargate which allow you to run thousands of containerised apps without any headaches at all. Plus they support you having unique IAM roles for each container.

belter 3y ago
Kubernetes? You mean the container orchestration system where they forgot to add Multi-tenancy? And no, Namespaces are not Multi-tenancy... https://www.vcluster.com/ https://www.vcluster.com/
- figmert 3y ago
  What kind of multi-tenancy are you trying to solve that Kubernetes namespaces cannot?
  - deleted 3y ago
    [deleted]
  - chippiewill 3y ago
    Kubernetes has a lot of limitations from a multi tenancy perspective. It's functional, but I think it's not as polished as the rest of Kubernetes which is why Kubernetes has a multi tenancy SIG that spawned the hierarchical namespace controller (https://github.com/kubernetes-sigs/hierarchical-namespaces https://github.com/kubernetes-sigs/hierarchical-namespaces) and virtual clusters (https://github.com/kubernetes-sigs/cluster-api-provider-nested/tree/main/virtualcluster https://github.com/kubernetes-sigs/cluster-api-provider-nest...)
  - belter 3y ago
    The one that the community recognizes never been there in the first place... "Multi-tenancy in Kubernetes: implementation and optimization" - https://www.cncf.io/blog/2022/11/09/multi-tenancy-in-kubernetes-implementation-and-optimization/ https://www.cncf.io/blog/2022/11/09/multi-tenancy-in-kuberne... "...As we can see from the above discussion, the sharing of Kubernetes clusters is not an easy task; multi-tenancy is not a built-in feature of Kubernetes and can only be implemented with other projects’ support in tenant isolation on both control and data planes. This has resulted in considerable learning and adaptation costs for the entire solution. As a result, we are seeing an increasing number of users adopt multi-cluster solutions instead..."