4 ms·

> That's dumb. Is it? It's certainly inefficient compared to dedicated hardware. But so is anything relying on a CPU - we could just use ASICs for everything

> That's dumb. Is it? It's certainly inefficient compared to dedicated hardware. But so is anything relying on a CPU - we could just use ASICs for everything. But then every logical change requires weeks/months/years of development and manufacturing. The goal of k8s, VMs, etc is flexibility. I can set up a 100-node k8s cluster with less-than-perfectly-efficient networking stack in mere minutes. Good luck matching that with dedicated hardware.

supahfly_remix 6y ago
For vCPUs on the same physical multicore CPU, packets are sent by shared memory constructs which is very fast. It's effectively sending a pointer to the memory. I have seen transfers like this up to 80 Gbps, much faster than the physical NIC.
- dahfizz 6y ago
  The transitions from host -> VM is not slow, it's the fact that the packet now has to touch the VM's CPU and go through it's whole routing and natting tables again. And then again for the container. The transition from physical host memory -> vnic is analogous to just sending the packet over a wire. That's not the slow part of modern networks.
dahfizz 6y ago
> But so is anything relying on a CPU - we could just use ASICs for everything. But then every logical change requires weeks/months/years of development and manufacturing. Right, except the asics already exist and you're actively choosing the less efficient, more expensive option. > The goal of k8s, VMs, etc is flexibility. I don't think this is necessarily bad, as long as you understand the tradeoff. You're choosing a fundamentally slower architecture to make management easier. It's a choice of prioritizing the developer experience over the user experience.
- oneplane 6y ago
  In this case it's not even user experience vs. developer experience as you can have both; it's just the cost for performance gets higher as the efficiency decreases. On the other hand: the cost for development goes down if you don't need to pay for extra steps taken by an extra person. If you take a 10-step process that is run by 5 people and reduce it to a 5-step process run by 3, you have 2 more people to do other stuff, or roles that you don't have to create/fill in the first place.
- CountSessine 6y ago
  Right, except the asics already exist and you're actively choosing the less efficient, more expensive option. It’s hard to imagine any dev task being more expensive than organizing and programming tables into a trident or tomahawk chip using the broadcom’s SDK.
gnufx 6y ago
Experimentally, you don't need luck to set up ~200 dedicated hardware nodes in minutes in an HPC cluster, for instance. (They do have to be connected up, with known or discoverable MAC addresses.)

supahfly_remix 6y ago

For vCPUs on the same physical multicore CPU, packets are sent by shared memory constructs which is very fast. It's effectively sending a pointer to the memory. I have seen transfers like this up to 80 Gbps, much faster than the physical NIC.

dahfizz 6y ago
The transitions from host -> VM is not slow, it's the fact that the packet now has to touch the VM's CPU and go through it's whole routing and natting tables again. And then again for the container. The transition from physical host memory -> vnic is analogous to just sending the packet over a wire. That's not the slow part of modern networks.

dahfizz 6y ago

> But so is anything relying on a CPU - we could just use ASICs for everything. But then every logical change requires weeks/months/years of development and manufacturing. Right, except the asics already exist and you're actively choosing the less efficient, more expensive option. > The goal of k8s, VMs, etc is flexibility. I don't think this is necessarily bad, as long as you understand the tradeoff. You're choosing a fundamentally slower architecture to make management easier. It's a choice of prioritizing the developer experience over the user experience.

oneplane 6y ago
In this case it's not even user experience vs. developer experience as you can have both; it's just the cost for performance gets higher as the efficiency decreases. On the other hand: the cost for development goes down if you don't need to pay for extra steps taken by an extra person. If you take a 10-step process that is run by 5 people and reduce it to a 5-step process run by 3, you have 2 more people to do other stuff, or roles that you don't have to create/fill in the first place.
CountSessine 6y ago
Right, except the asics already exist and you're actively choosing the less efficient, more expensive option. It’s hard to imagine any dev task being more expensive than organizing and programming tables into a trident or tomahawk chip using the broadcom’s SDK.

gnufx 6y ago

Experimentally, you don't need luck to set up ~200 dedicated hardware nodes in minutes in an HPC cluster, for instance. (They do have to be connected up, with known or discoverable MAC addresses.)