4 ms·

I think the article could do with a bit more information about _why_ you'd want to use macvlan vs a bridge. macvlan lets you offload switching to the network--

by yrro 5y ago

I think the article could do with a bit more information about _why_ you'd want to use macvlan vs a bridge. macvlan lets you offload switching to the network--so more of your CPU power can go towards running VMs. Since all traffic, even that between two VMs, goes via the external switch, that external switch can also perform auditing, accounting and policy enforcement. If you're a big enterprise with lots of expensive switches then you probably want to use them for these sorts of things, rather than push the job down to the hypervisors.

yokaze 5y ago
In my experience with said expensive hardware, it greatly depends on your use-case / scale. You have to pay attention to stay within the limits of the hardware the switches (TCAM, etc...). And correct me, if I am wrong, but most NICs are fairly constrained in their hardware support on the numbers of MACs they can filter for. And those two combined, you might end up with a lot of flooding, which needs to be handled by the CPU again. You also have to rely on the implementation of your vendor. And at least with one major vendor, who shall remain nameless, it is a major effort just to get acknowledged that there is a bug.
- dfox 5y ago
  The point about number of MAC filters is fairly moot as when you do bridging in hardware you are not going to utilize any kind of hardware MAC filtering capability anyway. Also the issue of flooding tends to be solved by the physical switches in the network.
  - yokaze 5y ago
    So, we are in agreement in that point: if you are going that way, you are relying on the physical switches. I'm not sure about the problem being solved, depending on the scale. If you are with your use-case (number of rules, macs, five-tuples, ...) outside the capabilities of the hardware, then, in my experience, it breaks down quickly. (I.e. flooding, or becoming unresponsive to configuration commands and needed to be _power cycled_, ...). And those are switches costing easily five figures, and from two different well known vendors. If you have practical experience with a setup, which scales more gracefully, please share. But I'm sceptical that it exists, considering Google, Facebook, AWS all went with custom hardware. I'm wondering, if at a certain scale of virtual NICs/flows per port, you are not better off with a hybrid approach, where software is part of a decision hierarchy. E.g Openflow. No practical experience though.
filleokus 5y ago
One use-case I’ve seen for MAC VLANs is specialized audio transmission (AES67 maybe?) where data is sent over multicast. Never really fully understood how it worked, but when developers wanted those applications to be hosted in containers, there was a lot of talk about MAC VLANs :-)
- bogomipz 5y ago
  Interesting. Could you or someone else say what benefits does MAC Vlan offer over IP multicasting?
navaati 5y ago
So it's kinda like doing VFIO, but with less things handled by the NIC, but externally from the switch point of view it looks like the same (several MACs going out of a single port), right ?
caskstrength 5y ago
> macvlan lets you offload switching to the network--so more of your CPU power can go towards running VMs. Since all traffic, even that between two VMs, goes via the external switch, that external switch can also perform auditing, accounting and policy enforcement. If you're a big enterprise with lots of expensive switches then you probably want to use them for these sorts of things, rather than push the job down to the hypervisors. The current trend is to push that to DPU/IPU-style NICs which have their own cores. Previous generation thing was to reduce CPU load on hypervisor by offloading the packet processing so (smart)NIC.

by yrro 5y ago

yokaze 5y ago

In my experience with said expensive hardware, it greatly depends on your use-case / scale. You have to pay attention to stay within the limits of the hardware the switches (TCAM, etc...). And correct me, if I am wrong, but most NICs are fairly constrained in their hardware support on the numbers of MACs they can filter for. And those two combined, you might end up with a lot of flooding, which needs to be handled by the CPU again. You also have to rely on the implementation of your vendor. And at least with one major vendor, who shall remain nameless, it is a major effort just to get acknowledged that there is a bug.

dfox 5y ago
The point about number of MAC filters is fairly moot as when you do bridging in hardware you are not going to utilize any kind of hardware MAC filtering capability anyway. Also the issue of flooding tends to be solved by the physical switches in the network.
- yokaze 5y ago
  So, we are in agreement in that point: if you are going that way, you are relying on the physical switches. I'm not sure about the problem being solved, depending on the scale. If you are with your use-case (number of rules, macs, five-tuples, ...) outside the capabilities of the hardware, then, in my experience, it breaks down quickly. (I.e. flooding, or becoming unresponsive to configuration commands and needed to be _power cycled_, ...). And those are switches costing easily five figures, and from two different well known vendors. If you have practical experience with a setup, which scales more gracefully, please share. But I'm sceptical that it exists, considering Google, Facebook, AWS all went with custom hardware. I'm wondering, if at a certain scale of virtual NICs/flows per port, you are not better off with a hybrid approach, where software is part of a decision hierarchy. E.g Openflow. No practical experience though.

filleokus 5y ago

One use-case I’ve seen for MAC VLANs is specialized audio transmission (AES67 maybe?) where data is sent over multicast. Never really fully understood how it worked, but when developers wanted those applications to be hosted in containers, there was a lot of talk about MAC VLANs :-)

bogomipz 5y ago
Interesting. Could you or someone else say what benefits does MAC Vlan offer over IP multicasting?

navaati 5y ago

So it's kinda like doing VFIO, but with less things handled by the NIC, but externally from the switch point of view it looks like the same (several MACs going out of a single port), right ?

caskstrength 5y ago

> macvlan lets you offload switching to the network--so more of your CPU power can go towards running VMs. Since all traffic, even that between two VMs, goes via the external switch, that external switch can also perform auditing, accounting and policy enforcement. If you're a big enterprise with lots of expensive switches then you probably want to use them for these sorts of things, rather than push the job down to the hypervisors. The current trend is to push that to DPU/IPU-style NICs which have their own cores. Previous generation thing was to reduce CPU load on hypervisor by offloading the packet processing so (smart)NIC.