3 ms·

Cilium lives below userspace which makes it perform better than istio. This article has more information about the differences from the cilium developer's point

Cilium lives below userspace which makes it perform better than istio. This article has more information about the differences from the cilium developer's point of view. https://cilium.io/blog/istio/ https://cilium.io/blog/istio/

hueving 8y ago
>Cilium lives below userspace which makes it perform better than istio. There are a lot of fast userspace networking projects that bypass the kernel precisely to be faster. Which approach is better is up for debate but the kernel is definitely not faster in all cases.
- woah 8y ago
  How can you bypass the kernel?
  - aseipp 8y ago
    One major difference between the kernel and any other piece of standard software is access control: the kernel has privileged access to most hardware peripherals. The software for the driver is nothing particularly special; it is simply run in a context in which it has control of the hardware. More concretely, most hardware devices are relatively easy to interface with: for example, you may simply set up a region of DMA memory, poke the hardware device with the address of this memory, and write into it, then read results back out. A NIC is a good example of such a model. This can be done with any block of memory, except normally the kernel is the only thing that can talk to the NIC (to tell it where to write to/read from). So the main thing you need to do is pass control of the hardware to a userspace process. For the NIC/DMA example, the easiest way is to just allocate some memory, make sure it's non-swappable, and then get its physical address. You then just need a small driver to connect userspace with the hardware -- it must give you a way to tell the hardware where to read/write. Maybe it exposes a sysfs-based file with normal unix permissions (a common method). Writing an address into this file is equivalent to telling the hardware to "read here, and write there". Now you can write to the memory you allocated (in userspace) to control the NIC. At this point, the kernel is more-or-less out of the loop completely. Of course, this is the easy part, since now you must write the rest of the hardware driver. :)
  - kjeetgill 8y ago
    The basic idea is that you get the NIC to write packets directly into your userspace applications' memory. There are plenty of projects out there to help get the kernel out of the critical path of a networking application. DPDK and PF_RING are the two I hear about most but here is a blog from cloudflare outlining a number of others: https://blog.cloudflare.com/kernel-bypass/ https://blog.cloudflare.com/kernel-bypass/
  - dward 8y ago
    You can map ingress/egress channels of a network device directly into a processes memory inuserspace. These are just memory pages in what's known as the DMA region that the device can write to without interacting with the CPU.
  - derphj 8y ago
    There is a proposal that has recently been merged into the kernel to implement AF_XDP (LWN intro: https://lwn.net/Articles/750845/ https://lwn.net/Articles/750845/), which is a long-term kernel built-in solution in combination with XDP/BPF to achieve speed of user space bypasses for use cases that require it. First, API pieces were recently merged and more work soon to be appear such as the zero-copy bits as another milestone.
perbu 8y ago
Cilium performs better because the instruction cache is kept well as opposed to netfilter, which needs to gather data from all over in order to do it's job which leads to low cache utilization. Kernel code doesn't run faster than userspace code.
- chatmasta 8y ago
  Isn’t kernel code considered “faster” when used in networking components, because of reduced context switching? The idea being that packets need to travel through kernel codepaths anyway, so any user space filtering will slow down the packets simply due to context switching.

hueving 8y ago

>Cilium lives below userspace which makes it perform better than istio. There are a lot of fast userspace networking projects that bypass the kernel precisely to be faster. Which approach is better is up for debate but the kernel is definitely not faster in all cases.

woah 8y ago
How can you bypass the kernel?
- aseipp 8y ago
  One major difference between the kernel and any other piece of standard software is access control: the kernel has privileged access to most hardware peripherals. The software for the driver is nothing particularly special; it is simply run in a context in which it has control of the hardware. More concretely, most hardware devices are relatively easy to interface with: for example, you may simply set up a region of DMA memory, poke the hardware device with the address of this memory, and write into it, then read results back out. A NIC is a good example of such a model. This can be done with any block of memory, except normally the kernel is the only thing that can talk to the NIC (to tell it where to write to/read from). So the main thing you need to do is pass control of the hardware to a userspace process. For the NIC/DMA example, the easiest way is to just allocate some memory, make sure it's non-swappable, and then get its physical address. You then just need a small driver to connect userspace with the hardware -- it must give you a way to tell the hardware where to read/write. Maybe it exposes a sysfs-based file with normal unix permissions (a common method). Writing an address into this file is equivalent to telling the hardware to "read here, and write there". Now you can write to the memory you allocated (in userspace) to control the NIC. At this point, the kernel is more-or-less out of the loop completely. Of course, this is the easy part, since now you must write the rest of the hardware driver. :)
- kjeetgill 8y ago
  The basic idea is that you get the NIC to write packets directly into your userspace applications' memory. There are plenty of projects out there to help get the kernel out of the critical path of a networking application. DPDK and PF_RING are the two I hear about most but here is a blog from cloudflare outlining a number of others: https://blog.cloudflare.com/kernel-bypass/ https://blog.cloudflare.com/kernel-bypass/
- dward 8y ago
  You can map ingress/egress channels of a network device directly into a processes memory inuserspace. These are just memory pages in what's known as the DMA region that the device can write to without interacting with the CPU.
- derphj 8y ago
  There is a proposal that has recently been merged into the kernel to implement AF_XDP (LWN intro: https://lwn.net/Articles/750845/ https://lwn.net/Articles/750845/), which is a long-term kernel built-in solution in combination with XDP/BPF to achieve speed of user space bypasses for use cases that require it. First, API pieces were recently merged and more work soon to be appear such as the zero-copy bits as another milestone.

perbu 8y ago

Cilium performs better because the instruction cache is kept well as opposed to netfilter, which needs to gather data from all over in order to do it's job which leads to low cache utilization. Kernel code doesn't run faster than userspace code.

chatmasta 8y ago
Isn’t kernel code considered “faster” when used in networking components, because of reduced context switching? The idea being that packets need to travel through kernel codepaths anyway, so any user space filtering will slow down the packets simply due to context switching.