7 ms·

Loxilb: eBPF based cloud-native service load-balancer

94 points by InitEnabler 4y ago

rektide 4y ago
> Optimized SRv6 implementation in eBPF Does anyone else do Segmemt Routing in kube? This particularly caught my eye. I wonder how much other software & setup users need to take advantage of this Loxilb. It's such a different paradigm, specifying much more of the route packets take!
alas44 4y ago
In case someone else seeks performance benchmarks https://loxilb-io.github.io/loxilbdocs/perf/ https://loxilb-io.github.io/loxilbdocs/perf/
- vbernat 4y ago
  The details for the bare metal benchmarks are sparse. I would have expected an eBPF solution to outperform the "aging" IPVS by an important margin. Moreover, the peak performance of IPVS is far better (115 vs 57 reqs/s). It would be interesting to know if it is an outlier. A benchmark with an increasing workload over time would be more precise on how to compare both solutions.
yogaBear 4y ago
My imaginary kingdom for native language libs that let me interact with eBPF to load balance logic forks. Then I can put k8s and containers behind me.
hujun 4y ago
This looks interesting, specially the support of GTP/SCTP; although it seems quite new, first commit on github is last year, wonder if anyone has used this in production?
cyberge99 4y ago
The name sounds like a pharmaceutical. Cloudrizi (loxilb-io)
- userbinator 4y ago
  ...or a Pokémon. https://i.redd.it/krnyqtpgdy221.png https://i.redd.it/krnyqtpgdy221.png
LinuxBender 4y ago
Is this a blind load balancer similar to the iptables statistics module or are there health checks? Are they active or passive health checks? Asking because I saw a comparison to HAProxy.
- nijave 4y ago
  Seems like it https://loxilb-io.github.io/loxilbdocs/cmd/#endpoint https://loxilb-io.github.io/loxilbdocs/cmd/#endpoint
- aeyes 4y ago
  On Kubernetes which they mainly target the Kubernetes control plane would update the list of active service endpoints according to health checks. Standalone you could do it with the API and a small daemon but out of the box there is no support for health checks (yet).
jiggawatts 4y ago
It talks a lot about performance, but the actual cloud load balancers such as AWS ELB or Azure Load Balancer are implemented in the software-defined network (SDN) and are accelerated in hardware. For example in Azure, only the first two packets in a VM->LB->VM flow will traverse the LB. Subsequent packets are direct from VM-to-VM and are rewritten in the host NICs to merely appear to go via the LB address. This enables staggering throughputs that no “software in a VM” can possibly hope to match. Personally I wish people would stop with the unnecessary middle boxes. It’s 2023 and there are cloud VMs now that can put out 200 Gbps! Anything in path of a few dozen such VMs will melt into slag. This is especially important for Kubernetes and microservices in general that are already very chatty and have layers of reverse proxies five deep in surprisingly common configurations already.
- betaby 4y ago
  > only the first two packets in a VM->LB->VM flow will traverse the LB. Subsequent packets are direct from VM-to-VM and are rewritten in the host NICs to merely appear to go via the LB address Do you have more details how that's done?
  - fnordpiglet 4y ago
    Generally you can do this if the network is software defined. The boundaries between networks aren’t actually real, which means you can load balance by simply determining if the packets should be able to route then letting them route for the rest of the flow.
    - baq 4y ago
      So, basically magic, just as I expected. Got it.
  - jvans 4y ago
    +1 would be great for more detailed explanation or links out to reading material. I assume there are some limitations if you actually skip the LB? How do the host NICs rewrite the LB address, does this imply there is hardware support for this kind of bypass routing?
travbrack 4y ago
Any benchmarks?
- victorbjorklund 4y ago
  https://loxilb-io.github.io/loxilbdocs/perf/ https://loxilb-io.github.io/loxilbdocs/perf/
goodpoint 4y ago
The build starts with "Install GoLang > v1.17". For a eBPF based application? Not good.
- tecleandor 4y ago
  Is it because of being Go, the version, or what?
  - egberts1 4y ago
    As a security vulnerability researcher, eBPF is a problematic one to this day. I instruct everyone to disable eBPF at kernel compile time. Unfortunately, one cannot completely compile eBPF out of their kernel

94 points by InitEnabler 4y ago

> Optimized SRv6 implementation in eBPF Does anyone else do Segmemt Routing in kube? This particularly caught my eye. I wonder how much other software & setup users need to take advantage of this Loxilb. It's such a different paradigm, specifying much more of the route packets take!

alas44 4y ago

In case someone else seeks performance benchmarks https://loxilb-io.github.io/loxilbdocs/perf/ https://loxilb-io.github.io/loxilbdocs/perf/

vbernat 4y ago
The details for the bare metal benchmarks are sparse. I would have expected an eBPF solution to outperform the "aging" IPVS by an important margin. Moreover, the peak performance of IPVS is far better (115 vs 57 reqs/s). It would be interesting to know if it is an outlier. A benchmark with an increasing workload over time would be more precise on how to compare both solutions.

yogaBear 4y ago

My imaginary kingdom for native language libs that let me interact with eBPF to load balance logic forks. Then I can put k8s and containers behind me.

hujun 4y ago

This looks interesting, specially the support of GTP/SCTP; although it seems quite new, first commit on github is last year, wonder if anyone has used this in production?

cyberge99 4y ago

The name sounds like a pharmaceutical. Cloudrizi (loxilb-io)

userbinator 4y ago
...or a Pokémon. https://i.redd.it/krnyqtpgdy221.png https://i.redd.it/krnyqtpgdy221.png

LinuxBender 4y ago

Is this a blind load balancer similar to the iptables statistics module or are there health checks? Are they active or passive health checks? Asking because I saw a comparison to HAProxy.

nijave 4y ago
Seems like it https://loxilb-io.github.io/loxilbdocs/cmd/#endpoint https://loxilb-io.github.io/loxilbdocs/cmd/#endpoint
aeyes 4y ago
On Kubernetes which they mainly target the Kubernetes control plane would update the list of active service endpoints according to health checks. Standalone you could do it with the API and a small daemon but out of the box there is no support for health checks (yet).

jiggawatts 4y ago

It talks a lot about performance, but the actual cloud load balancers such as AWS ELB or Azure Load Balancer are implemented in the software-defined network (SDN) and are accelerated in hardware. For example in Azure, only the first two packets in a VM->LB->VM flow will traverse the LB. Subsequent packets are direct from VM-to-VM and are rewritten in the host NICs to merely appear to go via the LB address. This enables staggering throughputs that no “software in a VM” can possibly hope to match. Personally I wish people would stop with the unnecessary middle boxes. It’s 2023 and there are cloud VMs now that can put out 200 Gbps! Anything in path of a few dozen such VMs will melt into slag. This is especially important for Kubernetes and microservices in general that are already very chatty and have layers of reverse proxies five deep in surprisingly common configurations already.

betaby 4y ago
> only the first two packets in a VM->LB->VM flow will traverse the LB. Subsequent packets are direct from VM-to-VM and are rewritten in the host NICs to merely appear to go via the LB address Do you have more details how that's done?
- fnordpiglet 4y ago
  Generally you can do this if the network is software defined. The boundaries between networks aren’t actually real, which means you can load balance by simply determining if the packets should be able to route then letting them route for the rest of the flow.
  - baq 4y ago
    So, basically magic, just as I expected. Got it.
- jvans 4y ago
  +1 would be great for more detailed explanation or links out to reading material. I assume there are some limitations if you actually skip the LB? How do the host NICs rewrite the LB address, does this imply there is hardware support for this kind of bypass routing?

travbrack 4y ago

Any benchmarks?

victorbjorklund 4y ago
https://loxilb-io.github.io/loxilbdocs/perf/ https://loxilb-io.github.io/loxilbdocs/perf/

goodpoint 4y ago

The build starts with "Install GoLang > v1.17". For a eBPF based application? Not good.

tecleandor 4y ago
Is it because of being Go, the version, or what?
- egberts1 4y ago
  As a security vulnerability researcher, eBPF is a problematic one to this day. I instruct everyone to disable eBPF at kernel compile time. Unfortunately, one cannot completely compile eBPF out of their kernel