6 ms·

> However, due to the fast, atomic operations involved at many places, HAProxy was previously limited to 64 threads, and therefore 64 CPU cores, on 64-bit machi

> However, due to the fast, atomic operations involved at many places, HAProxy was previously limited to 64 threads, and therefore 64 CPU cores, on 64-bit machines. This limit is now raised to 4096 threads by the introduction of thread groups. This is such a welcome change! I've been in more than one situation where I rediscovered this limitations when trying to scale things while trying to remain monolith (due to costs and performance reasons) but noticing HAProxy not being able to utilize the newly upgraded machine. This will make a huge difference!

ilyt 4y ago
Less than you think. You only need few cores to saturate tens of gigabits of traffic, HAProxy already is very performant. It's future-proofing basically. The page is ancient https://www.haproxy.org/10g.html https://www.haproxy.org/10g.html but they were doing 10Gbits on Core2Duo in '09 (sans SSL but back then AES acceleration wasn't very common either)
- wtarreau 4y ago
  Indeed, we've done 100Gbps about a year ago on a 8-core machine. Usually the only reason to use many cores is SSL, but since OpenSSL 3.0 that totally collapses under load, even then you're forced to significantly lower the number of threads (or to downgrade to 1.1.1 that about any high traffic site does).
  - ivanr 4y ago
    Do you know what's behind the performance degradation of OpenSSL 3.0? Has the problem been documented anywhere?
    - wtarreau 4y ago
      Horrible locking. 95% CPU spent in spinlocks. We're still doing measurements that we'll report with all data shortly. Anyway many of them were already collected by the project; there are so many that they created a meta-issue to link to them: https://github.com/openssl/openssl/issues/17627#issuecomment-1060123659 https://github.com/openssl/openssl/issues/17627#issuecomment... 3.1-dev is slightly less worse but still far behind 1.1.1. They made it too dynamic, and certain symbols that were constants or macroes have become functions running over lists under a lock. We noticed the worst degradation in client mode where the performance was divided by 200 for 48 threads, making it literally unusable.
toast0 4y ago
Maybe I'm biased by what I was doing with it and when, but HAProxy seemed better suited to processes than threads; there's not much state that's useful to share, and sharing the single process list of FDs is a scalability challenge (and presents opportunities for bugs around reuse; I fixed one of those). Also, I found it best to work with everything avoiding cross CPU communication; making sure everything happens on the CPU core that handles the RX queue for the incoming connection. My NICs only did 16 queues, so more cores than that wasn't useful for me. Do they make nics with 128 queues now? (Selecting outgoing ports so traffic to/from origin stays on the same CPU is tricky, but also solves the problem of making sure there's no port contention between the processes)
- xxpor 4y ago
  >Also, I found it best to work with everything avoiding cross CPU communication; making sure everything happens on the CPU core that handles the RX queue for the incoming connection. Yes, this is absolutely critical. Atomic ops (including taking a lock) when you have a large number of cores just completely kills performance. >Do they make nics with 128 queues now? Yep, the Intel E810 100gbit controller supports 256 queue pairs for just one example. Hope you can process that traffic pretty fast, the amount of memory dedicated to packet buffers with that many queues is making my head spin a bit.
  - toast0 4y ago
    Oh nice, in that case, time for 256 core haproxy box ;) My use case was just tcp proxy, inspecting the first few bytes to determine destination (http vs https vs proprietary) and add proxy protocol headers to the origin. Setup is somewhat expensive (tcp table manipulation isn't cheap), but data on established connections is simple and easy.
    - wtarreau 4y ago
      For pure TCP you rarely need threads, indeed! You may benefit from TCP splicing however if you're dealing with high bandwidth.

ilyt 4y ago

Less than you think. You only need few cores to saturate tens of gigabits of traffic, HAProxy already is very performant. It's future-proofing basically. The page is ancient https://www.haproxy.org/10g.html https://www.haproxy.org/10g.html but they were doing 10Gbits on Core2Duo in '09 (sans SSL but back then AES acceleration wasn't very common either)

wtarreau 4y ago
Indeed, we've done 100Gbps about a year ago on a 8-core machine. Usually the only reason to use many cores is SSL, but since OpenSSL 3.0 that totally collapses under load, even then you're forced to significantly lower the number of threads (or to downgrade to 1.1.1 that about any high traffic site does).
- ivanr 4y ago
  Do you know what's behind the performance degradation of OpenSSL 3.0? Has the problem been documented anywhere?
  - wtarreau 4y ago
    Horrible locking. 95% CPU spent in spinlocks. We're still doing measurements that we'll report with all data shortly. Anyway many of them were already collected by the project; there are so many that they created a meta-issue to link to them: https://github.com/openssl/openssl/issues/17627#issuecomment-1060123659 https://github.com/openssl/openssl/issues/17627#issuecomment... 3.1-dev is slightly less worse but still far behind 1.1.1. They made it too dynamic, and certain symbols that were constants or macroes have become functions running over lists under a lock. We noticed the worst degradation in client mode where the performance was divided by 200 for 48 threads, making it literally unusable.

toast0 4y ago

Maybe I'm biased by what I was doing with it and when, but HAProxy seemed better suited to processes than threads; there's not much state that's useful to share, and sharing the single process list of FDs is a scalability challenge (and presents opportunities for bugs around reuse; I fixed one of those). Also, I found it best to work with everything avoiding cross CPU communication; making sure everything happens on the CPU core that handles the RX queue for the incoming connection. My NICs only did 16 queues, so more cores than that wasn't useful for me. Do they make nics with 128 queues now? (Selecting outgoing ports so traffic to/from origin stays on the same CPU is tricky, but also solves the problem of making sure there's no port contention between the processes)

xxpor 4y ago
>Also, I found it best to work with everything avoiding cross CPU communication; making sure everything happens on the CPU core that handles the RX queue for the incoming connection. Yes, this is absolutely critical. Atomic ops (including taking a lock) when you have a large number of cores just completely kills performance. >Do they make nics with 128 queues now? Yep, the Intel E810 100gbit controller supports 256 queue pairs for just one example. Hope you can process that traffic pretty fast, the amount of memory dedicated to packet buffers with that many queues is making my head spin a bit.
- toast0 4y ago
  Oh nice, in that case, time for 256 core haproxy box ;) My use case was just tcp proxy, inspecting the first few bytes to determine destination (http vs https vs proprietary) and add proxy protocol headers to the origin. Setup is somewhat expensive (tcp table manipulation isn't cheap), but data on established connections is simple and easy.
  - wtarreau 4y ago
    For pure TCP you rarely need threads, indeed! You may benefit from TCP splicing however if you're dealing with high bandwidth.