6 ms·

I feel like we never see HAProxy in these reverse proxy comparisons. Lots of nginx, Apache, Caddy, Traefik, Envoy, etc. The HAProxy configuration is just as si

I feel like we never see HAProxy in these reverse proxy comparisons. Lots of nginx, Apache, Caddy, Traefik, Envoy, etc. The HAProxy configuration is just as simple as Caddy for a reverse proxy setup. It's also written in C which is a comparison the author makes between nginx and Caddy. And it seems to be available on every *nix OS.

RcouF1uZ4gsC 4y ago
> The HAProxy configuration is just as simple as Caddy for a reverse proxy setup. Does HAProxy have built in support for Let’s Encrypt? That is one of my favorite features. Caddy just automatically manages the certificates for https.
- gordian_NOT 4y ago
  It's not as turn-key as Caddy, that's for sure, but it's there: https://www.haproxy.com/blog/lets-encrypt-acme2-for-haproxy/ https://www.haproxy.com/blog/lets-encrypt-acme2-for-haproxy/
  - ei8ths 4y ago
    this is great, i'll implement this soon as my current cert is about to expire and have been wanting to get haproxy on lets encrypt.
- fullstop 4y ago
  Built-in? Not exactly, but there is an acmev2 implemention from haproxytech: https://github.com/haproxytech/haproxy-lua-acme https://github.com/haproxytech/haproxy-lua-acme
- abdusco 4y ago
  I use caddy mostly as a reverse proxy in front of an app. It's just one line in the caddy file: sub.domain.com { # transparent proxy + websocket support + letsencrypt TLS reverse_proxy 127.0.0.1:2345 } It's a fresh breath of air to have server with sensible defaults after dealing with apache and nginx (haproxy isn't much better in that regard).
  - mholt 4y ago
    If that's your whole Caddyfile, might as well not even use a config file: caddy reverse-proxy --from sub.domain.com --to :2345 Glad you like using Caddy!
    - bmurphy1976 4y ago
      Personally I still recommend the config file. Even when they are simple, it gives you one single source of truth that you can refer to, it will grow as you need it, and it can be stored in source control. Where and how parameters are configured is a bit more of a wild card and dependent on the environment you are running in.
      - francislavoie 4y ago
        That's something Matt and I tend to disagree on - I agree that a config file is better almost always because it gives you a better starting point to experiment with other features.
        mholt 4y ago
        Hey, I mean, I do agree that a config file is "better" most of the time -- but having the CLI is just so awesome! :D
  - CoolCold 4y ago
    I still cannot make myself to try Caddy.. in things like this looks sweet but just may be 5% of the functionality [I care about]. Not saying it's not possible, but with Nginx I already know how to do list of CORS, OPTIONS , per location & cookie name caching. Issuing certs is probably simplest and the last thing in config setups of reverse proxying.
- TimWolla 4y ago
  It does not, because HAProxy does not perform any disk access at runtime and thus would be unable to persist the certificates anywhere. Disks accesses can be unpredictably slow and would block the entire thread which is not something you want when handling hundreds of thousands of requests per second. See this issue and especially the comment from Lukas Tribus: https://github.com/haproxy/haproxy/issues/1864 https://github.com/haproxy/haproxy/issues/1864 Disclosure: Community contributor to HAProxy, I help maintain HAProxy's issue tracker.
  - mholt 4y ago
    That issue has some good explanation, thanks. I wonder if a disk-writing process could be spun out before dropping privileges? > Disks accesses can be unpredictably slow and would block the entire thread which is not something you want when handling hundreds of thousands of requests per second. This is not something I see mentioned in the issue, but I don't see why disk accesses need to block requests, or why they have to occur in the same thread as requests?
    - TimWolla 4y ago
      When reading along: Keep in mind that I'm not a core developer and thus are not directly involved in development, design decisions, or roadmap. I have some understanding of the internals and the associated challenges based on my contributions and discussions on the mailing list, but the following might not be entirely correct. > I wonder if a disk-writing process could be spun out before dropping privileges? I mean … it sure can and that appears the plan based on the last comment in that issue. However the “no disk access” policy is also useful for security. HAProxy can chroot itself to an empty directory to reduce the blast radius and that is done in the default configuration on at least Debian. > but I don't see why disk accesses need to block requests My understanding is that historically Linux disk IO was inherently blocking. A non-blocking interface (io_uring) only became available fairly recently: https://stackoverflow.com/a/57451551/782822 https://stackoverflow.com/a/57451551/782822. And even then it's a operating system specific interface. For the BSD's you need a different solution. If your process is blocked for even one millisecond when handling two million of requests per second (https://www.haproxy.com/de/blog/haproxy-forwards-over-2-million-http-requests-per-second-on-a-single-aws-arm-instance/ https://www.haproxy.com/de/blog/haproxy-forwards-over-2-mill...) then you drop 2k requests or increase latency. > or why they have to occur in the same thread as requests? “have” is a strong word, of course nothing “has” to be. One thing to keep in mind is that HAProxy is 20 years old and apart from possibly doing Let's Encrypt there was no real need for it to have disk access. HAProxy is a reverse proxy / load balancer, not a web server. Inter-thread communication comes with its own set of challenges and building something reliable for a narrow use case is not necessarily worth it, because you likely need to sacrifice something else. As an example at scale you can't even let your operating system schedule out one of the worker threads to schedule in the “disk writer” thread, because that will effectively result in a reduced processing capacity for some fractions of a second which will result in dropped requests or increased latency. This becomes even worse if the worker holds an important lock.
- deleted 4y ago
  [deleted]
fullstop 4y ago
I would also like to see benchmarks for reverse proxies with TLS termination.
- capableweb 4y ago
  Yeah, this tends to be (in my cases) where response times suffer the most, unless your bottleneck is I/O to/from the backend or further away
- mholt 4y ago
  I think one reason a lot of benchmarks don't include TLS termination is because it's often impractical in the real-world, where most clients reuse the connection and the TLS session for many requests, thus making them negligible in the long run. And given hardware optimizations for cryptographic functions combined with network round trips, you end up benchmarking the network and the protocol more than its actual implementation, which is often upstream from the server itself anyway. Go's TLS stack is set to be more efficient and safer in coming versions thanks to continued work by Filippo and team.
  - nerdponx 4y ago
    Maybe it would be a useful benchmark to simulate a scenario like "my site got posted on HN and now I'm getting a huge number of unique page views."
    - mholt 4y ago
      Sure, we've already done this very real test in production a number of times and Caddy doesn't even skip a beat. (IMO that's the best kind of benchmark right there. No need to simulate with pretend traffic!)
    - CoolCold 4y ago
      Any idea on how much traffic could be from HN? I doubt more than 100 rps or any other noticeable load
tempest_ 4y ago
I am not sure I would agree with the assertion that config for HAProxy is just as easy. In fact I use HAProxy in production pretty regularly because it is solid but its config one of the main reasons I would choose something else. A basic HAProxy config is fine but it feels like after a little bit each line is just a collection of tokens in a random order that I have to sit and think about to parse.
- bmurphy1976 4y ago
  I feel the same way. I'm not a fan of haproxy's configuration system. It's really difficult for me to understand it, whereas I feel I can read most nginx/apache configs and immediately know what is supposed to be happening. I still maintain servers under load in production that use all three to this day and I always go back to nginx because of the configuration alone.
  - kilburn 4y ago
    I can't comment on haproxy because I haven't used it enough, but I think that the "nginx's config is easy to grasp" posture has a bit of Stockholm syndrome in it. - Do you want to add a header in this "location" block? Great, you better remember to re-apply all the security headers you've defined at a higher level (server block for instance) because of course adding a new header will reset those. - Oh, you mixed prefix locations with exact location with regex locations. Great, let's see if you can figure out by which location block will a request end up being processed. The docs "clearly" explain what the priority rules for those are and they're easy to grasp [1]. - I see you used a hostname in a proxy_pass directive (e.g.: http://internal.thing.com http://internal.thing.com). Great, I will resolve it at startup and never check again, because this is the most sensible thing to do of course. - Oh... now you used a variable (e.g.: http://$internal_host http://$internal_host). That fundamentally changes things (how?) so I'll respect the DNS's TTL now. Except you'll have to set up a DNS resolver in my config because I refuse to use the system's normal resolver because reasons. - Here's an `if` directive for the configuration. It sounds extremely useful, doesn't it? Well.. "if is evil" [2] and you should NOT use it. There be dragons, you've been warned. I could go on... but I think I've proved my point already. Note that these are not complaints, it's just me pointing out that nginx's configuration has its _very_ significant warts too. [1] https://nginx.org/en/docs/http/ngx_http_core_module.html#location https://nginx.org/en/docs/http/ngx_http_core_module.html#loc... [2] https://www.nginx.com/resources/wiki/start/topics/depth/ifisevil/ https://www.nginx.com/resources/wiki/start/topics/depth/ifis...
snowwrestler 4y ago
I’m surprised Varnish is not mentioned much either. For a while there it had a reputation as the fastest reverse proxy. I think its popularity was harmed by complex config and refusal to handle TLS.
- pbowyer 4y ago
  It's always been blisteringly fast when we've used it, and I like the power of the configuration (it has its quirks but so do most powerful systems). But the overhead of setting it up and maintaining it due to having to handle TLS termination separately puts me off using it when other software is 'good enough'. If Varnish Enterprise was cheaper I would have bought it, but at their enterprise prices no way. I'm keeping a watching brief on https://github.com/darkweak/souin https://github.com/darkweak/souin and its Caddy integration to see if that can step up and replace Varnish for short-lived dynamic caching of web applications. Though I've lost track of its current status.
  - darkweak 4y ago
    Amazing that you're talking about Souin and it's possible usage to replace Varnish. Let me know if you have question about the configuration or implementation. ATM I'm working on the stabilization branch to have a more stable version and merge the improvements into the caddy's cache-handler module.
- rapind 4y ago
  Varnish is a must for my production apps. The grace period ability to serve stale cache while passing a request through to get the latest is just huge and I can’t live without it.
tylerjl 4y ago
FWIW I'm a big fan of HAProxy as well, but I was just constrained by the sheer volume of testing and how rigorous I intended to be. Maybe once my testing is a little more generalized I can fan out to additional proxies like HAProxy without too much hassle, as I'd love to know as well.
- tomohawk 4y ago
  Would love to see this
gog 4y ago
HAProxy does not serve static files (AFAIK), so for some stacks you need to add nginx or caddy after haproxy as well to serve static files and forward to a fastcgi backend.
- tomohawk 4y ago
  nginx started out as a web server and over time gained reverse proxy abilities. haproxy started out as a proxy and has gained some web server abilities, but is all about proxying. haproxy has less surprises as a reverse proxy than nginx does. Some of the defaults for nginx are appropriate for web serving, but not proxying.

RcouF1uZ4gsC 4y ago

> The HAProxy configuration is just as simple as Caddy for a reverse proxy setup. Does HAProxy have built in support for Let’s Encrypt? That is one of my favorite features. Caddy just automatically manages the certificates for https.

gordian_NOT 4y ago
It's not as turn-key as Caddy, that's for sure, but it's there: https://www.haproxy.com/blog/lets-encrypt-acme2-for-haproxy/ https://www.haproxy.com/blog/lets-encrypt-acme2-for-haproxy/
- ei8ths 4y ago
  this is great, i'll implement this soon as my current cert is about to expire and have been wanting to get haproxy on lets encrypt.
fullstop 4y ago
Built-in? Not exactly, but there is an acmev2 implemention from haproxytech: https://github.com/haproxytech/haproxy-lua-acme https://github.com/haproxytech/haproxy-lua-acme
abdusco 4y ago
I use caddy mostly as a reverse proxy in front of an app. It's just one line in the caddy file: sub.domain.com { # transparent proxy + websocket support + letsencrypt TLS reverse_proxy 127.0.0.1:2345 } It's a fresh breath of air to have server with sensible defaults after dealing with apache and nginx (haproxy isn't much better in that regard).
- mholt 4y ago
  If that's your whole Caddyfile, might as well not even use a config file: caddy reverse-proxy --from sub.domain.com --to :2345 Glad you like using Caddy!
  - bmurphy1976 4y ago
    Personally I still recommend the config file. Even when they are simple, it gives you one single source of truth that you can refer to, it will grow as you need it, and it can be stored in source control. Where and how parameters are configured is a bit more of a wild card and dependent on the environment you are running in.
    - francislavoie 4y ago
      That's something Matt and I tend to disagree on - I agree that a config file is better almost always because it gives you a better starting point to experiment with other features.
      - mholt 4y ago
        Hey, I mean, I do agree that a config file is "better" most of the time -- but having the CLI is just so awesome! :D
- CoolCold 4y ago
  I still cannot make myself to try Caddy.. in things like this looks sweet but just may be 5% of the functionality [I care about]. Not saying it's not possible, but with Nginx I already know how to do list of CORS, OPTIONS , per location & cookie name caching. Issuing certs is probably simplest and the last thing in config setups of reverse proxying.
TimWolla 4y ago
It does not, because HAProxy does not perform any disk access at runtime and thus would be unable to persist the certificates anywhere. Disks accesses can be unpredictably slow and would block the entire thread which is not something you want when handling hundreds of thousands of requests per second. See this issue and especially the comment from Lukas Tribus: https://github.com/haproxy/haproxy/issues/1864 https://github.com/haproxy/haproxy/issues/1864 Disclosure: Community contributor to HAProxy, I help maintain HAProxy's issue tracker.
- mholt 4y ago
  That issue has some good explanation, thanks. I wonder if a disk-writing process could be spun out before dropping privileges? > Disks accesses can be unpredictably slow and would block the entire thread which is not something you want when handling hundreds of thousands of requests per second. This is not something I see mentioned in the issue, but I don't see why disk accesses need to block requests, or why they have to occur in the same thread as requests?
  - TimWolla 4y ago
    When reading along: Keep in mind that I'm not a core developer and thus are not directly involved in development, design decisions, or roadmap. I have some understanding of the internals and the associated challenges based on my contributions and discussions on the mailing list, but the following might not be entirely correct. > I wonder if a disk-writing process could be spun out before dropping privileges? I mean … it sure can and that appears the plan based on the last comment in that issue. However the “no disk access” policy is also useful for security. HAProxy can chroot itself to an empty directory to reduce the blast radius and that is done in the default configuration on at least Debian. > but I don't see why disk accesses need to block requests My understanding is that historically Linux disk IO was inherently blocking. A non-blocking interface (io_uring) only became available fairly recently: https://stackoverflow.com/a/57451551/782822 https://stackoverflow.com/a/57451551/782822. And even then it's a operating system specific interface. For the BSD's you need a different solution. If your process is blocked for even one millisecond when handling two million of requests per second (https://www.haproxy.com/de/blog/haproxy-forwards-over-2-million-http-requests-per-second-on-a-single-aws-arm-instance/ https://www.haproxy.com/de/blog/haproxy-forwards-over-2-mill...) then you drop 2k requests or increase latency. > or why they have to occur in the same thread as requests? “have” is a strong word, of course nothing “has” to be. One thing to keep in mind is that HAProxy is 20 years old and apart from possibly doing Let's Encrypt there was no real need for it to have disk access. HAProxy is a reverse proxy / load balancer, not a web server. Inter-thread communication comes with its own set of challenges and building something reliable for a narrow use case is not necessarily worth it, because you likely need to sacrifice something else. As an example at scale you can't even let your operating system schedule out one of the worker threads to schedule in the “disk writer” thread, because that will effectively result in a reduced processing capacity for some fractions of a second which will result in dropped requests or increased latency. This becomes even worse if the worker holds an important lock.
deleted 4y ago
[deleted]

fullstop 4y ago

I would also like to see benchmarks for reverse proxies with TLS termination.

capableweb 4y ago
Yeah, this tends to be (in my cases) where response times suffer the most, unless your bottleneck is I/O to/from the backend or further away
mholt 4y ago
I think one reason a lot of benchmarks don't include TLS termination is because it's often impractical in the real-world, where most clients reuse the connection and the TLS session for many requests, thus making them negligible in the long run. And given hardware optimizations for cryptographic functions combined with network round trips, you end up benchmarking the network and the protocol more than its actual implementation, which is often upstream from the server itself anyway. Go's TLS stack is set to be more efficient and safer in coming versions thanks to continued work by Filippo and team.
- nerdponx 4y ago
  Maybe it would be a useful benchmark to simulate a scenario like "my site got posted on HN and now I'm getting a huge number of unique page views."
  - mholt 4y ago
    Sure, we've already done this very real test in production a number of times and Caddy doesn't even skip a beat. (IMO that's the best kind of benchmark right there. No need to simulate with pretend traffic!)
  - CoolCold 4y ago
    Any idea on how much traffic could be from HN? I doubt more than 100 rps or any other noticeable load

tempest_ 4y ago

I am not sure I would agree with the assertion that config for HAProxy is just as easy. In fact I use HAProxy in production pretty regularly because it is solid but its config one of the main reasons I would choose something else. A basic HAProxy config is fine but it feels like after a little bit each line is just a collection of tokens in a random order that I have to sit and think about to parse.

bmurphy1976 4y ago
I feel the same way. I'm not a fan of haproxy's configuration system. It's really difficult for me to understand it, whereas I feel I can read most nginx/apache configs and immediately know what is supposed to be happening. I still maintain servers under load in production that use all three to this day and I always go back to nginx because of the configuration alone.
- kilburn 4y ago
  I can't comment on haproxy because I haven't used it enough, but I think that the "nginx's config is easy to grasp" posture has a bit of Stockholm syndrome in it. - Do you want to add a header in this "location" block? Great, you better remember to re-apply all the security headers you've defined at a higher level (server block for instance) because of course adding a new header will reset those. - Oh, you mixed prefix locations with exact location with regex locations. Great, let's see if you can figure out by which location block will a request end up being processed. The docs "clearly" explain what the priority rules for those are and they're easy to grasp [1]. - I see you used a hostname in a proxy_pass directive (e.g.: http://internal.thing.com http://internal.thing.com). Great, I will resolve it at startup and never check again, because this is the most sensible thing to do of course. - Oh... now you used a variable (e.g.: http://$internal_host http://$internal_host). That fundamentally changes things (how?) so I'll respect the DNS's TTL now. Except you'll have to set up a DNS resolver in my config because I refuse to use the system's normal resolver because reasons. - Here's an `if` directive for the configuration. It sounds extremely useful, doesn't it? Well.. "if is evil" [2] and you should NOT use it. There be dragons, you've been warned. I could go on... but I think I've proved my point already. Note that these are not complaints, it's just me pointing out that nginx's configuration has its _very_ significant warts too. [1] https://nginx.org/en/docs/http/ngx_http_core_module.html#location https://nginx.org/en/docs/http/ngx_http_core_module.html#loc... [2] https://www.nginx.com/resources/wiki/start/topics/depth/ifisevil/ https://www.nginx.com/resources/wiki/start/topics/depth/ifis...

snowwrestler 4y ago

I’m surprised Varnish is not mentioned much either. For a while there it had a reputation as the fastest reverse proxy. I think its popularity was harmed by complex config and refusal to handle TLS.

pbowyer 4y ago
It's always been blisteringly fast when we've used it, and I like the power of the configuration (it has its quirks but so do most powerful systems). But the overhead of setting it up and maintaining it due to having to handle TLS termination separately puts me off using it when other software is 'good enough'. If Varnish Enterprise was cheaper I would have bought it, but at their enterprise prices no way. I'm keeping a watching brief on https://github.com/darkweak/souin https://github.com/darkweak/souin and its Caddy integration to see if that can step up and replace Varnish for short-lived dynamic caching of web applications. Though I've lost track of its current status.
- darkweak 4y ago
  Amazing that you're talking about Souin and it's possible usage to replace Varnish. Let me know if you have question about the configuration or implementation. ATM I'm working on the stabilization branch to have a more stable version and merge the improvements into the caddy's cache-handler module.
rapind 4y ago
Varnish is a must for my production apps. The grace period ability to serve stale cache while passing a request through to get the latest is just huge and I can’t live without it.

tylerjl 4y ago

FWIW I'm a big fan of HAProxy as well, but I was just constrained by the sheer volume of testing and how rigorous I intended to be. Maybe once my testing is a little more generalized I can fan out to additional proxies like HAProxy without too much hassle, as I'd love to know as well.

tomohawk 4y ago
Would love to see this

gog 4y ago

HAProxy does not serve static files (AFAIK), so for some stacks you need to add nginx or caddy after haproxy as well to serve static files and forward to a fastcgi backend.

tomohawk 4y ago
nginx started out as a web server and over time gained reverse proxy abilities. haproxy started out as a proxy and has gained some web server abilities, but is all about proxying. haproxy has less surprises as a reverse proxy than nginx does. Some of the defaults for nginx are appropriate for web serving, but not proxying.