10 ms·

Caddyhttp: Enable HTTP/3 by Default

284 points by fariszr 4y ago

mholt 4y ago
Oh hey HN. This is in the Caddy 2.6 beta already: https://github.com/caddyserver/caddy/releases/tag/v2.6.0-beta.3 https://github.com/caddyserver/caddy/releases/tag/v2.6.0-bet... - please try it out! Thanks to Marten Seemann for maintaining the quic-go library we use. (I still haven't heard whether Go will add HTTP/3 to the standard library.) Caddy 2.6 should be the first stable release of a general-purpose server to support and enable standardized HTTP/3 by default. HTTP versions can be toggled on or off. (Meaning you can serve only HTTP/3 exclusively if you're hard-core.) PS. Caddy 2.6 will be our biggest release since 2.0. My draft release notes are about 23 KB. We're looking at huge performance improvements and powerful new features like events, virtual file systems, HTTP 103 Early Hints, and a lot of other enhancements I'm excited to show off on behalf of our collaborators!
- asb 4y ago
  I've just moved over to Caddy and had a really good experience in doing so (thank you to you and all the other Caddy contributors!). One thing that stuck out to me during setup is that it seemed surprising to see gzip encoding not enabled by default. Wouldn't it make sense for a server that enables https by out of the box with zero configuration and soon supports http3 to also gzip by default?
  - mholt 4y ago
    Good question! When I first announced Caddy in 2015 and the server got busy, it started downloading everything as .gz files. (facepalm) (good ol' days) But that's just a fun story, not actually the reason we don't enable it by default... The tricky thing about enabling features by default is that turning them off is awkward in config. And while we like to have "magic" in Caddy, we don't like too much magic. Plus, gzip performance in Go is less good (but more memory-safe) than it is in C and assembly. Klauspost's flate implementation is very fast and that's the one we use. But even with a super-fast implementation, gzip requires memory and CPU that busy servers may be in short supply of. So it's opt-in for now. We can always change it later; going the other way is harder. (Caddy also supports serving pre-compressed files, if enabled! And starting with Caddy 2.6, those can be teleported at the speed of electricity to HTTP clients using sendfile. HTTPS also sees faster file serving in 2.6 due to optimized copying.) See also: https://serverfault.com/questions/296770/why-arent-features-like-gzip-enabled-by-default-on-many-webservers https://serverfault.com/questions/296770/why-arent-features-...
    - asb 4y ago
      Thanks for the response! I'd imagined reasoning along those lines. I'd perhaps expect that people anticipating such high load that gzipping output is problematic would be investing enough time in server setup to review and tweak these options. And for everyone else, compression by default is a good starting point. Moving from Nginx (HTTP2 off by default) to Caddy (HTTP2 on by default) you'd already need to review your setup if running a highly trafficked site to ensure any reverse proxied servers can handle the additional request load due to multiplexing. But it's obviously a trade-off and there's lots to weigh up...
    - hsbauauvhabzb 4y ago
      Why was gzipping everything a facepalm? Performance?
      - francislavoie 4y ago
        It was making the browser download files with the .gz extension, instead of serving the content in a way that the browser would display. TL;DR it was broken at the time. But it was a very long time ago.
    - layer8 4y ago
      If gzip is enabled, is the gzipped version of a static file cached in memory, or is it gzipped anew for each request?
      - mholt 4y ago
        Caddy doesn't cache responses by default, but you can enable caching with the cache plugin: https://github.com/caddyserver/cache-handler/ https://github.com/caddyserver/cache-handler/
        oynqr 4y ago
        Does this apply to static file serving? Although I suppose relying on the OS cache might be enough.
  - JZerf 4y ago
    In addition to the reasons that mholt mentioned in response to your question, enabling HTTP/GZIP compression could possibly be less secure for some web server configurations due to things like the BREACH attack. See https://en.wikipedia.org/wiki/HTTP_compression#Security_implications https://en.wikipedia.org/wiki/HTTP_compression#Security_impl... and https://en.wikipedia.org/wiki/BREACH https://en.wikipedia.org/wiki/BREACH for more info. I might be wrong but I don't think that current web serving protocols mitigate an attack like this. It might be better to default to safe settings that don't use HTTP/GZIP compression even if it might slow things down for the time being.
    - thadt 4y ago
      Eh, it's not just current web serving protocols. Any protocol where: - An application uses compression - An attacker is able to supply chosen data to it - The application compresses the attacker's data and static secret data together - The attacker is able to monitor the size of the compressed data - This can be repeated by the attacker a number of times will be vulnerable to having its secret data stolen by techniques like BREACH. If you want your secret data to stay secret, don't compress it with attacker chosen plaintext where the resulting size could be monitored.
- fariszr 4y ago
  Absolutely awesome! I have recently gave Caddy a shot, and immediately loved it!. I am a bit worried about the performance deficit compared to apache or Nginx, but none of my servers really have reached the scale where it actually matters. Its unfortunate that there is no dynamic brotli support, but oh well, a trade off I'm willing to make. One of things that aren't clear to me are Caddy plugins. How big of an affect do they have on peformance? Are plugins a core part of Caddy's usage? Or are they just exist to cover some edge cases? I'm thinking of using the caddy-filter and caddy-crowdsec-bouncer plugins.
  - francislavoie 4y ago
    Having plugins installed has no performance incursion at all (because they don't run unless configured), but using them will, of course affect performance. But it depends on the plugin and what it does. You have to do your own benchmarking to see what effect they have to performance. Everyone has different needs and configs, so it's not possible to give authoritative benchmarks. Just try it out and see. Regarding brotli, there is a plugin you can try if you'd like https://github.com/ueffel/caddy-brotli https://github.com/ueffel/caddy-brotli, but brotli's on-the-fly performance is not that good. It works well if you pre-compress you assets though, and you can serve those with the file_server directive's precompressed option.
    - deleted 4y ago
      [deleted]
  - mholt 4y ago
    Glad you liked it! Caddy's performance is competitive with nginx, but there are too many dimensions to discuss here. I guarantee that for most people, your web server will not be your bottleneck. Dynamic brotli is expensive, as brotli compression is inefficient. There are no optimized Go implementations of this yet. But Caddy can serve precompressed brotli files without any separate plugins. What do you mean by "what affect do [plugins] have on performance"? All features in Caddy are plugins, some just come installed by default: https://caddyserver.com/docs/architecture https://caddyserver.com/docs/architecture -- even its HTTP server is a plugin. (All plugins are compiled in natively.)
- purim 4y ago
  Does Caddy have the same issue with configuration spaghetti problem with Traefik? How does it differ from Traefik's model?
  - mholt 4y ago
    I'm not super familiar with Traefik, but if you're talking about using container labels for config, no, Caddy's config is given as a JSON document through a REST API. The CLI can let you use config files if you prefer. Or the Caddyfile is a more human-friendly format that Caddy supports by default. You can actually use any config format you want if there's a config adapter for it: https://caddyserver.com/docs/config-adapters https://caddyserver.com/docs/config-adapters
  - robk 4y ago
    Caddy is vastly vastly better in this regard. Traefik config files drive me insane when rewriting anything.
- todotask 4y ago
  Planning on trying Caddy 2.6 one of these day to see how much improvement it can make over other web servers, maybe CentmindMod would be interested in it. We all are aware local vs real world benchmark are differ.
  - francislavoie 4y ago
    Caddy v2.6 is making huge improvements here. See https://github.com/caddyserver/caddy/pull/5022 https://github.com/caddyserver/caddy/pull/5022 But also, please don't take benchmarks like this as gospel. Run your own benchmarks, on your own hardware, with your own config. Someone else's config may not be ideal for your usecase.
  - mholt 4y ago
    So you want to benchmark the network?
    - todotask 4y ago
      On a VPS, but no doubt that running Go app directly is more performant, I could use Go for serving static contents as well.
      - mholt 4y ago
        I have done that test. Go is not significantly faster than Caddy for serving static content.
michael_michael 4y ago
Thanks for a great piece of software that I use every day in production and just works. At first I was scared of how stupid simple it is. It feels like web servers are supposed to have giant config files with a hundred mysterious knobs to twiddle. Now I always default to Caddy, and have yet to find an instance where it didn't fit my needs. Congrats.
- lynndotpy 4y ago
  This is my experience as well. I've spent weekends trying to figure things out with Apache + Nginx + Let's Encrypt. Figured it out with Caddy within minutes. Really happy I found this software.
  - KronisLV 4y ago
    > I've spent weekends trying to figure things out with Apache + Nginx + Let's Encrypt. Curiously, new versions of Apache include mod_md, which lets you use Let's Encrypt for SSL, without the need for something like certbot: https://httpd.apache.org/docs/2.4/mod/mod_md.html https://httpd.apache.org/docs/2.4/mod/mod_md.html I actually wrote about it in a blog post of mine, "How and why to use Apache httpd in 2022": https://blog.kronis.dev/tutorials/how-and-why-to-use-apache-httpd-in-2022 https://blog.kronis.dev/tutorials/how-and-why-to-use-apache-... That said, while the capability is there, things like the DNS-01 validation need additional work and Caddy comes with more sane defaults out of the box and arguably a way easier config format. It would also be really cool if Nginx came with its own implementation of ACME certificate provisioning out of the box, following the trend of other servers, but certbot is also perfectly passable. Either way, it seems like most of the "popular" web servers are viable for most use cases nowadays and I can say that Caddy definitely belongs in that list!
howeyc 4y ago
does it work with "tls internal" or do the certs need to be signed by an outside CA for HTTP/3 to be enabled.
- mholt 4y ago
  Yes it works with self-signed certificates!
  - superkuh 4y ago
    I'm really glad to hear this is an option. But will browsers accessing the website accept the self-signed certs when setting up the TLS connection required by QUIC? I don't know how they behave. I'm genuinely asking.
    - francislavoie 4y ago
      Yes they do, as long as you installed Caddy's root CA cert in the browser's trust store. Caddy will attempt to install it automatically with https://github.com/smallstep/truststore https://github.com/smallstep/truststore if possible (usually requires root) but if it fails you can try again with "sudo caddy trust". You might be using a client or trust store that isn't supported though, in which case you'll need to install the root cert manually.
      - superkuh 4y ago
        What if I just want to generate and use a self signed cert without any CA involvement? I do that with HTTP all the time and it works for the public web even if the browsers like to scaremonger about it. Does a QUIC connection work without a CA (corporate) root cert being referenced?
        francislavoie 4y ago
        To be clear, Caddy itself manages its own internal CA. It's the same thing as self signing except you have a root cert you can trust, to trust every cert Caddy generates; that's instead of having individual self signed certificates. This makes it significantly easier if you have many subdomains for each of your services, etc. It has nothing to do with corporate CAs, no external entities. QUIC/HTTP3 use the same TLS stack as H1/H2, same things are trusted. So there's no difference there.
CodesInChaos 4y ago
Since address validation is about blocking senders which forge their IP address, I think the number of connection attempts which where the client doesn't eventually validate its address should factor into the decision to enable this feature. This should rarely happen for legitimate clients (e.g. connection loss/cancellation during the first roundtrip) but always happen for IP address forgers. Or perhaps simply use the number of half-open/embryonic connections as the metric.
- mholt 4y ago
  Good point. Our current metric (number of active requests) is a bit naive, but mostly does the job essentially for free. If your proposed metric doesn't cost too much in memory, it might be a good fit.
  - CodesInChaos 4y ago
    How does such an attacker influence your active requests metric? Is it incremented when the initial UDP packet arrives (even though the http request itself hasn't arrived yet) and decremented when it times out?
    - mholt 4y ago
      No, currently we increment it when the HTTP request is parsed. (It counts HTTP requests, not packets/connections.) This was a quick and easy way to count; but I am very open to improving this! Feel free to submit a proposal or a PR. Ref: https://github.com/caddyserver/caddy/blob/50748e19c34fc90882bbb268c95a2a0acb051752/modules/caddyhttp/server.go#L185-L186 https://github.com/caddyserver/caddy/blob/50748e19c34fc90882...
      - CodesInChaos 4y ago
        In that case, I don't think your metric works at all, since legitimate users will increase it, but attackers won't. During an attack it will either remain the same, or even go down as fewer legitimate users manage to connect. This attack is the QUIC equivalent of a SYN flood, which results in half-open connections, because the attacker is unable to complete the connection by responding to message the server sends. RequireAddressValidation corresponds enabling syn cookies. Unfortunately I don't use caddy myself, and don't have time to get involved deeper in improving this.
srameshc 4y ago
I very recently tried Caddy for the first time and it is one of the best software experiences I had especially because of it's ease of use.
- 41b696ef1113 4y ago
  My only complaint with it is this half-way json vs standard config file situation in which it finds itself. The documents push you to the json system, but most of the available documents are in the config system.
  - asb 4y ago
    There's also some handy features that are only available in the JSON format. e.g. `strip_path_suffix` and `http_redirect` for the `rewrite` handler (https://github.com/caddyserver/caddy/issues/2011#issuecomment-544203470 https://github.com/caddyserver/caddy/issues/2011#issuecommen...), which would be a more ergonomic way to set a rule to strip trailing slashes than handling via regex.
    - francislavoie 4y ago
      While it is true that some features are only possible via JSON config, the ones you mentioned are available in the Caddyfile. See the uri directive which lets you strip a suffix, and redir which performs Location redirects. Also, the issue you linked to is from an early beta of Caddy v2, and the discussion and config there no longer applies.
      - asb 4y ago
        The uri directive does expose stripping a suffix, but that's an internal rewrite, correct? For any /foo/ /bar/ /baz/ I want to redirect to /foo, /bar, or /baz. Unless I'm missing something, this needs to be done like so right now: @trailing_slash path_regexp trailing_slash ^/(.*)/$ redir @trailing_slash /{re.trailing_slash.1} 308
        4y ago
pachico 4y ago
Weirdly enough, among the reasons to switch from nginx to Caddy I never find the lack of observability nginx and its almost useless Prometheus exporter have. I see several posts discussing under which circumstances or use cases one might outperform the other but they never seem to care about having decent metrics. I might overrate the importance of this, who knows...
- nginxlegacy 4y ago
  - xet7 4y ago
    Aren't? But there is https://redbean.dev https://redbean.dev that uses Lua, and is faster than Go.
    - coder543 4y ago
      Redbean is a webserver. Go is not a webserver. Redbean cannot be "faster than Go". That's like saying that a car is faster than aluminum. Do you have benchmarks of Redbean versus Caddy that you can point to? I certainly haven't been able to find any. I also haven't been able to find any in-depth independent benchmarks of Redbean. About the only thing I could find was someone struggling to reproduce the benchmark results.[0] Personally, I've dealt with more than enough memory safety vulnerabilities caused by no programmer being good enough to write safe and correct C (or C++). Those vulnerabilities are easily prevented by choosing just about any other language people are likely to know about... so, I have no desire to deploy anything like Redbean when alternatives exist. It's cool to see the hacks they've been able to do with Cosmopolitan to make cross platform binaries, but hacks like that are not what I want for anything other than educational purposes, at least not until they've been more battle tested. [0]: https://github.com/jart/cosmopolitan/issues/72 https://github.com/jart/cosmopolitan/issues/72
      - mholt 4y ago
        Great points. I also think it's important to stop deploying C to the edge. PS. Go might someday compile for cosmopolitan if someone writes a port for it: https://github.com/golang/go/issues/51900 https://github.com/golang/go/issues/51900
0xEFF 4y ago
I've been using Caddy in a 3 node HA cluster sharing an anycast bgp address for about 18 months now and it's been fantastic. Certs "just work" across the cluster once consul is wired up. I recently added IPv6 which also "just works." greenpau/caddy-security is fantastic and "just works" for OIDC sso. mholt, thanks for recently adding the ability to bind to multiple specific IP addresses by default, this help me conserve precious public IPv4 addresses.
- jabart 4y ago
  This is a thing? Any docs on this as the top search is this comment.
  - francislavoie 4y ago
    For which part? The comment you replied to mentions a bunch of different features, and a plugin.
    - jabart 4y ago
      The HA Cluster with an Anycast IP.
      - francislavoie 4y ago
        They're essentially just saying they run multiple instances of Caddy, and those instances are reached by clients via Anycast. This isn't anything specific to Caddy, that's networking-level stuff, in front of Caddy.
        jabart 4y ago
        TCP needs a connection to be tracked or it counts as invalid session since the tuple wouldn't existing in the connection table. So there has to be something going on or something in front of Caddy doing ip based hashing to a backend caddy instance. Most anycast setups I have seen have been geographic which means unless you move you won't get routed to the wrong endpoint. *http3/quic gets around this be setting a connection id in the UDP packets so mobile phones jumping towers can still be tracked if they get a new IP.
nykolasz 4y ago
Will have to give Caddy a try. Been a long time nginx user, but they have been very slow to implement new features (including HTTP/3), unfortunately.
- fariszr 4y ago
  It should come in the next release > Two critical capabilities for security and scalability of web applications and traffic, HTTP3 and QUIC, are coming in the next version we ship https://www.nginx.com/blog/future-of-nginx-getting-back-to-our-open-source-roots/ https://www.nginx.com/blog/future-of-nginx-getting-back-to-o...
- number6 4y ago
  I tried caddy for my home setup. Replacing nginx and traefik. I am wondering if I should ditch them in production too. I am hesitant, cause I don't see caddy as "business" software: nginx has a learning curve and configuration to master, where caddy just does the job, sane defaults and not much to "work". It just doesn't feel like work using caddy...
  - francislavoie 4y ago
    Large companies like Stripe use Caddy. It's definitely not a toy. It is, and has been production ready for many years.
    - number6 4y ago
      I guess it's just me than; all the stuff I have learned: setting the right labels for traefik, debugging it for hours. Now it's just: ``` caddy: example.com caddy.reverse_proxy: {{ upstream 8080 }} ```
      - francislavoie 4y ago
        Yep! The CDP plugin https://github.com/lucaslorentz/caddy-docker-proxy https://github.com/lucaslorentz/caddy-docker-proxy makes configuration per docker service pretty easy!
westurner 4y ago
lucaslorentz/caddy-docker-proxy works like Traefik, in that Container metadata labels are added to the reverse proxy configuration which is reloaded upon container events, which you can listen to when you subscribe to a Docker/Podman_v3 socket (which is unfortunately not read only) So, with Caddy or Traefik, a container label can enable HTTP/3 (QUIC (UDP port 1704)) for just that container. "Labels to Caddyfile conversion" https://github.com/lucaslorentz/caddy-docker-proxy#labels-to-caddyfile-conversion https://github.com/lucaslorentz/caddy-docker-proxy#labels-to... From https://news.ycombinator.com/item?id=26127879 https://news.ycombinator.com/item?id=26127879 re: containersec : > > - [docker-socket-proxy] Creates a HAproxy container that proxies limited access to the [docker] socket
- mholt 4y ago
  This is a little-known awesome project! If you like how Traefik is configured with Docker labels but want to use Caddy, well now you can have your cake and eat it, too.
- francislavoie 4y ago
  The point of the link in OP is that now in v2.6, Caddy enables HTTP/3 by default, and doesn't need to be explicitly enabled by the user. So I'm not exactly sure the point you're trying to make. But yes, CDP is an awesome project!
  - westurner 4y ago
    That is a good point. Is there any way to disable HTTP/3 support with just config? The (unversioned?) docs have: https://caddyserver.com/docs/modules/http#servers/experimental_http3 https://caddyserver.com/docs/modules/http#servers/experiment... : > servers/experimental_http3: Enable experimental HTTP/3 support. Note that HTTP/3 is not a finished standard and has extremely limited client support. This field is not subject to compatibility promises TIL caddy has Prometheus metrics support (in addition to automatic LetsEncrypt X.509 Cert renewals)
    - mholt 4y ago
westurner 4y ago
For HTTP/3 support with python clients: - aioquic supports HTTP/3 only now https://github.com/aiortc/aioquic https://github.com/aiortc/aioquic - httpx is mostly requests-compatible, supports client-side caching, and HTTP/1.1 & HTTP/2, and here's the issue for HTTP/3 support: https://github.com/encode/httpx/issues/275 https://github.com/encode/httpx/issues/275
manishsharan 4y ago
I have been using HAproxy and it has been very performant but the lack of documentation for its APIs may be the reason I will start playing with Caddy.
- TimWolla 4y ago
  I'm surprised you find the documentation lacking. What type of API are you talking about? The internal C API? Lua? Configuration? Something else? Disclosure: I'm a community contributor to HAProxy and help maintain the issue tracker.
  - manishsharan 4y ago
    The documentation is extensive but I was unable to find tutorial that could help me dynamically add/remove backend servers using the dataplane API. Also being able to dynamically refresh certificate. These are pretty much the only reasons for why I am considering Caddy.
    - TimWolla 4y ago
      The dataplane API is an external product that itself uses the "stats socket". I personally don't have any experience with the dataplane API. Changing certificates (https://docs.haproxy.org/2.6/management.html#9.3-add%20ssl%20crt-list https://docs.haproxy.org/2.6/management.html#9.3-add%20ssl%2...) and adding servers (https://docs.haproxy.org/2.6/management.html#9.3-add%20server https://docs.haproxy.org/2.6/management.html#9.3-add%20serve...) using the stats socket is both officially documented. If you find the documentation insufficient or unclear, then this can be considered a documentation bug that I recommend filing here: https://github.com/haproxy/haproxy/issues https://github.com/haproxy/haproxy/issues
seabrookmx 4y ago
Tangential, but Google is serving HTTP/3 by default out of Google Cloud now: https://cloud.google.com/blog/products/networking/cloud-cdn-and-load-balancing-support-http3 https://cloud.google.com/blog/products/networking/cloud-cdn-... Pretty cool stuff.
Matthias247 4y ago
I'm curious about the implementation, and haven't looked at the source of quic-go yet: Does it use a single UDP socket to handle datagrams for all QUIC connections, does it use connected UDP sockets per connection, or does it use multiple UDP sockets, where each handle a certain set of connections - and where an external load balancer is required to redirect to the sockets? Unfortunately there's no best answer for this: Using a single socket will allow for connection migration, but it will end up being a bottleneck in terms of scalability since it will serialize access on a lot of kernel and driver datastructures (just a single transmit/receive queue). Connected sockets avoid that, but don't allow for address migration. And doing external load balancing gets far more complex than just starting a binary - even the most simple solution requires running XDP code.
- password4321 4y ago
  > connected UDP sockets per connection I think I'm stumbling over terminology here, but UDP is connectionless, right? I think the QUIC protocol re-implements connections on top of it, but I've never heard of "connected UDP". A socket pool does seem to make sense from a performance standpoint!
  - Matthias247 4y ago
    UDP is connectionless. Connected UDP sockets are a OS construct, and don't influence anything that happens on the wire. If you "connect" a UDP socket, you don't have to use sendto and recvfrom anymore, and all packets received and sent (via send/recv APIs) are targetting the same peer. It's kind of just sets up some routes in the kernel, but doesn't make anything stateful or reliable at the networking layer.
kgersen 4y ago
Beware of a performance hit (in term of bps not req/s) if you push big data with Caddy. Go implementation of HTTP/2 already took a /5 hit over http/1.1 (Go http/2 implementation is 5x slower than Go http/1.1) With HTTP/3 our early benchs indicate /2 ot /3 from HTTP/2 (so /10 from http/1.1)
- mholt 4y ago
  Marten is in the process of making a bunch of optimizations [0], as are we within Caddy, and 2.6 includes some of those. It will only get better with time. Most users do not operate at nearly the volume required to feel the impact, including enterprises. Would be interested in repeating your experiments and knowing your real world use case and seeing how similar they are. [0]: https://github.com/lucas-clemente/quic-go/issues/3526 https://github.com/lucas-clemente/quic-go/issues/3526
  - kgersen 4y ago
    We do speed tests hence we saw the issue. See https://github.com/golang/go/issues/47840 https://github.com/golang/go/issues/47840 Yeah I agree this is a edge case, 99,99% of Caddy users don't push so much data ;) But Caddy can't be used as a big files download server for instance, unless sticking with HTTP/1.1 (that and sendfile + kTLS not been supported last time I checked?) (at least with a single http instance). Another issue for HTTP/3 is https://github.com/lucas-clemente/quic-go/wiki/UDP-Receive-Buffer-Size https://github.com/lucas-clemente/quic-go/wiki/UDP-Receive-B... Current Linux in WSL for instance has this issue. This will probably resolves over time as more and more Linux distros will take HTTP/3 requirements into account.
    - mholt 4y ago
      Caddy 2.6 has some optimizations for serving large files, including sendfile, but other optimizations too. No ktls yet, but maybe someday. It's just... kinda gnarly.
      - kgersen 4y ago
rubatuga 4y ago
Bad idea to use HTTP/3, HTTP/2 should be good enough
- mholt 4y ago
  This is actually true some of the time.
deleted 4y ago
[deleted]

284 points by fariszr 4y ago

Oh hey HN. This is in the Caddy 2.6 beta already: https://github.com/caddyserver/caddy/releases/tag/v2.6.0-beta.3 https://github.com/caddyserver/caddy/releases/tag/v2.6.0-bet... - please try it out! Thanks to Marten Seemann for maintaining the quic-go library we use. (I still haven't heard whether Go will add HTTP/3 to the standard library.) Caddy 2.6 should be the first stable release of a general-purpose server to support and enable standardized HTTP/3 by default. HTTP versions can be toggled on or off. (Meaning you can serve only HTTP/3 exclusively if you're hard-core.) PS. Caddy 2.6 will be our biggest release since 2.0. My draft release notes are about 23 KB. We're looking at huge performance improvements and powerful new features like events, virtual file systems, HTTP 103 Early Hints, and a lot of other enhancements I'm excited to show off on behalf of our collaborators!

asb 4y ago
I've just moved over to Caddy and had a really good experience in doing so (thank you to you and all the other Caddy contributors!). One thing that stuck out to me during setup is that it seemed surprising to see gzip encoding not enabled by default. Wouldn't it make sense for a server that enables https by out of the box with zero configuration and soon supports http3 to also gzip by default?
- mholt 4y ago
  Good question! When I first announced Caddy in 2015 and the server got busy, it started downloading everything as .gz files. (facepalm) (good ol' days) But that's just a fun story, not actually the reason we don't enable it by default... The tricky thing about enabling features by default is that turning them off is awkward in config. And while we like to have "magic" in Caddy, we don't like too much magic. Plus, gzip performance in Go is less good (but more memory-safe) than it is in C and assembly. Klauspost's flate implementation is very fast and that's the one we use. But even with a super-fast implementation, gzip requires memory and CPU that busy servers may be in short supply of. So it's opt-in for now. We can always change it later; going the other way is harder. (Caddy also supports serving pre-compressed files, if enabled! And starting with Caddy 2.6, those can be teleported at the speed of electricity to HTTP clients using sendfile. HTTPS also sees faster file serving in 2.6 due to optimized copying.) See also: https://serverfault.com/questions/296770/why-arent-features-like-gzip-enabled-by-default-on-many-webservers https://serverfault.com/questions/296770/why-arent-features-...
  - asb 4y ago
    Thanks for the response! I'd imagined reasoning along those lines. I'd perhaps expect that people anticipating such high load that gzipping output is problematic would be investing enough time in server setup to review and tweak these options. And for everyone else, compression by default is a good starting point. Moving from Nginx (HTTP2 off by default) to Caddy (HTTP2 on by default) you'd already need to review your setup if running a highly trafficked site to ensure any reverse proxied servers can handle the additional request load due to multiplexing. But it's obviously a trade-off and there's lots to weigh up...
  - hsbauauvhabzb 4y ago
    Why was gzipping everything a facepalm? Performance?
    - francislavoie 4y ago
      It was making the browser download files with the .gz extension, instead of serving the content in a way that the browser would display. TL;DR it was broken at the time. But it was a very long time ago.
  - layer8 4y ago
    If gzip is enabled, is the gzipped version of a static file cached in memory, or is it gzipped anew for each request?
    - mholt 4y ago
      Caddy doesn't cache responses by default, but you can enable caching with the cache plugin: https://github.com/caddyserver/cache-handler/ https://github.com/caddyserver/cache-handler/
      - oynqr 4y ago
        Does this apply to static file serving? Although I suppose relying on the OS cache might be enough.
- JZerf 4y ago
  In addition to the reasons that mholt mentioned in response to your question, enabling HTTP/GZIP compression could possibly be less secure for some web server configurations due to things like the BREACH attack. See https://en.wikipedia.org/wiki/HTTP_compression#Security_implications https://en.wikipedia.org/wiki/HTTP_compression#Security_impl... and https://en.wikipedia.org/wiki/BREACH https://en.wikipedia.org/wiki/BREACH for more info. I might be wrong but I don't think that current web serving protocols mitigate an attack like this. It might be better to default to safe settings that don't use HTTP/GZIP compression even if it might slow things down for the time being.
  - thadt 4y ago
    Eh, it's not just current web serving protocols. Any protocol where: - An application uses compression - An attacker is able to supply chosen data to it - The application compresses the attacker's data and static secret data together - The attacker is able to monitor the size of the compressed data - This can be repeated by the attacker a number of times will be vulnerable to having its secret data stolen by techniques like BREACH. If you want your secret data to stay secret, don't compress it with attacker chosen plaintext where the resulting size could be monitored.
fariszr 4y ago
Absolutely awesome! I have recently gave Caddy a shot, and immediately loved it!. I am a bit worried about the performance deficit compared to apache or Nginx, but none of my servers really have reached the scale where it actually matters. Its unfortunate that there is no dynamic brotli support, but oh well, a trade off I'm willing to make. One of things that aren't clear to me are Caddy plugins. How big of an affect do they have on peformance? Are plugins a core part of Caddy's usage? Or are they just exist to cover some edge cases? I'm thinking of using the caddy-filter and caddy-crowdsec-bouncer plugins.
- francislavoie 4y ago
  Having plugins installed has no performance incursion at all (because they don't run unless configured), but using them will, of course affect performance. But it depends on the plugin and what it does. You have to do your own benchmarking to see what effect they have to performance. Everyone has different needs and configs, so it's not possible to give authoritative benchmarks. Just try it out and see. Regarding brotli, there is a plugin you can try if you'd like https://github.com/ueffel/caddy-brotli https://github.com/ueffel/caddy-brotli, but brotli's on-the-fly performance is not that good. It works well if you pre-compress you assets though, and you can serve those with the file_server directive's precompressed option.
  - deleted 4y ago
    [deleted]
- mholt 4y ago
  Glad you liked it! Caddy's performance is competitive with nginx, but there are too many dimensions to discuss here. I guarantee that for most people, your web server will not be your bottleneck. Dynamic brotli is expensive, as brotli compression is inefficient. There are no optimized Go implementations of this yet. But Caddy can serve precompressed brotli files without any separate plugins. What do you mean by "what affect do [plugins] have on performance"? All features in Caddy are plugins, some just come installed by default: https://caddyserver.com/docs/architecture https://caddyserver.com/docs/architecture -- even its HTTP server is a plugin. (All plugins are compiled in natively.)
purim 4y ago
Does Caddy have the same issue with configuration spaghetti problem with Traefik? How does it differ from Traefik's model?
- mholt 4y ago
  I'm not super familiar with Traefik, but if you're talking about using container labels for config, no, Caddy's config is given as a JSON document through a REST API. The CLI can let you use config files if you prefer. Or the Caddyfile is a more human-friendly format that Caddy supports by default. You can actually use any config format you want if there's a config adapter for it: https://caddyserver.com/docs/config-adapters https://caddyserver.com/docs/config-adapters
- robk 4y ago
  Caddy is vastly vastly better in this regard. Traefik config files drive me insane when rewriting anything.
todotask 4y ago
Planning on trying Caddy 2.6 one of these day to see how much improvement it can make over other web servers, maybe CentmindMod would be interested in it. We all are aware local vs real world benchmark are differ.
- francislavoie 4y ago
  Caddy v2.6 is making huge improvements here. See https://github.com/caddyserver/caddy/pull/5022 https://github.com/caddyserver/caddy/pull/5022 But also, please don't take benchmarks like this as gospel. Run your own benchmarks, on your own hardware, with your own config. Someone else's config may not be ideal for your usecase.
- mholt 4y ago
  So you want to benchmark the network?
  - todotask 4y ago
    On a VPS, but no doubt that running Go app directly is more performant, I could use Go for serving static contents as well.
    - mholt 4y ago
      I have done that test. Go is not significantly faster than Caddy for serving static content.

michael_michael 4y ago

Thanks for a great piece of software that I use every day in production and just works. At first I was scared of how stupid simple it is. It feels like web servers are supposed to have giant config files with a hundred mysterious knobs to twiddle. Now I always default to Caddy, and have yet to find an instance where it didn't fit my needs. Congrats.

lynndotpy 4y ago
This is my experience as well. I've spent weekends trying to figure things out with Apache + Nginx + Let's Encrypt. Figured it out with Caddy within minutes. Really happy I found this software.
- KronisLV 4y ago
  > I've spent weekends trying to figure things out with Apache + Nginx + Let's Encrypt. Curiously, new versions of Apache include mod_md, which lets you use Let's Encrypt for SSL, without the need for something like certbot: https://httpd.apache.org/docs/2.4/mod/mod_md.html https://httpd.apache.org/docs/2.4/mod/mod_md.html I actually wrote about it in a blog post of mine, "How and why to use Apache httpd in 2022": https://blog.kronis.dev/tutorials/how-and-why-to-use-apache-httpd-in-2022 https://blog.kronis.dev/tutorials/how-and-why-to-use-apache-... That said, while the capability is there, things like the DNS-01 validation need additional work and Caddy comes with more sane defaults out of the box and arguably a way easier config format. It would also be really cool if Nginx came with its own implementation of ACME certificate provisioning out of the box, following the trend of other servers, but certbot is also perfectly passable. Either way, it seems like most of the "popular" web servers are viable for most use cases nowadays and I can say that Caddy definitely belongs in that list!

howeyc 4y ago

does it work with "tls internal" or do the certs need to be signed by an outside CA for HTTP/3 to be enabled.

mholt 4y ago
Yes it works with self-signed certificates!
- superkuh 4y ago
  I'm really glad to hear this is an option. But will browsers accessing the website accept the self-signed certs when setting up the TLS connection required by QUIC? I don't know how they behave. I'm genuinely asking.
  - francislavoie 4y ago
    Yes they do, as long as you installed Caddy's root CA cert in the browser's trust store. Caddy will attempt to install it automatically with https://github.com/smallstep/truststore https://github.com/smallstep/truststore if possible (usually requires root) but if it fails you can try again with "sudo caddy trust". You might be using a client or trust store that isn't supported though, in which case you'll need to install the root cert manually.
    - superkuh 4y ago
      What if I just want to generate and use a self signed cert without any CA involvement? I do that with HTTP all the time and it works for the public web even if the browsers like to scaremonger about it. Does a QUIC connection work without a CA (corporate) root cert being referenced?
      - francislavoie 4y ago
        To be clear, Caddy itself manages its own internal CA. It's the same thing as self signing except you have a root cert you can trust, to trust every cert Caddy generates; that's instead of having individual self signed certificates. This makes it significantly easier if you have many subdomains for each of your services, etc. It has nothing to do with corporate CAs, no external entities. QUIC/HTTP3 use the same TLS stack as H1/H2, same things are trusted. So there's no difference there.

CodesInChaos 4y ago

Since address validation is about blocking senders which forge their IP address, I think the number of connection attempts which where the client doesn't eventually validate its address should factor into the decision to enable this feature. This should rarely happen for legitimate clients (e.g. connection loss/cancellation during the first roundtrip) but always happen for IP address forgers. Or perhaps simply use the number of half-open/embryonic connections as the metric.

mholt 4y ago
Good point. Our current metric (number of active requests) is a bit naive, but mostly does the job essentially for free. If your proposed metric doesn't cost too much in memory, it might be a good fit.
- CodesInChaos 4y ago
  How does such an attacker influence your active requests metric? Is it incremented when the initial UDP packet arrives (even though the http request itself hasn't arrived yet) and decremented when it times out?
  - mholt 4y ago
    No, currently we increment it when the HTTP request is parsed. (It counts HTTP requests, not packets/connections.) This was a quick and easy way to count; but I am very open to improving this! Feel free to submit a proposal or a PR. Ref: https://github.com/caddyserver/caddy/blob/50748e19c34fc90882bbb268c95a2a0acb051752/modules/caddyhttp/server.go#L185-L186 https://github.com/caddyserver/caddy/blob/50748e19c34fc90882...
    - CodesInChaos 4y ago
      In that case, I don't think your metric works at all, since legitimate users will increase it, but attackers won't. During an attack it will either remain the same, or even go down as fewer legitimate users manage to connect. This attack is the QUIC equivalent of a SYN flood, which results in half-open connections, because the attacker is unable to complete the connection by responding to message the server sends. RequireAddressValidation corresponds enabling syn cookies. Unfortunately I don't use caddy myself, and don't have time to get involved deeper in improving this.

srameshc 4y ago

I very recently tried Caddy for the first time and it is one of the best software experiences I had especially because of it's ease of use.

41b696ef1113 4y ago
My only complaint with it is this half-way json vs standard config file situation in which it finds itself. The documents push you to the json system, but most of the available documents are in the config system.
- asb 4y ago
  There's also some handy features that are only available in the JSON format. e.g. `strip_path_suffix` and `http_redirect` for the `rewrite` handler (https://github.com/caddyserver/caddy/issues/2011#issuecomment-544203470 https://github.com/caddyserver/caddy/issues/2011#issuecommen...), which would be a more ergonomic way to set a rule to strip trailing slashes than handling via regex.
  - francislavoie 4y ago
    While it is true that some features are only possible via JSON config, the ones you mentioned are available in the Caddyfile. See the uri directive which lets you strip a suffix, and redir which performs Location redirects. Also, the issue you linked to is from an early beta of Caddy v2, and the discussion and config there no longer applies.
    - asb 4y ago
      The uri directive does expose stripping a suffix, but that's an internal rewrite, correct? For any /foo/ /bar/ /baz/ I want to redirect to /foo, /bar, or /baz. Unless I'm missing something, this needs to be done like so right now: @trailing_slash path_regexp trailing_slash ^/(.*)/$ redir @trailing_slash /{re.trailing_slash.1} 308
      - 4y ago

pachico 4y ago

Weirdly enough, among the reasons to switch from nginx to Caddy I never find the lack of observability nginx and its almost useless Prometheus exporter have. I see several posts discussing under which circumstances or use cases one might outperform the other but they never seem to care about having decent metrics. I might overrate the importance of this, who knows...

nginxlegacy 4y ago
- xet7 4y ago
  Aren't? But there is https://redbean.dev https://redbean.dev that uses Lua, and is faster than Go.
  - coder543 4y ago
    Redbean is a webserver. Go is not a webserver. Redbean cannot be "faster than Go". That's like saying that a car is faster than aluminum. Do you have benchmarks of Redbean versus Caddy that you can point to? I certainly haven't been able to find any. I also haven't been able to find any in-depth independent benchmarks of Redbean. About the only thing I could find was someone struggling to reproduce the benchmark results.[0] Personally, I've dealt with more than enough memory safety vulnerabilities caused by no programmer being good enough to write safe and correct C (or C++). Those vulnerabilities are easily prevented by choosing just about any other language people are likely to know about... so, I have no desire to deploy anything like Redbean when alternatives exist. It's cool to see the hacks they've been able to do with Cosmopolitan to make cross platform binaries, but hacks like that are not what I want for anything other than educational purposes, at least not until they've been more battle tested. [0]: https://github.com/jart/cosmopolitan/issues/72 https://github.com/jart/cosmopolitan/issues/72
    - mholt 4y ago
      Great points. I also think it's important to stop deploying C to the edge. PS. Go might someday compile for cosmopolitan if someone writes a port for it: https://github.com/golang/go/issues/51900 https://github.com/golang/go/issues/51900

0xEFF 4y ago

I've been using Caddy in a 3 node HA cluster sharing an anycast bgp address for about 18 months now and it's been fantastic. Certs "just work" across the cluster once consul is wired up. I recently added IPv6 which also "just works." greenpau/caddy-security is fantastic and "just works" for OIDC sso. mholt, thanks for recently adding the ability to bind to multiple specific IP addresses by default, this help me conserve precious public IPv4 addresses.

jabart 4y ago
This is a thing? Any docs on this as the top search is this comment.
- francislavoie 4y ago
  For which part? The comment you replied to mentions a bunch of different features, and a plugin.
  - jabart 4y ago
    The HA Cluster with an Anycast IP.
    - francislavoie 4y ago
      They're essentially just saying they run multiple instances of Caddy, and those instances are reached by clients via Anycast. This isn't anything specific to Caddy, that's networking-level stuff, in front of Caddy.
      - jabart 4y ago
        TCP needs a connection to be tracked or it counts as invalid session since the tuple wouldn't existing in the connection table. So there has to be something going on or something in front of Caddy doing ip based hashing to a backend caddy instance. Most anycast setups I have seen have been geographic which means unless you move you won't get routed to the wrong endpoint. *http3/quic gets around this be setting a connection id in the UDP packets so mobile phones jumping towers can still be tracked if they get a new IP.

nykolasz 4y ago

Will have to give Caddy a try. Been a long time nginx user, but they have been very slow to implement new features (including HTTP/3), unfortunately.

fariszr 4y ago
It should come in the next release > Two critical capabilities for security and scalability of web applications and traffic, HTTP3 and QUIC, are coming in the next version we ship https://www.nginx.com/blog/future-of-nginx-getting-back-to-our-open-source-roots/ https://www.nginx.com/blog/future-of-nginx-getting-back-to-o...
number6 4y ago
I tried caddy for my home setup. Replacing nginx and traefik. I am wondering if I should ditch them in production too. I am hesitant, cause I don't see caddy as "business" software: nginx has a learning curve and configuration to master, where caddy just does the job, sane defaults and not much to "work". It just doesn't feel like work using caddy...
- francislavoie 4y ago
  Large companies like Stripe use Caddy. It's definitely not a toy. It is, and has been production ready for many years.
  - number6 4y ago
    I guess it's just me than; all the stuff I have learned: setting the right labels for traefik, debugging it for hours. Now it's just: ``` caddy: example.com caddy.reverse_proxy: {{ upstream 8080 }} ```
    - francislavoie 4y ago
      Yep! The CDP plugin https://github.com/lucaslorentz/caddy-docker-proxy https://github.com/lucaslorentz/caddy-docker-proxy makes configuration per docker service pretty easy!

westurner 4y ago

lucaslorentz/caddy-docker-proxy works like Traefik, in that Container metadata labels are added to the reverse proxy configuration which is reloaded upon container events, which you can listen to when you subscribe to a Docker/Podman_v3 socket (which is unfortunately not read only) So, with Caddy or Traefik, a container label can enable HTTP/3 (QUIC (UDP port 1704)) for just that container. "Labels to Caddyfile conversion" https://github.com/lucaslorentz/caddy-docker-proxy#labels-to-caddyfile-conversion https://github.com/lucaslorentz/caddy-docker-proxy#labels-to... From https://news.ycombinator.com/item?id=26127879 https://news.ycombinator.com/item?id=26127879 re: containersec : > > - [docker-socket-proxy] Creates a HAproxy container that proxies limited access to the [docker] socket

mholt 4y ago
This is a little-known awesome project! If you like how Traefik is configured with Docker labels but want to use Caddy, well now you can have your cake and eat it, too.
francislavoie 4y ago
The point of the link in OP is that now in v2.6, Caddy enables HTTP/3 by default, and doesn't need to be explicitly enabled by the user. So I'm not exactly sure the point you're trying to make. But yes, CDP is an awesome project!
- westurner 4y ago
  That is a good point. Is there any way to disable HTTP/3 support with just config? The (unversioned?) docs have: https://caddyserver.com/docs/modules/http#servers/experimental_http3 https://caddyserver.com/docs/modules/http#servers/experiment... : > servers/experimental_http3: Enable experimental HTTP/3 support. Note that HTTP/3 is not a finished standard and has extremely limited client support. This field is not subject to compatibility promises TIL caddy has Prometheus metrics support (in addition to automatic LetsEncrypt X.509 Cert renewals)
  - mholt 4y ago

westurner 4y ago

For HTTP/3 support with python clients: - aioquic supports HTTP/3 only now https://github.com/aiortc/aioquic https://github.com/aiortc/aioquic - httpx is mostly requests-compatible, supports client-side caching, and HTTP/1.1 & HTTP/2, and here's the issue for HTTP/3 support: https://github.com/encode/httpx/issues/275 https://github.com/encode/httpx/issues/275

manishsharan 4y ago

I have been using HAproxy and it has been very performant but the lack of documentation for its APIs may be the reason I will start playing with Caddy.

TimWolla 4y ago
I'm surprised you find the documentation lacking. What type of API are you talking about? The internal C API? Lua? Configuration? Something else? Disclosure: I'm a community contributor to HAProxy and help maintain the issue tracker.
- manishsharan 4y ago
  The documentation is extensive but I was unable to find tutorial that could help me dynamically add/remove backend servers using the dataplane API. Also being able to dynamically refresh certificate. These are pretty much the only reasons for why I am considering Caddy.
  - TimWolla 4y ago
    The dataplane API is an external product that itself uses the "stats socket". I personally don't have any experience with the dataplane API. Changing certificates (https://docs.haproxy.org/2.6/management.html#9.3-add%20ssl%20crt-list https://docs.haproxy.org/2.6/management.html#9.3-add%20ssl%2...) and adding servers (https://docs.haproxy.org/2.6/management.html#9.3-add%20server https://docs.haproxy.org/2.6/management.html#9.3-add%20serve...) using the stats socket is both officially documented. If you find the documentation insufficient or unclear, then this can be considered a documentation bug that I recommend filing here: https://github.com/haproxy/haproxy/issues https://github.com/haproxy/haproxy/issues

seabrookmx 4y ago

Tangential, but Google is serving HTTP/3 by default out of Google Cloud now: https://cloud.google.com/blog/products/networking/cloud-cdn-and-load-balancing-support-http3 https://cloud.google.com/blog/products/networking/cloud-cdn-... Pretty cool stuff.

Matthias247 4y ago

I'm curious about the implementation, and haven't looked at the source of quic-go yet: Does it use a single UDP socket to handle datagrams for all QUIC connections, does it use connected UDP sockets per connection, or does it use multiple UDP sockets, where each handle a certain set of connections - and where an external load balancer is required to redirect to the sockets? Unfortunately there's no best answer for this: Using a single socket will allow for connection migration, but it will end up being a bottleneck in terms of scalability since it will serialize access on a lot of kernel and driver datastructures (just a single transmit/receive queue). Connected sockets avoid that, but don't allow for address migration. And doing external load balancing gets far more complex than just starting a binary - even the most simple solution requires running XDP code.

password4321 4y ago
> connected UDP sockets per connection I think I'm stumbling over terminology here, but UDP is connectionless, right? I think the QUIC protocol re-implements connections on top of it, but I've never heard of "connected UDP". A socket pool does seem to make sense from a performance standpoint!
- Matthias247 4y ago
  UDP is connectionless. Connected UDP sockets are a OS construct, and don't influence anything that happens on the wire. If you "connect" a UDP socket, you don't have to use sendto and recvfrom anymore, and all packets received and sent (via send/recv APIs) are targetting the same peer. It's kind of just sets up some routes in the kernel, but doesn't make anything stateful or reliable at the networking layer.

kgersen 4y ago

Beware of a performance hit (in term of bps not req/s) if you push big data with Caddy. Go implementation of HTTP/2 already took a /5 hit over http/1.1 (Go http/2 implementation is 5x slower than Go http/1.1) With HTTP/3 our early benchs indicate /2 ot /3 from HTTP/2 (so /10 from http/1.1)

mholt 4y ago
Marten is in the process of making a bunch of optimizations [0], as are we within Caddy, and 2.6 includes some of those. It will only get better with time. Most users do not operate at nearly the volume required to feel the impact, including enterprises. Would be interested in repeating your experiments and knowing your real world use case and seeing how similar they are. [0]: https://github.com/lucas-clemente/quic-go/issues/3526 https://github.com/lucas-clemente/quic-go/issues/3526
- kgersen 4y ago
  We do speed tests hence we saw the issue. See https://github.com/golang/go/issues/47840 https://github.com/golang/go/issues/47840 Yeah I agree this is a edge case, 99,99% of Caddy users don't push so much data ;) But Caddy can't be used as a big files download server for instance, unless sticking with HTTP/1.1 (that and sendfile + kTLS not been supported last time I checked?) (at least with a single http instance). Another issue for HTTP/3 is https://github.com/lucas-clemente/quic-go/wiki/UDP-Receive-Buffer-Size https://github.com/lucas-clemente/quic-go/wiki/UDP-Receive-B... Current Linux in WSL for instance has this issue. This will probably resolves over time as more and more Linux distros will take HTTP/3 requirements into account.
  - mholt 4y ago
    Caddy 2.6 has some optimizations for serving large files, including sendfile, but other optimizations too. No ktls yet, but maybe someday. It's just... kinda gnarly.
    - kgersen 4y ago

rubatuga 4y ago

Bad idea to use HTTP/3, HTTP/2 should be good enough

mholt 4y ago
This is actually true some of the time.

deleted 4y ago

[deleted]