12 ms·

Caddy – Open-source web server with automatic HTTPS

149 points by 6581 5y ago

rgrmrts 5y ago
Big fan of Caddy as well. I recently listened to the Sourcegraph Podcast episode with Matt Holt. Pretty great episode, would recommend! https://about.sourcegraph.com/podcast/matt-holt/ https://about.sourcegraph.com/podcast/matt-holt/
BilalBudhani 5y ago
Caddy is a great piece of software. We, at hatchbox.io, have to moved to Caddy (in v2) for all our applications and honestly the experience so far has been great. It’s a bliss not to deal with SSL certificate generation for our customers’ apps and let Caddy take care of it.
- mholt 5y ago
  That's great to hear!
ElCapitanMarkla 5y ago
Caddy is fantastic. I do the odd freelance contract making touch screen interactives for museums here. These have to run on the local host machine which usually isn't networked and Caddy is perfect for this. The last one I did had a massive DeepZoom/OpenSeaDragon image tileset and Caddy managed to serve that flawlessly where my old go to server Mongoose couldn't keep up.
- mholt 5y ago
  Neat use case! First I've heard of it. Can you tell me more about the museums?
  - ElCapitanMarkla 5y ago
    Hey sorry I just saw your reply. I've been using it to serve what are effectively static sites but some have loads of media assets. I've noticed a massive gain in speed from using it to serve huge tilesets with https://openseadragon.github.io/ https://openseadragon.github.io/ I think I've used Caddy on the latest 3 or 4 projects I've worked on which are up at the Waitangi Treaty Grounds and The Maritime Museum up in Auckland. My email is in my profile, if you flick me a message I can share some links to the latest interactive builds.
bb1234 5y ago
Yes, Caddy is awesome! About 3-4 years ago it made it super-easy (and possible) for me to serve my websites with `https`. Before that, I was using nginx, and the process of obtaining certificates seemed quite complicated. Matt Holt, thank you for creating Caddy.
- mholt 5y ago
  You're welcome ^_^ Thanks for the nice comment
cpach 5y ago
I love Caddy! It’s so much nicer to configure compared to Nginx. No plugins needed for automatic TLS certs – it’s bulilt-in. And it’s written in a memory-safe language – Go. Which also means a single static binary to deploy. I can warmly recommend it.
- junon 5y ago
  Go is not memory safe, not sure where you got that idea. Go programs can absolutely succumb to nil pointer references, memory leaks, etc.
  - jrockway 5y ago
    Safety can include crashing. It's better than returning subtly incorrect results. Yes, there are nil pointers and unnecessary data structures that people keep around, but that's miles ahead of C where an HTTP request and a missing bounds check can inject new code into the application. If you have a Go program where user input can be executed as code (no cgo, no unsafe), I'd love to see it.
    - Zababa 5y ago
      > no cgo If you're not careful, you will use c code when using os/user and net according to https://www.arp242.net/static-go.html https://www.arp242.net/static-go.html.
    - junon 5y ago
      This is splitting hairs and arguing semantics. The absolutist statement "Go is memory safe" is simply incorrect.
      - mholt 5y ago
        > The absolutist statement "Go is memory safe" is simply incorrect. That absolutist statement is also incorrect. "Go has strong memory safety guarantees" is more accurate. But neither "Go is memory safe" nor "Go is not memory safe" are correct.
kalev 5y ago
Been using Caddy now for multiple years and will use it for all future projects. Highly recommended.
zamadatix 5y ago
Caddy is great, makes it easy to just set something up and go. For some reason I can't put my finger on I was really fighting with getting what I wanted to know out of the documentation initially but either it got better since Caddy 2 released or I just got used to it/Caddy and am no longer fighting "newness".
Tomte 5y ago
I learned only recently that Apache can do automatic Letsencrypt certificate management, not with some third-party module, but with the bundled mod_md.
- KronisLV 5y ago
  This is super interesting! Of course, it's still experimental: https://httpd.apache.org/docs/trunk/mod/module-dict.html#Status https://httpd.apache.org/docs/trunk/mod/module-dict.html#Sta... "Experimental" status indicates that the module is available as part of the Apache kit, but you are on your own if you try to use it. The module is being documented for completeness, and is not necessarily supported. Nonetheless, it would certainly be something to make Apache competitive with the likes of Caddy and Nginx again, if they manage to implement the functionality in a resilient and easy to use manner to the end.
  - ViViDboarder 5y ago
    > and Nginx Does nginx now have this too? I know Caddy and Traefik do.
    - KronisLV 5y ago
      Only though Certbot, as far as i know: https://certbot.eff.org/ https://certbot.eff.org/ That said, Nginx still blows most other web servers out of the water: it has excellent performance and the configuration file format is easier to grok than that of Apache2/httpd and can let you get more done in less time. Perhaps this is why it's often chosen as the default solution for implementing an ingress, e.g. in Kubernetes. And in my eyes that's a shame, since Apache2/httpd has served me faithfully for years and is a decent web server on its own, hence the addition of this new functionality has the potential to either make Nginx adopt it as well (which would be a net positive for everyone), or simply make more people consider Apache2/httpd for their deployments (which is good, because then its decline will be slower).
PragmaticPulp 5y ago
I love what the Caddy project is doing, but every time I look into it I can’t get a straight answer about performance. One of the top search results for Caddy performance is this Caddy forum thread with a response from the Caddy author: https://caddy.community/t/performance-compared-to-nginx/7993 https://caddy.community/t/performance-compared-to-nginx/7993 But it seems to be dodging the question and linking to the most misleading possible Tweets instead of providing any actual info: > Caddy is written in Go. It will perform just as well as any other Go web servers. > Google, Netflix, Cloudflare, Fastly, and other large network companies deploy Go on their edge. You can too. What a bizarre response. They then link to a Tweet from someone who claims that Caddy performed 20X better than nginx. An impressive claim! But further investigation shows that virtually every other benchmark I can find shows the opposite situation: Nginx beating Caddy by up to 8X. Performance isn’t everything and there are plenty of situations where Caddy’s ease of use might prevail, but I get nervous when project authors are giving cagey and misleading responses to straightforward (and important) questions.
- mholt 5y ago
  What are your performance requirements?
  - PragmaticPulp 5y ago
    > What are your performance requirements? If the requirements were such that web server performance didn’t matter, I wouldn’t care and wouldn’t be asking about it. But the problem is that it doesn’t make sense to hide the performance statistics and then debate each user’s performance requirements instead of just letting the performance be a known quantity so we can all decide for ourselves.
    - mholt 5y ago
      We aren't "hiding" any performance stats. They just don't exist. It's super hard to generate them in a way that applies perfectly to everyone's use case. I'm still waiting to hear your performance requirements btw.
zinxq 5y ago
Good lesson here really. Nginx is an awesome, powerful tool but quite often, you don't need something that sophisticated. Also, because of it's abilities (and legacy) it's a learned skill to configure it. Caddy just "works" (the advent of Certs that "just work" with LetsEncrypt helped enable that). That fills a real and important use case.
Hamcha 5y ago
I'm an avid user of Caddy but I think the more time it passes and the more opinionated it gets, the worse it becomes overall. I don't object to highly opinionated takes but when your biggest selling point is how neat the configuration is to write it becomes more of a pain when it just isn't anymore. Two examples from my experience upgrading from Caddy 1 to Caddy 2: Ex.1: Caddy 1 by default bound to an arbitrary port and served via HTTP, they now default to HTTPS, which doesn't work for me since I use traefik in front of it. Bypassing this means adding a ugly ":80" in front of every single vhost I have (which is 20+), before it was just about adding "ssl" if I wanted HTTPS. Ex.2: The reverse proxy is now transparent by default, in Caddy 1 it wasn't and you just added a "transparent" flag if you wanted it, but now you can't opt out of it and you have to manually specify every header in the config. I don't think my use-case is that unusual (file server behind edge router) and yet I feel like I have to work against Caddy every so many steps of the way, at one point I will have to ask myself how much more work would it take to just go back to nginx instead.
- cunthorpe 5y ago
  To be fair, HTTPS by default makes perfect sense for a server. Of course the more complex the setup, the more you’re going against the defaults. It’s impossible to have defaults that work for everyone, so they chose the most common/desirable/sellable default.
  - mook 5y ago
    I'd argue that HTTPS by default makes perfect sense for a _public facing_ server. For something on an internal network it doesn't make as much sense, especially since that makes using the default path for getting certificates difficult (can't get certs if you don't have DNS). I think what I'm trying to say is that Caddy was initially great for me (serving static files on the local network by IP address as an easy way to transfer files), but hasn't been that for ages.
    - francislavoie 5y ago
      FWIW, Caddy can act as its own CA, so it can issue certs for any private site, and you can add the root CA cert to any machines necessary. > but hasn't been that for ages Huh? It still is. This is all you need: :80 { root * /path/to/files file_server }
fabiospampinato 5y ago
I can't recommend Caddy enough. I switched to that for a simple server of mine and I was able to throw away a whole bunch of junk as a result. The only thing I wasn't so sure about was the DSL that should be used for configuring Caddy, but after using it I have to say it's very well designed, and rewriting the same configuration in something like JSON would only complicate things in the end.
tiffanyh 5y ago
What makes Caddy setup of Lets Encrypt so much easier? I frequently hear that being a huge reason for Caddy use but looking at the documentation for NGINX [0] and Caddy [1], it doesn't seem to be much different. [0] https://www.nginx.com/blog/using-free-ssltls-certificates-from-lets-encrypt-with-nginx/ https://www.nginx.com/blog/using-free-ssltls-certificates-fr... [1] https://caddyserver.com/docs/automatic-https https://caddyserver.com/docs/automatic-https
- nullwarp 5y ago
  It has no third party tool required
- mholt 5y ago
  NGINX requires external tooling (e.g. certbot), Caddy's auto-HTTPS is built-in. Caddy's auto-HTTPS logic is much smarter and more robust than certbot or any other ACME client. In general, Caddy sites stay up when other sites will go down due to HTTPS/certificate issues. Certbot+cron is much more brittle than Caddy's embedded handling of certificates. Plus, Caddy has stronger memory safety guarantees than a C program like NGINX. Caddy automatically staples and caches OCSP responses by default, as well. OCSP responder outages? No problem. This sometimes brings sites down (esp. Must-Staple certs in Firefox) because other servers' OCSP stapling implementations are not as robust. I remember when gnu.org went down for a while because of this. Certificate got revoked? No problem, Caddy will automatically detect that via OCSP and replace it for you. Internal throttling and retry logic makes it more resilient to domain validation problems. Caddy will fall back to a secondary CA if your first CA is having trouble issuing a certificate (i.e. multiple default CAs). Not to mention, when using Let's Encrypt, Caddy will retry with its staging endpoint to help avoid rate limit issues. Plus, deploy multiple Caddy instances as a cluster and they will automatically share and coordinate management of certificates and OCSP staples. (Doing this is as simple as configuring each Caddy instance to use the same storage backend.) It's very quick to get started and try out HTTPS: https://caddyserver.com/docs/quick-starts/https https://caddyserver.com/docs/quick-starts/https
  - cwaffles 5y ago
    Nginx has an excellent track record for CVEs: https://www.cvedetails.com/product/17956/Nginx-Nginx.html?vendor_id=10048 https://www.cvedetails.com/product/17956/Nginx-Nginx.html?ve...
sneak 5y ago
Remember when you had to put a cdrom into a special plastic tray before you could put it into the cd drive?
Shadonototra 5y ago
Go such a wonderful language and stack
Maskawanian 5y ago
I tried to use caddy, but I wasn't able to find a currently supported module for IP filtering just a bunch of old unsubscribed ones, does anyone know if this is currently possible?
- mholt 5y ago
  Yes, this is very common if I understand you correctly. You want the remote_ip matcher: https://caddyserver.com/docs/caddyfile/matchers#remote-ip https://caddyserver.com/docs/caddyfile/matchers#remote-ip
- francislavoie 5y ago
  Depends what you mean by "IP filtering", but there's a built-in 'remote_ip' matcher that can let you handle requests from specific IPs/CIDRs differently (such as aborting/forcefully closing the connection): https://caddyserver.com/docs/caddyfile/matchers#remote-ip https://caddyserver.com/docs/caddyfile/matchers#remote-ip
ksec 5y ago
>We already have a number of features to land in the upcoming 2.1 release: Well it turns out the latest release is v2.4.6 on Github already. I wish there is a changlog / release note link on the homepage somewhere. Basically I think they need to delete the whole V2 section. I find it quite confusing.
- mholt 5y ago
  Yeah, sorry. Been so focused on dev that sometimes parts of the website get left behind.
  - ksec 5y ago
    Its all good. No need to apologise :) Apart from Release Note, and V2 section. You may want to include a section of Big Name Clients / users that are already using Caddy under the fold or right before the fewer moving parts.
buybackoff 5y ago
With acme.sh, 80% time is spent on configuring Cloudflare API keys, then the rest 20% it's just editing and running a couple of commands (so both steps are fast). The certificates are installed in any dir, e.g. `etc/nginx/ssl/mysite.com`, and auto renewed, with a custom shell command after, e.g. to restart NGINX. Tried Caddy before learning acme.sh. Even looked in the source code. Nice idea, but it's a new custom thing. Spent a very small learning time on NGINX, everything is googleable, multiple tutorials, Q&As, samples on every possible subject and usecase. E.g. homelab Guacamole was freezing, the first search result was the fix - not trivial, but just copy-paste several config lines. It's hard to beat the performance and simplicity of NGINX. It does not require to learn all possible tweaks from the start, but learn as you go from 5-lines config SSL reverse proxy and up to whatever complexity is required.
crummy 5y ago
I rave about Caddy. Worth it just to never have to think about HTTPS again. We used to have an Nginx+certbot docker container and it was fragile at best, but migration to Caddy was very easy. One feature it has that I love: If a request comes in but a backend is down, instead of returning an error it'll wait a set amount of time for the backend to come up, then forward the request. This means you can do zero-downtime deploys with nothing more complicated than "docker-compose up -d". I know that there are better ways to do true seamless deploys but they all introduce enough complexity to a system that we want to keep as simple as possible.
- number6 5y ago
  For docker I use traefik2 and i really like how it integrates; you can also set up a zero downtime setup
- mholt 5y ago
  I like that feature too! Someone else just wrote about it this week: https://til.simonwillison.net/caddy/pause-retry-traffic https://til.simonwillison.net/caddy/pause-retry-traffic
zwarag 5y ago
Anyone switch from traefik to caddy and can describe why they did it?
- mholt 5y ago
  https://twitter.com/_stoakes/status/1425700401022705669 https://twitter.com/_stoakes/status/1425700401022705669
- qbasic_forever 5y ago
  Better documentation for Caddy 2 vs. Traefik 2 IMHO. I still don't understand half of the weird labels and incantations Traefik wants for simple container proxy scenarios. Caddy's config is just a straightforward file if you want, like nginx or apache.
joshstrange 5y ago
I love using caddy for my self-hosted apps, it's so easy to setup and with a little work I got SSO working (v1) for all my apps which is awesome. I no longer have to maintain a different password for each service I run and once I log into 1 of them, I'm logged into all of them. I need to port my setup to v2 but I've been lazy about doing it since some of the plugins/config I use has changed or been removed in v2. I've never used Caddy for production because I've not been in the position to dictate using it or, before switching off nginx/lighttpd, benchmarking it but the ease of use for SSL alone is bar-none.
cosmotic 5y ago
In addition to the auto HTTPS which many mentioned, it also has simplified PHP support. I also find the configuration files are easier to write with more consistent syntax. They are smaller than similar nginx rules, and it's easy to test them because caddy will automatically reload when the file is saved.
KronisLV 5y ago
In my eyes, Caddy is a lovely web server that works pretty well as ingress for container clusters (e.g. Nomad, Docker Swarm etc.). That said, i can't help but to feel that v1 was easier and in some ways nicer to use than v2, even though it's abandoned at this point. That said, i have certain grievances with most of the web servers out there. Apache2/httpd - actually decently usable even nowadays, but if the fragmentation of service names (httpd vs apache2, with additional scripts like a2enmod) between different distros doesn't hurt it, then the configuration format and how it does reverse proxying and path rewriting most certainly will. The performance is still passable, no matter what anyone says, my applications still have been the bottleneck in approx. 95% of the cases, though that might change with frameworks like Vert.X or such. The further down you scroll, the less user friendly it becomes: https://httpd.apache.org/docs/2.4/rewrite/remapping.html https://httpd.apache.org/docs/2.4/rewrite/remapping.html Admittedly, the docs themselves are good, though, despite the syntax that you're stuck with. Nginx - recently migrated my ingress to it at work, seems pretty okay so far, the configuration format seems to make a bit more sense and probably lies somewhere between Apache and Caddy as far as its ease of use and pleasantness goes. I no longer even need rewrite rules to get websockets working properly, which is nice. And my containers can have all of the necessary config in a single file vs the unnecessary boilerplate fragmentation that httpd forces upon me. For example, both of these seem more passable to me when compared to Apache2: https://docs.nginx.com/nginx/admin-guide/web-server/reverse-proxy/ https://docs.nginx.com/nginx/admin-guide/web-server/reverse-... and https://www.nginx.com/blog/creating-nginx-rewrite-rules/ https://www.nginx.com/blog/creating-nginx-rewrite-rules/ Currently, my biggest gripe is that Nginx kills itself when it cannot resolve an upstream host, for example, while Docker containers are still starting, their health checks haven't passed and therefore their DNS records also haven't been created: https://stackoverflow.com/questions/42720618/docker-nginx-stopped-emerg-11-host-not-found-in-upstream https://stackoverflow.com/questions/42720618/docker-nginx-st... The worst part is that none of the suggested answers actually work for me, so i can't have a single Nginx instance be in front of the development environment with about 20 containers, a few of those being down when Nginx is being restarted will not let many of them be used until the startup finishes. Unacceptable. Caddy - as stated before, i liked v1 more than v2, though the project itself is pretty close to as good as a web server might get. What i don't enjoy is them taking the old docs offline, merely letting you download an archive, nor am i a fan of the current docs, since at the current point in time they are a bit like running "tar --usage": https://caddyserver.com/docs/caddyfile/directives/reverse_proxy https://caddyserver.com/docs/caddyfile/directives/reverse_pr... It's nice that there are a few examples for the common use cases, but there probably could be even more, just look at what the PHP documentation has at the bottom for a good example: https://www.php.net/manual/en/function.str-replace.php https://www.php.net/manual/en/function.str-replace.php (crowd sourced, but i like the idea of letting the community contribute useful information like that). Apart from that, some of the behavior is weird and you will get a 200 when you'd expect to get a 502/404 in most other web servers: https://caddy.community/t/why-does-caddy-return-an-empty-200-response-for-a-non-configured-host/7685 https://caddy.community/t/why-does-caddy-return-an-empty-200... which will sometimes be misleading ("Huh, i'm not getting any data in the response to my request, even though the status is 200 in my log, weird...") Also, i remember when v1 had this "fail-fast" habit of shutting down the entire server when renewing/obtaining a certificate failed, something that i utterly hate when web servers do: https://github.com/caddyserver/caddy/issues/642 https://github.com/caddyserver/caddy/issues/642 Admittedly, things are a bit better now: https://caddyserver.com/docs/automatic-https#errors https://caddyserver.com/docs/automatic-https#errors I just don't understand why web servers can be so opinionated about these things and not provide something like "failure_action" in Docker Compose (https://docs.docker.com/compose/compose-file/compose-file-v3/#update_config https://docs.docker.com/compose/compose-file/compose-file-v3...) so that people can choose between either stopping everything as soon as problems manifest, or continuing with a "best effort" strategy. If i'm hosting 100 sites behind a reverse proxy, i don't want 99 to be taken down just because 1 of them was misconfigured, the web server should be able to throw out a warning about that one host if i tell it to, and proceed to run the rest 99 as instructed. When no web server forces me to cope with such brittleness will be a good day.
doteka 5y ago
Caddy is the best. I use it as the static file server and reverse proxy for several side projects running in docker-compose. What I like most about it is how little config you need for reasonable defaults that would require 300 lines of nginx boilerplate. If I could wish for one thing though, I’d really like the functionality to get let’s encrypt certs while being proxied through cloudflare to be built in. Right now it requires building a custom caddy with a plug-in, which is a lot of hassle for such a vanilla setup.
- mholt 5y ago
  Thanks for your feedback. We try to avoid tight integration with specific, third-party providers as much as possible to keep Caddy light and flexible. Cloudflare is popular, but is also not a majority use case for Caddy users, in our experience. The Cloudflare plugin has only 16,000 downloads (and Route53 has 10,000)... out of over a million custom builds (not to mention 100M+ Docker pulls, or other ways of installing Caddy). So it's not enough of a standout to merit inclusion by default.
  - doteka 5y ago
    Thanks for the reply, mholt! I would have expected way higher numbers there, but going by that ratio you’re absolutely right. Regardless, thanks for the great work on Caddy, it’s rock solid.
orware 5y ago
I'm not sure I understand the criticism in the top comment in this thread about Caddy's performance. I don't think I ever really decided to use Caddy solely based on performance itself (although already fairly familiar with Go's built-in libraries for creating HTTP servers gave me confidence that it should be fairly quickly...I wasn't overly concerned about how it directly compared to Nginx/Apache/IIS/etc.). For me, the biggest selling point was the Caddyfile format, which felt a lot more human readable than the .htaccess or Nginx configuration options I had seen over the years (I know with Caddy v2 there's the possibly now more preferred JSON option, but I'm still a sucker for the original Caddyfile format myself :-). There's still a lot I don't know about for all of the web server options out there in general so I don't claim to be an expert in any of them, but Caddy was the first one I felt comfortable using new situations to simplify things in my environment. For example, Caddy's reverse proxy functionality (and the way it is setup within the Caddyfile) is what finally made that particularly capability "click" for myself over 5 years ago when I first came across Caddy (even though I had seen info how Nginx + Apache were used together in the years prior, with Nginx providing reverse proxy access to Apache in those hybrid setups, it wasn't something that had "clicked" for me being aware of those situations). Since my environment is fairly simple (no real super high demand situations requiring lots of load balancing) we mainly use Caddy to provide automatic SSL capabilities and act as the publicly accessible endpoint in our environment, and use the reverse proxy functionality to tie Caddy into the various other internal servers that need that SSL termination. I've also been able to take advantage of the built-in static file server for an internal website need and it's actually something I really think is a nice/neat feature to have available (especially due to its ability for simple server side includes, allowing for relatively easy simple templating). Separate from that, I've also experienced first hand the personal responses that Matt Holt has provided to myself and to many others in the Caddy Community, and I'm definitely appreciative of the work he's put into Caddy (along with others of course) and I hope some of negativity in this thread is able to be easily shrugged off by Matt, because I could imagine it can be exhausting to deal with when he genuinely does care and puts a lot of thought/effort into the responses he provides.
- mholt 5y ago
  Thank you for sharing your experiences. Lots of positive feedback here has helped to offset the negative, so it's only a little exhausting today. :)
itsjloh 5y ago
I really like Caddy and use it on a few projects. Its a good piece of software that generally just works. Their latest security incident leaves me feeling somewhat uneasy though. The only announcement for it was on twitter[1] and a footnote on one of the releases[2]. I don't believe there was ever a mention of the incident on the official forum so if you weren't following them on Twitter you'd never find out. The general messaging around it was "a GitHub bug caused it" and its never really been followed up on publicly :\ [1]: https://twitter.com/caddyserver/status/1338324878441603073 https://twitter.com/caddyserver/status/1338324878441603073 [2]: https://github.com/caddyserver/caddy/releases/tag/v2.2.1 https://github.com/caddyserver/caddy/releases/tag/v2.2.1
- francislavoie 5y ago
  It wasn't a security incident, actually. It's true that "a GitHub bug caused it". It wasn't malicious. TLDR, a contributor made a tag on their own fork of Caddy, and for some reason our next release used their tag, because it turns out forks in GitHub aren't actual separate repos, but rather "still technically the same repo". It's really strange. Wasn't that contributor's fault either, they had no idea that would happen. All that happened is that the v2.2.2 git tag wasn't properly signed with Matt's signing key. There was no problem with the code at all. We've put in place checks during our CI actions to ensure that releases are always verified to be signed by Matt's key. See https://github.com/caddyserver/caddy/pull/3932 https://github.com/caddyserver/caddy/pull/3932 Specifically, commit https://github.com/caddyserver/caddy/commit/1d473ae924f0d52cb5a8f525e3e02b5167a4a5f1 https://github.com/caddyserver/caddy/commit/1d473ae924f0d52c... (which you'll notice is _not_ part of the Caddy repo, it was actually from a fork which was later deleted) triggered this GitHub Actions job for the release https://github.com/caddyserver/caddy/actions/runs/392345801 https://github.com/caddyserver/caddy/actions/runs/392345801 which we cancelled when we noticed it was happening. But we can't remove the tag from Go's caching server https://pkg.go.dev/github.com/caddyserver/caddy/v2@v2.2.2 https://pkg.go.dev/github.com/caddyserver/caddy/v2@v2.2.2 so it's kinda there forever. More context: https://twitter.com/mholt6/status/1337879764317564928 https://twitter.com/mholt6/status/1337879764317564928, https://twitter.com/danlsgiga/status/1338859470227529732 https://twitter.com/danlsgiga/status/1338859470227529732, that whole twitter thread has many branches, so click around to get the whole conversation.

149 points by 6581 5y ago

Big fan of Caddy as well. I recently listened to the Sourcegraph Podcast episode with Matt Holt. Pretty great episode, would recommend! https://about.sourcegraph.com/podcast/matt-holt/ https://about.sourcegraph.com/podcast/matt-holt/

BilalBudhani 5y ago

Caddy is a great piece of software. We, at hatchbox.io, have to moved to Caddy (in v2) for all our applications and honestly the experience so far has been great. It’s a bliss not to deal with SSL certificate generation for our customers’ apps and let Caddy take care of it.

mholt 5y ago
That's great to hear!

ElCapitanMarkla 5y ago

Caddy is fantastic. I do the odd freelance contract making touch screen interactives for museums here. These have to run on the local host machine which usually isn't networked and Caddy is perfect for this. The last one I did had a massive DeepZoom/OpenSeaDragon image tileset and Caddy managed to serve that flawlessly where my old go to server Mongoose couldn't keep up.

mholt 5y ago
Neat use case! First I've heard of it. Can you tell me more about the museums?
- ElCapitanMarkla 5y ago
  Hey sorry I just saw your reply. I've been using it to serve what are effectively static sites but some have loads of media assets. I've noticed a massive gain in speed from using it to serve huge tilesets with https://openseadragon.github.io/ https://openseadragon.github.io/ I think I've used Caddy on the latest 3 or 4 projects I've worked on which are up at the Waitangi Treaty Grounds and The Maritime Museum up in Auckland. My email is in my profile, if you flick me a message I can share some links to the latest interactive builds.

bb1234 5y ago

Yes, Caddy is awesome! About 3-4 years ago it made it super-easy (and possible) for me to serve my websites with `https`. Before that, I was using nginx, and the process of obtaining certificates seemed quite complicated. Matt Holt, thank you for creating Caddy.

mholt 5y ago
You're welcome ^_^ Thanks for the nice comment

cpach 5y ago

I love Caddy! It’s so much nicer to configure compared to Nginx. No plugins needed for automatic TLS certs – it’s bulilt-in. And it’s written in a memory-safe language – Go. Which also means a single static binary to deploy. I can warmly recommend it.

junon 5y ago
Go is not memory safe, not sure where you got that idea. Go programs can absolutely succumb to nil pointer references, memory leaks, etc.
- jrockway 5y ago
  Safety can include crashing. It's better than returning subtly incorrect results. Yes, there are nil pointers and unnecessary data structures that people keep around, but that's miles ahead of C where an HTTP request and a missing bounds check can inject new code into the application. If you have a Go program where user input can be executed as code (no cgo, no unsafe), I'd love to see it.
  - Zababa 5y ago
    > no cgo If you're not careful, you will use c code when using os/user and net according to https://www.arp242.net/static-go.html https://www.arp242.net/static-go.html.
  - junon 5y ago
    This is splitting hairs and arguing semantics. The absolutist statement "Go is memory safe" is simply incorrect.
    - mholt 5y ago
      > The absolutist statement "Go is memory safe" is simply incorrect. That absolutist statement is also incorrect. "Go has strong memory safety guarantees" is more accurate. But neither "Go is memory safe" nor "Go is not memory safe" are correct.

kalev 5y ago

Been using Caddy now for multiple years and will use it for all future projects. Highly recommended.

zamadatix 5y ago

Caddy is great, makes it easy to just set something up and go. For some reason I can't put my finger on I was really fighting with getting what I wanted to know out of the documentation initially but either it got better since Caddy 2 released or I just got used to it/Caddy and am no longer fighting "newness".

Tomte 5y ago

I learned only recently that Apache can do automatic Letsencrypt certificate management, not with some third-party module, but with the bundled mod_md.

KronisLV 5y ago
This is super interesting! Of course, it's still experimental: https://httpd.apache.org/docs/trunk/mod/module-dict.html#Status https://httpd.apache.org/docs/trunk/mod/module-dict.html#Sta... "Experimental" status indicates that the module is available as part of the Apache kit, but you are on your own if you try to use it. The module is being documented for completeness, and is not necessarily supported. Nonetheless, it would certainly be something to make Apache competitive with the likes of Caddy and Nginx again, if they manage to implement the functionality in a resilient and easy to use manner to the end.
- ViViDboarder 5y ago
  > and Nginx Does nginx now have this too? I know Caddy and Traefik do.
  - KronisLV 5y ago
    Only though Certbot, as far as i know: https://certbot.eff.org/ https://certbot.eff.org/ That said, Nginx still blows most other web servers out of the water: it has excellent performance and the configuration file format is easier to grok than that of Apache2/httpd and can let you get more done in less time. Perhaps this is why it's often chosen as the default solution for implementing an ingress, e.g. in Kubernetes. And in my eyes that's a shame, since Apache2/httpd has served me faithfully for years and is a decent web server on its own, hence the addition of this new functionality has the potential to either make Nginx adopt it as well (which would be a net positive for everyone), or simply make more people consider Apache2/httpd for their deployments (which is good, because then its decline will be slower).

PragmaticPulp 5y ago

I love what the Caddy project is doing, but every time I look into it I can’t get a straight answer about performance. One of the top search results for Caddy performance is this Caddy forum thread with a response from the Caddy author: https://caddy.community/t/performance-compared-to-nginx/7993 https://caddy.community/t/performance-compared-to-nginx/7993 But it seems to be dodging the question and linking to the most misleading possible Tweets instead of providing any actual info: > Caddy is written in Go. It will perform just as well as any other Go web servers. > Google, Netflix, Cloudflare, Fastly, and other large network companies deploy Go on their edge. You can too. What a bizarre response. They then link to a Tweet from someone who claims that Caddy performed 20X better than nginx. An impressive claim! But further investigation shows that virtually every other benchmark I can find shows the opposite situation: Nginx beating Caddy by up to 8X. Performance isn’t everything and there are plenty of situations where Caddy’s ease of use might prevail, but I get nervous when project authors are giving cagey and misleading responses to straightforward (and important) questions.

mholt 5y ago
What are your performance requirements?
- PragmaticPulp 5y ago
  > What are your performance requirements? If the requirements were such that web server performance didn’t matter, I wouldn’t care and wouldn’t be asking about it. But the problem is that it doesn’t make sense to hide the performance statistics and then debate each user’s performance requirements instead of just letting the performance be a known quantity so we can all decide for ourselves.
  - mholt 5y ago
    We aren't "hiding" any performance stats. They just don't exist. It's super hard to generate them in a way that applies perfectly to everyone's use case. I'm still waiting to hear your performance requirements btw.

zinxq 5y ago

Good lesson here really. Nginx is an awesome, powerful tool but quite often, you don't need something that sophisticated. Also, because of it's abilities (and legacy) it's a learned skill to configure it. Caddy just "works" (the advent of Certs that "just work" with LetsEncrypt helped enable that). That fills a real and important use case.

Hamcha 5y ago

I'm an avid user of Caddy but I think the more time it passes and the more opinionated it gets, the worse it becomes overall. I don't object to highly opinionated takes but when your biggest selling point is how neat the configuration is to write it becomes more of a pain when it just isn't anymore. Two examples from my experience upgrading from Caddy 1 to Caddy 2: Ex.1: Caddy 1 by default bound to an arbitrary port and served via HTTP, they now default to HTTPS, which doesn't work for me since I use traefik in front of it. Bypassing this means adding a ugly ":80" in front of every single vhost I have (which is 20+), before it was just about adding "ssl" if I wanted HTTPS. Ex.2: The reverse proxy is now transparent by default, in Caddy 1 it wasn't and you just added a "transparent" flag if you wanted it, but now you can't opt out of it and you have to manually specify every header in the config. I don't think my use-case is that unusual (file server behind edge router) and yet I feel like I have to work against Caddy every so many steps of the way, at one point I will have to ask myself how much more work would it take to just go back to nginx instead.

cunthorpe 5y ago
To be fair, HTTPS by default makes perfect sense for a server. Of course the more complex the setup, the more you’re going against the defaults. It’s impossible to have defaults that work for everyone, so they chose the most common/desirable/sellable default.
- mook 5y ago
  I'd argue that HTTPS by default makes perfect sense for a _public facing_ server. For something on an internal network it doesn't make as much sense, especially since that makes using the default path for getting certificates difficult (can't get certs if you don't have DNS). I think what I'm trying to say is that Caddy was initially great for me (serving static files on the local network by IP address as an easy way to transfer files), but hasn't been that for ages.
  - francislavoie 5y ago
    FWIW, Caddy can act as its own CA, so it can issue certs for any private site, and you can add the root CA cert to any machines necessary. > but hasn't been that for ages Huh? It still is. This is all you need: :80 { root * /path/to/files file_server }

fabiospampinato 5y ago

I can't recommend Caddy enough. I switched to that for a simple server of mine and I was able to throw away a whole bunch of junk as a result. The only thing I wasn't so sure about was the DSL that should be used for configuring Caddy, but after using it I have to say it's very well designed, and rewriting the same configuration in something like JSON would only complicate things in the end.

tiffanyh 5y ago

What makes Caddy setup of Lets Encrypt so much easier? I frequently hear that being a huge reason for Caddy use but looking at the documentation for NGINX [0] and Caddy [1], it doesn't seem to be much different. [0] https://www.nginx.com/blog/using-free-ssltls-certificates-from-lets-encrypt-with-nginx/ https://www.nginx.com/blog/using-free-ssltls-certificates-fr... [1] https://caddyserver.com/docs/automatic-https https://caddyserver.com/docs/automatic-https

nullwarp 5y ago
It has no third party tool required
mholt 5y ago
NGINX requires external tooling (e.g. certbot), Caddy's auto-HTTPS is built-in. Caddy's auto-HTTPS logic is much smarter and more robust than certbot or any other ACME client. In general, Caddy sites stay up when other sites will go down due to HTTPS/certificate issues. Certbot+cron is much more brittle than Caddy's embedded handling of certificates. Plus, Caddy has stronger memory safety guarantees than a C program like NGINX. Caddy automatically staples and caches OCSP responses by default, as well. OCSP responder outages? No problem. This sometimes brings sites down (esp. Must-Staple certs in Firefox) because other servers' OCSP stapling implementations are not as robust. I remember when gnu.org went down for a while because of this. Certificate got revoked? No problem, Caddy will automatically detect that via OCSP and replace it for you. Internal throttling and retry logic makes it more resilient to domain validation problems. Caddy will fall back to a secondary CA if your first CA is having trouble issuing a certificate (i.e. multiple default CAs). Not to mention, when using Let's Encrypt, Caddy will retry with its staging endpoint to help avoid rate limit issues. Plus, deploy multiple Caddy instances as a cluster and they will automatically share and coordinate management of certificates and OCSP staples. (Doing this is as simple as configuring each Caddy instance to use the same storage backend.) It's very quick to get started and try out HTTPS: https://caddyserver.com/docs/quick-starts/https https://caddyserver.com/docs/quick-starts/https
- cwaffles 5y ago
  Nginx has an excellent track record for CVEs: https://www.cvedetails.com/product/17956/Nginx-Nginx.html?vendor_id=10048 https://www.cvedetails.com/product/17956/Nginx-Nginx.html?ve...

sneak 5y ago

Remember when you had to put a cdrom into a special plastic tray before you could put it into the cd drive?

Shadonototra 5y ago

Go such a wonderful language and stack

Maskawanian 5y ago

I tried to use caddy, but I wasn't able to find a currently supported module for IP filtering just a bunch of old unsubscribed ones, does anyone know if this is currently possible?

mholt 5y ago
Yes, this is very common if I understand you correctly. You want the remote_ip matcher: https://caddyserver.com/docs/caddyfile/matchers#remote-ip https://caddyserver.com/docs/caddyfile/matchers#remote-ip
francislavoie 5y ago
Depends what you mean by "IP filtering", but there's a built-in 'remote_ip' matcher that can let you handle requests from specific IPs/CIDRs differently (such as aborting/forcefully closing the connection): https://caddyserver.com/docs/caddyfile/matchers#remote-ip https://caddyserver.com/docs/caddyfile/matchers#remote-ip

ksec 5y ago

>We already have a number of features to land in the upcoming 2.1 release: Well it turns out the latest release is v2.4.6 on Github already. I wish there is a changlog / release note link on the homepage somewhere. Basically I think they need to delete the whole V2 section. I find it quite confusing.

mholt 5y ago
Yeah, sorry. Been so focused on dev that sometimes parts of the website get left behind.
- ksec 5y ago
  Its all good. No need to apologise :) Apart from Release Note, and V2 section. You may want to include a section of Big Name Clients / users that are already using Caddy under the fold or right before the fewer moving parts.

buybackoff 5y ago

With acme.sh, 80% time is spent on configuring Cloudflare API keys, then the rest 20% it's just editing and running a couple of commands (so both steps are fast). The certificates are installed in any dir, e.g. `etc/nginx/ssl/mysite.com`, and auto renewed, with a custom shell command after, e.g. to restart NGINX. Tried Caddy before learning acme.sh. Even looked in the source code. Nice idea, but it's a new custom thing. Spent a very small learning time on NGINX, everything is googleable, multiple tutorials, Q&As, samples on every possible subject and usecase. E.g. homelab Guacamole was freezing, the first search result was the fix - not trivial, but just copy-paste several config lines. It's hard to beat the performance and simplicity of NGINX. It does not require to learn all possible tweaks from the start, but learn as you go from 5-lines config SSL reverse proxy and up to whatever complexity is required.

crummy 5y ago

I rave about Caddy. Worth it just to never have to think about HTTPS again. We used to have an Nginx+certbot docker container and it was fragile at best, but migration to Caddy was very easy. One feature it has that I love: If a request comes in but a backend is down, instead of returning an error it'll wait a set amount of time for the backend to come up, then forward the request. This means you can do zero-downtime deploys with nothing more complicated than "docker-compose up -d". I know that there are better ways to do true seamless deploys but they all introduce enough complexity to a system that we want to keep as simple as possible.

number6 5y ago
For docker I use traefik2 and i really like how it integrates; you can also set up a zero downtime setup
mholt 5y ago
I like that feature too! Someone else just wrote about it this week: https://til.simonwillison.net/caddy/pause-retry-traffic https://til.simonwillison.net/caddy/pause-retry-traffic

zwarag 5y ago

Anyone switch from traefik to caddy and can describe why they did it?

mholt 5y ago
https://twitter.com/_stoakes/status/1425700401022705669 https://twitter.com/_stoakes/status/1425700401022705669
qbasic_forever 5y ago
Better documentation for Caddy 2 vs. Traefik 2 IMHO. I still don't understand half of the weird labels and incantations Traefik wants for simple container proxy scenarios. Caddy's config is just a straightforward file if you want, like nginx or apache.

joshstrange 5y ago

I love using caddy for my self-hosted apps, it's so easy to setup and with a little work I got SSO working (v1) for all my apps which is awesome. I no longer have to maintain a different password for each service I run and once I log into 1 of them, I'm logged into all of them. I need to port my setup to v2 but I've been lazy about doing it since some of the plugins/config I use has changed or been removed in v2. I've never used Caddy for production because I've not been in the position to dictate using it or, before switching off nginx/lighttpd, benchmarking it but the ease of use for SSL alone is bar-none.

cosmotic 5y ago

In addition to the auto HTTPS which many mentioned, it also has simplified PHP support. I also find the configuration files are easier to write with more consistent syntax. They are smaller than similar nginx rules, and it's easy to test them because caddy will automatically reload when the file is saved.

KronisLV 5y ago

In my eyes, Caddy is a lovely web server that works pretty well as ingress for container clusters (e.g. Nomad, Docker Swarm etc.). That said, i can't help but to feel that v1 was easier and in some ways nicer to use than v2, even though it's abandoned at this point. That said, i have certain grievances with most of the web servers out there. Apache2/httpd - actually decently usable even nowadays, but if the fragmentation of service names (httpd vs apache2, with additional scripts like a2enmod) between different distros doesn't hurt it, then the configuration format and how it does reverse proxying and path rewriting most certainly will. The performance is still passable, no matter what anyone says, my applications still have been the bottleneck in approx. 95% of the cases, though that might change with frameworks like Vert.X or such. The further down you scroll, the less user friendly it becomes: https://httpd.apache.org/docs/2.4/rewrite/remapping.html https://httpd.apache.org/docs/2.4/rewrite/remapping.html Admittedly, the docs themselves are good, though, despite the syntax that you're stuck with. Nginx - recently migrated my ingress to it at work, seems pretty okay so far, the configuration format seems to make a bit more sense and probably lies somewhere between Apache and Caddy as far as its ease of use and pleasantness goes. I no longer even need rewrite rules to get websockets working properly, which is nice. And my containers can have all of the necessary config in a single file vs the unnecessary boilerplate fragmentation that httpd forces upon me. For example, both of these seem more passable to me when compared to Apache2: https://docs.nginx.com/nginx/admin-guide/web-server/reverse-proxy/ https://docs.nginx.com/nginx/admin-guide/web-server/reverse-... and https://www.nginx.com/blog/creating-nginx-rewrite-rules/ https://www.nginx.com/blog/creating-nginx-rewrite-rules/ Currently, my biggest gripe is that Nginx kills itself when it cannot resolve an upstream host, for example, while Docker containers are still starting, their health checks haven't passed and therefore their DNS records also haven't been created: https://stackoverflow.com/questions/42720618/docker-nginx-stopped-emerg-11-host-not-found-in-upstream https://stackoverflow.com/questions/42720618/docker-nginx-st... The worst part is that none of the suggested answers actually work for me, so i can't have a single Nginx instance be in front of the development environment with about 20 containers, a few of those being down when Nginx is being restarted will not let many of them be used until the startup finishes. Unacceptable. Caddy - as stated before, i liked v1 more than v2, though the project itself is pretty close to as good as a web server might get. What i don't enjoy is them taking the old docs offline, merely letting you download an archive, nor am i a fan of the current docs, since at the current point in time they are a bit like running "tar --usage": https://caddyserver.com/docs/caddyfile/directives/reverse_proxy https://caddyserver.com/docs/caddyfile/directives/reverse_pr... It's nice that there are a few examples for the common use cases, but there probably could be even more, just look at what the PHP documentation has at the bottom for a good example: https://www.php.net/manual/en/function.str-replace.php https://www.php.net/manual/en/function.str-replace.php (crowd sourced, but i like the idea of letting the community contribute useful information like that). Apart from that, some of the behavior is weird and you will get a 200 when you'd expect to get a 502/404 in most other web servers: https://caddy.community/t/why-does-caddy-return-an-empty-200-response-for-a-non-configured-host/7685 https://caddy.community/t/why-does-caddy-return-an-empty-200... which will sometimes be misleading ("Huh, i'm not getting any data in the response to my request, even though the status is 200 in my log, weird...") Also, i remember when v1 had this "fail-fast" habit of shutting down the entire server when renewing/obtaining a certificate failed, something that i utterly hate when web servers do: https://github.com/caddyserver/caddy/issues/642 https://github.com/caddyserver/caddy/issues/642 Admittedly, things are a bit better now: https://caddyserver.com/docs/automatic-https#errors https://caddyserver.com/docs/automatic-https#errors I just don't understand why web servers can be so opinionated about these things and not provide something like "failure_action" in Docker Compose (https://docs.docker.com/compose/compose-file/compose-file-v3/#update_config https://docs.docker.com/compose/compose-file/compose-file-v3...) so that people can choose between either stopping everything as soon as problems manifest, or continuing with a "best effort" strategy. If i'm hosting 100 sites behind a reverse proxy, i don't want 99 to be taken down just because 1 of them was misconfigured, the web server should be able to throw out a warning about that one host if i tell it to, and proceed to run the rest 99 as instructed. When no web server forces me to cope with such brittleness will be a good day.

doteka 5y ago

Caddy is the best. I use it as the static file server and reverse proxy for several side projects running in docker-compose. What I like most about it is how little config you need for reasonable defaults that would require 300 lines of nginx boilerplate. If I could wish for one thing though, I’d really like the functionality to get let’s encrypt certs while being proxied through cloudflare to be built in. Right now it requires building a custom caddy with a plug-in, which is a lot of hassle for such a vanilla setup.

mholt 5y ago
Thanks for your feedback. We try to avoid tight integration with specific, third-party providers as much as possible to keep Caddy light and flexible. Cloudflare is popular, but is also not a majority use case for Caddy users, in our experience. The Cloudflare plugin has only 16,000 downloads (and Route53 has 10,000)... out of over a million custom builds (not to mention 100M+ Docker pulls, or other ways of installing Caddy). So it's not enough of a standout to merit inclusion by default.
- doteka 5y ago
  Thanks for the reply, mholt! I would have expected way higher numbers there, but going by that ratio you’re absolutely right. Regardless, thanks for the great work on Caddy, it’s rock solid.

orware 5y ago

I'm not sure I understand the criticism in the top comment in this thread about Caddy's performance. I don't think I ever really decided to use Caddy solely based on performance itself (although already fairly familiar with Go's built-in libraries for creating HTTP servers gave me confidence that it should be fairly quickly...I wasn't overly concerned about how it directly compared to Nginx/Apache/IIS/etc.). For me, the biggest selling point was the Caddyfile format, which felt a lot more human readable than the .htaccess or Nginx configuration options I had seen over the years (I know with Caddy v2 there's the possibly now more preferred JSON option, but I'm still a sucker for the original Caddyfile format myself :-). There's still a lot I don't know about for all of the web server options out there in general so I don't claim to be an expert in any of them, but Caddy was the first one I felt comfortable using new situations to simplify things in my environment. For example, Caddy's reverse proxy functionality (and the way it is setup within the Caddyfile) is what finally made that particularly capability "click" for myself over 5 years ago when I first came across Caddy (even though I had seen info how Nginx + Apache were used together in the years prior, with Nginx providing reverse proxy access to Apache in those hybrid setups, it wasn't something that had "clicked" for me being aware of those situations). Since my environment is fairly simple (no real super high demand situations requiring lots of load balancing) we mainly use Caddy to provide automatic SSL capabilities and act as the publicly accessible endpoint in our environment, and use the reverse proxy functionality to tie Caddy into the various other internal servers that need that SSL termination. I've also been able to take advantage of the built-in static file server for an internal website need and it's actually something I really think is a nice/neat feature to have available (especially due to its ability for simple server side includes, allowing for relatively easy simple templating). Separate from that, I've also experienced first hand the personal responses that Matt Holt has provided to myself and to many others in the Caddy Community, and I'm definitely appreciative of the work he's put into Caddy (along with others of course) and I hope some of negativity in this thread is able to be easily shrugged off by Matt, because I could imagine it can be exhausting to deal with when he genuinely does care and puts a lot of thought/effort into the responses he provides.

mholt 5y ago
Thank you for sharing your experiences. Lots of positive feedback here has helped to offset the negative, so it's only a little exhausting today. :)

itsjloh 5y ago

I really like Caddy and use it on a few projects. Its a good piece of software that generally just works. Their latest security incident leaves me feeling somewhat uneasy though. The only announcement for it was on twitter[1] and a footnote on one of the releases[2]. I don't believe there was ever a mention of the incident on the official forum so if you weren't following them on Twitter you'd never find out. The general messaging around it was "a GitHub bug caused it" and its never really been followed up on publicly :\ [1]: https://twitter.com/caddyserver/status/1338324878441603073 https://twitter.com/caddyserver/status/1338324878441603073 [2]: https://github.com/caddyserver/caddy/releases/tag/v2.2.1 https://github.com/caddyserver/caddy/releases/tag/v2.2.1

francislavoie 5y ago
It wasn't a security incident, actually. It's true that "a GitHub bug caused it". It wasn't malicious. TLDR, a contributor made a tag on their own fork of Caddy, and for some reason our next release used their tag, because it turns out forks in GitHub aren't actual separate repos, but rather "still technically the same repo". It's really strange. Wasn't that contributor's fault either, they had no idea that would happen. All that happened is that the v2.2.2 git tag wasn't properly signed with Matt's signing key. There was no problem with the code at all. We've put in place checks during our CI actions to ensure that releases are always verified to be signed by Matt's key. See https://github.com/caddyserver/caddy/pull/3932 https://github.com/caddyserver/caddy/pull/3932 Specifically, commit https://github.com/caddyserver/caddy/commit/1d473ae924f0d52cb5a8f525e3e02b5167a4a5f1 https://github.com/caddyserver/caddy/commit/1d473ae924f0d52c... (which you'll notice is _not_ part of the Caddy repo, it was actually from a fork which was later deleted) triggered this GitHub Actions job for the release https://github.com/caddyserver/caddy/actions/runs/392345801 https://github.com/caddyserver/caddy/actions/runs/392345801 which we cancelled when we noticed it was happening. But we can't remove the tag from Go's caching server https://pkg.go.dev/github.com/caddyserver/caddy/v2@v2.2.2 https://pkg.go.dev/github.com/caddyserver/caddy/v2@v2.2.2 so it's kinda there forever. More context: https://twitter.com/mholt6/status/1337879764317564928 https://twitter.com/mholt6/status/1337879764317564928, https://twitter.com/danlsgiga/status/1338859470227529732 https://twitter.com/danlsgiga/status/1338859470227529732, that whole twitter thread has many branches, so click around to get the whole conversation.