8 ms·
Build a tiny certificate authority for your homelab
- mmalone 6y agoI work at smallstep and we're partnering with yubico to give away five build kits for this project. DM us at @smallsteplabs on twitter to enter. See also: https://twitter.com/smallsteplabs/status/1341800787291168768 https://twitter.com/smallsteplabs/status/1341800787291168768
- toomuchtodo 6y agoThere an email address to enter for those who don’t use Twitter?
- superkuh 6y agoThis is technically neat, but I really wish that browsers would just have a config option that would allow me to turn on trust on first use for certs. Also, all other software that use TLS. It works fine for mailservers. No on expects a mailserver cert to be from an authority, you can self sign and it works fine.
- mmalone 6y agoI think TOFU would be really problematic for browsers & the Web PKI trust model. At this point Web PKI is dealing with attacks from nation-states and other advanced threats that end-users aren't really in a position to handle themselves. Like, just last week the browsers had to remove a certificate authority from their root cert programs because Kazakhstan was issuing certificates to MiTM traffic. A TOFU model would make it a lot harder to detect and remediate this sort of attack and lots of other relevant attack vectors. We'd also need to re-solve a bunch of adjacent problems like revocation, renewal/rotation, and transparency, which would probably mean re-introducing the sorts of centralized architectural components and processes that I'm assuming you're trying to eliminate with TOFU.
- xg15 6y agoAll those points can be summed up as: The point of web PKI is that the decision of who to trust and who not is not supposed to rest at the end user but at some central authority. Then however, we get to the political question who exactly that central authority should be and why. > Like, just last week the browsers had to remove a certificate authority from their root cert programs because Kazakhstan was issuing certificates to MiTM traffic. I may have misunderstood the incident, but wasn't it such that the CA was not even one of the built-ins, but a "custom" root CA that all users were required to install on their systems? As such, the block was more equivalent to block a specific to TOFU key. Of course, blocking the MITM CA won't magically turn off the ISP's MITM proxy. It will simply make it so that kazhakh citizens can't access any web sites at all until the government hopefully caves and turns off the proxy.
- mmalone 6y agoYea you’re correct. They were forcing people to add a CA. So this was not a great example. I wouldn’t say the centralization of Web PKI is by design so much as it is (was?) by necessity. There’s a crypto conjecture called Zooko’s Triangle that says there are three desirable properties for a naming system: human-meaningful, secure, and decentralized. Zooko’s conjecture is that you can only have two. Web PKI picks secure & human-meaningful. Simple PKI (like TOFU) picks secure & decentralized (the names aren’t actually human-meaningful since you’re really trusting a public key which is a big random number, not a domain name). DNS picks human-meaningful and decentralized. More recently, Aaron Schwartz realized you can “square the triangle” using blockchain. So it appears to be technically possible to have all three now, but there are other hurdles. In any case, simple public keys aren’t a silver bullet. Just a different set of compromises.
- xen2xen1 6y agoWould also give you an indication that the cert changed. It's done for SSH all the time, and if my SSH key changes, I darn well want to know.
- mmalone 6y agoCounterpoint: this is also a big problem with SSH. When a host key changes with SSH you'll get a "Host Key Verification Failure" that basically says "yo, the key I expected for this host wasn't actually presented, you're probably being mitm'd". Then you have to go into your `~/.ssh/known_hosts` and delete a line. Then you get a new TOFU warning that you'll just type "yes" to and proceed. So most SSH users (who, on average, are way more technically competent than browser users) will automatically 1) panic and think they're being hacked, or 2) blindly trust the new key when a key change occurs with SSH. Both of these responses are dangerous. The result is that, unless you have fancy stuff for managing known hosts for all of your users on all of your endpoints, you're probably just avoiding this scenario by not rotating host keys at all for SSH. Which is also problematic. By using a CA you're delegating the key binding to a trusted piece of infrastructure that can be locked down and monitored by experts. You can't do that easily with key-bindings written to files on a bunch of different endpoints. With a CA, end users shouldn't need to care about key changes. The fact that the CA can issue a new certificate for some entity is a benefit: it makes credential rotation easier. If you do want to know when credentials rotate there are ways to monitor that yourself (and Web PKI has ways like key pinning and cert transparency).
- downut 6y ago"...that can be locked down and monitored by experts." In my homelab? I guess I should also kerberize my NFS mounts, after I solve the SPOF problem for kerberos's dependent pieces. I might have a little time after doing all this for working on my homelab projects. How much time could this configuration, maintenance and monitoring possibly take? (Just between you and me I resolve that on January 1st I will again start reading all daily/weekly/monthly logs, and this year I WILL NOT FAIL. It's only dozen give or take few boxen.) If the security infrastructure needs to designed, configured, monitored, and maintained by "experts" in an unfunded environment, the security infrastructure is doomed to fail. IOW, it's security theatre.
- jlgaddis 6y ago> No on expects a mailserver cert to be from an authority, you can self sign and it works fine. Yes, but that's A Bad Thing(TM) -- it means that MITM'ing a mail server also "works fine"! Now, personally, I'm not a big fan of DNSSEC (especially considering how widespread 512- and 1024-bit keys were!) but I was hopeful that "DNS-based Authentication of Named Entities" (DANE) [0] would become widely supported once it was fully standardized. Unfortunately, that didn't happen -- especially in the browsers! After the explosion in growth of "HTTPS everywhere", transport security for e-mail was the next big (unencrypted) problem that needed to be addressed in my opinion. Luckily, both Exim and Postfix (my MTA of choice) gained support for DANE so while progress was technically made, widespread adoption never really happened there either (mostly due to the dependency on DNSSEC, AFAICT). In the meantime, though, "SMTP MTA Strict Transport Security (MTA-STS)" appeared on the scene and has since been formalized as RFC8461 [1]. In many ways, it's technically superior to DANE and, importantly, does not rely on DNSSEC. It does have the same reliance on the "WebPKI" as the browsers so it's certainly not perfect; it's still a huge improvement, though, and it's certainly better than opportunistic encryption which is trivial to MITM. Anyways, as MTA-STS was designed by folks at Google, Microsoft, Comcast, and Oath, I'm hopeful, once again, that we'll eventually get to the point where (at least) the overwhelming majority of e-mail is encrypted in transit as it passes from MTA to MTA on its way to the recipient's mailbox. One of the few good things about so much of the e-mail nowadays being handled by Google and Microsoft is the huge volume of mail that will suddenly just start being encrypted in transit, overnight, when just those two enable MTA-STS in their mail infrastructure. I haven't kept up with progress on MTA-STS since leaving my previous job (which included responsibility for the mail infrastructure) about two years ago -- shortly after it became a standard -- but I'd love to find out that it's already been widely rolled out by the big players in the meantime! I suppose it's about time to catch up on the last two year's worth of messages sitting unread in the "mailops" folder in my mailbox! -- [0]: https://en.wikipedia.org/wiki/DNS-based_Authentication_of_Named_Entities https://en.wikipedia.org/wiki/DNS-based_Authentication_of_Na... [1]: https://tools.ietf.org/html/rfc8461 https://tools.ietf.org/html/rfc8461
- superkuh 6y agoDNSSEC doesn't provide anything valuable in terms of privacy. All it does it make it so central governments can more easily censor their controlled tld. If DNSSEC had been rolled out in Libya back in 2010 I doubt the Libyan Revolution would've happened. TLA cert authorities are bad enough. MTA-STS is would make running your own mailsever harder and leash you to one of a dozen cert authorities. If you don't think that's a problem then remember what happened to dot org, and think what will happen to these cert authorities given enough money and time. I've run my own mailserver for a decade now and I am strongly against everything you suggest. But it doesn't matter what I care about. If everyone uses only one of a handful of email providers then they'll just slowly close off their walled gardens anyway. It's already happening no matter how many superfluous cargo-cult standards independent mailserver operators implement.
- wwarner 6y agoI feel woefully out of step here. If you can create a CA that browsers respect, have you really improved on cleartext transmission? And isn't the CA is now a new attack vector?
- mmalone 6y agoSure the CA is a new attack vector. That doesn't make it inherently insecure or the same as cleartext transmission. Is running an internal OAuth OIDC identity provider to issue signed identity tokens the same as using cleartext passwords?
- sethhochberg 6y agoYou're definitely shifting your attack surface towards the CA, but in doing so you've (probably) given yourself a lot of additional control over the weakest link in your system. With cleartext, vulnerabilities tend to range from simple things like "find the switch, plug in, sniff traffic over the wire" to slightly more complex, but still viable things like "sniff wifi traffic now, worry about cracking WPA-2 later". Your typical TLS certificate is going to be more robust than your typical consumer / prosumer LAN security, and you typically have better tools (SSH certificates, 2FA, encryption at rest, etc, etc) for locking down a machine running a CA than network gear you either don't control or bought from a consumer-focused vendor. Its a typical example of defense-in-depth. Hopefully your network is trustworthy, but if for whatever reason it becomes insecure/vulnerable you now have genuine encryption between client and server at application layer.
- deleted 6y ago[deleted]
- xg15 6y agoI may be wrong, but I think you still have to manually install your CA cert as a custom root CA on any device that you want to use inside your network. You can't use this tutorial to make a new CA that is automatically trusted.
- mmalone 6y agoThat is correct.
- hadcomplained 6y agoAlthough I do not deny the necessity for certificate authorities for convenience, I do not understand why using CAs is the only option. Why does the TLS protocol not allow for using a key pair which is agreed-upon between a server and a client beforehand, like in SSH connections where a public key to be used in a connection is placed in a server prior to the connection? There are many CAs out there and in the event of China or Russia hacking into one of them, it would enable them to perform man-in-the-middle attacks. I'd like to eliminate such a possibility, but the TLS protocol requires me to trust a certificate authority. I might just be a conspiracy theorist, but I am suspecting why it's impossible to use TLS without trusting a third-party called certifcate authority is exactly because someone needed to leave a way to do MITM attacks.
- tashian 6y agoTLS does have a mode that uses a pre-shared key, but in TLSv1.3 I believe it's only used for session resumption. Edit: Also, I recently learned about DNS Certification Authority Authorization (CAA) records. You can specify which of the public certificate authorities a browser is allowed to respect, for your domain. I don't think it's verified by all browsers yet, but it's a step.
- mmalone 6y agoPretty sure CAA is supposed to be enforced by CAs, not by browsers. So, for instance, Let's Encrypt should refuse to issue a cert for your domain if you have CAA setup for digicert.
- JonathonW 6y agoCAA records (which specify which CAs are allowed to issue certificates for a given domain) are intended to be enforced by CAs, not by browsers. This prevents unintentional misissuance of a certificate, but not deliberate MITM attempts if the CA is actively involved. Their counterpart for browsers are TLSA records, which associate specific keys or certificates with a domain name. This is the part that actually prevents MITM attacks on the client side (assuming the client's getting a complete and accurate DNS response, which is a whole other issue), since it'll cause a compliant client to reject any other keys or certificates. (No idea how widespread the implementation of this is on the client side, though.)
- guessmyname 6y agoFilippo Valsorda, from the Go security team, created mkcert some time ago → https://github.com/FiloSottile/mkcert https://github.com/FiloSottile/mkcert which makes this process much easier. I setup a home laboratory for several HTTPS websites and it all works as expected without too much hassle.
- kissgyorgy 6y agoThe two are nowhere near the same thing.
- greymadder 6y agohttps://github.com/jsha/minica https://github.com/jsha/minica has been around longer and has met all my home lab needs so far. It's even plugged on LetsEncrypt-- https://letsencrypt.org/docs/certificates-for-localhost https://letsencrypt.org/docs/certificates-for-localhost.
- RedShift1 6y agoXCA (https://hohnstaedt.de/xca/ https://hohnstaedt.de/xca/) is another way to manage a small CA, though it doesn't have ACME.
- rad_gruchalski 6y agoThere is hashicorp vault pki, square certstrap, cloudflare’s cfssl... but none comes with acme.
- tialaramex 6y agoIn one sense this is actually what you want. The whole point of ACME is to facilitate proof of control. For the Web PKI this is mandatory, certificate automation without proof of control wouldn't really be deployable. In a local environment you likely don't need that, and may even not have a way to do it even if you wanted to. So ACME's secret sauce is irrelevant. Any of the older protocols that just do certificate issuance with no proof of control, like SCEP, would be fine. But because ACME has been so successful (good!) lots of newer or updated software incorporates ACME support out of the box, and so it is attractive to be able to interface to that support. We could have had a situation where most server software did SCEP and you needed an adaptor to reflect the SCEP requests into a system that would use ACME to get them a certificate if they face the public Internet. But that isn't what we got, so, fine, this works, I'll take it.
- dr-ando 6y agoWhy not use Let’s Encrypt, e.g. with DNS validation so they do not need to hit your HTTP server?
- denkmoon 6y agoThey won't validate my internal domains (obviously). I have all my infra on .lan and using this they all get ACME certs and I never have to see another "insecure connection" page. Also had my old workplace on .dev until those bastards at Google stole it and added the entire tld to the hsts preload list!!
- tazard 6y agoWith DNS validation it's fairly straight forward to get a wildcard cert for all your internal domains (*.my.domain)
- notpiika 6y agoHow would one set this up? Why is DNS validation needed?
- jrockway 6y agoThose are simply the rules. You can do ACME with an HTTP challenge or a DNS challenge. The HTTP challenge is adequate for proving that you control x.example.com, but serving a website on x.example.com doesn't prove that you own y.example.com. But, being able to create example.com DNS records does, so that is what's required to get a wildcard certificate. I imagine you are confused because the proposal above sounds like "just get *.example.com, then copy that cert to everything that will ever serve traffic for example.com", which doesn't sound like a great idea to me.
- vxNsr 6y agoI was looking into something like for my homelab but for a cert noob I got lost somewhere between trying to use intra.mydomain.com and not screwing up my public address Can you recommend a good book or blog series that covers this topic in depth?
- maxk42 6y agoI've been trying to figure out how to set up a local CA on my (linux) dev machine for a while now. Every time I get started I seem to run into problems. Does anyone have a simple guide for people who don't plan on investing in a raspberry pi?
- _jal 6y agoFollow the instructions on whatever machine you want. A CA is about configuration and setup, not the hardware.
- sneak 6y agoIn general, it is a good idea to stick to keeping the timezone on all machines (excluding only the one 30cm in front of you) set to UTC.
- lmilcin 6y agoIf you have a dedicated piece of hardware like Raspberry Pi and you are going to be the only person ever using it, for private lab purposes, I don't see any reason to complicate it further with Yubikeys or hardware RNGs.
- miked85 6y agoRaspberry Pi already has its own hardware RNG as well, I suppose the one mentioned in the article may be better?
- lmilcin 6y agoBut why? Do you have any specific knowledge to the effect that a better hardware RNG would somehow benefit your private lab? As it is Linux already has completely acceptable source of entropy. The most significant drawbacks are that it could potentially be observed by somebody else with access to the machine and that it provides only very limited amount of entropy. Both are not a problem when you use private RPi to generate a dozen keys for your home lab.
- miked85 6y ago> Do you have any specific knowledge to the effect that a better hardware RNG would somehow benefit your private lab? I was not arguing that a hardware RNG was needed or not, just questioning if adding a second one to a Pi made sense.
- mvuksano 6y agoI think this is great read. The point of this is learning - as you probably would have seen in the article yourself. I don't know what's the point of questioning his design - if you don't like you always have option change it and/or implement it as you think is best. IMHO Carl did some really great work here.
- znpy 6y agoI am very happy I learned how to use vault from hashicorp. It's great for having a rest-enabled CA and can do a lot of other stuff too. And if you're into kubernetes and/or run your own little kubernetes cluster at home, cert-manager can issue certificates off vault too.
- danmur 6y agoVault is ridiculously easy to use for this.
- visualphoenix 6y agoVault as a CA is also ridiculously painful. This comes from the battlefield.
- danmur 6y agoI haven't used it seriously for that purpose so I can't comment :). Just my home setup. Their other products have their rough edges though so I can understand.
- znpy 6y agoWe use it in prod for secret storage and management (and maybe someday for CA) and I use at home for my own ca and btw it's surprisingly simple and effective.
- jrockway 6y agoI use cert-manager for internal mTLS and am satisfied with it. I found it surprisingly easy to use compared to things that try to be automatic, like linkerd and istio. (Plus, I can verify the authenticity of the connection at the application layer, and make appropriate decisions rather than having to trust "oh yeah, the sidecar has it under control, nothing to see here".) It's a downgrade in security compared to pressing a physical button a YubiKey every so often... but I am OK with things being automatically renewed. (It basically means, if you own the apiserver, you own the network, which is a tradeoff I can live with. Compare this to "if you can intercept any packet, you own the network", which is what non-mTLS is.)
- paulsmith 6y agoA year ago I would have found this appealing in a #homelab context, but now I WireGuard/Tailscale each host and bind LAN-only services to tailscale0 interface.
- npiit 6y agoIt's theoretically the same idea at the node level instead of the application level except that the WireGuard curve25519 keys now cannot be verified since they are published by a 3rd party that you have zero control on. This 3rd party can simply connect to your machines anytime by injecting its public keys into your nodes and have complete access into your private network. That's the power of owing your own CA as opposed to letting others injecting peer public keys as if there is nothing to verify.
- henvic 6y agoNice! In the past, I've tried to use Let's Encrypt with ESXi, but didn't see a clear and simple way with Let's Encrypt (and that itself might be overkill)... as I just needed it to admin my own servers, I decided to use https://github.com/FiloSottile/mkcert https://github.com/FiloSottile/mkcert for that. I posted what I did in https://henvic.dev/posts/homelab/ https://henvic.dev/posts/homelab/
- LordGrey 6y agoFor anyone needing a refresher on PKI, CA, etc, the article references an earlier blog post that is very well-written and informative: https://smallstep.com/blog/everything-pki/ https://smallstep.com/blog/everything-pki/