10 ms·
The database servers powering Let's Encrypt (2021)
- distract8901 3y agoInteresting that their previous server is just one or two models up from the server I recently grabbed foe $150 on eBay. I'm just now getting serious about a homelab and it's shocking how much compute you can get for peanuts from enterprise surplus. I've had this machine (1U poweredge r620) for about a year and I'm already itching for an upgrade. Hopefully something with a lot of 3.5" drive bays for long-term data hoarding.
- dorfsmay 3y agoIt's because if you don't use that compute power, it's just an expensive loud heating system that's going to run up your electricity bill!
- sgarland 3y agoI have three R620s. They are shockingly quiet after being booted, even under pretty decent load. If you tune C-states for energy conservation in the BIOS, they idle at just under 100W each. With moderate load (like you'd have in a homelab), around 125W. According to iDRAC, the peak they've seen is about 350W. For better apples-to-apples, each of them has 2x E5-2650v2, 3x SATA SSDs, 8x 16 GiB PC3-12800R sticks, and 1x NVMe SSD (mounted in a PCIe adapter). As a comparison, I also have a Supermicro X11SSH-F with a bunch (9) of spinning disks. It has a single E3-1220 v5, 1x NVMe SSD, 2x 32 GiB PC4-17000U, and averages 125W. Given the relative power consumption of HDDs, that's a pretty impressive difference IMO. My entire rack (the aforementioned servers, and some Unifi networking gear) pulls around 500-600W in total. When the backup server (Supermicro X9DRi-LN4F+; dual-socket something-or-other Xeons, some DDR3 ram, and 8x spinning drives) kicks on daily to ingest ZFS snapshots, the total consumption is around 700W, I think. Maybe 750W.
- p1necone 3y ago> E5-2650v2 Those are pretty awful by current standards - you could buy a midrange minipc or laptop from a few cpu generations ago and get similar/better compute performance for much lower power draw. If you're just running a homelab and you don't have poor quality wall power or something you aren't really gaining anything from ECC ram either. Although you said you were thinking of upgrading, so you're probably aware of this already.
- ThePowerOfFuet 3y ago>If you're just running a homelab and you don't have poor quality wall power or something you aren't really gaining anything from ECC ram either. It has nothing to do with the quality or reliability of the mains supply; ECC increases reliability via protecting data integrity from bitflips caused by cosmic rays etc.
- distract8901 3y agoHm, I keep my server in the basement, I wonder if 6" of concrete makes any difference to cosmic rays?
- sgarland 3y ago> Those are pretty awful by current standards Oh, fully agree. My base spec M1 Air destroys them in anything single-threaded. I got them because I wanted three nodes to play around with various things requiring HA, and I didn't want to just virtualize that. That said, they _are_ virtualized. Every server I have runs Proxmox, with templated VMs built from Packer + Ansible. I'm currently in the process of shifting K8s providers; right now they're running k3os but since Rancher has killed that, it's forever stuck at v1.21, so I'm shifting to Talos. I have the Terraform for them working fine, just need to set up a PXE server so they'll pick up the desired static IP when being created. > you aren't really gaining anything from ECC ram either Sibling comment to mine addressed this; to add on, I disagree. Cosmic rays aside, there's a ton of EMI inside a computer, and anything I can do to help prevent data corruption is a win in my book. Torvalds agrees, if that carries any weight with you. [0] As far as cost, it costs me ~$30-40/month to run in electricity, and some small amount of A/C load (roughly equivalent to 5 extra people existing in the house) [1]. [0]: https://www.realworldtech.com/forum/?threadid=198497&curpostid=198647 https://www.realworldtech.com/forum/?threadid=198497&curpost... [1]: https://www.engineeringtoolbox.com/metabolic-heat-persons-d_706.html https://www.engineeringtoolbox.com/metabolic-heat-persons-d_...
- distract8901 3y agoThe absolute maximum I've ever seen it hit is 400W. At my electricity rate, the theoretical maximum is something like $15/month. I'm perfectly fine with that, it's well within my hobby budget. Really, that's what it is to me, a hobby. It doesn't have to be useful or profitable, just an interesting way to spend my free time tinkering
- sgarland 3y ago> We increase target & max IOPS above the defaults. We still use conservative values to avoid excessive SSD wear, but the defaults were tuned for spinning disks: innodb_io_capacity=1000, innodb_io_capacity_max=2500. I'd be interested to see if they actually needed this. Those parameters affect the baseline and burst IOPS InnoDB is allowed to use for background tasks like flushing logs, respectively, and generally you don't need to raise them. They default to 200 and 2000; those defaults been perfectly adequate for me on a MySQL 8.x instance serving 120K+ QPS.
- FpUser 3y agoWhat? No k8, no microservices, no cloud ? What is the world coming to?
- tyingq 3y agoThey were issuing 2M certs/day when this happened in 2021, now up to 3.4M/day.[2] Looks like it scaled pretty well so far. [2] https://letsencrypt.org/stats/ https://letsencrypt.org/stats/
- jimaek 3y agoRefreshing to see big projects running bare metal rather than overpaying for AWS. Especially one relying on donations and sponsors.
- deleted 3y ago[deleted]
- sho 3y agoI bet that compared to an equivalent load hosted on AWS, that lovely box pays for itself in full every month, if not every week...
- deleted 3y ago[deleted]
- d_silin 3y agoAbsolutely. Never understood why people are so infatuated with "cloud" options. Yes, it is convinient, but you are absolutely paying at least order of magnitude more for the same amount of compute/storage.
- civilitty 3y agoIt's gotten absolutely bonkers with all the demand for GPUs and the cloud companies are just raking it in hand over fist. On AWS reserved instances with 8x A100s or H100s (if you can even get them) cost more per year than the total upfront retail price of the equivalent pods from Lambda Labs. The on-demand price is even more absurd.
- qwertox 3y agoIf you don't need to host something big, where you can get away with one good VPS, the cloud has the benefit of offering cheap bandwidth. Once you have the bandwidth at your location and you don't need to be present in multiple locations, it's cheaper to self-host. Next step would be colocation, but for a start, using cloud offerings is a cheap way to be a part of the internet.
- nacs 3y agoCloud bandwidth is definitely not cheap. If anything, it's where they rip you off the most. You can get "baremetal"/dedicated servers from places like Hetzner and OVH that give unmetered gigabit connections for like $50.
- 3y ago
- gloyoyo 3y agoAlways nice to get an upgrade.
- mark242 3y agoSomeone reading the headline of this might think "oh, they're using Cockroach, or Fauna, or Planetscale" -- nope, this is about next-gen hardware powering their single-write (with a number of read replicas) MariaDB instance.
- PaywallBuster 3y agoIs there an update to this article? Since the article was written: - AMD has gen4 available - Genoa - there's NVMe drives 15tb capacity each
- rz2k 3y agoAnd, it looks like with PCIe 5.0 x4 performance exceeds 2,500,000 IOPS reads, and 14 GBps sequential read rate.
- tyingq 3y agoI would guess they try to get 3-5 years out of a new server, and this was done in 2021...so still a bit early for a refresh. The servers prior to this one had E5-2650 processors, so they would have been bought sometime between 2013-2016. Meaning their last refresh cycle was at least 5 years.
- wmf 3y agoThose 2021 servers will probably last them 4-5 years so they haven't upgraded yet.
- mcpherrinm 3y agoThe same hardware is still in use today, and I expect will continue to be used for some time. The biggest thing that’s happened in the meantime is reducing load on the databases by pushing some of the OCSP load out to redis caches: https://letsencrypt.org/2022/12/15/ocspcaching.html https://letsencrypt.org/2022/12/15/ocspcaching.html By not storing OCSP in the MariaDBs, we reduced write volume and storage size significantly. The next big thing is going to be sharding the database so multiple servers can share the write loads, and some queries moved out to redis caches. (I work for Let’s Encrypt)
- efxhoy 3y agoWhy add caching and sharding? Is the current architecture not going to keep up with demand?
- 3y ago
- candiddevmike 3y agoDoesn't mention if they're running Linux/what flavor they're running. I personally would be wary of running OpenZFS in production on Linux, especially ZFS on root. It has bit me in the ass too many times on Debian with an update breaking DKMS and rendering my system unbootable. Also, it's very, very strange/worrying to see no mention of disk encryption anywhere in the post or the tuning guide. For a company with encrypt in the name, that is responsible for the majority of trust on the internet, WTF? That should be highlighted in their benchmarking. ZFS supports native encryption, MariaDB does encryption, how are they encrypting at rest/transit/use?
- vorpalhex 3y agoYou can always boot off ext4 and then just run data off OpenZFS pools. The benefits of booting off ZFS are extremely minimal compared to having your working data on ZFS.
- deleted 3y ago[deleted]
- deleted 3y ago[deleted]
- shrubble 3y agoZFS has been very reliable for me since 2006 or so when I first started using it on SPARC hardware with the Solaris 10 beta; I assume that since they have a backup server and a primary server, they don't update and reboot them both at the same time.
- pmarreck 3y ago> I personally would be wary of running OpenZFS in production on Linux A ton of people in the enterprise have been doing this for years without issue; https://openzfs.org/wiki/Companies https://openzfs.org/wiki/Companies > especially ZFS on root I've been running ZFS on root on NixOS since this excellent guide (https://openzfs.github.io/openzfs-docs/Getting%20Started/NixOS/Root%20on%20ZFS.html https://openzfs.github.io/openzfs-docs/Getting%20Started/Nix...) for about 2 years. Zero issues. (Actually, I see they've updated it, I need to look at that. Also, they default to encrypted root. I turn it off, because the slight hit to performance and extra risk of recovery impossibility is not worth it to me.) > It has bit me in the ass too many times on Debian with an update breaking DKMS and rendering my system unbootable Well, I think you've found your problem, then. (That also might be why FreeNAS, which I also have running as my NAS, switched to Linux from Debian when they re-branded as TrueNAS Core/Enterprise.) Come over to NixOS, where you can simply reboot into any of N previous instances after an update that borks something (which almost never happens, anyway, because you can actually specify, in your machine configuration, "use the latest kernel that is compatible with ZFS"). No USB boot key needed. Here's the magic line from my own declarative configuration: kernelPackages = config.boot.zfs.package.latestCompatibleLinuxPackages; Aaaaand... DONE. ;) > Also, it's very, very strange/worrying to see no mention of disk encryption anywhere in the post or the tuning guide. For a company with encrypt in the name, that is responsible for the majority of trust on the internet, WTF? You're assuming they're not doing it, without evidence. Also, if they're already managing the security around their certs and cert generation properly, they might not need FDE. FDE is overrated IMHO, frankly, and also incurs a performance cost, as well as an extra risk cost (try recovering an encrypted drive to know what I mean). In short, religions are bad, even in technological choices; there is no single technological configuration choice that is 100% better than all possible alternative configurations. > That should be highlighted in their benchmarking. ZFS supports native encryption, MariaDB does encryption, how are they encrypting at rest/transit/use? Multiple layers of encryption incur an extra performance cost with almost no gain in extra security.
- KronisLV 3y ago> We currently use MariaDB, with the InnoDB database engine. Oh hey, it's not often that you hear about MySQL/MariaDB on HN, so this is a nice change. For what it's worth, it's a pretty decent database for getting things done, even if not as advanced as PostgreSQL in some respects.
- turtles3 3y agoI think it is also important to acknowledge that there are things innodb does better than Postgres, eg. For most workloads an undo log is a far better data structure than implementing MVCC by duplicating rows. Autovacuum and vacuum can be an absolute nightmare, plus the extra disk traffic the duplication generates. Maybe one day OrioleDb will bring this to postgres too.
- linsomniac 3y agoI have had a MariaDB + Galera multi-master 3-node cluster running for several years, and prior to that Percona+Galera for ~5 years, and it's been just great. It stores lookup tables for our postfix mail server, fairly small databases and low hits, but it's great to just be able to reboot nodes for updates or migration to other VMs without having to do any of the old clustering gyrations. I almost switched to CockroachDB in the last refresh, until I found that Postfix required Latin-1 encoding or something, and CockroachDB only supported UTF8. Postfix has more recently gotten a config option for changing that.
- deleted 3y ago[deleted]
- hyc_symas 3y agoStill an order of magnitude slower / less efficient than OpenLDAP / LMDB.
- lelouch11 3y agoThe same argument came up in a big company, incidentally for the same use case, certificate store in a CA. This led to benchmarks, where OpenLDAP was significantly slower, like three orders of magnitudes. Databases have gotten really faster in the last couple of years while OpenLDAP has stagnated.
- d_silin 3y agoShows how much bang for buck you can get when hosting your own hardware.
- tda 3y agoI recently bought my first server, just about the cheapest Dell had available (still €2000). I was completely undewhelmed by the specs, it even has spinning rust. The upgrade costs to SSD's were like 250 per disk. So reading these specs, how much does such a server cost? Or do you put the disks in yourself? Can you negotiate a 50% discount? So many questions I have as a newbie in the entireprise server world.
- barkingcat 3y agoNo one buys dell servers at list price.
- Neil44 3y agoWhere possible I go third party on disks and ram.
- milesdyson_phd 3y agoIf you are an enterprise (or just big enough) the prices on Dell’s site are meaningless, they are just conversation starters if that.
- Dachande663 3y agoFrom old knowledge, you’re out by about 2 orders of magnitude.
- dang 3y agoDiscussed at the time: The database servers powering Let's Encrypt - https://news.ycombinator.com/item?id=25861422 https://news.ycombinator.com/item?id=25861422 - Jan 2021 (226 comments)
- nunez 3y agoThese performance improvements are absolutely insane, and churning out certs at speed is the _perfect_ use case for these CPUs. Kudos the LE team!
- hu3 3y agoIndeed. If I had to bet, I would say doubling the memory allowed some/all indexes to fit in RAM. RAM is usually a big improvement when it comes to Relational Database Management Systems (RMDBS).
- nologic01 3y agoI bet if one could develop an index of useful computation per kwh (or $) this project would be near the top. Imagine if that kind of hyperleveraged impact was the norm rather than the exception.
- dmurray 3y agoIt's kind of hard for it to be. If the only software projects that existed were the elite hyperefficient ones, the ten guys who ran them could just meet up every year for a key signing party - no need for a scalable service to automatically issue them security certificates. Computers being commoditized means the median project is very low-value, but it also means there are millions of cool projects out there.