5 ms·
Healthchecks.io now uses self-hosted object storage
- _joel 6mo agoI'm sure it's a lot better now but everytime I see btrfs I get PTSD.
- poly2it 6mo agoCare to elaborate? I've heard good things about it, but am personally a ZFS user.
- metadat 6mo agoYears of serious corruption bugs.
- dundercoder 6mo agoGluster was that for me
- sigio 6mo agoYup, still get nightmares about glusterfs.... still have one customer running on it.
- dundercoder 6mo agoI heard it got better, but we ran into the BOTF (billions of tiny files) issue around 2016. (For a genealogy startup this was a serious issue)
- _joel 6mo agoAh, another one! Yep, also same, before ceph days at least (although I've had my own, albeit self-inflicted, nightmare there too).
- deleted 6mo ago[deleted]
- uroni 6mo agoI'd worry about file create, write, then fsync performance with btrfs, but not about reliability or data-loss. But a quick grep across versitygw tells me they don't use Sync()/fsync, so not a problem... Any data loss occurring from that is obviously not btrfs fault.
- 060880 6mo agoSame here. Had a production node running btrfs under heavy write load (lots of small files, frequent creates) and spent two days debugging what turned out to be filesystem-level corruption. Switched to ext4 and never looked back. The article doesn't mention what filesystem sits under Versitygw here, which seems like a pretty relevant omission for anyone thinking of replicating the setup.
- __turbobrew__ 6mo agoI hit a panic in btrfs using an ubuntu 24 LTS kernel. The trauma is still well and alive.
- stackskipton 6mo agoSame, and reasoning around inodes feels easy fixed by just upping inode per KB from 16k to 4k which is likely block size anyways.
- NelsonMinar 6mo agoI'm a little surprised it's not ZFS. Too difficult to add to their Linux environment? That's still a problem here in 2026.
- tobilg 6mo agoI don't get it, if it's running on the same (mentioning "local") machine, why does it even need the S3 API? Could just be plain IO on the local drive(s)
- esafak 6mo agoSo you don't need to refactor your code?
- zipy124 6mo agoseperate machine I think given the quoted point at the end: > The costs have increased: renting an additional dedicated server costs more than storing ~100GB at a managed object storage service. But the improved performance and reliability are worth it.
- VHRanger 6mo agoThe S3 API doesn't work like normal filesystem APIs. Part of it is that it follows the object storage model, and part of it is just to lock people into AWS once they start working with it.
- tobilg 6mo agoI'm 100% aware of how S3 works. I was questioning why the S3 API is needed when the service is using local storage.
- 6mo ago
- lsb 6mo agoSelf Hosted object storage looks neat! For this project, where you have 120GB of customer data, and thirty requests a second for ~8k objects (0.25MB/s object reads), you’d seem to be able to 100x the throughput vertically scaling on one machine with a file system and an SSD and never thinking about object storage. Would love to see why the complexity
- jakewins 6mo agoThe complexity for that is almost always for redundancy and for ease of deploys.
- cuu508 6mo ago(Author here) that's more or less what I have right now – one machine with a file system and an SSD. S3 API on top is there to give multiple web servers shared access to the same storage. I could have used something else instead of S3 – say, NFS – but there was a feature request for S3 [1] and S3 has a big ecosystem around it already. [1] https://github.com/healthchecks/healthchecks/issues/609 https://github.com/healthchecks/healthchecks/issues/609
- smjburton 6mo ago> In March 2026, I migrated to self-hosted object storage powered by Versity S3 Gateway. Thanks for sharing this, I wasn't even aware of Versity S3 from my searches and discussions here. I recently migrated my projects from MinIO to Garage, but this seems like another viable option to consider.
- zenoprax 6mo agoFirst time hearing about Versity for me too. I thought "S3 Gateways" were an Amazon-only service rather than something mere mortals could set up. I've been trying to give some containers (LXC/D and OCI) unprivileged access to a network-accessible ZFS filesystem and this might be what I need. Managing UID/GID through bind-mounts from the host to the container (ie NFS on host) has been trickier than I was expecting.
- nyrikki 6mo agoNot really s3, and I haven't touched LXC in a long but this may help on the OCI side. I apologize if this is redundant to you. Remember that UID mapping on namespaces is just a facade with an offset and a range typically based on subuid[0] and subgid[1] today. In the container `cat /proc/self/uid_map` or by looking at the pid from the host `cat /proc/$PID/uid_map` you can tell what those offsets are. $ cat /proc/self/uid_map 0 1000 1 1 100000 65536 Here you know that PID =0 in the container maps to PID 1000 in the host, with a length of 1 The Container PID offset of 1 maps to the host offset of 100000 for a length of 65536 With subuid/subgid you can assign ranges to the user that is instantiating the container, in the flowing I have two users that launch containers. $ cat /etc/subuid debian:100000:65536 runner:165536:65536 $ cat /etc/subgid debian:100000:65536 runner:165536:65536 Assuming you pass in the host UID/GID that is how you can configure a compatible user at the entrypoint. But note that only highly trusted containers should ever really use host bind mounts, it is often much safer to use mount NFS internally. Host bind mounts of network filesystems, if that is what you are doing, is also fragile as far as dataloss goes. I am an object store fan, but just wanted to give you the above info as it seems hard for people to find. I would highly encourage you to look into the history of security problems with host bind mounts to see the wack-a-mole that is required with them to see if it fits in with your risk appitite. But if you choose to use them, setting up dedicated uid/gid mappings and setting the external host to the expected effective ID of the container users is a better way than using Idmapped mounts etc... [0] https://man7.org/linux/man-pages/man5/subuid.5.html https://man7.org/linux/man-pages/man5/subuid.5.html [1] https://man7.org/linux/man-pages/man5/subgid.5.html https://man7.org/linux/man-pages/man5/subgid.5.html
- iamcreasy 6mo agoGiven the individual file size and total volume, I'd argue it make sense to use move to local only storage. On a separate note, what tool is the final benchmark screenshot form?
- choilive 6mo agoMoved object storage from AWS to CloudFlare and have been pretty happy. No problems with performance so far. Bills were 90% cheaper too (free bandwidth)
- chasd00 6mo ago> The costs have increased: renting an additional dedicated server costs more than storing ~100GB at a managed object storage service. But the improved performance and reliability are worth it. Were your users complaining about reliability and performance? If it cost more, adds more work (backup/restore management), and the users aren't happier then why make the change in the first place?
- encoderer 6mo agoNot the OP but I have some… similar experience. When you run a high availability service without a full ops team, reliable infrastructure is non-negotiable. Burn out has to be managed.
- kikoreis 6mo agoAs a user for over a decade, just here to submit my appreciation. healthchecks.io is fantastic.
- bityard 6mo agoAs someone who has dealt with wacky storage issues/designs, a lot of this "felt" strange to me. Btrfs? Rsync? Then I got to the bottom and saw that they were only handling about 100 GB of data! At that scale, nearly anything will work great and TFA was right to just pick the thing with the fewest knobs. At a previous job years ago, we had a service that was essentially a file server for something like 50TB of tiny files. We only backed it up once a week because just _walking_ the whole filesystem with something like `du` took more than a day. Yes, we should have simply thrown money at the problem and just bought the right solution from an enterprise storage vendor or dumped them all into S3. Unfortunately, these were not options. Blame management. A close second would have been to rearchitect dependent services to speak S3 instead of a bespoke REST-ish API, deploy something like SeaweedFS, and call it a day. SeaweedFS handles lots of small files gracefully because it doesn't just naively store one object per file on the filesystem like most locally-hosted S3 solutions (including Versity) do. And we'd get replication/redundancy on top of it. Unfortunately, I didn't get buy-in from the other teams maintaining the dependent services ("sorry, we don't have time to refactor our code, guess that makes it a 'you' problem"). What I did instead was descend into madness. Instead of writing each file to disk, all new files were written to a "cache" directory which matched the original filesystem layout of the server. And then every hour, that directory was tarred up and archived. When a read was required, the code would check the cache first. If the file wasn't there, it would figure out which tarball was needed and extract the file from there instead. This only worked because all files had a timestamp embedded in the path. Read performance sucked, but that didn't matter because reads were very rare. But the data absolutely had to be there when needed. Most importantly, backups took less than an hour for the first time in years.
- winterbloom 6mo agowhat happens if you suffer catastrophic failure, you'd lose up to one hour of data?
- bityard 6mo agoThe system had RAID, so disks could be replaced without taking anything down. We never had a catastrophic failure, but yeah, and hour of lost data wouldn't be the end of the world. For regular maintenance we would just coordinate with the QA team to pause testing temporarily. (Testing was only around-the-clock near a product release.)
- jeffrallen 6mo ago> Our S3 API is now served by Versity S3 Gateway and backed by a plain simple Btrfs filesystem. With apologies to the SRE Book ("hope is not a strategy")... Btrfs is not a strategy.
- arend321 6mo agoIt's interesting that cloud providers are unable to provide stable S3 as a service. Hetzner is unable to deliver stable object storage, but given the article neither are OVHCloud and UpCloud.
- jerf 6mo ago"Our current (April 2026) object usage is: 14 million objects, 119GB" I mean, I appreciate the openness about the scale, but for context, my home's personal backup managed via restic to S3 is 370GB. Fewer objects, but still, we're not talking a big install here. This is pretty much like that story of, if it fits on your laptop, it's not big data.
- eichin 6mo agoMy phone has 327.1G of storage in use right now :-) (years ago a friend of mine who was also old enough to complain about "640k" suggested that we should just denominate storage in dollars, so "reasonable" amounts scaled with Moore's law...)
- rapnie 6mo agoFor S3 self-hosters check out Garage, who are developing with NLnet funding: https://garagehq.deuxfleurs.fr https://garagehq.deuxfleurs.fr
- russelg 6mo agoCovered by the article: >I ran local experiments with Minio, SeaweedFS, and Garage. My primary objection to all of them was the operational complexity
- johntash 6mo agoOut of the listed options, Garage is actually really simple for something that runs on multiple servers. It was also the only easy option I found that can replicate over WAN and isn't super latency sensitive.
- tonymet 6mo agogreat writeup. Is s3 a customer or internal requirement? Why not write to your disks? Easily 1/3 - 1/5 the price and better performance.
- Unsponsoredio 6mo agolove this. one person, 119GB, two drives, rsync. no kubernetes, no distributed cluster, no nonsense. just works. this is the kind of setup that lets you actually go to bed without checking your phone every 20 minutes.
- sigmonsays 6mo agohow do you provide highly available fault tolerant storage? versity does not include any erasure coding or replication...
- CoolCold 6mo agoYou can wire lsyncd for near real-time syncs and btrfs snapshots for offline storage