16 ms·
What One-person SaaS Healthchecks.io uses for hosting, hardware and software
- taf2 4y agoYou should consider backblaze for backups it is much less expensive and has an s3 compatible api…
- herodoturtle 4y agoThis is such a great blog post. Succinct and to the point. Read it from start to finish without pause even though the stack used (which is great!) is only remotely applicable to my workflow.
- altdataseller 4y agoHow much does it all cost?
- davidodio 4y agoThe monthly Hetzner bill is €484.
- str34m 4y ago
- deleted 4y ago[deleted]
- cuu508 4y agoHi, the author of the blog post here. I don't have a precise number, but somewhere in the €800/mo region.
- skunkworker 4y agoIs the decision not to use Patroni for HA Postgres in this case, so that you don’t add more complexity?
- cuu508 4y agoYes. Plus, from reading database outage postmortems, I was not comfortable making the "do we failover now?" decision automatic. Think about the brownouts, where the primary is still up, but slow. Or it experiences intermittent packet loss. I've automated the mechanics of the failover, but it still must be initiated manually.
- tpetry 4y agoThat‘s a good decision! Developers nowadays fear a failing machine so much because it‘s happening more often on the cloud than dedicated servers. I wouldn‘t use automatic failover too. Cleaning up the mess can take hours and days, a small downtime of a few minutes/hours is better. I‘ve seen one developer using telegram for ha-failover which is great! Just a message and the scripts are executed for a failover. You can do it in seconds without even being on a computer.
- CoolCold 4y agoProject I relate to, doing switch to Patroni managed DB (first of many) right today, in like a 5 hours, I may share observations from almost first hand experience later, if you care.
- BilalBudhani 4y agoI'm someone who likes configuring & running my own servers, it is always a pleasure reading how others small / solo businesses are running their show. It is so cool to see one-person SaaS business running on old school bare metal servers without needing any fancy devOps / containerize tooling.
- altdataseller 4y agoThat fancy devops/containers tooling is easier to maintain and work on vs running on bare metal servers.
- VWWHFSfQ 4y agoIs that some kind of universal truth or just what you think
- dolmen 4y agoWhen there are just 9 servers and a single developer like in the article your opinion doesn't matter.
- hakube 4y agoIt's not easy to maintain I can tell you
- altdataseller 4y agoI said easier, not easy.
- BilalBudhani 4y agoI don't see how it is easier to maintain. I follow a simple rule of thumb - the more abstractions you introduce the more complex it gets to maintain Life is easier if you can get away with lesser abstractions.
- holoduke 4y agoThe more tools you have the more difficult it is to maintain yourself. In a larger team with dedicated DevOps you might be right. But in my case where I maintain frontend, backend, databases, security, hosting and the servers I cannot have too much complexity. And i find container tools too complex to fully understand as a full stack software developer (not an expert in anything). Same for heavy weight frontend or backend frameworks. I like to keep it simple without too many abstractions on all layers. So far I managed to build apps used by millions of daily users for about 10 years.
- supz_k 4y agoGreat stuff. Off topic, any pgDash alternatives for MYSQL?
- rtpg 4y agoVery cool setup. I know it's a bit silly, but anyone building out cloud services should look at this post and think about how easy it is to set up this. This is just slightly more complex than a static site, has a bit of heterogenity, but otherwise is a lot of known tools. Can your stack provide all of this without breaking the bank for the people signing up? Can it be done without having to learn a bunch of bespoke systems? I think that Heroku's success in particular is totally down to managing this bespoke-ness balance (that is still kinda missing from container-based setups).
- threeseed 4y ago> This is just slightly more complex than a static site That's because their product is only slightly more complex than a static site. I wish every project was as basic as a web server and database but it's pretty rare these days.
- rjh29 4y agoAlso pragmatically choosing data centres in one physical location (even though this is a health checking service) rather than immediately going for highly available Aurora Serverless multi-AZ or something that would inflate the bill 10x.
- nojito 4y agoMost saas companies are simple crud apps.
- threeseed 4y agoNo they aren't. Even seemingly basic ones like Notion hide a lot of complexity. What you're seeing is survivorship bias where the SaaS companies that are doing well are the ones that can provide a simple, clean UI whilst adding the functionality that you need for the product to be useful.
- capableweb 4y ago> I wish every project was as basic as a web server and database but it's pretty rare these days. Is it? What functionality are you thinking about that absolutely require more than that? I find that most ideas even today can be solved with a web server + DB, while developers today like to over-engineer things and think about scaling a service for a million users while they haven't even figured out the value proposition of their product/service yet.
- thunfisch 4y agoI've been using healthchecks.io for years now. Awesome tool. Small scope, does the things it intends to do really well. Can highly recommend. I'm surprised that the machines are that large, there must be a bunch of people using the service nowadays :) I remember the setup being even smaller in previous blog posts. But I really like it. Small, easy to reason about. Makes debugging when stuff goes wrong quite easy. I'm a bit surprised that there's no config management listed though - seems like it really is just Fabric + a bunch of scripts. But hey, if it works, great!
- tothrowaway 4y agoCan you share more details about what the 4 HAProxy servers are doing? I run a SaaS too, but on a single beefy machine handling web and DB stuff. I'm curious what I'm missing out on not using HAProxy in front.
- russellendicott 4y agoIn the post he says they're for SSL termination and for rolling upgrades of the servers behind them.
- cuu508 4y agoTo answer the question of why four of them: * the traffic from the monitored systems comes with spikes. Looking at netdata graphs, currently the baseline is 600 requests/s, but there is a 2000 requests/s spike every minute, and 4000 requests/s spike every 10 minutes. * want to maintain redundancy and capacity even when a load balancer is removed from DNS rotation (due to network problems, or for upgrade) There are spare resources on the servers, especially RAM, and I could pack more things on fewer hosts. But, with Hetzner prices, why bother? :-)
- tothrowaway 4y agoI didn't realize you were getting that many requests per second. That's quite remarkable when you only have 18 app cores available. In my experience, you're lucky if you can get Django to service a request in 10ms. So I'd expect closer to 40 cores to handle that level of traffic. I suppose since it is spiky, and not a particularly latency sensitive application, you can get by with fewer cores.
- cuu508 4y agoI've got a small program written in Golang for handling the HTTP "ping" requests. There's also rate limiting, not every request makes it past NGINX.
- deleted 4y ago[deleted]
- mherrmann 4y agoAnother happy healthchecks.io user here, for many many years. The service has always worked great for me and I root for Peteris. Can happily recommend.
- mfontani 4y ago> On app servers: [...] Are "app" servers the haproxy ones or the "web servers", or other ones not mentioned?
- cuu508 4y agoApp servers are the ones running NGINX + uwsgi + the Django app.
- piccachew 4y agoI wonder how much time do you need to spend in order to maintain such infrastructure?
- tpetry 4y agoWith ubuntu and the automatic security updates you don‘t have to spend any time maintaining it. I haven‘t done anything in months, the system is applying security updates automatically.
- UmbertoNoEco 4y agoIs this on by default? I just installed Ubuntu 22 in a new VPS and was just thinking about this.
- Nextgrid 4y agoSee https://askubuntu.com/a/204 https://askubuntu.com/a/204 - also applies to Debian (and most likely other derivatives) too.
- UmbertoNoEco 4y agoThanks, I, in fact, did that already, so good to know it is setup.
- sys42590 4y agoHardware redundancy aside (and the good cost efficiency at Hetzner), reading the title, I'm getting the impression, the bus factor at healthchecks.io seems to be 1. If I'd run a one man show type of business, I'd love to have some kind of plan B in case I'd be incapacitated for more than half a day. THAT would be an interesting read to me.
- cuu508 4y agoYes, the bus factor is 1, and it's bugging me too. I think any realistic plan B involves expanding the team.
- florinutz 4y ago
- albertgoeswoof 4y agoWhat volume of data are you storing in postgres? Any reason not to use a hosted postgres provider? At first glance it seems like a lot of compute for the service but curious about scale
- cuu508 4y agoAround 200 write tx/s as a baseline. Spikes to 2000 write tx/s at the start of every minute, and 4000 write tx/s every 10 minutes. > Any reason not to use a hosted postgres provider? * Cost * Schrems II * From what I remember, both Google Cloud SQL and AWS RDS used to have mandatory maintenance windows. The fail-over was not instant, so there was some unavoidable downtime every month. This was a while ago – maybe it is different now.
- avrionov 4y agoI don't think that this will work differently than you manually updating your databases. GCP claims that maintenance downtime is less than 60 seconds.
- euph0ria 4y agoWhat software do you use to manage/develop against the postgres db?
- anyfactor 4y agoI respect that they use a combination of git tools just for version controlling. I understand that you can do everything with the git cli but my experience is that the more I tried to limit myself to the cli the messy my version controlling become.
- anyfactor 4y agoQuestion for OP. For the healthcheck's integration solution (email, signal, discord....), how did you go about doing that? I would love to hear about all that. I did find a section about Signal's integration.
- cuu508 4y agoStarted with just the email integration, and added other integration types over time, one by one. A few were contributed as GitHub PRs. The Signal one took by far the most effort to get going. But, for ideological reasons, I really wanted to have it :-) Unlike most other services, Signal doesn't have public HTTP API for sending messages. Instead you have to run your own local Signal client and send messages through it. Healthchecks is using signal-cli: https://github.com/AsamK/signal-cli https://github.com/AsamK/signal-cli
- comprev 4y ago>A dedicated laptop inside a dedicated backpack, for dealing with emergencies while away from the main PC. This line here shows much thought has gone into "what if...." situations. Bravo! I too have a similar setup - I have a contractual responsibility to my clients, and if anything goes wrong with my main equipment, I can resume ASAP. This includes a dedicated smartphone for hotspotting if my internet connection dies.
- thelittleone 4y agoA business bug out bag or BCP in a back pack.
- fsloth 4y agoI notice the author is using Braintree for payments. Is it a good option for accepting payments from global audiences?
- influx 4y agoI'm also curious why this over stripe?
- cuu508 4y agoWhen I started, Stripe was not yet available in my country, Latvia (it now is). Personally I've had good experience with Braintree. Particularly their support has been impressively good – they take time to respond, but you can tell the support agents have deep knowledge of their system, they have access to tools to troubleshoot problems, and they don't hesitate to escalate to engineering.
- fasteo 4y ago> HAProxy 2.2 for terminating TLS connections, and load balancing between app servers. Enables easy rolling updates of application servers. Wonder why he is not using hetzner's load balancer [1]. At least cost-wise, the savings are huge (4xAX41-NVME are around 160€; LB31 - the most expensive one - is 36€) [1] https://www.hetzner.com/cloud/load-balancer https://www.hetzner.com/cloud/load-balancer
- baobob 4y agoThat product has a serious lack of docs, and I know from experience some off-cloud vendor balancers have crappy architectures with low overall throughput limits. Their product could be a shared CPU VPS under the hood or something deeply baked into their network architecture, guilty until proven innocent
- withinboredom 4y agoI believe the load balancer is pretty new and it is probably the only service that allows connecting to dedicated servers -- which is separate from their cloud offering.
- fnomnom 4y agoi tried to use it out of curiosity but the documentation was really lacking and the workflow of creating the networks is bad
- tyingq 4y agoIt's kind of interesting that there's more horsepower/cores in the load balancers than in the downstream web/app servers...4xAX41 vs 3xAX41. So I imagine something is different about the workload there versus a typical load balancer.
- dolmen 4y agoThe load balancer is at the core of the product reliability. That seems a good reason to completely control it instead of relying on a black box.
- matteocontrini 4y agoHetzner's LB doesn't really scale. LB31 should be based on CX31 which is just 2 vCPU compared to the 12 threads of a single AX41 dedicated server.
- 8fingerlouie 4y agoI never realized that Healthchecks.io was a one man show. I'm almost ashamed to say that i've been using the free tier for years. I moved from self hosted to the free tier when i killed off all of self hosted services, and i don't have much legitimate use for it, having it attached to a backup of a Raspberry Pi running HomeBridge in my summerhouse :) I do use it for a purpose it probably was not designed for :) In my summerhouse, one of the circuit breakers trips every now and then (1-2 times per year) for no apparent reason, and since the fridge is connected to that breaker, i very much fear arriving at a summer house where the fridge has been unpowered for a couple of weeks. The solution to that of course, was to setup a systemd timer on a Raspberry Pi (that also runs HomeBridge), which pings Healthchecks.io every 15 minutes. If it misses 2 pings it raises an alert. So far i've had 2 notifications in the 2+ years it's been running, and both have been internet connection related, so monitoring the situation has apparently changed something :)
- stavros 4y agoI use it for the same purpose as well, a friend left on a trip and his fridge lost power, spoiling thousands of dollars worth of his small baking business materials. I set up a phone with Tasker, if the phone stops charging or fails to check in, it sends an alert. Luckily, and as you say, monitoring changed the situation, as it never failed since.
- finnlab 4y agoNo fair! You changed the outcome by measuring it!
- stavros 4y agoPipe down, Farnsworth!
- dicknuckle 4y ago"The doctor is in" or "It doesn't do that when a mechanic looks at it"
- 4y ago
- freemint 4y agoDo you use Hetzner for DNS too?
- gbourant 4y agoI'm really proud of this project. Personally, it has been a case study for a long time. It's the absolute epitome of pragmatic engineering solutions that actually works and turn them into a SaaS as an open source project. Well done!
- iosjunkie 4y agoGreat write up. I would love to hear more detail how WireGuard is setup.
- cuu508 4y agoI use vanilla Wireguard (the "wg" command and the "wg-quick" service). I set up new hosts and update peer configuration using Fabric tasks. It may sound messy, but works fine in practice. For example, to set up Wireguard on a new host: * On the new host, I run a Fabric task which generates a key pair and spits out the public key. The private key never leaves the server. * I paste the public key in a peer configuration template. * On every host that must be able to contact the new host, I run another Fabric task which updates the peer configuration from the template ("wg syncconf"). One thing to watch out is any services that bind to the Wireguard network interface. I had to make sure on reboot they start after wg-quick.
- iosjunkie 4y agoThanks for taking the time to respond!
- artemonster 4y agoSveiki! @cuu508 how do you think opensourcing selfhosted version of your product impacted your sales? Positively, negatively?
- cuu508 4y agoI can't say definitely, but my gut feeling is positively. A. What if another operator takes the source code, and starts a competing commercial service? I've seen very few (I think 1 or 2) instances of somebody attempting a commercial product based on Healthchecks open-source code. I think that's because it's just a lot of work to run the service professionally, and then even more work to find users and get people to pay for it. B. What if a potential customer decides to self-host instead? I do see a good amount of enthusiasts and companies self-hosting their private Healthchecks instance. I'm fine with that. For one thing, the self-hosting users are all potential future clients of the hosted service. They are already familiar and happy with the product, I just need to sell the "as a service" part.
- CoolCold 4y agoI'd like to here more on your usage of SSLMate and SOPs. For SSLMate I hardly understand what problem it solves (even if we forget about having Let's Encrypt free certs). For SOPs - how it's integrated in your flow. Thanks in advance.
- cuu508 4y agoSSLMate it is a certificate reseller with a convenient (for me) interface – a CLI program. It's no fun copy-pasting certificates from email attachments. I'm using both RSA and ECDSA certificates (RSA for compatibility with old clients, ECDSA for efficiency). I'm not sure but looks like ECDSA is not yet generally available from Lets Encrypt. On sops: the secrets (passwords, API keys, access tokens) are sitting in an encrypted file ("vault"). When a Fabric task needs secrets to fill in a configuration file template, it calls sops to decrypt the vault. My Yubikey starts flashing, I tap the key, Fabric task receives the secrets and can continue.
- CoolCold 4y agoThanks for clarification!
- upupandup 4y agomy concerns for using Hetzner is the latency. Servers are cheap but they are located in Scandinavia or Germany. So by the time your browser requests an API call to a Hetzner server, there is a 100~170ms overhead. Compare that to a hosting provider that has datacenters in US West and Asia
- Mo3 4y agoNot only that. It's single-DC, and Hetzner is somewhat known for having serious DC-wide network failures from time to time. The hardware used is consumer hardware too. This is literally the opposite of what I would expect for a SaaS with uptime requirement... I feel like OP/the developers behind this might have made a really terrible choice and could be paying more in manpower than the savings they have by using a budget server discounter Don't get me wrong, I have nothing against Hetzner, but this seems like an unfit usecase. This is literally what you use cloud platforms for..
- upupandup 4y agoI didn't know that thanks for sharing. Has there been major cases recently? really puzzling that a healthcheck provider itself is not accounting for this potential issue. Is there some sort of edgy reverse proxy here? Doesn't seem like it. Even then there would be a big latency.
- Mo3 4y agohttps://status.hetzner.com/ https://status.hetzner.com/ We haven't been using them for anything in production, so I am not 100% sure - they have a backbone outage right now however.. There are no edge / reverse proxies being described in the OP article, just a HAProxy configuration within the same DC.
- upupandup 4y agointeresting....im also concerned that with the recent geopolitical developments, Hetzner could already be at crosshairs
- andreisbc 4y agoCame, saw, read, bookmarked
- BilalBudhani 4y agoOne thing I just realise is OP can replace > HAProxy > NGINX > SSLMate with Caddy [0] which would reduce the number of dependencies and complexity involve in infra. I've been using Caddy for a very long time now and it has been working out so well. [0] https://caddyserver.com https://caddyserver.com
- ddorian43 4y agoHe can replace them with NGINX too I believe.
- BilalBudhani 4y agoNGINX doesn't have first class support for auto generating SSL, you would still have to rely on third party service/dependency for that. Whereas, with Caddy you would solve all 3 problems with single tool.
- mardix 4y ago(Shameless plug) Here's a tiny PAAS that can be used if Caddy is too complex. https://github.com/mardix/sailor https://github.com/mardix/sailor Sailor is a tiny PaaS to install on your servers/VPS that uses git push to deploy micro-apps, micro-services, sites with SSL, on your own servers or VPS.
- BilalBudhani 4y agoSailor looks really cool! Just took a peak at the codebase and it already seems very interesting compared to other self hosting PaaS solutions.
- dmje 4y agoMinor usability suggestion that literally no one will likely care about but I notice it all the time: If you're writing a blog post about your service then chances are you're interested in driving people to the main website about that service? So...make it easy for those potential users to get there. At the top of your blog have a prominent "what is [service]" link to the marketing home / about page or a call out box. This guy does it as a CTA after the post which is a start but many, many people don't. That then leaves me and other potential users to have to figure out how to get to the homepage. I'll do it but you'll lose many others.
- Nextgrid 4y agoIndeed it's very annoying when company blogs have their logo or homepage nav link go to the blog's home page instead of their main landing page.
- andyjohnson0 4y agoSeconded. I'd never heard of healthchecks.io and was curious enough to strip off the blog-related parts of the url to get to their main site (not totally trivial on a phone browser). I almost didnt bother but I'm glad I did because it looks like a really useful service that i might have uses for. I'd assumed that it was a healthcare-related business tbh. (Also: super-informative post.)
- rdoherty 4y agoThis is fascinating. I am curious how sites like this (and others such as Pingdom, 'up time' checkers, etc) handle scheduled tasks that have to run at high frequencies? Cron on one machine? Celery beat?
- cuu508 4y agoHealthchecks runs a loop of 10 send any due notifications 20 SLEEP 2 30 GOTO 10 The actual loop [1] is of course a little more complicated, and is being run concurrently on several machines. [1] https://github.com/healthchecks/healthchecks/blob/09a99d3e9c8329d20505e058a6514ba94cb17bb7/hc/api/management/commands/sendalerts.py#L151 https://github.com/healthchecks/healthchecks/blob/09a99d3e9c...
- adrienjarthon 4y agoIn my case (updown.io), I use a "scheduler" process (one per machine) which runs the following loop: every 5 seconds it fetches the checks that needs to execute in the next 5 seconds, ordered by time of run ascending. Then it iterates through them one by one, wait with a small sleep until the precise running time, and then push the job into a background job queue (sidekiq / redis in my case). This allows for a very precise timing of execution even if there is hundreds jobs per seconds, and a good distribution over time (instead of firing 100 jobs every seconds in spikes, it can schedule one every 10ms for example)
- sdevonoes 4y ago> Wireguard for private networking between the servers. I'm curious: doesn't Hetzner provide some sort of private virtual cloud? (Digitalocean calls that VPC)
- metafunctor 4y agoThey do provide private networks now, but AFAIK they didn't in 2017. Today, one could use “vSwitch” or “Cloud Network”, or both, to set up private networking. But, if you want to easily encrypt all traffic, Wireguard might still be a good idea.
- sdevonoes 4y agoGood to know. > But, if you want to easily encrypt all traffic, Wireguard might still be a good idea. But, if one is already using https I guess encrypting the traffic would be unnecessary, no?
- adamckay 4y agoThey do for their cloud instances [0], but not their dedicated server offerings. I've seen on their custom solutions page [1] you can pay extra to have your own private interconnect or manged switches, but I've never used them nor heard much about them. [0] https://www.hetzner.com/cloud https://www.hetzner.com/cloud - see "FEATURES" [1] https://www.hetzner.com/custom-solutions https://www.hetzner.com/custom-solutions
- sdevonoes 4y ago> Elastic Email for sending transactional email. > Fastmail for sending and receiving support email. As a SaaS newbie, is it preferred to have two different email providers for transactional and support email? Why wouldn't one use Elastic Email for support email?
- andyjohnson0 4y agoA saas platform usually generates transactional email by calling an http api, and it is usually outgoing-only (smtp). Its not designed for sending ad hoc messages. Also, transactional email services police their usage to ensure that they are trusted as sources of legitimate bulk email. It would be hard to get away with sending much transactional email via a normal email provider without having your account shut down. In contrast. Support email is handled by people and is bidirectional and ad-hoc. Fastmail provides smtp/imap/webmail, spam filtering, mailboxes, etc. and is a good match for handling support.
- gituliar 4y agoGreat post, thanks for sharing your experience. Could you also share what is you strategy for marketing and acquiring new users ?