4 ms·

Monit for automated restarts. Hardware raid cards. Plus an architecture that is robust. In my experience, good dedicated servers practically never crash. You

Monit for automated restarts. Hardware raid cards. Plus an architecture that is robust. In my experience, good dedicated servers practically never crash. You might lose a HDD every few years, but that is not urgent to fix if you have a good raid. Avoid most cloud services. Heroku, Rackspace, AWS all had much more outages than Hetzner. Plus they'll sometimes force reboot or force migrate ( =pause) your instances. So if you go cloud, you'll need failover, distributed database, all that messy and complicated stuff. If you go dedicated, it's much easier and you only need to keep that one box running. Plus, honestly, would your customers really mind if you're offline for 5 minutes? My dedicated hoster also has a service where they will monitor standard services like Apache, Postgresql, Rails for you and restart as needed. They have 5-10 minutes response time in my experience and I belive its good enough :) Also, going dedicated makes it affordable to overprovision 10x the hardware you need, so you will practically never have a traffic spike high enough to cause issues. With Heroku / AWS on the other hand, everyone else will also be scaling up when their cloud has hiccups, so your on-demand instances might not start when you need them. Anyway, Hetzner dedicated + raid + monit is how I've been running my SaaS company for 10+ years. And I don't even remember which year I last had an issue that was both urgent and required my attention. The Hetzner ppl can exchange HDDs just fine without me. C++ core, Ruby website, Postgresql and RabbitMQ. 100GB database, 5TB customer data.

cddotdotslash 6y ago
Ignoring the fact that this sounds like an ad for Hetzner, I'm not sure this is good _generic_ advice. It may be good for _some_, but the vast majority of single SaaS founders have access to platforms now (mostly via big IaaS providers) that allow them to build, develop, and deploy without ever worrying about RAID, Apache servers or Postgresql restarts. > Plus they'll sometimes force reboot or force migrate ( =pause) your instances. Extremely rare, but probably happens at a similar rate as your "single box" dedicated provider losing an HDD or having a datacenter blip. My point here is that what you described sounds like the kinds of things SaaS developers needed to worry about ~10 years ago. The platforms of today aren't perfect, but they abstract away 90% of that and allow you to focus on business logic, which is exactly what a single SaaS developer should be doing.
- heliodor 6y ago
  The topic is reliability, not ease of use. What was described is not a big deal to pull off. If Hetzner fits your use case (now and future case) then it's a great way to go.
- Silhouette 6y ago
  As someone else who has been happily running a small business on dedicated hardware with a managed hosting service for years, I share some of the GP's scepticism about modern cloud hosting. Modern platforms should remove most of the complexity of operating routine infrastructure and allow you to focus on your business logic, but it doesn't always work out that way. Just the fact that your VMs have a significant chance of being forcibly shut down with little notice is a significant downside, for example. As a solo operator, you now have to arrange all the automatic scaling and failover configuration on your cloud host as well (possibly at considerable extra cost for capacity you might not be using 99% of the time) and you have all the 24/7/365 monitoring problems that OP was asking about. Cloud services are also notorious for obfuscating their pricing so it's hard to work out the TCO. In my experience, arguments that cloud hosting works out much cheaper overall tend to be based on rather optimistic assumptions. It might be true if you lease some VMs at carefully chosen sizes and then set everything up yourself including scaling things down again any time you don't need them. However, once you start using the automatic services that actually do something for you beyond supplying a machine on demand, the prices might jump 3x or more (sometimes much more, like orders of magnitude) compared to ordering the equivalent basic resources and setting the same functionality up manually. Then there are the security and compatibility updates. The basic cloud services tend to be provided as-is and it's up to you to ensure everything gets updated when it needs to be. Or again, you might be able to get a more automated service that does some of this for you, but it will come with a pricing premium. Meanwhile, a solo operator using a more traditional managed service probably doesn't have to worry much about any of this, because those services will often be happy to take care of things like setting up your redundant database servers or monitoring the security mailing lists and applying emergency patches very quickly so you don't have to. That's the level of individual service and advice they tend to offer to distinguish themselves from the generic cloud hosting services. Obviously you do pay extra for that management service compared to just a basic hosting arrangement, but whether you pay more than you'd have paid trying to do all of it yourself on AWS or Azure or even DO is another question.
leesalminen 6y ago
> Plus, honestly, would your customers really mind if you're offline for 5 minutes? Idk what kind of software you work on but in my slice of the B2B SaaS world, 5 minutes of downtime during business hours would generate 100s of support tickets with very angry users letting us know they couldn’t do their job.
- kevinconroy 6y ago
  > 5 minutes of downtime during business hours would generate 100s of support tickets Perhaps part of the answer is that if you're a solo founder who is staying solo, try not to create a business where you have these kinds of dynamics. OR, if you are creating one that does, scale up past being solo ASAP.
  - leesalminen 6y ago
    Fair point and agree 100%.
    - o-__-o 6y ago
      I would argue there are some services that need 5 nines (or more) of availability. Your support ticketing system and company web presence are two I can think of. I provide the ticketing service myself but I outsource my web presence (the landing page not the app) to a third party service. My thought is the web site should never go down. If the ticketing system is down it means things are beyond hosed :)
- Silhouette 6y ago
  If you are a solo SAAS operator and have enough customers to generate hundreds of support tickets in 5 minutes, maybe hiring more help several years ago would have been a good idea? I doubt you're trying to solve the same problems as the GP at that scale, or probably even the same types of problems.
  - leesalminen 6y ago
    That’s fair. I forgot all this was in context of solo founder. Back when I was solo we definitely didn’t have that many users.
benevol 6y ago
> If you go dedicated, it's much easier and you only need to keep that one box running. Can/should you really run everything on just 1 box, with rather huge projects? Why not gain redundancy/uptime/peace of mind by having multiple (redundant) dedicated boxes?
- wongarsu 6y ago
  Everything else being equal, having more boxes leads to more failures. If each server fails on average every 10 years, then with ten servers you expect on average one failure every year.
  - benevol 6y ago
    True. But a failure of a redundant server (say, 1 out of 3 application servers) would then not force you to cancel your night/weekend/vacation.
  - vishnugupta 6y ago
    Even day to day operations/maintenance becomes that much painful. Think of all those security patches, zero down time deployments, failovers, log aggregation, monitoring setup so on and so forth.
- toast0 6y ago
  Two servers with manual failover is probably the sweetspot; especially if you exercise the failover often. Confirming after every release is best, but once a month is probably fine. Then your incident response can be verify the lead server is dead, or ensure it's dead and switch to the alternate. But one server is way more convenient, until it isn't.
  - jlokier 6y ago
    To avoid configuring every service for high-availability on two servers, you can do surprisingly well with a replicated VM and DRBD or equivalent. In the event of unplanned failover, it looks to the VM like an unplanned abrupt reboot took place. In reality it reboots, usually very fast, on the other host. All services running inside can recover in the usual way (journalled filesystems, databases, programs restart), and don't need any high-availability configuration or replicas configured. You do need to ensure I/O is committed durably across the network, including I/O barriers. This is a combination of VM host, filesystem and DRBD config. (It is actually possible to do this with the VM not even seeming to reboot, so network connections and processes are unaffected by the fail-triggered instant migration. This is done by running VMs in synchronised tandem and is a rather more advanced technique. I've never used it.)
e12e 6y ago
I agree with most of the points here, but it's interesting that you mention: > In my experience, good dedicated servers practically never crash One of my toy servers (ecc ram/xeon cpu - but bought "second hand" via hetzner's auction) disappeared the other day. I thought maybe a disk had failed - but I couldn't bring it up in their network booted rescue mode - and requested a "hands on" power cycle - and after a few minutes the server was up again: > Dear Client. > A fault in your neighbor servers PSU tripped the fuse of the small rack segment which your server is located in too. We have fixed the issue and now your OS is back online. Now, I think that box had a 700-900 days up-time before - I didn't really have to do anything (or pay) to get it back up. But it was kind of surprising. I guess all I'm saying is that I do like cheap, dedicated servers from hetzner - but if you need to guarantee five nines uptime, the architecture part is important.
- Silhouette 6y ago
  I guess all I'm saying is that I do like cheap, dedicated servers from hetzner - but if you need to guarantee five nines uptime, the architecture part is important. Five-nines is less than 10 minutes of downtime per year. I doubt anyone is really guaranteeing that without 24/7 active monitoring and maintaining extensive automated failover systems, which is already several full-time jobs. No solo operator is credibly providing that level of service.
  - sokoloff 6y ago
    Agreed. I doubt most people/services should build to 'guarantee' even .9999. .9990 or .9995 is much cheaper, much easier, and probably closer to what your end user's network connectivity is anyway. (Yes, they're multiplicative, but if your user is connecting from a single-path residential connection and a $50 router, your 5th nine isn't needed to demolish their local 5-10 hours of downtime per year.)
    - CodesInChaos 6y ago
      That argument works pretty well in b2c, but in b2b your customers often insist on high uptimes, even if they benefit little from them. Though the promised uptime might not matter that much in practice, since the penalties for a couple of hours of downtime are often affordable.
jlokier 6y ago
I had a dedicated server at Hetzner that would completely lock up every few weeks to months randomly, so they aren't always reliable. I ran it for about a year as I always had other things to deal with, but the abrupt loss of my services was kind of embarrassing and I should have reported it sooner than I did. To be fair to Hetzner, eventually when I reported it they immediately took the initiative to replace it, no questions, and gave me options for when I'd like that to happen. Never had any problems since.

cddotdotslash 6y ago

Ignoring the fact that this sounds like an ad for Hetzner, I'm not sure this is good _generic_ advice. It may be good for _some_, but the vast majority of single SaaS founders have access to platforms now (mostly via big IaaS providers) that allow them to build, develop, and deploy without ever worrying about RAID, Apache servers or Postgresql restarts. > Plus they'll sometimes force reboot or force migrate ( =pause) your instances. Extremely rare, but probably happens at a similar rate as your "single box" dedicated provider losing an HDD or having a datacenter blip. My point here is that what you described sounds like the kinds of things SaaS developers needed to worry about ~10 years ago. The platforms of today aren't perfect, but they abstract away 90% of that and allow you to focus on business logic, which is exactly what a single SaaS developer should be doing.

heliodor 6y ago
The topic is reliability, not ease of use. What was described is not a big deal to pull off. If Hetzner fits your use case (now and future case) then it's a great way to go.
Silhouette 6y ago
As someone else who has been happily running a small business on dedicated hardware with a managed hosting service for years, I share some of the GP's scepticism about modern cloud hosting. Modern platforms should remove most of the complexity of operating routine infrastructure and allow you to focus on your business logic, but it doesn't always work out that way. Just the fact that your VMs have a significant chance of being forcibly shut down with little notice is a significant downside, for example. As a solo operator, you now have to arrange all the automatic scaling and failover configuration on your cloud host as well (possibly at considerable extra cost for capacity you might not be using 99% of the time) and you have all the 24/7/365 monitoring problems that OP was asking about. Cloud services are also notorious for obfuscating their pricing so it's hard to work out the TCO. In my experience, arguments that cloud hosting works out much cheaper overall tend to be based on rather optimistic assumptions. It might be true if you lease some VMs at carefully chosen sizes and then set everything up yourself including scaling things down again any time you don't need them. However, once you start using the automatic services that actually do something for you beyond supplying a machine on demand, the prices might jump 3x or more (sometimes much more, like orders of magnitude) compared to ordering the equivalent basic resources and setting the same functionality up manually. Then there are the security and compatibility updates. The basic cloud services tend to be provided as-is and it's up to you to ensure everything gets updated when it needs to be. Or again, you might be able to get a more automated service that does some of this for you, but it will come with a pricing premium. Meanwhile, a solo operator using a more traditional managed service probably doesn't have to worry much about any of this, because those services will often be happy to take care of things like setting up your redundant database servers or monitoring the security mailing lists and applying emergency patches very quickly so you don't have to. That's the level of individual service and advice they tend to offer to distinguish themselves from the generic cloud hosting services. Obviously you do pay extra for that management service compared to just a basic hosting arrangement, but whether you pay more than you'd have paid trying to do all of it yourself on AWS or Azure or even DO is another question.

leesalminen 6y ago

> Plus, honestly, would your customers really mind if you're offline for 5 minutes? Idk what kind of software you work on but in my slice of the B2B SaaS world, 5 minutes of downtime during business hours would generate 100s of support tickets with very angry users letting us know they couldn’t do their job.

kevinconroy 6y ago
> 5 minutes of downtime during business hours would generate 100s of support tickets Perhaps part of the answer is that if you're a solo founder who is staying solo, try not to create a business where you have these kinds of dynamics. OR, if you are creating one that does, scale up past being solo ASAP.
- leesalminen 6y ago
  Fair point and agree 100%.
  - o-__-o 6y ago
    I would argue there are some services that need 5 nines (or more) of availability. Your support ticketing system and company web presence are two I can think of. I provide the ticketing service myself but I outsource my web presence (the landing page not the app) to a third party service. My thought is the web site should never go down. If the ticketing system is down it means things are beyond hosed :)
Silhouette 6y ago
If you are a solo SAAS operator and have enough customers to generate hundreds of support tickets in 5 minutes, maybe hiring more help several years ago would have been a good idea? I doubt you're trying to solve the same problems as the GP at that scale, or probably even the same types of problems.
- leesalminen 6y ago
  That’s fair. I forgot all this was in context of solo founder. Back when I was solo we definitely didn’t have that many users.

benevol 6y ago

> If you go dedicated, it's much easier and you only need to keep that one box running. Can/should you really run everything on just 1 box, with rather huge projects? Why not gain redundancy/uptime/peace of mind by having multiple (redundant) dedicated boxes?

wongarsu 6y ago
Everything else being equal, having more boxes leads to more failures. If each server fails on average every 10 years, then with ten servers you expect on average one failure every year.
- benevol 6y ago
  True. But a failure of a redundant server (say, 1 out of 3 application servers) would then not force you to cancel your night/weekend/vacation.
- vishnugupta 6y ago
  Even day to day operations/maintenance becomes that much painful. Think of all those security patches, zero down time deployments, failovers, log aggregation, monitoring setup so on and so forth.
toast0 6y ago
Two servers with manual failover is probably the sweetspot; especially if you exercise the failover often. Confirming after every release is best, but once a month is probably fine. Then your incident response can be verify the lead server is dead, or ensure it's dead and switch to the alternate. But one server is way more convenient, until it isn't.
- jlokier 6y ago
  To avoid configuring every service for high-availability on two servers, you can do surprisingly well with a replicated VM and DRBD or equivalent. In the event of unplanned failover, it looks to the VM like an unplanned abrupt reboot took place. In reality it reboots, usually very fast, on the other host. All services running inside can recover in the usual way (journalled filesystems, databases, programs restart), and don't need any high-availability configuration or replicas configured. You do need to ensure I/O is committed durably across the network, including I/O barriers. This is a combination of VM host, filesystem and DRBD config. (It is actually possible to do this with the VM not even seeming to reboot, so network connections and processes are unaffected by the fail-triggered instant migration. This is done by running VMs in synchronised tandem and is a rather more advanced technique. I've never used it.)

e12e 6y ago

I agree with most of the points here, but it's interesting that you mention: > In my experience, good dedicated servers practically never crash One of my toy servers (ecc ram/xeon cpu - but bought "second hand" via hetzner's auction) disappeared the other day. I thought maybe a disk had failed - but I couldn't bring it up in their network booted rescue mode - and requested a "hands on" power cycle - and after a few minutes the server was up again: > Dear Client. > A fault in your neighbor servers PSU tripped the fuse of the small rack segment which your server is located in too. We have fixed the issue and now your OS is back online. Now, I think that box had a 700-900 days up-time before - I didn't really have to do anything (or pay) to get it back up. But it was kind of surprising. I guess all I'm saying is that I do like cheap, dedicated servers from hetzner - but if you need to guarantee five nines uptime, the architecture part is important.

Silhouette 6y ago
I guess all I'm saying is that I do like cheap, dedicated servers from hetzner - but if you need to guarantee five nines uptime, the architecture part is important. Five-nines is less than 10 minutes of downtime per year. I doubt anyone is really guaranteeing that without 24/7 active monitoring and maintaining extensive automated failover systems, which is already several full-time jobs. No solo operator is credibly providing that level of service.
- sokoloff 6y ago
  Agreed. I doubt most people/services should build to 'guarantee' even .9999. .9990 or .9995 is much cheaper, much easier, and probably closer to what your end user's network connectivity is anyway. (Yes, they're multiplicative, but if your user is connecting from a single-path residential connection and a $50 router, your 5th nine isn't needed to demolish their local 5-10 hours of downtime per year.)
  - CodesInChaos 6y ago
    That argument works pretty well in b2c, but in b2b your customers often insist on high uptimes, even if they benefit little from them. Though the promised uptime might not matter that much in practice, since the penalties for a couple of hours of downtime are often affordable.

jlokier 6y ago

I had a dedicated server at Hetzner that would completely lock up every few weeks to months randomly, so they aren't always reliable. I ran it for about a year as I always had other things to deal with, but the abrupt loss of my services was kind of embarrassing and I should have reported it sooner than I did. To be fair to Hetzner, eventually when I reported it they immediately took the initiative to replace it, no questions, and gave me options for when I'd like that to happen. Never had any problems since.