5 ms·

Hetzner is definitely an interesting option. I’m a bit scared of managing the services on my own (like Postgres, Site2Site VPN, …) but the price difference make

Hetzner is definitely an interesting option. I’m a bit scared of managing the services on my own (like Postgres, Site2Site VPN, …) but the price difference makes it so appealing. From our financial models, Hetzner can win over AWS when you spend over 10~15K per month on infrastructure and you’re hiring really well. It’s still a risk, but a risk that definitely can be worthy.

adamcharnock 9mo ago
You sum it up very neatly. We've heard this from quite a few companies, and that's kind of why we started our ours. We figured, "Okay, if we can do this well, reliably, and de-risk it; then we can offer that as a service and just split the difference on the cost savings" (plus we include engineering time proportional to cluster size, and also do the migration on our own dime as part of the de-risking)
iso1631 9mo ago
> I’m a bit scared of managing the services on my own (like Postgres, Site2Site VPN, …) Out of interest, how old are you? This was quite normal expectation of a technical department even 15 years ago.
- christophilus 9mo ago
  I’m curious to know the answer, too. I used to deploy my software on-prem back in the day, and that always included an installation of Microsoft SQL Server. So, all of my clients had at least one database server they had to keep operational. Most of those clients didn’t have an IT staff at all, so if something went wrong (which was exceedingly rare), they’d call me and I’d walk them through diagnosing and fixing things, or I’d Remote Desktop into the server if their firewalls permitted and fix it myself. Backups were automated and would produce an alert if they failed to verify. It’s not rocket science, especially when you’re talking about small amounts of data (small credit union systems in my example).
- infecto 9mo ago
  No it was not. 15 years ago Heroku was the rage. Even the places that had bare metal usually had someone running something similar to devops and at least core infrar was not being touched. I am sure places existed but 15 years while far away was already pretty far along from what you describe. At least in SV.
  - acdha 9mo ago
    Heroku was popular with startups who didn’t have infrastructure skills but the price was high enough that anyone who wasn’t in that triangle of “lavish budget, small team, limited app diversity” wasn’t using it. Things like AWS IaaS were far more popular due to the lower cost and greater flexibility but even that was far from a majority service class.
    - infecto 9mo ago
      I am not sure if you are trying to refute my lived experience or what exactly the point is. Heroku was wildly popular with startups at the time, not just those with lavish budgets. I was already touching RDS at this point and even before RDS came around no organization I worked at had me jumping on bare metal to provision services myself. There always a system in place where someone helped out engineering to deploy systems. I know this was not always the case but the person I was responding to made it sound like 15 years ago all engineers were provisioning their own database and doing other times of dev/sys ops on a regular basis. It’s not true at least in SV.
      - sanderjd 9mo ago
        A tricky thing on this site is that there are lots of different people with very different kinds of experience, which often results in people talking past each other. A lot of people here have experience as zero-to-one early startup engineers, and yep, I share your experience that Heroku was very popular in that space. A lot of other people have experience at later growth and infrastructure focused startups, and they have totally different experiences. And other people have experience as SREs at big tech, or doing IT / infrastructure for non-tech fortune 500 businesses. All of these are very different experiences, and very different things have been popular over the last couple decades depending on which kind of experience you have.
- Lucasoato 9mo ago
  Ahah I'm 31, but deciding if it makes sense to manage your own db doesn't depend on the age of the CTO. See, turning up a VM, installing and running Postgres is easy. The hard part is keeping it updated, keeping the OS updated, automate backups, deploying replicas, encrypting the volumes and the backups, demonstrating to a third party auditor all of the above... and mind that there might be many other things I honestly ignore! I'm not saying I won't go that path, it might be a good idea after a certain scale, but in the first and second year of a startup your mind should 100% be on "How can I make my customer happy" rather than "We failed again the audit, we won't have the SOC 2 Type I certification in time to sign that new customer". If deciding between Hetzner and AWS was so easy, one of them might not be pricing its services correctly.
baby 9mo ago
I’m wondering if it makes sense to distribute your architecture so that workers who do most of the heavy lifting are in hetzner, while the other stuff is in costly AWS. On the other hand this means you don’t have easy access to S3, etc.
- rockwotj 9mo ago
  networking costs are so high in AWS I doubt this makes sense
  - mattbillenstein 8mo ago
    Depends on how data-heavy the work is. We run a bunch of gpu training jobs on other clouds with the data ending up in S3 - the extra transfer costs wrt what we save on getting the gpus from the cheapest cloud available, it makes a lot of sense. Also, just availability of these things on AWS has been a real pain - I think every startup got a lot of credits there, so flood of people trying to then use them.
wulfstan 9mo ago
I've just shifted my SWE infrastructure from AWS to Hetzner (literally in the last month). My current analysis looks like it will be about 15-20% of the cost - £240 vs 40-50 euros. Expect a significant exit expense, though, especially if you are shifting large volumes of S3 data. That's been our biggest expense. I've moved this to Wasabi at about 8 euros a month (vs about $70-80 a month on S3), but I've paid transit fees of about $180 - and it was more expensive because I used DataSync. Retrospectively, I should have just DIYed the transfer, but maybe others can benefit from my error...
- adamcharnock 9mo ago
  FYI, AWS offers free Egress when leaving them (because they were forced to be EU regulation, but they chose to offer it globally): https://aws.amazon.com/blogs/aws/free-data-transfer-out-to-internet-when-moving-out-of-aws/ https://aws.amazon.com/blogs/aws/free-data-transfer-out-to-i... But. Don't leave it until the last minute to talk to them about this. They don't make it easy, and require some warning (think months, IIRC)
  - wulfstan 9mo ago
    Extremely useful information - unfortunately I just assumed this didn't apply to me because I am in the UK and not the EU. Another mistake, though given it's not huge amounts of money I will chalk it up to experience. Hopefully someone else will benefit from this helpful advice.
  - sciencejerk 9mo ago
    Thank God for the EU regulations. USA has been too lax about cracking down on anti-competitive market practices
- deleted 9mo ago
  [deleted]
mrweasel 9mo ago
> I’m a bit scared of managing the services on my own I see it from the other direction, when if something fails, I have complete access to everything, meaning that I have a chance of fixing it. That's down to hardware even. Things get abstracted away, hidden behind APIs and data lives beyond my reach, when I run stuff in the cloud. Security and regular mistakes are much the same in the cloud, but I then have to layer whatever complications the cloud provide comes with on top. If cost has to be much much lower if I'm going to trust a cloud provider over running something in my own data center.
- iso1631 8mo ago
  Do you want the power to fix or do you want the paper to wave so you aren't held accountable. The main benefit of outsourcing to aws etc is that the CEO isn't yelling at you when it breaks, because their golf buddies are in the same situation.
objektif 9mo ago
No amount of money will make me maintain my own dbs. We tried it at first and it was a nightmare.
- g8oz 9mo ago
  It's worth becoming good at.
  - sanderjd 9mo ago
    Is it though? This is a genuine question. My intuition is that the investment of time / stress / risk to become good at this is unlikely to have high ROI to either the person putting in that time or to the business paying them to do so. But maybe that's not right.
    - Symbiote 9mo ago
      Managing the PostgreSQL databases is a medium to low complexity task as I see it. Take two equivalent machines, set up with streaming replication exactly as described in the documentation, add Bacula for backups to an off-site location for point-in-time recovery. We haven't felt the need to set up auto fail-over to the hot spare; that would take some extra effort (and is included with AWS equivalents?) but nothing I'd be scared of. Add monitoring that the DB servers are working, replication is up-to-date and the backups are working.
      - sanderjd 9mo ago
        This sounds medium to high complexity to me. You need to do all those things, and also have multiple people who know how to do them, and also make sure that you don't lose all the people who know how to do them, and have one of those people on call to be able to troubleshoot and fix things if they go wrong, and have processes around all that. (At least if you are running in production with real customers depending on you, you should have all those things.) With a managed solution, all of that is amortized into your monthly payment, and you're sharing the cost of it across all the customers of the provider of the managed offering. Personally, I would rather focus on things that are in or at least closer to the core competency of our business, and hire out this kind of thing.

adamcharnock 9mo ago

You sum it up very neatly. We've heard this from quite a few companies, and that's kind of why we started our ours. We figured, "Okay, if we can do this well, reliably, and de-risk it; then we can offer that as a service and just split the difference on the cost savings" (plus we include engineering time proportional to cluster size, and also do the migration on our own dime as part of the de-risking)

iso1631 9mo ago

> I’m a bit scared of managing the services on my own (like Postgres, Site2Site VPN, …) Out of interest, how old are you? This was quite normal expectation of a technical department even 15 years ago.

christophilus 9mo ago
I’m curious to know the answer, too. I used to deploy my software on-prem back in the day, and that always included an installation of Microsoft SQL Server. So, all of my clients had at least one database server they had to keep operational. Most of those clients didn’t have an IT staff at all, so if something went wrong (which was exceedingly rare), they’d call me and I’d walk them through diagnosing and fixing things, or I’d Remote Desktop into the server if their firewalls permitted and fix it myself. Backups were automated and would produce an alert if they failed to verify. It’s not rocket science, especially when you’re talking about small amounts of data (small credit union systems in my example).
infecto 9mo ago
No it was not. 15 years ago Heroku was the rage. Even the places that had bare metal usually had someone running something similar to devops and at least core infrar was not being touched. I am sure places existed but 15 years while far away was already pretty far along from what you describe. At least in SV.
- acdha 9mo ago
  Heroku was popular with startups who didn’t have infrastructure skills but the price was high enough that anyone who wasn’t in that triangle of “lavish budget, small team, limited app diversity” wasn’t using it. Things like AWS IaaS were far more popular due to the lower cost and greater flexibility but even that was far from a majority service class.
  - infecto 9mo ago
    I am not sure if you are trying to refute my lived experience or what exactly the point is. Heroku was wildly popular with startups at the time, not just those with lavish budgets. I was already touching RDS at this point and even before RDS came around no organization I worked at had me jumping on bare metal to provision services myself. There always a system in place where someone helped out engineering to deploy systems. I know this was not always the case but the person I was responding to made it sound like 15 years ago all engineers were provisioning their own database and doing other times of dev/sys ops on a regular basis. It’s not true at least in SV.
    - sanderjd 9mo ago
      A tricky thing on this site is that there are lots of different people with very different kinds of experience, which often results in people talking past each other. A lot of people here have experience as zero-to-one early startup engineers, and yep, I share your experience that Heroku was very popular in that space. A lot of other people have experience at later growth and infrastructure focused startups, and they have totally different experiences. And other people have experience as SREs at big tech, or doing IT / infrastructure for non-tech fortune 500 businesses. All of these are very different experiences, and very different things have been popular over the last couple decades depending on which kind of experience you have.
Lucasoato 9mo ago
Ahah I'm 31, but deciding if it makes sense to manage your own db doesn't depend on the age of the CTO. See, turning up a VM, installing and running Postgres is easy. The hard part is keeping it updated, keeping the OS updated, automate backups, deploying replicas, encrypting the volumes and the backups, demonstrating to a third party auditor all of the above... and mind that there might be many other things I honestly ignore! I'm not saying I won't go that path, it might be a good idea after a certain scale, but in the first and second year of a startup your mind should 100% be on "How can I make my customer happy" rather than "We failed again the audit, we won't have the SOC 2 Type I certification in time to sign that new customer". If deciding between Hetzner and AWS was so easy, one of them might not be pricing its services correctly.

baby 9mo ago

I’m wondering if it makes sense to distribute your architecture so that workers who do most of the heavy lifting are in hetzner, while the other stuff is in costly AWS. On the other hand this means you don’t have easy access to S3, etc.

rockwotj 9mo ago
networking costs are so high in AWS I doubt this makes sense
- mattbillenstein 8mo ago
  Depends on how data-heavy the work is. We run a bunch of gpu training jobs on other clouds with the data ending up in S3 - the extra transfer costs wrt what we save on getting the gpus from the cheapest cloud available, it makes a lot of sense. Also, just availability of these things on AWS has been a real pain - I think every startup got a lot of credits there, so flood of people trying to then use them.

wulfstan 9mo ago

I've just shifted my SWE infrastructure from AWS to Hetzner (literally in the last month). My current analysis looks like it will be about 15-20% of the cost - £240 vs 40-50 euros. Expect a significant exit expense, though, especially if you are shifting large volumes of S3 data. That's been our biggest expense. I've moved this to Wasabi at about 8 euros a month (vs about $70-80 a month on S3), but I've paid transit fees of about $180 - and it was more expensive because I used DataSync. Retrospectively, I should have just DIYed the transfer, but maybe others can benefit from my error...

adamcharnock 9mo ago
FYI, AWS offers free Egress when leaving them (because they were forced to be EU regulation, but they chose to offer it globally): https://aws.amazon.com/blogs/aws/free-data-transfer-out-to-internet-when-moving-out-of-aws/ https://aws.amazon.com/blogs/aws/free-data-transfer-out-to-i... But. Don't leave it until the last minute to talk to them about this. They don't make it easy, and require some warning (think months, IIRC)
- wulfstan 9mo ago
  Extremely useful information - unfortunately I just assumed this didn't apply to me because I am in the UK and not the EU. Another mistake, though given it's not huge amounts of money I will chalk it up to experience. Hopefully someone else will benefit from this helpful advice.
- sciencejerk 9mo ago
  Thank God for the EU regulations. USA has been too lax about cracking down on anti-competitive market practices
deleted 9mo ago
[deleted]

mrweasel 9mo ago

> I’m a bit scared of managing the services on my own I see it from the other direction, when if something fails, I have complete access to everything, meaning that I have a chance of fixing it. That's down to hardware even. Things get abstracted away, hidden behind APIs and data lives beyond my reach, when I run stuff in the cloud. Security and regular mistakes are much the same in the cloud, but I then have to layer whatever complications the cloud provide comes with on top. If cost has to be much much lower if I'm going to trust a cloud provider over running something in my own data center.

iso1631 8mo ago
Do you want the power to fix or do you want the paper to wave so you aren't held accountable. The main benefit of outsourcing to aws etc is that the CEO isn't yelling at you when it breaks, because their golf buddies are in the same situation.

objektif 9mo ago

No amount of money will make me maintain my own dbs. We tried it at first and it was a nightmare.

g8oz 9mo ago
It's worth becoming good at.
- sanderjd 9mo ago
  Is it though? This is a genuine question. My intuition is that the investment of time / stress / risk to become good at this is unlikely to have high ROI to either the person putting in that time or to the business paying them to do so. But maybe that's not right.
  - Symbiote 9mo ago
    Managing the PostgreSQL databases is a medium to low complexity task as I see it. Take two equivalent machines, set up with streaming replication exactly as described in the documentation, add Bacula for backups to an off-site location for point-in-time recovery. We haven't felt the need to set up auto fail-over to the hot spare; that would take some extra effort (and is included with AWS equivalents?) but nothing I'd be scared of. Add monitoring that the DB servers are working, replication is up-to-date and the backups are working.
    - sanderjd 9mo ago
      This sounds medium to high complexity to me. You need to do all those things, and also have multiple people who know how to do them, and also make sure that you don't lose all the people who know how to do them, and have one of those people on call to be able to troubleshoot and fix things if they go wrong, and have processes around all that. (At least if you are running in production with real customers depending on you, you should have all those things.) With a managed solution, all of that is amortized into your monthly payment, and you're sharing the cost of it across all the customers of the provider of the managed offering. Personally, I would rather focus on things that are in or at least closer to the core competency of our business, and hire out this kind of thing.