6 ms·

I used to be CTO of an Uber Eats style service that ran on hosting costs of about £100pm. Thousands of orders every few days, real time driver tracking, self ho

I used to be CTO of an Uber Eats style service that ran on hosting costs of about £100pm. Thousands of orders every few days, real time driver tracking, self hosted invoice generation, the works. Hetzner servers, Docker Swarm and Cloudflare for the tech.

nordsieck 3y ago
> Hetzner servers It is truly amazing to me the value that is provided by renting hardware instead of VPSes. As long as you're willing to roll your own infrastructure instead of buy into a cloud provider's infrastructure.
- niemandhier 3y ago
  I did that once and the data center burned down. Sure I could have the service spread over several centers and build distributed backups etc. In the end self hosting and self managing is a money / time trade off, especially for a side gig I’d us SAS and managed solutions. The one thing one has to make sure off is to not get locked in with a particular provider, so knowing how to do everything yourself is a very valuable skill.
  - number6 3y ago
    To be fair, Datacenter burning down is pretty much on the bad luck side of risk management. It's in the same category as your distribution center being hit by an earthquake... I'd guess it wouldn't happen again, but who knows...
  - nik736 3y ago
    A Google Cloud region in France was also unavailable for months (or still is?) due to flooding/electricity. But no one is talking about it.
    - withinboredom 3y ago
      Didn't that happen from someone leaving a sink on or something, causing flooding and then the UPS batteries to short and explode? I was there when someone at AWS accidentally unplugged an entire region. Shit happens, it doesn't matter where it's hosted. People act like the cloud is infallible or something. You're literally sending lightning bolts worth of electricity through bricks of metal. Anything can happen.
- dangus 3y ago
  > As long as you're willing to roll your own infrastructure Uh huh. This is why DevOps/SRE roles pay more than development roles. “Step 2: draw the rest of the owl”
  - Propelloni 3y ago
    They do? That's interesting news!
    - callalex 3y ago
      It pays more because the on-call/pager requirements can be absolutely soul-crushing.
      - dangus 3y ago
        If you have a terrible manager, team, and product/platform. The reality is that most outages are caused by 9-5 changes made by humans.
    - mrweasel 3y ago
      Also more open positions, at least in some market.
- intelVISA 3y ago
  If you know what you're doing it's easy enough to roll out a multi-region distributed system with HA and backups on a pretty modest (<100pcm) budget that can handle competitive QPS. However, most people do not - some will learn, but most will fall for the cloud marketing depts and become infra renters for life. Teach a man to fish, and so on.
  - Balladeer 3y ago
    > some will learn, but most will fall for the cloud marketing depts and become infra renters for life Do you have any learning recommendations for someone looking to start down this path? I've only ever worked in an infra-renter context, and I've begun exploring the 'rent from Hetzner, manage your own infra' for personal projects, but I would love to learn from the paths of experts where possible.
    - arrowsmith 3y ago
      I would also like to hear some recommendations as I've been considering making a similar switch.
    - intelVISA 3y ago
      I'm no expert but hopefully I can still point you toward the happy path: start very small and increase distributed complexity at your own pace until you can fully appreciate the entire end-to-end system and all the processes involved. The book: DDIA is a well-known 101, if a little primitive, and has references you can dig into as well. Ideally, you also have some exposure to this at $job as simply building DIY infra horrors without seeing the real-world context, tradeoffs, etc. in which they typically operate will be misleading.
    - freefaler 3y ago
      For your basic needs you'd need: 1. a DNS monitoring with failover (DNSMadeeasy has a decent solution) 2. a Haproxy setup with health checks for switching to a working upstream service 3. a distributed filesystem 4. a master-slave replication with monitoring (something like Mariadb + orchestrator service) and nightly backups for all this. Database and FS are latency sensitive so they shouldn't be too far apart.
- hardwaresofton 3y ago
  I'm working on a middle-tier solution for this gap -- I call it Nimbus, but basically the idea is to provide managed services at low cost cloud prices. There's no reason someone should have to run a service like Chatwoot themselves, the software is so good that it's mostly set & forget for most small use cases. That's where I come in. Unfortunately I don't have ChatWoot yet, but I have (and use) Umami for page view tracking extensively on my own projects now, with Nimbus[1]. The dogfood tastes decent so far. [1]: https://nimbusws.com/managed/umami/ https://nimbusws.com/managed/umami/
  - nextaccountic 3y ago
    do you anticipate that hosted umami will have features unavailable in the open source version?
    - hardwaresofton 3y ago
      Yeah, but all improvements will likely be upstreamed/made open source (their license isn't AGPL or anything but just makes sense to me since it's MIT). The first thing I want to do is add a backup mechanism that isn't just take a snapshot -- There are a bunch of similar tools to Umami and I don't think that any of them have a really good cross-project way of taking backups. Feels like there should be a page view/analytics backup standard, so you can easy move from a tool like Plausible or Umami and try out a new one, like Fugu. But outside of advanced functionality I think my platform is just a lot closer on cost. The instance costs don't go up per traffic served (especially since 99% of people won't need that) -- it's more like parts + maintenance (and since Umami is good software it doesn't need a TON of maintenance either, just regular patches and some monitoring/extremely light resilience engineering).
- harshitaneja 3y ago
  Hetzner is great and I love it and use it for all my dev needs. But there availability is in limited zones and for production I do suffer from the latency. Haven't found any other provider which costs anywhere near the same(every comparable server is at least 2x the cost). I use their AX41-NVME. Any suggestions of an alternate?
  - dlisboa 3y ago
    Vultr has some very cheap offerings and maybe worth a look. I can't speak to their reliability or service, but they have DCs everywhere even in South America and Africa where there are few cloud options outside of AWS. They're more expensive than Hetzner, maybe even a little over your 2x range depending on your load, but if latency is your issue they probably got you covered.
  - Tijdreiziger 3y ago
    Maybe OVH or Scaleway?
    - freefaler 3y ago
      Both of them are french companies and from my experience with them if you need support is a hit or miss.
      - tracker1 3y ago
        I've been pretty happy with my OVH server, but agreed on support... had better luck with chatgpt and a lot of additional reading getting my CIDR block of addresses configured under ProxMox (which made me far less worried about ChatGPT taking my job any time soon). In the end, was an interesting learing experience, that I hope to not have to repeat. Only went for a single server as I had several smaller VPSes on DigitalOcean and wanted to add a mail server in the mix, and couldn't reliably send via DO or Linode, so was easier to consolidate and run on a single/larger host for hobby projects.
- qingcharles 3y ago
  You'd be amazed how much you can get out of one of the Hetzner $3/mo ARM servers with the right code. I use a $6/mo box for my primary business hosting, but I have a $3/mo one that I'm using to build v2, really just to prove what's possible. If you set up your DB and caching right you can do so much with so little...
  - ljm 3y ago
    Those ARM boxes are incredible for the price, I'm using them to document a hobby K8S cluster because the overall cost is low enough that it won't price a hobbyist out. You get a lot of bang for your buck out of them.
    - esafak 3y ago
      Which k8s are you using and how are you managing it?
      - suralind 3y ago
        Talos Linux is great.
      - ljm 3y ago
        k3s, and nothing fancy except using a splash of helm to set some things up. The cluster has an auto-updater for unattended upgrades.
  - dlisboa 3y ago
    Similarly, but on the opposite spectrum, I also wonder how far you can go on vertical scaling nowadays. For 200 EUR/month on Hetzner you get a dedicated 80-core ARM CPU, 128GB ECC RAM, 2TB SSD...using a good performance multi-threaded language, what _can't_ you run on that? It's ridiculous value.
    - nordsieck 3y ago
      > what _can't_ you run on that? It's ridiculous value. Yeah - I remember a StackOverflow talk[1] where they basically said that they just vertically scale their database. The fact that they were able to make it work tells me that most businesses should probably just go that route and avoid the headache of distributed systems[2]. --- 1. https://www.infoq.com/presentations/stack-exchange/ https://www.infoq.com/presentations/stack-exchange/ 2. Obviously a business should probably invest in redundancy when it comes to data (as did StackOverflow), but a pure "Raid 1" setup is the easiest of distributed systems to understand.
esafak 3y ago
What's the best way to build and run a containerized PaaS on Hetzner?
- slig 3y ago
  CapRover / Dokku.

nordsieck 3y ago

> Hetzner servers It is truly amazing to me the value that is provided by renting hardware instead of VPSes. As long as you're willing to roll your own infrastructure instead of buy into a cloud provider's infrastructure.

niemandhier 3y ago
I did that once and the data center burned down. Sure I could have the service spread over several centers and build distributed backups etc. In the end self hosting and self managing is a money / time trade off, especially for a side gig I’d us SAS and managed solutions. The one thing one has to make sure off is to not get locked in with a particular provider, so knowing how to do everything yourself is a very valuable skill.
- number6 3y ago
  To be fair, Datacenter burning down is pretty much on the bad luck side of risk management. It's in the same category as your distribution center being hit by an earthquake... I'd guess it wouldn't happen again, but who knows...
- nik736 3y ago
  A Google Cloud region in France was also unavailable for months (or still is?) due to flooding/electricity. But no one is talking about it.
  - withinboredom 3y ago
    Didn't that happen from someone leaving a sink on or something, causing flooding and then the UPS batteries to short and explode? I was there when someone at AWS accidentally unplugged an entire region. Shit happens, it doesn't matter where it's hosted. People act like the cloud is infallible or something. You're literally sending lightning bolts worth of electricity through bricks of metal. Anything can happen.
dangus 3y ago
> As long as you're willing to roll your own infrastructure Uh huh. This is why DevOps/SRE roles pay more than development roles. “Step 2: draw the rest of the owl”
- Propelloni 3y ago
  They do? That's interesting news!
  - callalex 3y ago
    It pays more because the on-call/pager requirements can be absolutely soul-crushing.
    - dangus 3y ago
      If you have a terrible manager, team, and product/platform. The reality is that most outages are caused by 9-5 changes made by humans.
  - mrweasel 3y ago
    Also more open positions, at least in some market.
intelVISA 3y ago
If you know what you're doing it's easy enough to roll out a multi-region distributed system with HA and backups on a pretty modest (<100pcm) budget that can handle competitive QPS. However, most people do not - some will learn, but most will fall for the cloud marketing depts and become infra renters for life. Teach a man to fish, and so on.
- Balladeer 3y ago
  > some will learn, but most will fall for the cloud marketing depts and become infra renters for life Do you have any learning recommendations for someone looking to start down this path? I've only ever worked in an infra-renter context, and I've begun exploring the 'rent from Hetzner, manage your own infra' for personal projects, but I would love to learn from the paths of experts where possible.
  - arrowsmith 3y ago
    I would also like to hear some recommendations as I've been considering making a similar switch.
  - intelVISA 3y ago
    I'm no expert but hopefully I can still point you toward the happy path: start very small and increase distributed complexity at your own pace until you can fully appreciate the entire end-to-end system and all the processes involved. The book: DDIA is a well-known 101, if a little primitive, and has references you can dig into as well. Ideally, you also have some exposure to this at $job as simply building DIY infra horrors without seeing the real-world context, tradeoffs, etc. in which they typically operate will be misleading.
  - freefaler 3y ago
    For your basic needs you'd need: 1. a DNS monitoring with failover (DNSMadeeasy has a decent solution) 2. a Haproxy setup with health checks for switching to a working upstream service 3. a distributed filesystem 4. a master-slave replication with monitoring (something like Mariadb + orchestrator service) and nightly backups for all this. Database and FS are latency sensitive so they shouldn't be too far apart.
hardwaresofton 3y ago
I'm working on a middle-tier solution for this gap -- I call it Nimbus, but basically the idea is to provide managed services at low cost cloud prices. There's no reason someone should have to run a service like Chatwoot themselves, the software is so good that it's mostly set & forget for most small use cases. That's where I come in. Unfortunately I don't have ChatWoot yet, but I have (and use) Umami for page view tracking extensively on my own projects now, with Nimbus[1]. The dogfood tastes decent so far. [1]: https://nimbusws.com/managed/umami/ https://nimbusws.com/managed/umami/
- nextaccountic 3y ago
  do you anticipate that hosted umami will have features unavailable in the open source version?
  - hardwaresofton 3y ago
    Yeah, but all improvements will likely be upstreamed/made open source (their license isn't AGPL or anything but just makes sense to me since it's MIT). The first thing I want to do is add a backup mechanism that isn't just take a snapshot -- There are a bunch of similar tools to Umami and I don't think that any of them have a really good cross-project way of taking backups. Feels like there should be a page view/analytics backup standard, so you can easy move from a tool like Plausible or Umami and try out a new one, like Fugu. But outside of advanced functionality I think my platform is just a lot closer on cost. The instance costs don't go up per traffic served (especially since 99% of people won't need that) -- it's more like parts + maintenance (and since Umami is good software it doesn't need a TON of maintenance either, just regular patches and some monitoring/extremely light resilience engineering).
harshitaneja 3y ago
Hetzner is great and I love it and use it for all my dev needs. But there availability is in limited zones and for production I do suffer from the latency. Haven't found any other provider which costs anywhere near the same(every comparable server is at least 2x the cost). I use their AX41-NVME. Any suggestions of an alternate?
- dlisboa 3y ago
  Vultr has some very cheap offerings and maybe worth a look. I can't speak to their reliability or service, but they have DCs everywhere even in South America and Africa where there are few cloud options outside of AWS. They're more expensive than Hetzner, maybe even a little over your 2x range depending on your load, but if latency is your issue they probably got you covered.
- Tijdreiziger 3y ago
  Maybe OVH or Scaleway?
  - freefaler 3y ago
    Both of them are french companies and from my experience with them if you need support is a hit or miss.
    - tracker1 3y ago
      I've been pretty happy with my OVH server, but agreed on support... had better luck with chatgpt and a lot of additional reading getting my CIDR block of addresses configured under ProxMox (which made me far less worried about ChatGPT taking my job any time soon). In the end, was an interesting learing experience, that I hope to not have to repeat. Only went for a single server as I had several smaller VPSes on DigitalOcean and wanted to add a mail server in the mix, and couldn't reliably send via DO or Linode, so was easier to consolidate and run on a single/larger host for hobby projects.
qingcharles 3y ago
You'd be amazed how much you can get out of one of the Hetzner $3/mo ARM servers with the right code. I use a $6/mo box for my primary business hosting, but I have a $3/mo one that I'm using to build v2, really just to prove what's possible. If you set up your DB and caching right you can do so much with so little...
- ljm 3y ago
  Those ARM boxes are incredible for the price, I'm using them to document a hobby K8S cluster because the overall cost is low enough that it won't price a hobbyist out. You get a lot of bang for your buck out of them.
  - esafak 3y ago
    Which k8s are you using and how are you managing it?
    - suralind 3y ago
      Talos Linux is great.
    - ljm 3y ago
      k3s, and nothing fancy except using a splash of helm to set some things up. The cluster has an auto-updater for unattended upgrades.
- dlisboa 3y ago
  Similarly, but on the opposite spectrum, I also wonder how far you can go on vertical scaling nowadays. For 200 EUR/month on Hetzner you get a dedicated 80-core ARM CPU, 128GB ECC RAM, 2TB SSD...using a good performance multi-threaded language, what _can't_ you run on that? It's ridiculous value.
  - nordsieck 3y ago
    > what _can't_ you run on that? It's ridiculous value. Yeah - I remember a StackOverflow talk[1] where they basically said that they just vertically scale their database. The fact that they were able to make it work tells me that most businesses should probably just go that route and avoid the headache of distributed systems[2]. --- 1. https://www.infoq.com/presentations/stack-exchange/ https://www.infoq.com/presentations/stack-exchange/ 2. Obviously a business should probably invest in redundancy when it comes to data (as did StackOverflow), but a pure "Raid 1" setup is the easiest of distributed systems to understand.

esafak 3y ago

What's the best way to build and run a containerized PaaS on Hetzner?

slig 3y ago
CapRover / Dokku.