12 ms·

We moved from AWS to Hetzner, saved 90%, kept ISO 27001 with Ansible

• OutOfHere 1y ago
  Hetzner's biggest problem is that they can and do terminate a user's account without warning if the user starts using CPU resources very heavily or for any reason. This is for very legal usage, of course. This can and does happen to people in months. When this happens, consider your data lost and your account blocked. They will offer no explanation whatsoever, and will even send you a bill for the full month. Hetzner simply cannot be trusted, not even a little bit. As for OVH, they don't do the above, but they have week-long unplanned downtimes, so using them is okay only as an optional resource. Even so, there are lots of providers that are cheaper than Amazon and won't screw you over.
  • winrid 1y ago
    Don't depend on a single provider. Always do live replication across providers. OVH downtime is only an issue if your entire company is running on a single server. Split it across a couple zones if this is actually paying bills.
    • OutOfHere 1y ago
      At the very least I backup data to a different provider, so I can restore the services elsewhere, although there may still exist some residual data loss since the last backup. I guess it depends on just how well it's paying the bills.
  • sksjvsla 1y ago
    I have not experienced this in spite of rumours online. As I mention in these two comments, given these we decided to design our way around it by assuming that they would both go down at some point of time (but not at the same time). 1. https://news.ycombinator.com/item?id=44335920#44339234 https://news.ycombinator.com/item?id=44335920#44339234 2. https://news.ycombinator.com/item?id=44335920#44337619 https://news.ycombinator.com/item?id=44335920#44337619
• deleted 1y ago
  [deleted]
• everfrustrated 1y ago
  How do you do disk encryption? Coming from AWS this is simple but I haven't seen how to do it well on hosting providers. Obviously one can't write the disk encryption key to a boot partition or that undermines the point of it...
  • adamcharnock 1y ago
    ISO27001 doesn't specifically require disk encryption. Rather it requires data on disks be protected according to how it is classified. Disk encryption is one way to achieve this, especially in a shared-hardware environment. In this case, the disks being in a ISO27001 data centre with processes in place to ensure erasure during de-provisioning (which Hetzner is, and has), may well also meet this criteria.
• znpy 1y ago
  So op is now spending 10% of their "$24,000 annual bill" which would be 2400 $/year. Which in turn would be $200/month on infrastructure. If the whole company can run on 200$/month in VPSes, they probably went on AWS too early.
  • sksjvsla 1y ago
    As I wrote elsewhere in this thread: Being able to stay compliant and protect revenue is worth far more than quibbling over which cloud costs a little less. The real ratio to look at is cloud spend vs. the revenue. For me, switching from AWS to European providers wasn’t just about saving on cloud bills (though that was a nice bonus). It was about reducing risk and enabling revenue. Relying on U.S. hyperscalers in Europe is becoming too risky — what happens if Safe Harbor doesn’t get renewed? Or if Schrems III (or whatever comes next) finally forces regulators to act?
• pabs3 1y ago
  Did you consider on-premises hardware instead of EU Cloud?
  • sksjvsla 1y ago
    We did not consider that because we are a lean, small startup. We barely have a premise to put anything on. We talked to a few premium hosting vendors in Denmark and to build our own redundancy beyond what they guarantee, it actually became more expensive than AWS.
• mollerhoj 1y ago
  This stack appears to be a solid choice for building generic CRUD applications, regardless of immediate ISO certification needs. Would it be feasible to package this as a ready-to-use solution for greenfield projects that may pursue ISO certification in the future? Which components would still require manual setup, and why?
  • sksjvsla 1y ago
    Yes, web apps all need logging, performance dashboard, redundancy, DB backups and such. This could be a stack that could be parametrised with sound defaults just requiring some terraform provider credentials as well as a path to an executable web app and a choice of database engine. ISO readiness built-in and abstracted at the OS level rather than programming language level. If anyone wants to "assetize" what I built, reach out at jk@datapult.dk. I bring a battle-tested setup that has been ISO certified by independent auditors. You bring clients directly or indirectly with marketing/growth hacking mindset.
• Sander8888 1y ago
  [dead]
• Sander8888 1y ago
  [dead]
• Keyframe 1y ago
  I think the most often mentioned problems mentioned are pollution of Hetzner addresses by shady people (might be addressed with "exits" from AWS / Cloudflare) and you are running on hardware which does tend to fail / needs upgrades. Were there some concerns on those from you? Also, Loki! How do you handle memory hunger on loki reader for those pesky long range queries, and are there alternatives?
  • sksjvsla 1y ago
    Pollution: We front everything user-facing through Cloudflare, so external users (and bots) don’t interact directly with our Hetzner/OVH IPs. We lock down our IPs at the firewall (ufw + Cloudflare IP allowlisting) so only trusted sources can even connect at L4. Failures/upgrades: We provision with Terraform, so spinning up replacements or adding capacity is fast and deterministic. We monitor hardware metrics via Prometheus and node exporter to get early warnings. So far (9 months in) no hardware failure, but it’s a risk we offset through this automation + design. Apps are mostly data-less and we have (frequently tested) disaster recovery for the database. Loki: We’re handling the memory hunger by • Distinguishing retention limits and index retention • Tuning query concurrency and max memory usage via Loki'’'s config + systemd resource limits. • Use Promtail-style labels + structured logging so queries can filter early rather than regex the whole log content. • Where we need true deep history search, we offload to object store access tools or simple grep of backups — we treat Loki as operational logs + nearline, not as an archive search engine.
    • Keyframe 1y ago
      Thanks for thorough answer! Seems like you've platformized(!) yourself to an extent, have you considered going full on with k8s on top of metal (their machines) to offset some of the concerns about hardware?
      • sksjvsla 1y ago
        Thanks for the compliment. We used AWS EKS in the old days and we never liked the extreme complexity of it. With two Spring Boot apps, a database and Redis running across Ubuntu servers, we found simpler tools to distribute and scale workloads. Since compute is dirt cheap, we over-provision and sleep well. We have live alerts and quarterly reviews (just looking at a dashboard!) to assess if we balance things well. K8s on EKS was not pleasant, I wanna make sure I never learn how much worse it can get across European VPS providers.
• jordanbeiber 1y ago
  Same here, but Azure. About 90% saved, with a very similar stack. It is a great big cloud play to make enterprises reliant on the competency in their weird service abstractions, which is slowly draining the quite simple ops story an enterprise usually needs.
  • ed_mercer 1y ago
    Can you please elaborate how Azure is cheaper?
    • miyuru 1y ago
      I think the parent meant that they moved from Azure to Hetzner.
    • jordanbeiber 1y ago
      ”Same here” meaning moving to Hetzner, but from Azure - could’ve made it less ambiguous! Might throw together a post on it eventually: https://news.ycombinator.com/context?id=43216847 https://news.ycombinator.com/context?id=43216847
• sokoloff 1y ago
  Might be interesting, but doesn’t seem to be a valid “Show HN” * - https://news.ycombinator.com/showhn.html https://news.ycombinator.com/showhn.html
• nopakos 1y ago
  I think a European CloudFlare would be nice to exist.
  • sksjvsla 1y ago
    Yes, it would be nice. Given Cloudflare's dev-friendly branding for some reason, I did not mind keeping it.
  • abc123abc123 1y ago
    No problem! https://bunny.net/about/ https://bunny.net/about/ Enjoy!
    • miyuru 1y ago
      bunny still don't support IPv6 to origin, or else I would have switched.
      • AtNightWeCode 1y ago
        I never seen IPv6 used for origins in CF. Why do you need that?
        • everfrustrated 1y ago
          If you're using a cheap vps provider who charges for ipv4 but gives ipv6 for free it helps.
        • miyuru 1y ago
          Cloudflare even has a blog post on it. https://blog.cloudflare.com/amazon-2bn-ipv4-tax-how-avoid-paying/ https://blog.cloudflare.com/amazon-2bn-ipv4-tax-how-avoid-pa...
          • AtNightWeCode 1y ago
            Ok, I see. Thanks.
• saltysalt 1y ago
  I love Hetzner, I run my Internet search engine from there: bare metal FTW.
• louwrentius 1y ago
  I'm involved with a cloud migration myself so I like the topic, but the Medium article contains less information than this "Shown HN" post. The Medium post is mostly fluff and a lead generator.
  • sksjvsla 1y ago
    The Medium post is more of a high-level case study for a mixed audience (including non-technical decision makers). I intentionally kept the details lighter there, partly to avoid overwhelming readers and partly because the real “meat” (like our Ansible/Terraform patterns, Prometheus config, etc.) is harder to convey in that format without turning it into a giant technical appendix. I’m happy to share specific configs, diagrams, or lessons learned here on HN if people want — and actually I’m finding this thread a much better forum for that kind of deep dive. I'll dive into other aspects elsewhere: You can't doubt that given what I am sharing here. Any particular area you’d like me to expand on? (e.g. how we structured Terraform modules, Ansible hardening, Prometheus alerting, Loki tuning?)
    • clcaev 1y ago
      More detail how you tie ISO and Terraform/Ansible would be welcome.
      • sksjvsla 1y ago
        Here you go. Write me at jk@datapult.dk, if you need more. A.5.25 Security in development and support processes: Safe rolling deploy, rollback mechanisms, NGINX health checks, code versioning, Prometheus alerting for deployment issues A.6.1.2 Segregation of duties: Separate roles for database, monitoring, web apps; distinct system users A.8.1.1 Inventory of assets: Inventory management through Ansible inventory.ini and groups A.8.2.3 Handling of assets: Backup management with OVH S3 storage; retention policy for backups A.8.16 Monitoring activities (audit logging, monitoring): auditd installed with specific rule sets; Prometheus + Grafana Agent + Loki for system/application/audit log monitoring A.9.2.1 User registration and de-registration: ansible_user, restricted SSH access (no root login, pubkey auth), AllowUsers, DenyUsers enforced A.9.2.3 Management of privileged access rights: Controlled sudo, audit rules track use of sudo/su; no direct root access A.9.4.2 Secure log-on procedures: SSH hardening (no password login, no root, key-based access) A.9.4.3 Password management system: Uses Ansible Vault and variables; A.10.1.1 Cryptographic controls policy: SSL/TLS certificate generation with Cloudflare DNS-01 challenge, enforced TLS on Loki, Prometheus A.12.1.1 Security requirements analysis and specification: Tasks assert required variables and configurations before proceeding A.12.4.1 Event logging: auditd, Prometheus metrics, Grafana Agent shipping logs to Loki A.12.4.2 Protection of log information: Logs shipped securely via TLS to Loki, audit logs with controlled permissions A.12.4.3 Administrator and operator logs: auditd rules monitor privileged command usage, config changes, login records A.12.4.4 Clock synchronization: chrony installed and enforced on all hosts A.12.6.1 Technical vulnerability management: Lynis, Wazuh, vulnerability scans for Prometheus metrics A.13.1.1 Network controls: UFW with strict defaults, Cloudflare whitelisting, inter-server TCP port controls A.13.1.2 Security of network services: SSH hardening, NGINX SSL, Prometheus/Alertmanager access control A.13.2.1 Information transfer policies and procedures: Secure database backups to OVH S3 (HTTPS/S3 API) A.14.2.1 Secure development policy: Playbooks enforce strict hardening as part of deploy processes A.15.1.1 Information security policy for supplier relationships: OVH S3, Cloudflare services usage with access key/secret controls; external endpoint defined A.16.1.4 Assessment of and decision on information security events: Prometheus alert rules (e.g., high CPU, low disk, instance down, SSL expiry, failed backups) A.16.1.5 Response to information security incidents: Alertmanager routes critical/security alerts to email/webhook; plans for security incident log webhook A.17.1.2 Implementing information security continuity: Automated DB backups, Prometheus backup job monitoring, retention enforcement A.18.1.3 Protection of records: Loki retention policy, S3 bucket storage with rotation; audit logs secured on disk
• ArtTimeInvestor 1y ago
  How did you decide on Hetzner and OVH and why do you need both? Have you looked into others as well, like IONOS and Scaleway?
  • sksjvsla 1y ago
    Great question. Technically speaking I might not need both, but I have a gut feeling that one of these cloud providers might not be as hardened as the hyperscalers, and that Russia is just waiting to put one of these two services down. So for maximal resiliency I chose to design from a multi-cloud setup from the beginning. Scaleway came up but is more expensive. IONOS did not come up in our research. Part of what we tried to do was to make ourselves independent from traditional cloud services and be really good at doing stuff on a VPS. Once you start doing that, you can actually allow yourself to look more at uptimes and at costs. Also, since we wanted everything to be fully automated, Terraform support was important for us, and OVHcloud and Hetzner had that. I'm sure there's many great cloud providers out in Europe, but it's hard to vet them to understand if they can meet demand and if they are financially stable. We would want not to keep switching cloud providers. So picking two of the major ones seemed like a safe choice.
    • handfuloflight 1y ago
      What would Russia's interests be in putting these ISPs down, specifically?
      • sksjvsla 1y ago
        Without making it too political and speculating on things I don't know, I, like many other Europeans, have seen plenty of cases of Russia ruining infrastructure projects in Europe, everything from internet cables on the ocean bed, telcos, water supplies, railways and more. Authorities are asking civilians in Scandinavia to be prepare their hiused with. Good and water and are actively hardening security around critical infrastructure, including their software. I won't comment more on this because it's gonna derail this discussion.
• jillesvangurp 1y ago
  > We rebuilt key AWS features ourselves At what cost? People usually exclude the cost of DIY style hosting. Which usually is the most expensive part. Providing 24x7 support for the stuff that you've home grown alone is probably going to make large dent into any savings you got by not outsourcing that to amazon. > $24,000 annual bill felt disproportionate That's around 1-2 months of time for a decent devops freelancer. If you underpay your devs, about 1/3rd of an FTE per year. And you are not going to get 24x7 support with such a budget. This still could make sense. But you aren't telling the full story here. And I bet it's a lot less glamorous when you factor in development time for this. Don't get me wrong; I'm actually considering making a similar move but more for business reasons (some of our German customers really don't like US hosting companies) than for cost savings. But this will raise cost and hassle for us and I probably will need some re-enforcements on my team. As the CTO, my time is a very scarce commodity. So, the absolute worst use of my time would be doing this myself. My focus should be making our company and product better. Your techstack is fine. Been there done that. IMHO Terraform is overkill for small setups like this; fits solidly in the YAGNI category. But I like Ansible.
  • uncircle 1y ago
    I feel there is a lot of FUD spread whenever someone moves off the cloud, with the inane comparison to the annual wage of a dedicated sysadmin, trying to discourage you from doing a “reckless” migration which will bite you in the ass, your servers will catch fire every day and that it is better to stay within the golden handcuffs of AWS and GCP. I wonder if it’s both stockholm syndrome and learned helplessness of developers that cannot imagine having to spend a little more effort and save, like OP, 90% off their monthly bill. Yeah sure for some use cases AWS is the market leader, but let’s not kid ourselves, 9/10 companies on AWS don’t require more than a few servers and a database.
    • sksjvsla 1y ago
      Well said. It reminds me of a story I heard in a podcast once. A database administrator for a drug cartel became an informant for the police. His cartel boss called him in on a weekend due to a server errors. He said in the podcast "I knew I've been found out because a database running Linux never crashes" Makes you wonder what everyone is telling themselves about the need for RDS..
• hk1337 1y ago
  Does anybody care, besides you, that you’re ISO 27001 compliant? I thought SSAE 16 and other SSAE standards were the main things people were concerned with having?
  • Freak_NL 1y ago
    Pff… You wish. Depending on the sector you are in, ISO 27001 can either be a hard requirement (either directly or through national standards built upon it, like the Dutch healthcare NEN 7510) or completely irrelevant. If this company needs it, you can bet their customers need it — usually because they in turn are required to do so because of regulations.
  • hbnjgf 1y ago
    Seems to depend on industry and/or region. Most of our customers have a hard requirement on ISO 9001. Many on ISO 27001, too. The rest strongly prefers a partner having a plan to get ISO 27001
  • folmar 1y ago
    You must be looking from US perspective. In EU I don't think I've seen any SSAE provided or wanted, and I've seen a bit of medium to big industry.
• anticodon 1y ago
  I'm not surprised about 90% of savings. I remember that initially AWS was promoted everywhere as being "cheaper" than your own hardware, colocation or VPS/VDS hosting. Once I was working in a quite small company (around 100 employees) that hosted everything on AWS. Due to high bills (it's a small company that resided in Asia) and other problems, I migrated everything to DigitalOcean (we still used AWS for things like SES), and the monthly bill for hosting became like 10 times lower. With no other consequences (in other words, it haven't become less reliable). I still wonder who calculated that AWS is cheaper than everything else. It's definitely one of the most expensive providers.
  • RainyDayTmrw 1y ago
    AWS is cheaper under certain, favorable (to them) assumptions, primarily around needing fewer employees to maintain the hardware.
    • const_cast 1y ago
      In my mind it's very similar to how people sometimes frame the cost-effectiveness of Apple products. Like when the 5K iMac originally came out, there was a lot of people claiming it was a good value. Because if you bought a 5K display and then built a PC, that would end up being more expensive. So, like for like, Apple was cheaper. But... that assumed you even needed a 5K display, which were horribly overpriced and rare at the time. As soon as you say "4K is good enough", the cost advantage disappears, and it's not even close.
  • iLoveOncall 1y ago
    A single developer in Denmark (it's a Danish company) would easily cost around $100K a year to the company. They might save 90% of their $24K on hardware, but just spend probably double the amount on salaries. This is why AWS is in the end cheaper if it is costs more for the same (let's be real it's not at all the same actually) software.
    • hermanzegerman 1y ago
• sam_lowry_ 1y ago
  I did a successful AWS to Hetzner migration myself once, and I'd like to make a business of "back-to-earth migrations" but clients are hard to find. Everyone talks about it but none wants to be the first mover.
  • phillipseamore 1y ago
    Offer a contingent payment of 10% (or similar) of the savings pr. month for one year after the migration. If a migration doesn't make sense then it won't happen and the client doesn't pay (would be very rare!) but when a migration happens 10% of the savings is probably more than you would have charged by the hour or quoted as a fixed price.
  • dijit 1y ago
    When the money really starts drying up it will be on the table again, for now it's "someone elses money" at worst and "less profits for the shareholders" at best (which is not an incentive for an engineer on the ground). There's also a lot of FUD regarding hiring more staff, my observed experience is that hyperscalers need an equivalent number of people on rotation- it's just different skills (learning the intricacies/quirks of different product offerings on the hyperscaler vs CS/Operational fundamentals) - so everyone is scared to overload their teams with work and potentially need to hire people -- you can couple this with the fact that all migrations are up-front expensive and change is bad by default. There will come a day where there simply isn't enough money to spend 10x the cost on these systems. It will be a sad day for everyone because salaries will be depressed too, and we will opine the days of shiny tools where we could make lots of work disappear by saying that our time is too expensive to work with such peasant issues.
• sgt 1y ago
  For those wondering about ISO 27001 - it's a standard for international security management, and popular in Europe. However in the US it's not very relevant or even interesting to companies, and some European companies fail to understand that. SOC 2 is the default and the preferred standard in the US - it's more domestic and less rigid than ISO 27001.
  • AtNightWeCode 1y ago
    Name one big US cloud provider or similar that is not ISO 27001 compliant.
    • sksjvsla 1y ago
      The cloud provider can be compliant with your app being so.. Most apps will not pass an ISO audit unless designed to do so.
  • ozim 1y ago
    ISO27001 I wouldn’t call rigid, most of the stuff you should be doing anyway if you use any software. checking for evidence that you are doing those things I would call ridgit. SOC2 as attestation doesn’t require so much documentation.
    • sgt 1y ago
      Sure, it depends on the implementation of your ISMS. Ideally you want to follow the control guidance in 27002. They've done a lot of thinking on this.
  • candiddevmike 1y ago
    Having been through both, I much prefer the "rigid" ISO 27001 as the SOC2 audits seem to be based on how well you vibed with the auditor and the auditors competency more than anything. The things they are auditing seem overly broad and open to interpretation, and the auditors descriptions of your controls can easily be twisted.
• cataflam 1y ago
  Happy for you, don't get me wrong, but your post is not particularly news, I'm guessing everyone on HN knows bare metal/VPS providers are cheaper than AWS/Azure/GCP. And also lacking a bit in details: - both technical (e.g. how are you dealing with upgrades or multi-data center fallback for your postgresql), and - especially business, e.g. what's the total cost analysis including the supplemental labor cost to set this up but mostly to maintain it. Maybe if you shared your scripts and your full cost analysis, that would be quite interesting.
  • sksjvsla 1y ago
    > Techincal I'm trying to share as much technical across this thread as for your two examples: System upgrades: Keep in mind that as per the ISO specification, system upgrades should be applied but in a controlled manner. This lends itself perfectly to the following case that is manually triggered. Since we take steps to make applications stateless, and Ansible scripts are immutable: We spin up a new machine with the latest packages and once ready it join the Cloudflare load balancer. The old machines are drained and deprovisioned. we spin up a new machine We have a playbook that iterates through our machines and does it per machine before proceeding. Since we have redundancy on components, this creates no downtime. The redundancy in the web application is easy to achieve using the load balancer in Cloudflare. For the Postgres database, it does require that we switch the read-only replica to become the main database. DB failover: The database is only written and read from by our web applications. We have a second VM on a different cloud that has a streaming replication of the Postgres database. It is a hot standby that can be promoted. You can use something like PG Bouncer or HAProxy to route traffic from your apps. But our web framework allows for changing the database at runtime. > Business Before migration (AWS): We had about 0.1 FTE on infra — most of the time went into deployment pipelines and occasional fine-tuning (the usual AWS dance). After migration (Hetzner + OVHCloud + DIY stack): After stabilizing it is still 0.1 FTE (but I was 0.5 FTE for 3-4 months), but now it rests with one person. We didn’t hire a dedicated ops person. On scaling — if we grew 5-10×: * For stateless services, we’re confident we’d stay DIY — Hetzner + OVHCloud + automation scales beautifully. * For stateful services, especially the Postgres database, I think we'd investigate servicing clients out of their own DBs in a multi-tenant setup, and if too cumbersome (we would need tenant-specific disaster recovery playbooks), we'd go back to a managed solution quickly. I can't speak for cloud FTE toll vs a series of VPS servers in the big boys league ($ million in monthly consumption) and in the tiny league but at our league it turns out that it is the same FTE requirement. Anyone want to see my scripts, hit me up at jk@datapult.dk. I'm not sure it'd be great security posture to hand it out on a public forum.
• BrandoElFollito 1y ago
  Any reasons to go for certbot instead of Traefik or Caddy?
  • sksjvsla 1y ago
    We use cloudflare as the WAF and loadbalancer, which makes traefik less relevant and Certbort easier to couple.
• nnurmanov 1y ago
  I moved from managed AWS to unmanaged AWS (lightsail), decreasing the cost significantly and still staying in AWS ecosystem. I use S3, Route53, SES and other cheap services, you could consider this path
• martypitt 1y ago
  > A combination of Prometheus, Grafana, and Loki allowed us to replicate — and in some ways exceed — the visibility we had on AWS Given these existence of these tools, which are fantastic, I'm often stunned at how sluggish, expensive and how lacklustre the UX is of the AWS monitoring stack. Monitoring quickly became the most expensive, and most unpleasant part of our AWS experience.
  • mystifyingpoi 1y ago
    When I discovered that Live Tail (an equivalent of looking at the logs with `tail -f ...`) is paid, I laughed out loud. The most obvious functionality for everyday looking at logs is not free. CW is pain.
    • iLoveOncall 1y ago
      I don't mean that in an offensive way, but if your amount of logs is so small that you can live tail them, you don't operate at the scale that AWS cares about. It's paid because operating that feature at AWS' scale is expensive as hell. Maybe not for your project, but for 90% of their customers it is.
      • sksjvsla 1y ago
        If you were to divide the AWS customer base into a 10% bucket and a 90% bucket, a 90% bucket would not be the ones needing the infinite scale of AWS.
      • mystifyingpoi 1y ago
        You can specify filters in Live Tail, if I read the docs right. So seeing live logs from one transaction id or one user should be possible. This is definitely useful. Searching through gigs/s of logs in order to perform that could be expensive - sure. This just feels wrong to me anyway. CW is already pricy.
• yread 1y ago
  I don't get the numbers. It used to be 24000$/year. You saved 90%. So you're spending 200$ a month at Hetzner? That's literally one EPYC server. You really don't need distributed systems for that. Can you talk a bit more about requests per second or number of users?
  • robin_reala 1y ago
    $2400, not $200.
    • enronmusk 1y ago
      $2400/year is $200/month. He is correct.
  • sksjvsla 1y ago
    You can't do single server setup for your workloads if you are ISO 27001 compliant and, further, you must have a separate server for logging and monitoring. No matter load, there is a need for complexity for this certification. Not all employees log in daily. For a scheduling app, most people check a few times a week, but not every day. Daily active users (DAU) = around 10,000 to 20,000 Peak concurrency (users on at the exact same time) = generally between 1,500 to 2,000 at busy times (like when new schedules drop or at shift start/end times) Average concurrent users at any random time = maybe 50 to 150 Why cloud costs can add up even for us: Extensive use of real-time features and complex labour rules mean the app needs to handle a lot of data processing and ultimately sync into salary systems. An example: Being assigned to a shift has different implications for every user. It may trigger a nuisance bonus, and such a bonus could further only be triggered in certain cases, such as when you had the shifts assigned compared to when it start time. Lastly, there is the optimizing of a schedule why is computationally expensive.
    • yread 1y ago
      Thanks for the answer makes sense. So you can have a a few smaller app and db dedi servers plus a few hetzner cloud vps instances to handle backups and monitoring and object storage to store it. It would be interesting to read more about your policy on logging and monitoring and how you've implemented it. Our app is a lot more demanding (I put 0.5 cores/user, 300 iops/user and 20Mb/s/user as requirements) and I forgot that there are also lighter use cases. We blew thru the thousands in free credits on aws in like 2 months and went immediately to Hetzner
• FutureSeasons 1y ago
  [dead]
• liampulles 1y ago
  Part of what I expect to get when I pay AWS is that it reduces my operational burden, and this has been true in my experience. I've almost forgotten about all the prep, the stress, etc. that comes from upgrading deprecated mysql clusters now that I've gotten used to using the AWS managed equivalents. That is not to say that this aspect alone justifies huge fees, but it does have significant value.
  • sksjvsla 1y ago
    This comment is great. Upgrade processes should be part of your internal processes if you want to get ISO 27001 certification that is not just checking the boxes but actually something you use for more control of your development and release cycle. AWS RDS does not upgrade major or minor versions of Postgres or, as you mentioned, MySQL. In that case, they might patch update it. But these patch updates are easy to do yourself and does not take long to be reminded of in your ISMS and then subsequently carry out. The purpose of this post is not to justify cloud hyperscalers versus European servers. It is actually a post on how to manage a highly regulated, compliant, and certified server setup yourself outside AWS because so many people just have their ISO certification on AWS infrastructure and once they got that they are never able to leave AWS again. If you have no client demand and no real need to work on updating your infrastructure yourself, then you can go ahead and not go for an ISO 27001 certification and let AWS RDS update as it pleases. But if you operate a complex beast in a regulated industry such as employment law, finance, and such, then you get some more fun challenges and higher need for control.
• 0xjunhao 1y ago
  With the rise of Agentic AI, this increasingly feels like the right move, unless AWS drastically lowers their prices.
  • sksjvsla 1y ago
    Agreed, LLMs helped us with this.
• ksec 1y ago
  I know OVH and Hetzner gets mentioned a lot as European Cloud, but I thought I should bring UpCloud [1] for HN's attention. I believe their CPU core are actual CPU core and not vCPU as in a single thread ( Although I cant find reference to it which is annoying ) I also sometimes think OVH and Hetzner are not a fair comparison as much as I want competition to HyperScaler. Hetzner uses consumer grade component with a few server grade selections. [1] https://upcloud.com https://upcloud.com
  • mdaniel 1y ago
    Do I have Stockholm Syndrome for being just utterly confused how ever single one of these budget providers [that I've seen so far] has no meaningful IaaS IAM offering? I don't mean "yeah, I can login to the console with username and password," I mean Permissions, Roles, Machine Identity, ... you know, who can do what to what and be able to see those actions in an audit log What I find especially bizarre is that OpenStack seems to tick many of those boxes, including https://docs.openstack.org/keystone/2025.1/admin/manage-services.html#service-users https://docs.openstack.org/keystone/2025.1/admin/manage-serv... and https://docs.openstack.org/keystone/2025.1/admin/federation/federated_identity.html https://docs.openstack.org/keystone/2025.1/admin/federation/... so it's not like those providers would have to roll such a control plane from scratch as they seem to have done now As a concrete example for your link, they cite Crossplane (and good for them) but then the Crossplane provider gets what I can only presume is some random person's console creds https://upcloud.com/docs/guides/getting-started-crossplane/#:~:text=create%20a%20secret%20with%20your%20upcloud%20api%20credentials%20and%20a%20providerconfig%20that%20will%20use%20them%20to%20provision%20your%20infra https://upcloud.com/docs/guides/getting-started-crossplane/#... and their terraform provider auths the same way I do see the https://developers.upcloud.com/1.3/3-accounts/#add-subaccount https://developers.upcloud.com/1.3/3-accounts/#add-subaccoun... and some of its built-in filtering but it doesn't have a good explanation for how it interacts with https://developers.upcloud.com/1.3/18-permissions/#normal-response-for-a-subaccount-sub_account_user1 https://developers.upcloud.com/1.3/18-permissions/#normal-re... . But don't worry, you can't manage any of that with their terraform provider, so I guess upcli it is
    • everfrustrated 1y ago
• mollerhoj 1y ago
  Did you look into prepackaged solutions such as kamal/dokku/caprover for parts of this? What were you missing from those?
  • sksjvsla 1y ago
    I looked at Kamal, Dokku, and CapRover — all great tools if you want to abstract away server management. But for a HIPAA/ISO 27001 certifiable app, I need a higher level of control and auditability across the entire stack. With Ansible, I can version everything — from server hardening to DB backups — and ensure idempotent, transparent provisioning. I don’t have to reverse-engineer how a PaaS layer configures things under the hood, or worry about opaque defaults that might not meet compliance requirements. There's nothing wrong with these tools, but once you're in the mood for the ISO certification, and once you start doing these things yourself, they actually seem like a step backwards or add very little value. I also prefer running my own DB backups rather than relying on magic snapshots — it's easier to integrate with encrypted offsite storage and disaster recovery policies that align with ISO requirements. This lets me lock down the environment exactly as needed, with no surprise moving parts. Tools like Kamal/Dokku/CapRover shine for fast, developer-friendly deploys, but for regulated workloads, I’ll take boring, explicit, and auditable any day.
    • mollerhoj 1y ago
      Interesting point about db recovery: I guess your db is small enough that you can do multiple full backups without issue? Or do you backup the WAL only?