30 ms·

We cut our Mongo DB costs by 90% by moving to Hetzner

261 points by arbol 11mo ago

CyanLite2 11mo ago
"We replaced a cluster of virtualized servers with a single bare metal server. Nothing has gone wrong, yet."
- usrnm 11mo ago
  There are many cases when some downtime is perfectly ok. Or, at least, worth the savings
  - tayo42 11mo ago
    They saved a little under 3k and were motivated by the aws outage
- arbol 11mo ago
  Lol yep that could've been the headline. We plan to add replica servers at some point. This DB is not critical to our product hence the relaxed interim setup.
- NorwegianDude 11mo ago
  To be fair, a single server is way more reliable than cloud clusters. Just look at the most recent many hour long Azure downtime where Microsoft could not even get microsoft.com back. With that much downtime you could physically move drives between servers multiple times each year, and still have less downtime. Servers are very reliable, cloud software is not. I'm not saying people should use a single server if they can avoid it, but using a single cloud provider is just as bad. "We moved to the cloud, with managed services and redundancy, nothing has gone wrong...today"
rmoriz 11mo ago
I‘m a big fan of owning the stack but why not spend the money on redundancy? At least a couple of machines in a different data center at Hetzner or another provider (OVH, Scaleway, Vultr, …) can easily fit your budget.
- arbol 11mo ago
  We will be adding additional db servers and running our own replica set eventually. We're just not there yet. Thanks for reading!
  - hinkley 11mo ago
    But then you’ll be tripling your costs. Business people are weird about numbers. You should have claimed 70% even if the replicas do nothing and made them work later on. This is highly likely to bite you on the ass.
    - mystifyingpoi 11mo ago
      +1 this is so true. You've lost, you've already publicly praised yourself that you saved 90%. They won't like the idea of tripling the costs, even if it is still below the previous costs.
    - rmoriz 11mo ago
      Exactly, this is junior mistake I made too many times. There is a wisdom: Never tell anyone, when you’ve won the lottery. In technical terms you need to plan ahead. The legacy mistakes are caused by actions in the past and will likely be made again, when you can’t change the strategy or approach to problems. You won‘t get budget for this AFTER you successfully made a change. „It‘s all solved now, we are good“. No.
petcat 11mo ago
> The more keen eyed among you will have noticed the huge cost associated with data transfer over the internet - its as much as the servers! We're building Prosopo to be resilient to outages, such as the recent massive AWS outage, so we use many different cloud providers. I mean, you're connecting to your primary database potentially on another continent? I imagine your costs will be high, but even worse, your performance will be abysmal. > When you migrate to a self-hosted solution, you're taking on more responsibility for managing your database. You need to make sure it is secure, backed up, monitored, and can be recreated in case of failure or the need for extra servers arises. > ...for a small amount of pain you can save a lot of money! I wouldn't call any of that "a small amount of pain." To save $3,000/month you've now required yourself to become experts in a domain that maybe is out of your depth. So whatever cost saved is now tech debt and potentially having to hire someone else to manage your homemade solution for you. However, I self-host, and applaud other self-hosters. But sometimes it really has to make business sense for your team.
- arbol 11mo ago
  > I mean, you're connecting to your primary database potentially on another continent? Atlas AWS was actually setup in Ireland. The data transfer costs were coming from extracting data for ML modelling. We don't get charged for extracting data with the new contract. > experts in a domain that maybe is out of your depth We're in the bot detection space so we need to be able to run our own infra in order to inspect connections for patterns of abuse. We've built up a fair amount of knowledge because of this and we're lucky enough to have a guy in our team who just understands everything related to computers. He's also pretty good at disseminating information. Thanks for reading!
  - goastler 11mo ago
    aww shucks ;)
kachapopopow 11mo ago
Always consider if 12 hours of lost revenue is worth the savings. Recently hetzner has been flakey with minimum or no response for support or even status updates that anything was wrong. My favorite was them blaming an issue on my side just to have a maintenance status update the day after about congestion.
- arbol 11mo ago
  Atlas wasn't giving us any support for $3K per month. Hetzner at least have some channel to contact them, which is an improvement. That said, if their uptime is rubbish them we'll probably migrate again. Moving back to Atlas is not an option as we were getting hammered by the data transfer costs and this was only going to increase due to our architecture. Thanks for reading!
  - zamalek 11mo ago
    OVH is allegedly pretty good. I host all my personal stuff on Hetzner right now so I can't speak to it personally.
    - arbol 11mo ago
      We also use OVH and have so far not had any downtime in about 6 months.
  - kosherhurricane 11mo ago
    500GB isn't a lot of data, and $3K/month seems like an extortion for that little data. Having said that, MongoDB pricing page promises 99.995% uptime, which is outstanding, and would probably be hard to beat that doing it oneself, even after adding redundancy. But maybe you don't need that much uptime for your particular use case.
    - arbol 11mo ago
      Its more like 700GB now on the new server and we were about to have to migrate to a higher tier on Atlas. > maybe you don't need that much uptime for your particular use case. Correct. Thanks for reading!
CodesInChaos 11mo ago
MongoDB Atlas is so overpriced that you can probably save already 90% by moving to AWS.
- computerfan494 11mo ago
  Most of the cost in their bill wasn't from MongoDB, it was cost passed on from AWS
  - CodesInChaos 11mo ago
    I don't remember the numbers (90% is probably a bit exaggerated) but our savings of going from Atlas to MongoDB Community on EC2 several years ago were big. In addition to direct costs, Atlas had also expensive limitations. For example we often spin up clone databases from a snapshot which have lower performance and no durability requirements, so a smaller non-replicated server suffices, but Atlas required those to be sized like the replicated high performance production cluster.
  - CodesInChaos 11mo ago
    Was it? Assuming an M40 cluster consists of 3 m6g.xlarge machines, that's $0.46/hr on-demand compared to Atlas's $1.04/hr for the compute. Savings plans or reserved instances reduce that cost further.
    - computerfan494 11mo ago
      There's definitely MongoDB markup, but a full 33% of their bill was AWS networking costs that have nothing to do with Atlas.
  - darth_avocado 11mo ago
    Highly doubt that. MongoDB has 5000 well paid employees and is not a big loss making enterprise. If most of the cost was pass through to AWS, they’d not be able to do that. Their quarterly revenue is $500M+ but also spend $200M in sales and marketing and $180M in R&D. (All based on their filings)
cnkk 11mo ago
Are you sure you went with RAID1 with 4x disks instead of RAID10?
- arbol 11mo ago
  Good spot - this is wrong. It should've been 4 x 3.84 TB NVMe SSD RAID 5. My colleague set this bit up so I'm not entirely up to speed on the terminology.
CodesInChaos 11mo ago
How long does mongodump take on that database? My experience was that incremental filesystem/blockdevice snapshots were the only realistic way of backing up (non sharded) mongodb. In our case EBS snapshots, but I think you can achieve the same using LVM or filesystems like XFS and ZFS.
- goastler 11mo ago
  It takes ~21hrs to dump the entire db (~500gb), but I'm limited by my internet speed (100mbps, seeing 50-100mbps during dump). Interestingly, the throughput is faster than doing a db dump from atlas which used to max around 30mbps
0xbadcafebee 11mo ago
> Here's how we managed to cut our costs by 90% You could cut your MongoDB costs by 100% by not using it ;) > without sacrificing performance or reliability. You're using a single server in a single datacenter. MongoDB Atlas is deployed to VMs on 2-3 AZs. You don't have close to the same reliability. (I'm also curious why their M40 instance costs $1000, when the Pricing Calculator (https://www.mongodb.com/pricing https://www.mongodb.com/pricing) says M40 is $760/month? Was it the extra storage?) > We're building Prosopo to be resilient to outages, such as the recent massive AWS outage, so we use many different cloud providers This means you're going to have multiple outages, AND incur more cross-internet costs. How does going to Hetzner make you more resilient to outages? You have one server in one datacenter. Intelligent, robust design at one provider (like AWS) is way more resilient, and intra-zone transfer is cheaper than going out to the cloud ($0.02/GB vs $0.08/GB). You do not have a centralized or single point of failure design with AWS. They're not dummies; plenty of their services are operated independently per region. But they do expect you to use their infrastructure intelligently to avoid creating a single point of failure. (For example, during the AWS outage, my company was in us-east-1, and we never had any issues, because we didn't depend on calling AWS APIs to continue operating. Things already running continue to run.) I get it; these "we cut bare costs by moving away from the cloud" posts are catnip for HN. But they usually don't make sense. There's only a few circumstances where you really have to transfer out a lot of traffic, or need very large storage, where cloud pricing is just too much of a premium. The whole point of using the cloud is to use it as a competitive advantage. Giving yourself an extra role (sysadmin) in addition to your day job (developer, data scientist, etc) and more maintenance tasks (installing, upgrading, patching, troubleshooting, getting on-call, etc) with lower reliability and fewer services, isn't an advantage.
- goastler 11mo ago
  > you're going to have multiple outages us: 0, aws: 1. Looking good so far ;) > AND incur more cross-internet costs hetzner have no bandwidth traffic limit (only speed) on the machine, we can go nuts. I understand you point wrt the cloud, but I spend as much time debugging/building a cloud deployment (atlas :eyes: ) as I do a self-hosted solution. Aws gives you all the tools to build a super reliable data store, but many people just chuck something on us-east-1 and go. There's you single point of failure. Given we're constructing a many-node decentralised system, self-hosted actually makes more sense for us because we've already had to become familiar enough to create a many-node system for our primary product. When/if we have a situation where we need high data availability I would strongly consider the cloud, but in the situations where you can deal with a bit of downtime you're massively saving over cloud offerings. We'll post a 6-month and 1-year follow-up to update the scoreboard above
euph0ria 11mo ago
You probably want to store the backup somewhere else, ie. not Hetzner. They are known to just cancel accounts and cut access.
- sdoering 11mo ago
  Any proof of that? I am a Hetzner customer and had never heard of this before. Would be good to know what I got into.
  - ch2026 11mo ago
    A few years back I launched an io game and used hetzner as my backend. an hour into launch day they null routed my account because their anti-abuse system thought my sudden surge in websocket connections was an attack (unclear if they thought it was inbound or outbound doing the attacking). I had paid for advertising on a few game curation sites plus youtubers and streamers. Lovely failure all thanks to Hetzner. Took 3 days and numerous emails with the most arrogant Germans you’ve ever met before my account was unlocked. I switched to OVH and while they’re not without their own faults (reliability is a big one), it’s been a far better experience.
    - __turbobrew__ 11mo ago
      OVH also null routes, it has happened to me. It seems like you have to go to one of the big boys like hurricane electric where you are allowed to use the bandwidth you paid for without someone sticking their fingers in it.
  - arcanemachiner 11mo ago
    There are a lot of such stories if you go digging around HN and reddit threads. Haven't seen a lot of these stories in a while, so it may be happening less now.
- arbol 11mo ago
  Good shout. I think we'll also run replicas on other providers. We've got some complex geo-fencing stuff to do with regards to data hence why we're just on Hetzner right now.
PeterZaitsev 11mo ago
Note, if you're looking for MongoDB Enterprise features you can find many of them with Percona Server for MongoDB, which you can use for free the same way as MongoDB Community
- arbol 11mo ago
  Nice, thanks for the tip!
ianberdin 11mo ago
I’m starting to worry about this Hetzner trend. It can end up to get the price skyrocketing.
- goastler 11mo ago
  There's other providers (OVH, etc) so I'm sure the price will remain competitive
- arbol 11mo ago
  Hopefully not. Their console is pretty bad so I reckon that will put a lot of people off.
  - patrickmcnamara 11mo ago
    The new console is completely fine.
  - the_duke 11mo ago
    The cloud console is pretty good though? Even does live sync! The old one for dedicated servers (robot) is horribly outdated though.
    - arbol 11mo ago
      Ah right, we're on robot so I've not seen the cloud one. Robot is old! :)
- righthand 11mo ago
  We’re just going to end up with everyone moving from Amazon to Hetzner and the same issue will remain. High prices, lockin, etc will appear. We need an American “get off American big tech” movement. Differentiate people! Reading “we moved from X to Y” does not mean everyone move from X to Y, it means start considering the Y values and research other Y’s around you.
  - arbol 11mo ago
poszlem 11mo ago
As in so many of these stories, what gets glossed over is just how much complexity there is in setting up your own server securely. You set up your server. Harden it. Follow all the best practices for your firewall with ufw. Then you run a Docker container. Accidentally, or simply because you don’t know any better, you bind it to 0.0.0.0 by doing 5432:5432. Oops. Docker just walked right past your firewall rules, ignored ufw, and now port 5432 is exposed with default Postgres credentials. Congratulations. Say hello to Kinsing. And this is just one of many possible scenarios like that. I’m not trying to spread FUD, but this really needs to be stressed much more clearly. EDIT. as always - thank you HN for downvoting instead of actually addressing the argument.
- mkesper 11mo ago
  I don't see the point of using ufw at all as Hetzner provides an external firewall.
  - poszlem 11mo ago
    If you use a dedicated hetzner machine you only get a stateless firewall. That would be one reason.
  - tracker1 11mo ago
    UFW doesn't add much overhead given the implementation in Linux is already in place, it's mostly just a convenient front-end. That said, you also need to be concerned with internal/peer threats as well as external ones... Clearly defining your boundaries is important for both internal and external vectors of attack.
- isaacvando 11mo ago
  There are also an enormous number of ways to build insecure apps on AWS. I think the difficulty of setting up your own server is massively overblown. And that should be unsurprising given that there are so many companies that benefit from developers thinking it's too hard.
cpursley 11mo ago
Why in the world do people choose Mongo over Postgres? I'm legit curious. Is it inexperience? Javascript developers who don't know backend or proper data modeling (or about jsonb)? Is this type of decision coming down from non-technical management? Are VCs telling their portfolio companies what to use so they have something to burn their funding on? It's just really confounding, especially when there's even mongo-api compatible Postgres solutions now. Perhaps I'm just not webscale and too cranky.
- a13n 11mo ago
  maybe instead of communicating how dumb you think people are for choosing mongo, communicate why you think it’s so dumb
  - cpursley 11mo ago
    Why mongo is dumb has been written up about ad nauseam - from data modeling and quality issues, out of control costs, etc. It's been a known toxic dumpsterfire for well over a decade...
  - williamdclt 11mo ago
    I've read a lot more about "how dumb it is to use mongo over PG" than the opposite, I think the burden of proof is on the mongo-lovers these days (not that anyone has to prove anything to randos on the internet)
- nalekberov 11mo ago
  IMHO it's because so many people take decisions in rush. e.g. let's not design database, put whatever data shape we came ip in alpha version and see where it goes. Sometimes people favor one particular technology because every other startup chose it. To be quite honest today's software engineering sadly is mostly about addressing 'how complex can we go' rather than 'what problem are we trying to solve'.
- tgv 11mo ago
  I'll repeat it again: you don't always want a relational database. Sometimes you need a document-oriented one. It matches quite a lot of use cases, e.g. when there aren't really interesting relations, or when the structures are very deep. That can be really annoying in SQL. > when there's even mongo-api compatible Postgres solutions With their own drawbacks.
trustmeimhuman 11mo ago
[dead]
zkmon 11mo ago
Atlas is plain robbery. I see companies paying 600K USD/month on a few clusters, mostly used for testing. The problem is they got locked into this, by doing a huge migration of their apps and switching to a different tech would easily take 2 to 5 years.
- mathattack 11mo ago
  I’ve seen this happen many times. It looks cheap and easy to spin up, then it grows out of hand and they kill you on the renewals.
- nuschk 11mo ago
  Would a company paying 600k per month not also be able to employ a couple of devs to improve the situation? Sure, effort is required, but with the right people they could save a ton and have a very good ROI. I think it's just more complicated than that. No hostage situation, just good old incentives.
zzzeek 11mo ago
it's getting hard to ignore Hetzner (as a Linode user). Thing is, Linode was great 10-15 years ago, then enshittification ensued (starting with Akamai buying them). So what does enshittification for Hetzner look like? I've already got migration scripts pointed at their servers but can't wait for the eventual letdown.
- tracker1 11mo ago
  IMO, virtual servers and dedicated server hosting is really commoditized at this point. So you have a lot of options... assuming you have appropriate orchestration and management scripted out, with good backup procedures in place, you should be able to shift to any other provider relatively easily. The pain points are when you're also intwined with specific implementations for services from a given provider... Sure, you can shift from PostgreSQL on a hosted provider to another without much pain... but say SQS to Azure Simple Queues or Service Bus is a lot more involved. And that is just one example. The is a large reason to keep your services to those with self-hosted options and/or self-hosting from the start... that said, I'm happy to outsource things that are easier to (re) integrate or replace.
mads_quist 11mo ago
OK guys, running on a single instance is REALLY a BAD IDEA for non-pet-projects. Really bad! Change it as fast as you can. I love Hetzner for what they offer but you will run into huge outages pretty soon. At least you need two different network zones on Hetzner and three servers. It's not hard to setup, but you need to do it.
- antoniojtorres 11mo ago
  agree on single instance, but for hetzner, I run 100+ large bare metal servers in hetzner, have for at least 5 years and there’s only been one significant outage they had, we do spread across all their datacenter zones and replicate, so it’s all been manageable. It’s worth it for us, very worth it.
- MaKey 11mo ago
  I think you're being overly dramatic. In practice I've seen complexity (which HA setups often introduce) causing downtimes far more often than a service being hosted only on a single instance.
  - lewiscollard 11mo ago
    Yes, any time someone says "I'm going to make a thing more reliable by adding more things to it" I either want to buy them a copy of Normal Accidents or hit them over the head with mine.
    - smartbit 11mo ago
      Normal Accidents https://en.wikipedia.org/wiki/Normal_Accidents https://en.wikipedia.org/wiki/Normal_Accidents
    - immibis 11mo ago
      How bad are the effects of an interruption for you? Google has servers running every day, but you with one server can afford to gamble on it, since it probably won't fail for years - no matter the hardware though, keep a backup, because data loss is permanent. Would you lose millions of dollars a minute, or would you just have to send an email to customers saying "oops"? Risk management is a normal part of business - every business does it. Typically the risk is not brought down all the way to zero, but to an acceptable level. The milk truck may crash and the grocery store will be out of milk that day - they don't send three trucks and use a quorum. If you want to guarantee above-normal uptime, feel free, but it costs you. Google has servers failing every day just because they have so many, but you are not Google and you most likely won't experience a hardware failure for years. You should have a backup because data loss is permanent, but you might not need redundancy for your online systems. Depending on what your business does.
Glamklo 11mo ago
[dead]
lunias 11mo ago
Just host on a server in your basement. Put another instance in someone else's basement. I'm only half joking - track the downtime.
tracker1 11mo ago
As much as I like MongoDB as a developer, the last thing I ever want to do is manage a deployment again. I feel like some of these articles miss a few points, even in this one. The monthly cost of the MongoDB hosting was around $2k... that's less than a FT employee salary, and if it can spare you the cost of an employee, that's not a bad thing. On the flip side, if you have employee talent that is already orchestrating Kubernetes across multiple clouds, then sure it makes sense to internalize services that would otherwise be external if it doesn't add too much work/overhead to your team(s). In either case, I don't think the primary driver in this is cost at all. Because that 90% quoted reduction in hosting costs is balanced by the ongoing salary of the person or people who maintain those systems.
mv4 11mo ago
"I cut my healthcare costs by 90% by canceling insurance and doctor visits." In all seriousness, this is a recurring pattern on HN and it sends the wrong message. It's almost as bad as vibecoding a paid service and losing private customer data. There was a thread here awhile ago, 'How We Saved $500,000 Per Year by Rolling Our Own “S3' [1]. Then they promptly got hacked. [2] [1] https://engineering.nanit.com/how-we-saved-500-000-per-year-by-rolling-our-own-s3-6caec1ee1143 https://engineering.nanit.com/how-we-saved-500-000-per-year-... [2] https://www.cbsnews.com/colorado/news/colorado-mom-stranger-talking-baby-monitor/ https://www.cbsnews.com/colorado/news/colorado-mom-stranger-...
- csunoser 11mo ago
  Even after reading the source, it doesn’t seem like they were hacked? Or if they were, they were not accused of such. I do think hand rolling your own thing is fraught. But it is very confusing to equate one mother’s complaint to “they have been hacked”. PS: The people who made their own s3 rans a baby monitor company. News article is about a mother reporting hearing a weird voice from the baby monitour.
  - mv4 11mo ago
    Multiple reports on reddit suggest people making this baby cam do not understand security. https://www.reddit.com/r/NewParents/comments/1ocgmoi/nanit_camera_hacked/ https://www.reddit.com/r/NewParents/comments/1ocgmoi/nanit_c... https://www.reddit.com/r/Nanit/comments/1ffc051/nanit_hacked_again_another_strange_voicenanit/ https://www.reddit.com/r/Nanit/comments/1ffc051/nanit_hacked... https://www.reddit.com/r/Nanit/comments/1dyaph6/heard_a_voice_on_our_nanit/ https://www.reddit.com/r/Nanit/comments/1dyaph6/heard_a_voic...
- raxxorraxor 11mo ago
  You just need to vibe configure your server too so that it matches your application. Seriously, I think for most services Hetzner is the better option. No provider lock-in, easier configuration (you cannot tell me AWS/Azure configuration is easier than system administration, these services change every 3 months and use non-standard tools). Most services can stomach a technical fault. Recoverability is more important. There are some exceptions to this and that highly depends on the nature of the service. Nobody here described the nature of their services, so we can only speculate.
pdyc 11mo ago
hetzner routinely refuses to accept you as customer so while u can cut costs its a privilege.
tuhgdetzhh 11mo ago
I recently did a total cost of ownership analysis for moving off AWS to Hetzner: https://beuke.org/hetzner-aws/ https://beuke.org/hetzner-aws/
TZubiri 11mo ago
I hope you don't mind if I hijack this post to ask: Is there a provider similar to Hetzner but US based?
- christophilus 11mo ago
  I’ve never heard of or used them, but this was linked in a previous Hetzner thread: https://ioflood.com/ https://ioflood.com/
niffydroid 11mo ago
Having run a small mongo database and having it hosted in 3 different places at one point. The last point was atlas, yes it was expensive but we got replication, we could have an analytical node, we even had data residency. If I remember correctly you can have your replicas in different providers at the same time. One of the biggest issues was cost, but we were treated like first class citizens, the support was good, we saw constant updates and features. Using atlas search was fantastic because we didn't have to replicate the data to another resource for quick searching. Before atlas we were on Compose.io and well mongo there just withered and we were plagued by performance issues
bmcahren 11mo ago
MongoDB Atlas was around 500% more expensive than in-house every time I evaluated it (at almost every scale they offer as well). They also leaned too heavily on sharding as a universal solution to scaling as opposed to leveraging the minimal cost of terabytes of RAM. The p99 latency increase, risk of major re-sharding downtime, increased restore times, and increased operational complexity weren't worth it for ~1 TB datasets.
- winrid 11mo ago
  That's because sharding is way more likely to make them more money with their licensing model.
jjwiseman 11mo ago
Docker/TypeS cript Node
mathattack 11mo ago
I experienced some cutthroat commercial behavior from MongoDB. It scared us enough to avoid Atlas, and ultimately move to Cosmos on Azure. Massive savings. I moved to another employer that was using Atlas, and the bill rivaled AWS. Unfortunately it was too complex to untangle.
cmsbobcatsx2025 11mo ago
[flagged]
rglover 11mo ago
I love MongoDB's query language (JS/Node.js developer so the syntax fits my mental model well), but running a production replica set without spending tons of cash is a nightmare. Doubly so if you have any unoptimized queries (it's easy to trick yourself into thinking throwing more hardware at the problem will help). Lord help you if you use a hosted/managed service. Just fixed a bug on my MongoDB instance last night that, due to a config error w/ self-signed certs (the hostname in the replica set config has to match the CN on the cert), that caused MongoDB to rocket to 400% CPU utilization (3x, 8GB, 4VCPU dedicated boxes on DO) due to a weird election loop in the replica set process. Fixing that and adding a few missing indexes brought it down to ~12% on average. Simple mistakes, sure, but the real-world cost of those mistakes is brutal.
Too 11mo ago
The dump, restore and custom scripts to synchronize the new instance sound a bit odd. You could just add the instance as a secondary to your cluster and mongo itself handles synchronization. Then removing the old instances automatically promotes the new to primary.
speedgoose 11mo ago
I wonder how many companies are running databases on non-encrypted storage on Hetzner. Their bare-metal servers don't have storage encryption by default, and I don't know for sure about the VM hosts, as I don't have access, but Hetzner never claims that it is encrypted at rest. And there is no mention of storage encryption in their data protection agreement. https://www.hetzner.com/AV/DPA_en.pdf https://www.hetzner.com/AV/DPA_en.pdf Also, their data privacy FAQ mentions "you as the customer are responsible for both the data that is stored on your rented server and for the encryption of that data." https://docs.hetzner.com/general/general-terms-and-conditions/data-privacy-faq/ https://docs.hetzner.com/general/general-terms-and-condition... I would recommend, just in case, to set up LUKS on your server. You will find many guides for Hetzner. If you don't do that, seeing your data in the wild is a real scenario. A few years ago, a Youtuber bought some used hard-drives in the hope to recover data to illustrate the risks of not erasing a hard-drive correctly. He eventually bought a hard-drive containing non-encrypted VM disks from Scaleway, a Hetzner competitor. My guess is that some hard drives disappeared before destruction after being decommissioned. Some customers got their shitty source code exposed on a 1.4M views video. Here is the first one: https://www.youtube.com/watch?v=vt8PyQ2PGxI https://www.youtube.com/watch?v=vt8PyQ2PGxI So, use LUKS.
- immibis 11mo ago
  Indeed, this is something to be aware of. They are not selling you a magical SaaS solution. They are renting to you a particular physical server on a particular physical shelf. In principle, you could break into the DC and steal your server and lay it on your desk at home and it would operate exactly as it did inside the DC. Some people accustomed to cloud expect magic from their dedicated servers, which does not exist.
lisbbb 11mo ago
Hetnzer, lol. Reminds me of all those weird Chinese brands for nvme and sdd hard drive enclosures on Amazon--Amaloo, Ugreen, Fideco, Orico, and on and on. All trash.
dvfjsdhgfv 11mo ago
As much as I love Hetzner, the article is misleading. Using a single server today makes no sense whatsoever unless it's for hobby projects. It will fail. My servers at Hetzner routinely fail every few years (4-5 maybe), usually it's a hard drive, but sometimes motherboard or PSU. If it's a drive, you need to take it offline to rebuild the array, it can take a few hours. Like honestly, this article blew up my mind. I'd never use such setup in production. Just add the damn second server (or two), it's dirt cheap!
- raxxorraxor 11mo ago
  I can deal with an outage every 4-5 years. I doubt you will get around that in a managed server environment, because you will fail configuration at some point when the service will inevitably change in the same timeframe.
MagicMoonlight 11mo ago
You could cut it even more by moving to on-prem instead. Personally I’d rather have it in AWS so that I’m not responsible for keeping it working 24/7.
urbandw311er 11mo ago
Can somebody explain to me how you would retain vector indexes in this migration to Hetzner? We recently began using Atlas Cloud and I’m concerned about these rising costs — but my understanding was that, if you self-host, you lose the ability to create vector indexes.
- stopthe 11mo ago
  Available in community edition 8.2+ https://www.mongodb.com/company/blog/product-release-announcements/supercharge-self-managed-apps-search-vector-search-capabilities https://www.mongodb.com/company/blog/product-release-announc...
  - urbandw311er 11mo ago
    Thank you!
game_the0ry 11mo ago
I feel like we see stories like this more and more. Makes you wonder just how durable the revenue of cloud providers are when self hosting on VMs has never been easier and more cost effective. Then again, nextjs + vercel + gihtub are awfully convenient.
- loloquwowndueo 11mo ago
  Self hosting requires a different skill set - full blown sysadmin / SRE vs. “The application starts, just deploy it and the PaaS takes care of load balancing, scaling, healing, observability etc”. I’m not defending cloud-esque paas here (I would totally prefer to manage VMs directly) but it should be recognized that it gives some value depending on what you’re comfortable doing with infrastructure.
  - game_the0ry 11mo ago
    True, but with the right web framework, much of this comes out of the box.
lightningspirit 11mo ago
In that case, can you reduce pricing for customers, too?
stego-tech 11mo ago
What most of the commenters here are missing is the reality that not every system, function, or business needs the sort of uptime that AWS offers - and that's fine. It's something a lot of newer entrants into the technology field fail to grasp, because they've never had to actually deal with an outage before - or a time when the internet itself was ephemeral and temporary, available only as long as your connection remained active. The number one thing people poo-pooing these "We saved $XXX by getting off public cloud" posts is that each business has different calculus for its risk tolerances, business needs, and opportunity costs. Once a function reaches some form of stability or homeostasis, then hosting it in the public cloud can become a net liability rather than a net asset. Being able to make those decisions impartially is what separates the genuinely good talent from those who conflate TC with wisdom. Even when public cloud is the right decision, using managed services increasingly isn't. MongoDB Atlas is a managed service with a corresponding price tag to match. Running it in a VPS like Hetzner may shift some of the maintenance and support tasks onto your team, but let's be real - modern databases are designed to be bulletproof, and huge companies operated just fine with a single database instance on bare metal for decades, even with the odd downtime along the way. We ran a MongoDB CE database at a PriorCo on a single VM in a single datacenter for nearly a decade, and it underpinned a substantial chunk of our operations - operations we could do by hand, if needed, during downtime or outages (that never happened). We eventually moved it to AWS DocumentDB not out of cost-savings or necessity, but because a higher-up demanded we do so. If anything, the visceral rebuke of anyone daring to move off public cloud feels very reminiscent of my own collegial douchebagginess in the 2000s, loudly mocking Linux stans and proclaiming closed source (Microsoft) would run the planet. Past-me was a douchebag then, and the same applies to the AWS-stans of today.
CodeCrusader 11mo ago
It does look like Hetzner is getting a lot more popular, I see such articles every month
anonymid 11mo ago
$2700/mo is about 1/3 of an engineers' salary (cost to the business of a mid-level engineer in the UK)... But, there's the time to set all of this up (which admittedly is a one-time investment and would amortize). And there's the risk of having made a mistake in your backups or recovery system (Will you exercise it? Will you continue to regularly exercise it?). And they're a 3-person team... is it really worth your limited time/capacity to do this, rather than do something that's likely to attract $3k/mo of new business? If the folks who wrote the blog see this, please share how much time (how many devs, how many weeks) this took to set up, and how the ongoing maintenance burden shapes up.
- tester756 11mo ago
  You can get decent eastern eu engineer for 2700$ (after tax) salary
ritcgab 11mo ago
Looking at the root server hardware page on Hetzner [1], it is not clear if their server is using ECC memory. It would be pretty bad if not. [1] https://docs.hetzner.com/robot/dedicated-server/general-information/root-server-hardware/ https://docs.hetzner.com/robot/dedicated-server/general-info...
- walletdrainer 11mo ago
  https://www.hetzner.com/dedicated-rootserver/ax42/ https://www.hetzner.com/dedicated-rootserver/ax42/ Go to the product pages (Yes all the normal Hetzner servers use ECC)
  - ritcgab 11mo ago
    > Yes all the normal Hetzner servers use ECC This is wrong - or define "normal"? In their current product line, at least the default configuration of AX52/EX44/EX63/GEX44 doesn't have ECC. It is an upgrade option only. The blog post says their server has "8 cores Intel Xeon W-2145", which is PX92 or its variant and its base configuration can be without ECC.
    - walletdrainer 11mo ago
      Apparently I was wrong yeah, just looked at a lower specced server and foolishly extrapolated from there.
ribtoks 11mo ago
Keeping time series in Mongo…

261 points by arbol 11mo ago

CyanLite2 11mo ago

"We replaced a cluster of virtualized servers with a single bare metal server. Nothing has gone wrong, yet."

usrnm 11mo ago
There are many cases when some downtime is perfectly ok. Or, at least, worth the savings
- tayo42 11mo ago
  They saved a little under 3k and were motivated by the aws outage
arbol 11mo ago
Lol yep that could've been the headline. We plan to add replica servers at some point. This DB is not critical to our product hence the relaxed interim setup.
NorwegianDude 11mo ago
To be fair, a single server is way more reliable than cloud clusters. Just look at the most recent many hour long Azure downtime where Microsoft could not even get microsoft.com back. With that much downtime you could physically move drives between servers multiple times each year, and still have less downtime. Servers are very reliable, cloud software is not. I'm not saying people should use a single server if they can avoid it, but using a single cloud provider is just as bad. "We moved to the cloud, with managed services and redundancy, nothing has gone wrong...today"

rmoriz 11mo ago

I‘m a big fan of owning the stack but why not spend the money on redundancy? At least a couple of machines in a different data center at Hetzner or another provider (OVH, Scaleway, Vultr, …) can easily fit your budget.

arbol 11mo ago
We will be adding additional db servers and running our own replica set eventually. We're just not there yet. Thanks for reading!
- hinkley 11mo ago
  But then you’ll be tripling your costs. Business people are weird about numbers. You should have claimed 70% even if the replicas do nothing and made them work later on. This is highly likely to bite you on the ass.
  - mystifyingpoi 11mo ago
    +1 this is so true. You've lost, you've already publicly praised yourself that you saved 90%. They won't like the idea of tripling the costs, even if it is still below the previous costs.
  - rmoriz 11mo ago
    Exactly, this is junior mistake I made too many times. There is a wisdom: Never tell anyone, when you’ve won the lottery. In technical terms you need to plan ahead. The legacy mistakes are caused by actions in the past and will likely be made again, when you can’t change the strategy or approach to problems. You won‘t get budget for this AFTER you successfully made a change. „It‘s all solved now, we are good“. No.

petcat 11mo ago

> The more keen eyed among you will have noticed the huge cost associated with data transfer over the internet - its as much as the servers! We're building Prosopo to be resilient to outages, such as the recent massive AWS outage, so we use many different cloud providers. I mean, you're connecting to your primary database potentially on another continent? I imagine your costs will be high, but even worse, your performance will be abysmal. > When you migrate to a self-hosted solution, you're taking on more responsibility for managing your database. You need to make sure it is secure, backed up, monitored, and can be recreated in case of failure or the need for extra servers arises. > ...for a small amount of pain you can save a lot of money! I wouldn't call any of that "a small amount of pain." To save $3,000/month you've now required yourself to become experts in a domain that maybe is out of your depth. So whatever cost saved is now tech debt and potentially having to hire someone else to manage your homemade solution for you. However, I self-host, and applaud other self-hosters. But sometimes it really has to make business sense for your team.

arbol 11mo ago
> I mean, you're connecting to your primary database potentially on another continent? Atlas AWS was actually setup in Ireland. The data transfer costs were coming from extracting data for ML modelling. We don't get charged for extracting data with the new contract. > experts in a domain that maybe is out of your depth We're in the bot detection space so we need to be able to run our own infra in order to inspect connections for patterns of abuse. We've built up a fair amount of knowledge because of this and we're lucky enough to have a guy in our team who just understands everything related to computers. He's also pretty good at disseminating information. Thanks for reading!
- goastler 11mo ago
  aww shucks ;)

kachapopopow 11mo ago

Always consider if 12 hours of lost revenue is worth the savings. Recently hetzner has been flakey with minimum or no response for support or even status updates that anything was wrong. My favorite was them blaming an issue on my side just to have a maintenance status update the day after about congestion.

arbol 11mo ago
Atlas wasn't giving us any support for $3K per month. Hetzner at least have some channel to contact them, which is an improvement. That said, if their uptime is rubbish them we'll probably migrate again. Moving back to Atlas is not an option as we were getting hammered by the data transfer costs and this was only going to increase due to our architecture. Thanks for reading!
- zamalek 11mo ago
  OVH is allegedly pretty good. I host all my personal stuff on Hetzner right now so I can't speak to it personally.
  - arbol 11mo ago
    We also use OVH and have so far not had any downtime in about 6 months.
- kosherhurricane 11mo ago
  500GB isn't a lot of data, and $3K/month seems like an extortion for that little data. Having said that, MongoDB pricing page promises 99.995% uptime, which is outstanding, and would probably be hard to beat that doing it oneself, even after adding redundancy. But maybe you don't need that much uptime for your particular use case.
  - arbol 11mo ago
    Its more like 700GB now on the new server and we were about to have to migrate to a higher tier on Atlas. > maybe you don't need that much uptime for your particular use case. Correct. Thanks for reading!

CodesInChaos 11mo ago

MongoDB Atlas is so overpriced that you can probably save already 90% by moving to AWS.

computerfan494 11mo ago
Most of the cost in their bill wasn't from MongoDB, it was cost passed on from AWS
- CodesInChaos 11mo ago
  I don't remember the numbers (90% is probably a bit exaggerated) but our savings of going from Atlas to MongoDB Community on EC2 several years ago were big. In addition to direct costs, Atlas had also expensive limitations. For example we often spin up clone databases from a snapshot which have lower performance and no durability requirements, so a smaller non-replicated server suffices, but Atlas required those to be sized like the replicated high performance production cluster.
- CodesInChaos 11mo ago
  Was it? Assuming an M40 cluster consists of 3 m6g.xlarge machines, that's $0.46/hr on-demand compared to Atlas's $1.04/hr for the compute. Savings plans or reserved instances reduce that cost further.
  - computerfan494 11mo ago
    There's definitely MongoDB markup, but a full 33% of their bill was AWS networking costs that have nothing to do with Atlas.
- darth_avocado 11mo ago
  Highly doubt that. MongoDB has 5000 well paid employees and is not a big loss making enterprise. If most of the cost was pass through to AWS, they’d not be able to do that. Their quarterly revenue is $500M+ but also spend $200M in sales and marketing and $180M in R&D. (All based on their filings)

cnkk 11mo ago

Are you sure you went with RAID1 with 4x disks instead of RAID10?

arbol 11mo ago
Good spot - this is wrong. It should've been 4 x 3.84 TB NVMe SSD RAID 5. My colleague set this bit up so I'm not entirely up to speed on the terminology.

CodesInChaos 11mo ago

How long does mongodump take on that database? My experience was that incremental filesystem/blockdevice snapshots were the only realistic way of backing up (non sharded) mongodb. In our case EBS snapshots, but I think you can achieve the same using LVM or filesystems like XFS and ZFS.

goastler 11mo ago
It takes ~21hrs to dump the entire db (~500gb), but I'm limited by my internet speed (100mbps, seeing 50-100mbps during dump). Interestingly, the throughput is faster than doing a db dump from atlas which used to max around 30mbps

0xbadcafebee 11mo ago

> Here's how we managed to cut our costs by 90% You could cut your MongoDB costs by 100% by not using it ;) > without sacrificing performance or reliability. You're using a single server in a single datacenter. MongoDB Atlas is deployed to VMs on 2-3 AZs. You don't have close to the same reliability. (I'm also curious why their M40 instance costs $1000, when the Pricing Calculator (https://www.mongodb.com/pricing https://www.mongodb.com/pricing) says M40 is $760/month? Was it the extra storage?) > We're building Prosopo to be resilient to outages, such as the recent massive AWS outage, so we use many different cloud providers This means you're going to have multiple outages, AND incur more cross-internet costs. How does going to Hetzner make you more resilient to outages? You have one server in one datacenter. Intelligent, robust design at one provider (like AWS) is way more resilient, and intra-zone transfer is cheaper than going out to the cloud ($0.02/GB vs $0.08/GB). You do not have a centralized or single point of failure design with AWS. They're not dummies; plenty of their services are operated independently per region. But they do expect you to use their infrastructure intelligently to avoid creating a single point of failure. (For example, during the AWS outage, my company was in us-east-1, and we never had any issues, because we didn't depend on calling AWS APIs to continue operating. Things already running continue to run.) I get it; these "we cut bare costs by moving away from the cloud" posts are catnip for HN. But they usually don't make sense. There's only a few circumstances where you really have to transfer out a lot of traffic, or need very large storage, where cloud pricing is just too much of a premium. The whole point of using the cloud is to use it as a competitive advantage. Giving yourself an extra role (sysadmin) in addition to your day job (developer, data scientist, etc) and more maintenance tasks (installing, upgrading, patching, troubleshooting, getting on-call, etc) with lower reliability and fewer services, isn't an advantage.

goastler 11mo ago
> you're going to have multiple outages us: 0, aws: 1. Looking good so far ;) > AND incur more cross-internet costs hetzner have no bandwidth traffic limit (only speed) on the machine, we can go nuts. I understand you point wrt the cloud, but I spend as much time debugging/building a cloud deployment (atlas :eyes: ) as I do a self-hosted solution. Aws gives you all the tools to build a super reliable data store, but many people just chuck something on us-east-1 and go. There's you single point of failure. Given we're constructing a many-node decentralised system, self-hosted actually makes more sense for us because we've already had to become familiar enough to create a many-node system for our primary product. When/if we have a situation where we need high data availability I would strongly consider the cloud, but in the situations where you can deal with a bit of downtime you're massively saving over cloud offerings. We'll post a 6-month and 1-year follow-up to update the scoreboard above

euph0ria 11mo ago

You probably want to store the backup somewhere else, ie. not Hetzner. They are known to just cancel accounts and cut access.

sdoering 11mo ago
Any proof of that? I am a Hetzner customer and had never heard of this before. Would be good to know what I got into.
- ch2026 11mo ago
  A few years back I launched an io game and used hetzner as my backend. an hour into launch day they null routed my account because their anti-abuse system thought my sudden surge in websocket connections was an attack (unclear if they thought it was inbound or outbound doing the attacking). I had paid for advertising on a few game curation sites plus youtubers and streamers. Lovely failure all thanks to Hetzner. Took 3 days and numerous emails with the most arrogant Germans you’ve ever met before my account was unlocked. I switched to OVH and while they’re not without their own faults (reliability is a big one), it’s been a far better experience.
  - __turbobrew__ 11mo ago
    OVH also null routes, it has happened to me. It seems like you have to go to one of the big boys like hurricane electric where you are allowed to use the bandwidth you paid for without someone sticking their fingers in it.
- arcanemachiner 11mo ago
  There are a lot of such stories if you go digging around HN and reddit threads. Haven't seen a lot of these stories in a while, so it may be happening less now.
arbol 11mo ago
Good shout. I think we'll also run replicas on other providers. We've got some complex geo-fencing stuff to do with regards to data hence why we're just on Hetzner right now.

PeterZaitsev 11mo ago

Note, if you're looking for MongoDB Enterprise features you can find many of them with Percona Server for MongoDB, which you can use for free the same way as MongoDB Community

arbol 11mo ago
Nice, thanks for the tip!

ianberdin 11mo ago

I’m starting to worry about this Hetzner trend. It can end up to get the price skyrocketing.

goastler 11mo ago
There's other providers (OVH, etc) so I'm sure the price will remain competitive
arbol 11mo ago
Hopefully not. Their console is pretty bad so I reckon that will put a lot of people off.
- patrickmcnamara 11mo ago
  The new console is completely fine.
- the_duke 11mo ago
  The cloud console is pretty good though? Even does live sync! The old one for dedicated servers (robot) is horribly outdated though.
  - arbol 11mo ago
    Ah right, we're on robot so I've not seen the cloud one. Robot is old! :)
righthand 11mo ago
We’re just going to end up with everyone moving from Amazon to Hetzner and the same issue will remain. High prices, lockin, etc will appear. We need an American “get off American big tech” movement. Differentiate people! Reading “we moved from X to Y” does not mean everyone move from X to Y, it means start considering the Y values and research other Y’s around you.
- arbol 11mo ago

poszlem 11mo ago

As in so many of these stories, what gets glossed over is just how much complexity there is in setting up your own server securely. You set up your server. Harden it. Follow all the best practices for your firewall with ufw. Then you run a Docker container. Accidentally, or simply because you don’t know any better, you bind it to 0.0.0.0 by doing 5432:5432. Oops. Docker just walked right past your firewall rules, ignored ufw, and now port 5432 is exposed with default Postgres credentials. Congratulations. Say hello to Kinsing. And this is just one of many possible scenarios like that. I’m not trying to spread FUD, but this really needs to be stressed much more clearly. EDIT. as always - thank you HN for downvoting instead of actually addressing the argument.

mkesper 11mo ago
I don't see the point of using ufw at all as Hetzner provides an external firewall.
- poszlem 11mo ago
  If you use a dedicated hetzner machine you only get a stateless firewall. That would be one reason.
- tracker1 11mo ago
  UFW doesn't add much overhead given the implementation in Linux is already in place, it's mostly just a convenient front-end. That said, you also need to be concerned with internal/peer threats as well as external ones... Clearly defining your boundaries is important for both internal and external vectors of attack.
isaacvando 11mo ago
There are also an enormous number of ways to build insecure apps on AWS. I think the difficulty of setting up your own server is massively overblown. And that should be unsurprising given that there are so many companies that benefit from developers thinking it's too hard.

cpursley 11mo ago

Why in the world do people choose Mongo over Postgres? I'm legit curious. Is it inexperience? Javascript developers who don't know backend or proper data modeling (or about jsonb)? Is this type of decision coming down from non-technical management? Are VCs telling their portfolio companies what to use so they have something to burn their funding on? It's just really confounding, especially when there's even mongo-api compatible Postgres solutions now. Perhaps I'm just not webscale and too cranky.

a13n 11mo ago
maybe instead of communicating how dumb you think people are for choosing mongo, communicate why you think it’s so dumb
- cpursley 11mo ago
  Why mongo is dumb has been written up about ad nauseam - from data modeling and quality issues, out of control costs, etc. It's been a known toxic dumpsterfire for well over a decade...
- williamdclt 11mo ago
  I've read a lot more about "how dumb it is to use mongo over PG" than the opposite, I think the burden of proof is on the mongo-lovers these days (not that anyone has to prove anything to randos on the internet)
nalekberov 11mo ago
IMHO it's because so many people take decisions in rush. e.g. let's not design database, put whatever data shape we came ip in alpha version and see where it goes. Sometimes people favor one particular technology because every other startup chose it. To be quite honest today's software engineering sadly is mostly about addressing 'how complex can we go' rather than 'what problem are we trying to solve'.
tgv 11mo ago
I'll repeat it again: you don't always want a relational database. Sometimes you need a document-oriented one. It matches quite a lot of use cases, e.g. when there aren't really interesting relations, or when the structures are very deep. That can be really annoying in SQL. > when there's even mongo-api compatible Postgres solutions With their own drawbacks.

trustmeimhuman 11mo ago

[dead]

zkmon 11mo ago

Atlas is plain robbery. I see companies paying 600K USD/month on a few clusters, mostly used for testing. The problem is they got locked into this, by doing a huge migration of their apps and switching to a different tech would easily take 2 to 5 years.

mathattack 11mo ago
I’ve seen this happen many times. It looks cheap and easy to spin up, then it grows out of hand and they kill you on the renewals.
nuschk 11mo ago
Would a company paying 600k per month not also be able to employ a couple of devs to improve the situation? Sure, effort is required, but with the right people they could save a ton and have a very good ROI. I think it's just more complicated than that. No hostage situation, just good old incentives.

zzzeek 11mo ago

it's getting hard to ignore Hetzner (as a Linode user). Thing is, Linode was great 10-15 years ago, then enshittification ensued (starting with Akamai buying them). So what does enshittification for Hetzner look like? I've already got migration scripts pointed at their servers but can't wait for the eventual letdown.

tracker1 11mo ago
IMO, virtual servers and dedicated server hosting is really commoditized at this point. So you have a lot of options... assuming you have appropriate orchestration and management scripted out, with good backup procedures in place, you should be able to shift to any other provider relatively easily. The pain points are when you're also intwined with specific implementations for services from a given provider... Sure, you can shift from PostgreSQL on a hosted provider to another without much pain... but say SQS to Azure Simple Queues or Service Bus is a lot more involved. And that is just one example. The is a large reason to keep your services to those with self-hosted options and/or self-hosting from the start... that said, I'm happy to outsource things that are easier to (re) integrate or replace.

mads_quist 11mo ago

OK guys, running on a single instance is REALLY a BAD IDEA for non-pet-projects. Really bad! Change it as fast as you can. I love Hetzner for what they offer but you will run into huge outages pretty soon. At least you need two different network zones on Hetzner and three servers. It's not hard to setup, but you need to do it.

antoniojtorres 11mo ago
agree on single instance, but for hetzner, I run 100+ large bare metal servers in hetzner, have for at least 5 years and there’s only been one significant outage they had, we do spread across all their datacenter zones and replicate, so it’s all been manageable. It’s worth it for us, very worth it.
MaKey 11mo ago
I think you're being overly dramatic. In practice I've seen complexity (which HA setups often introduce) causing downtimes far more often than a service being hosted only on a single instance.
- lewiscollard 11mo ago
  Yes, any time someone says "I'm going to make a thing more reliable by adding more things to it" I either want to buy them a copy of Normal Accidents or hit them over the head with mine.
  - smartbit 11mo ago
    Normal Accidents https://en.wikipedia.org/wiki/Normal_Accidents https://en.wikipedia.org/wiki/Normal_Accidents
  - immibis 11mo ago
    How bad are the effects of an interruption for you? Google has servers running every day, but you with one server can afford to gamble on it, since it probably won't fail for years - no matter the hardware though, keep a backup, because data loss is permanent. Would you lose millions of dollars a minute, or would you just have to send an email to customers saying "oops"? Risk management is a normal part of business - every business does it. Typically the risk is not brought down all the way to zero, but to an acceptable level. The milk truck may crash and the grocery store will be out of milk that day - they don't send three trucks and use a quorum. If you want to guarantee above-normal uptime, feel free, but it costs you. Google has servers failing every day just because they have so many, but you are not Google and you most likely won't experience a hardware failure for years. You should have a backup because data loss is permanent, but you might not need redundancy for your online systems. Depending on what your business does.

Glamklo 11mo ago

[dead]

lunias 11mo ago

Just host on a server in your basement. Put another instance in someone else's basement. I'm only half joking - track the downtime.

tracker1 11mo ago

As much as I like MongoDB as a developer, the last thing I ever want to do is manage a deployment again. I feel like some of these articles miss a few points, even in this one. The monthly cost of the MongoDB hosting was around $2k... that's less than a FT employee salary, and if it can spare you the cost of an employee, that's not a bad thing. On the flip side, if you have employee talent that is already orchestrating Kubernetes across multiple clouds, then sure it makes sense to internalize services that would otherwise be external if it doesn't add too much work/overhead to your team(s). In either case, I don't think the primary driver in this is cost at all. Because that 90% quoted reduction in hosting costs is balanced by the ongoing salary of the person or people who maintain those systems.

mv4 11mo ago

"I cut my healthcare costs by 90% by canceling insurance and doctor visits." In all seriousness, this is a recurring pattern on HN and it sends the wrong message. It's almost as bad as vibecoding a paid service and losing private customer data. There was a thread here awhile ago, 'How We Saved $500,000 Per Year by Rolling Our Own “S3' [1]. Then they promptly got hacked. [2] [1] https://engineering.nanit.com/how-we-saved-500-000-per-year-by-rolling-our-own-s3-6caec1ee1143 https://engineering.nanit.com/how-we-saved-500-000-per-year-... [2] https://www.cbsnews.com/colorado/news/colorado-mom-stranger-talking-baby-monitor/ https://www.cbsnews.com/colorado/news/colorado-mom-stranger-...

csunoser 11mo ago
Even after reading the source, it doesn’t seem like they were hacked? Or if they were, they were not accused of such. I do think hand rolling your own thing is fraught. But it is very confusing to equate one mother’s complaint to “they have been hacked”. PS: The people who made their own s3 rans a baby monitor company. News article is about a mother reporting hearing a weird voice from the baby monitour.
- mv4 11mo ago
  Multiple reports on reddit suggest people making this baby cam do not understand security. https://www.reddit.com/r/NewParents/comments/1ocgmoi/nanit_camera_hacked/ https://www.reddit.com/r/NewParents/comments/1ocgmoi/nanit_c... https://www.reddit.com/r/Nanit/comments/1ffc051/nanit_hacked_again_another_strange_voicenanit/ https://www.reddit.com/r/Nanit/comments/1ffc051/nanit_hacked... https://www.reddit.com/r/Nanit/comments/1dyaph6/heard_a_voice_on_our_nanit/ https://www.reddit.com/r/Nanit/comments/1dyaph6/heard_a_voic...
raxxorraxor 11mo ago
You just need to vibe configure your server too so that it matches your application. Seriously, I think for most services Hetzner is the better option. No provider lock-in, easier configuration (you cannot tell me AWS/Azure configuration is easier than system administration, these services change every 3 months and use non-standard tools). Most services can stomach a technical fault. Recoverability is more important. There are some exceptions to this and that highly depends on the nature of the service. Nobody here described the nature of their services, so we can only speculate.

pdyc 11mo ago

hetzner routinely refuses to accept you as customer so while u can cut costs its a privilege.

tuhgdetzhh 11mo ago

I recently did a total cost of ownership analysis for moving off AWS to Hetzner: https://beuke.org/hetzner-aws/ https://beuke.org/hetzner-aws/

TZubiri 11mo ago

I hope you don't mind if I hijack this post to ask: Is there a provider similar to Hetzner but US based?

christophilus 11mo ago
I’ve never heard of or used them, but this was linked in a previous Hetzner thread: https://ioflood.com/ https://ioflood.com/

niffydroid 11mo ago

Having run a small mongo database and having it hosted in 3 different places at one point. The last point was atlas, yes it was expensive but we got replication, we could have an analytical node, we even had data residency. If I remember correctly you can have your replicas in different providers at the same time. One of the biggest issues was cost, but we were treated like first class citizens, the support was good, we saw constant updates and features. Using atlas search was fantastic because we didn't have to replicate the data to another resource for quick searching. Before atlas we were on Compose.io and well mongo there just withered and we were plagued by performance issues

bmcahren 11mo ago

MongoDB Atlas was around 500% more expensive than in-house every time I evaluated it (at almost every scale they offer as well). They also leaned too heavily on sharding as a universal solution to scaling as opposed to leveraging the minimal cost of terabytes of RAM. The p99 latency increase, risk of major re-sharding downtime, increased restore times, and increased operational complexity weren't worth it for ~1 TB datasets.

winrid 11mo ago
That's because sharding is way more likely to make them more money with their licensing model.

jjwiseman 11mo ago

Docker/TypeS cript Node

mathattack 11mo ago

I experienced some cutthroat commercial behavior from MongoDB. It scared us enough to avoid Atlas, and ultimately move to Cosmos on Azure. Massive savings. I moved to another employer that was using Atlas, and the bill rivaled AWS. Unfortunately it was too complex to untangle.

cmsbobcatsx2025 11mo ago

[flagged]

rglover 11mo ago

I love MongoDB's query language (JS/Node.js developer so the syntax fits my mental model well), but running a production replica set without spending tons of cash is a nightmare. Doubly so if you have any unoptimized queries (it's easy to trick yourself into thinking throwing more hardware at the problem will help). Lord help you if you use a hosted/managed service. Just fixed a bug on my MongoDB instance last night that, due to a config error w/ self-signed certs (the hostname in the replica set config has to match the CN on the cert), that caused MongoDB to rocket to 400% CPU utilization (3x, 8GB, 4VCPU dedicated boxes on DO) due to a weird election loop in the replica set process. Fixing that and adding a few missing indexes brought it down to ~12% on average. Simple mistakes, sure, but the real-world cost of those mistakes is brutal.

Too 11mo ago

The dump, restore and custom scripts to synchronize the new instance sound a bit odd. You could just add the instance as a secondary to your cluster and mongo itself handles synchronization. Then removing the old instances automatically promotes the new to primary.

speedgoose 11mo ago

I wonder how many companies are running databases on non-encrypted storage on Hetzner. Their bare-metal servers don't have storage encryption by default, and I don't know for sure about the VM hosts, as I don't have access, but Hetzner never claims that it is encrypted at rest. And there is no mention of storage encryption in their data protection agreement. https://www.hetzner.com/AV/DPA_en.pdf https://www.hetzner.com/AV/DPA_en.pdf Also, their data privacy FAQ mentions "you as the customer are responsible for both the data that is stored on your rented server and for the encryption of that data." https://docs.hetzner.com/general/general-terms-and-conditions/data-privacy-faq/ https://docs.hetzner.com/general/general-terms-and-condition... I would recommend, just in case, to set up LUKS on your server. You will find many guides for Hetzner. If you don't do that, seeing your data in the wild is a real scenario. A few years ago, a Youtuber bought some used hard-drives in the hope to recover data to illustrate the risks of not erasing a hard-drive correctly. He eventually bought a hard-drive containing non-encrypted VM disks from Scaleway, a Hetzner competitor. My guess is that some hard drives disappeared before destruction after being decommissioned. Some customers got their shitty source code exposed on a 1.4M views video. Here is the first one: https://www.youtube.com/watch?v=vt8PyQ2PGxI https://www.youtube.com/watch?v=vt8PyQ2PGxI So, use LUKS.

immibis 11mo ago
Indeed, this is something to be aware of. They are not selling you a magical SaaS solution. They are renting to you a particular physical server on a particular physical shelf. In principle, you could break into the DC and steal your server and lay it on your desk at home and it would operate exactly as it did inside the DC. Some people accustomed to cloud expect magic from their dedicated servers, which does not exist.

lisbbb 11mo ago

Hetnzer, lol. Reminds me of all those weird Chinese brands for nvme and sdd hard drive enclosures on Amazon--Amaloo, Ugreen, Fideco, Orico, and on and on. All trash.

dvfjsdhgfv 11mo ago

As much as I love Hetzner, the article is misleading. Using a single server today makes no sense whatsoever unless it's for hobby projects. It will fail. My servers at Hetzner routinely fail every few years (4-5 maybe), usually it's a hard drive, but sometimes motherboard or PSU. If it's a drive, you need to take it offline to rebuild the array, it can take a few hours. Like honestly, this article blew up my mind. I'd never use such setup in production. Just add the damn second server (or two), it's dirt cheap!

raxxorraxor 11mo ago
I can deal with an outage every 4-5 years. I doubt you will get around that in a managed server environment, because you will fail configuration at some point when the service will inevitably change in the same timeframe.

MagicMoonlight 11mo ago

You could cut it even more by moving to on-prem instead. Personally I’d rather have it in AWS so that I’m not responsible for keeping it working 24/7.

urbandw311er 11mo ago

Can somebody explain to me how you would retain vector indexes in this migration to Hetzner? We recently began using Atlas Cloud and I’m concerned about these rising costs — but my understanding was that, if you self-host, you lose the ability to create vector indexes.

stopthe 11mo ago
Available in community edition 8.2+ https://www.mongodb.com/company/blog/product-release-announcements/supercharge-self-managed-apps-search-vector-search-capabilities https://www.mongodb.com/company/blog/product-release-announc...
- urbandw311er 11mo ago
  Thank you!

game_the0ry 11mo ago

I feel like we see stories like this more and more. Makes you wonder just how durable the revenue of cloud providers are when self hosting on VMs has never been easier and more cost effective. Then again, nextjs + vercel + gihtub are awfully convenient.

loloquwowndueo 11mo ago
Self hosting requires a different skill set - full blown sysadmin / SRE vs. “The application starts, just deploy it and the PaaS takes care of load balancing, scaling, healing, observability etc”. I’m not defending cloud-esque paas here (I would totally prefer to manage VMs directly) but it should be recognized that it gives some value depending on what you’re comfortable doing with infrastructure.
- game_the0ry 11mo ago
  True, but with the right web framework, much of this comes out of the box.

lightningspirit 11mo ago

In that case, can you reduce pricing for customers, too?

stego-tech 11mo ago

What most of the commenters here are missing is the reality that not every system, function, or business needs the sort of uptime that AWS offers - and that's fine. It's something a lot of newer entrants into the technology field fail to grasp, because they've never had to actually deal with an outage before - or a time when the internet itself was ephemeral and temporary, available only as long as your connection remained active. The number one thing people poo-pooing these "We saved $XXX by getting off public cloud" posts is that each business has different calculus for its risk tolerances, business needs, and opportunity costs. Once a function reaches some form of stability or homeostasis, then hosting it in the public cloud can become a net liability rather than a net asset. Being able to make those decisions impartially is what separates the genuinely good talent from those who conflate TC with wisdom. Even when public cloud is the right decision, using managed services increasingly isn't. MongoDB Atlas is a managed service with a corresponding price tag to match. Running it in a VPS like Hetzner may shift some of the maintenance and support tasks onto your team, but let's be real - modern databases are designed to be bulletproof, and huge companies operated just fine with a single database instance on bare metal for decades, even with the odd downtime along the way. We ran a MongoDB CE database at a PriorCo on a single VM in a single datacenter for nearly a decade, and it underpinned a substantial chunk of our operations - operations we could do by hand, if needed, during downtime or outages (that never happened). We eventually moved it to AWS DocumentDB not out of cost-savings or necessity, but because a higher-up demanded we do so. If anything, the visceral rebuke of anyone daring to move off public cloud feels very reminiscent of my own collegial douchebagginess in the 2000s, loudly mocking Linux stans and proclaiming closed source (Microsoft) would run the planet. Past-me was a douchebag then, and the same applies to the AWS-stans of today.

CodeCrusader 11mo ago

It does look like Hetzner is getting a lot more popular, I see such articles every month

anonymid 11mo ago

$2700/mo is about 1/3 of an engineers' salary (cost to the business of a mid-level engineer in the UK)... But, there's the time to set all of this up (which admittedly is a one-time investment and would amortize). And there's the risk of having made a mistake in your backups or recovery system (Will you exercise it? Will you continue to regularly exercise it?). And they're a 3-person team... is it really worth your limited time/capacity to do this, rather than do something that's likely to attract $3k/mo of new business? If the folks who wrote the blog see this, please share how much time (how many devs, how many weeks) this took to set up, and how the ongoing maintenance burden shapes up.

tester756 11mo ago
You can get decent eastern eu engineer for 2700$ (after tax) salary

ritcgab 11mo ago

Looking at the root server hardware page on Hetzner [1], it is not clear if their server is using ECC memory. It would be pretty bad if not. [1] https://docs.hetzner.com/robot/dedicated-server/general-information/root-server-hardware/ https://docs.hetzner.com/robot/dedicated-server/general-info...

walletdrainer 11mo ago
https://www.hetzner.com/dedicated-rootserver/ax42/ https://www.hetzner.com/dedicated-rootserver/ax42/ Go to the product pages (Yes all the normal Hetzner servers use ECC)
- ritcgab 11mo ago
  > Yes all the normal Hetzner servers use ECC This is wrong - or define "normal"? In their current product line, at least the default configuration of AX52/EX44/EX63/GEX44 doesn't have ECC. It is an upgrade option only. The blog post says their server has "8 cores Intel Xeon W-2145", which is PX92 or its variant and its base configuration can be without ECC.
  - walletdrainer 11mo ago
    Apparently I was wrong yeah, just looked at a lower specced server and foolishly extrapolated from there.

ribtoks 11mo ago

Keeping time series in Mongo…