5 ms·

> Here's how we managed to cut our costs by 90% You could cut your MongoDB costs by 100% by not using it ;) > without sacrificing performance or reliability.

> Here's how we managed to cut our costs by 90% You could cut your MongoDB costs by 100% by not using it ;) > without sacrificing performance or reliability. You're using a single server in a single datacenter. MongoDB Atlas is deployed to VMs on 2-3 AZs. You don't have close to the same reliability. (I'm also curious why their M40 instance costs $1000, when the Pricing Calculator (https://www.mongodb.com/pricing https://www.mongodb.com/pricing) says M40 is $760/month? Was it the extra storage?) > We're building Prosopo to be resilient to outages, such as the recent massive AWS outage, so we use many different cloud providers This means you're going to have multiple outages, AND incur more cross-internet costs. How does going to Hetzner make you more resilient to outages? You have one server in one datacenter. Intelligent, robust design at one provider (like AWS) is way more resilient, and intra-zone transfer is cheaper than going out to the cloud ($0.02/GB vs $0.08/GB). You do not have a centralized or single point of failure design with AWS. They're not dummies; plenty of their services are operated independently per region. But they do expect you to use their infrastructure intelligently to avoid creating a single point of failure. (For example, during the AWS outage, my company was in us-east-1, and we never had any issues, because we didn't depend on calling AWS APIs to continue operating. Things already running continue to run.) I get it; these "we cut bare costs by moving away from the cloud" posts are catnip for HN. But they usually don't make sense. There's only a few circumstances where you really have to transfer out a lot of traffic, or need very large storage, where cloud pricing is just too much of a premium. The whole point of using the cloud is to use it as a competitive advantage. Giving yourself an extra role (sysadmin) in addition to your day job (developer, data scientist, etc) and more maintenance tasks (installing, upgrading, patching, troubleshooting, getting on-call, etc) with lower reliability and fewer services, isn't an advantage.

goastler 1y ago
> you're going to have multiple outages us: 0, aws: 1. Looking good so far ;) > AND incur more cross-internet costs hetzner have no bandwidth traffic limit (only speed) on the machine, we can go nuts. I understand you point wrt the cloud, but I spend as much time debugging/building a cloud deployment (atlas :eyes: ) as I do a self-hosted solution. Aws gives you all the tools to build a super reliable data store, but many people just chuck something on us-east-1 and go. There's you single point of failure. Given we're constructing a many-node decentralised system, self-hosted actually makes more sense for us because we've already had to become familiar enough to create a many-node system for our primary product. When/if we have a situation where we need high data availability I would strongly consider the cloud, but in the situations where you can deal with a bit of downtime you're massively saving over cloud offerings. We'll post a 6-month and 1-year follow-up to update the scoreboard above
- MobileVet 1y ago
  Thanks for sharing the story and committing to a 6-month and 1 year follow up. We will definitely be interested to hear further how it went over time. In the mean time, I am curious where the time was spent debugging and building Atlas deployments? It certainly isn't the cheapest option, but it has been quite a '1 click' solution for us.
- runako 1y ago
  > many people just chuck something on us-east-1 and go Even dropping something on a single EC2 node in us-east-1 (or at Google Cloud) is going to be more reliable over time than a single dedicated machine elsewhere. This is because they run with a layer that will e.g. live migrate your running apps in case of hardware failures. The failure modes of dedicated are quite different than those of the modern hyperscaler clouds.
  - chubot 1y ago
    It's not an apples-to-apples comparison, because EC2 and Google Cloud have ephemeral disk - persistent disk is an add-on, which is implemented with a complex and frequently changing distributed storage system On the other hand, a Hetzner machine I just rented came with Linux software RAID enabled (md devices in the kernel) --- I'm not aware of any comparisons, but I'd like to see see some It's not straightforward, and it's not obvious the cloud is more reliable The cloud introduces many other single points of failure, by virtue of being more complex e.g. human administration failure, with the Unisuper incident https://news.ycombinator.com/item?id=40366867 https://news.ycombinator.com/item?id=40366867 https://arstechnica.com/gadgets/2024/05/google-cloud-accidentally-nukes-customer-account-causes-two-weeks-of-downtime/ https://arstechnica.com/gadgets/2024/05/google-cloud-acciden... - “Unprecedented” Google Cloud event wipes out customer account and its backups Of course, dedicated hardware could have a similar type of failure, but I think the simplicity means there is less variety in the errors. e.g. A distributed system is one in which the failure of a computer you didn't even know existed can render your own computer unusable - Leslie Lamport
    - travisgriggs 1y ago
Glamklo 1y ago
[dead]
mnutt 1y ago
> we never had any issues, because we didn't depend on calling AWS APIs to continue operating. Things already running continue to run. I think it was just luck of the draw that the failure happened in this way and not some other way. Even if APIs falling over but EC2 instances remaining up is a slightly more likely failure mode, it means you can't run autoscaling, can't depend on spot instances which in an outage you can lose and can't replace.
- 0xbadcafebee 1y ago
  > it means you can't run autoscaling, can't depend on spot instances which in an outage you can lose and can't replace Yes, this is part of designing for reliability. If you use spot or autoscaling, you can't assume you will have high availability in those components. They're optimizations, like a cache. A cache can disappear, and this can have a destabilizing effect on your architecture if you don't plan for it. This lack of planning is pretty common, unfortunately. Whether it's in a software component or system architecture, people often use a thing without understanding the implications of it. Then when AWS API calls become unavailable, half the internet falls over... because nobody planned for "what happens when the control plane disappears". (This is actually a critical safety consideration in other systems)
  - mnutt 1y ago
    Sure, you can only use EC2, not use autoscaling or spot and instead just provision to your highest capacity needs, and not use any other AWS service that relies on dynamo as a dependency. We still take some steps to mitigate control plane issues in what I consider a reasonable AWS setup (attempt to lock ASGs to prevent scale-down) but I place the control plane disappearing on the same level as the entire region going dark, and just run multi-region.
toast0 1y ago
> Intelligent, robust design at one provider (like AWS) is way more resilient, and intra-zone transfer is cheaper than going out to the cloud ($0.02/GB vs $0.08/GB). If traffic cost is relevant (which it is for a lot of use cases), Hetzner's price of $1.20/TB ($0.0012 / GB) for internet traffic [1] is an order of magnitude less than what AWS charges between AWS locations in the same metro. If you host only at providers with reasonable bandwidth charges, most likely all of your bandwidth will be billed at less than what AWS charges for inter-zone traffic. That's obscene. As far as I can tell, clouds are balancing their budgets on the back of traffic charges, but nothing else feels under cost either. > For example, during the AWS outage, my company was in us-east-1, and we never had any issues, because we didn't depend on calling AWS APIs to continue operating. Things already running continue to run. This doesn't always work out. During the GCP outage, my service was running fine, but other similar services were having trouble, so we attracted more usage, which we would have scaled up for, except that the GCP outage prevented that. Cloud makes it very expensive to run scaled beyond current needs and promises that scale out will be available to do just in time... [1] https://docs.hetzner.com/robot/general/traffic/ https://docs.hetzner.com/robot/general/traffic/
- precommunicator 1y ago
  keep in mind, for dedicated servers, traffic is free and unlimited - see the page you've linked
  - toast0 1y ago
    Not if you're running at 10G...
dspillett 1y ago
> You're using a single server in a single datacenter. This is a common problem with “bare metal saved us $000/mo” articles. Bare metal is cheaper than cloud by any measure, but the comparisons given tend to be misleadingly exaggerated as they don't compare like-for-like in terms of redundancy and support, and after considering those factors it can be a much closer result (sometimes down as far as familiarity and personal preference being more significant). Of course unless you are paying extra for multi-region redundancy things like the recent us-east-1 outage will kill you, and that single point of failure might not really matter if there are several others throughout your systems anyway, as is sometimes the case.
- PenguinCoder 1y ago
  Premature optimization. Not every single service needs or require 5 nines.
  - withinboredom 1y ago
    and each additional nine increases complexity geometrically.
  - dpkirchner 1y ago
    It's true, but I'm woken up more frequently if there are fewer 9s, which is unpleasant. It's worth the extra cost to me.
  - chasd00 1y ago
    you have to look at all the factors, a simple server in a simple datacenter can be very very stable. When we were all doing bare metal servers back in the day server uptimes measured in years wasn't that rare.
  - hdgvhicv 1y ago
    Hence you can use AWS to host them.
  - bastawhiz 1y ago
    What does that mean, though? If I'm storing data on a NAS, and I keep backups on a tape, a simple hardware failure that causes zero downtime on S3 might take what, hours to recover? Days? If my database server dies and I need to boot a new one, how long will that take? If I'm on RDS, maybe five minutes. If it's bare metal and I need to install software and load my data into it, perhaps an hour or more. Being able to recover from failure isn't a premature optimization. "The site is down and customers are angry" is an inevitability. If you can't handle failure modes in a timely manner, you aren't handling failure modes. That's not an optimization, that's table stakes. It's not about five nines, it's about four nines or even three nines.
koakuma-chan 1y ago
> You could cut your MongoDB costs by 100% by not using it ;) I cut my Mongo DB costs by 100% by piping my data to /dev/null.
- philote 1y ago
  https://github.com/dcramer/mangodb https://github.com/dcramer/mangodb
- diffuse_l 1y ago
  At least it's ACID compliant
canucktrash669 1y ago
At some point our cross-AZ traffic for Elasticsearch replication at AWS was more expensive than what we'd pay to host the whole cluster replicated across multiple baremetal Hetzner servers. Could we have done better with more sensible configs? Was it silly to cluster ES cross-AZ? Maybe. Point is that if you don't police every single detail of your platform at AWS/GCP and the like, their made-up charges will bleed your startup and grease their stock price.
- canucktrash669 1y ago
  turns out cross AZ is recommended for ES. perhaps our data team was rewritting the indices too often. but it was an internal requirement. so I think the data schema could have been more efficient to append deltas instead of reindexing all. but none of that will inflate your bill significantly at Hetzner. of course it will at AWS as that's how they incentivise clients to optimize and reduce their impact. and that's how you cut your runway by 3-6 months in compute heavy startups
lxe 1y ago
I think you underestimate how reduction in complexity can increase reliability. becoming a sysadmin for a single inexpensive server instance carries almost the same operational burden as operating an unavoidably very complicated cluster using a cloud provider.
- gervwyk 1y ago
  not if you are using Atlas. Its as simple as it can be with way more functionality you can ever admin in yourself. As others have said unless the scale of the data is the issue, if your switching because of cost, perhaps you should be going back to your business model instead.
- hdgvhicv 1y ago
  Nowhere near the same. Admining a few servers is far easier than a mix of AWS cloud services, especially when they are either metal as a service or plain VMs.
navigate8310 1y ago
Usually AWS is pretty good at hiding all the reliability and robustness that goes onto into making a customer's managed service. Customers are not made aware what it takes.
- chasd00 1y ago
  An interesting experiment would be doing the equivalent at the scale of the median saas company. Setup mongodb (or any database) so that you have geographically distributed nodes with replication+whatever else and maintain the same SLA as one of the big hyperscalers. Blog about how long did it take to setup, how hard is it to maintain, and how much are the ongoing costs. My hunch is a setup on the scale of the median saas company is way more simple and cost effective than you'd think.
port11 1y ago
These types of posts make for excellent karma farming, but this one does present all the issues you've mentioned. Heck, Scaleway has managed Mongo for a bit more money and with redundancy and multi-AZ to boot. Were they trying to go as cheap as possible?
celsoazevedo 1y ago
> You have one server in one datacenter. It doesn't have to be only one server in one datacenter though. It's more work, but you can have replicas ready to go at other Hetzner DCs (they offer bare metal at 3 locations in 2 different countries) or at other cheaper providers like OVH. Two or three $160 servers is still cheaper than what they're paying right now.
whatever1 1y ago
AWS and Azure were down for a full day in the past month. No way I cannot spin up my infra in a full day even if the current datacenter burns to the ground. So we have the same reliability.
- hsbauauvhabzb 1y ago
  Not if you don’t have hot replicated user data etc, assuming that matters, which it will unless you outsource auth and if you do that you’re back at square 1
  - walletdrainer 1y ago
    Just have backups, you could literally sync everything to rsync.net unless you have a ridiculous amount of users
KaiserPro 1y ago
> MongoDB Atlas is deployed to VMs on 2-3 AZs I've not actually seen an AZ go down in isolation, so whilst I agree its technically a less "robust" deployment, in practice its not that much of a difference. > these "we cut bare costs by moving away from the cloud" posts are catnip for HN. But they usually don't make sense. We moved away from atlas because they couldn’t cope with the data growth that we had(4tb is the max per DB). Turns out that its a fuck load cheaper even hosting on amazon (as in 50%). We haven't moved to hertzner because that would be more effort than we really want to expend, but its totally doable, with not that much extra work. > more maintenance tasks (installing, upgrading, patching, troubleshooting, getting on-call, etc) with lower reliability and fewer services, isn't an advantage. Depends right, firstly its not that much of an overhead, and if it saves you significant cash, then it increases your run rate.
- korkybuchek 1y ago
  > I've not actually seen an AZ go down in isolation Counterpoint: I have. Maybe not completely down, but degraded, or out of capacity on an instance type, or some other silly issue that caused an AZ drain. It happens.
  - dvfjsdhgfv 1y ago
    While I agree, I remember we once had cross-region replication for some product but when AWS was down the service was down anyway because of some dependency. Things were working fine during our DR exercises, but when the actual failure arrived, cross-region turned out useless.
hsbauauvhabzb 1y ago
> For example, during the AWS outage, my company was in us-east-1, and we never had any issues, because we didn't depend on calling AWS APIs to continue operating. Things already running continue to run. Naïve. If the network infrastructure is down, your computer goes down, it just happens that the functionality that went down you didn’t rely on. You could not rely on any functions at all by turning the server off, too.
- tekno45 1y ago
  yeah loadbalancers definitely removed instances during that outage. Watched it happen.
deleted 1y ago
[deleted]
torginus 1y ago
How often do the AZs matter? - I feel like there's a major global outage on every cloud provider of choice, at least every other year, yet I don't remember any outage where only a single AZ went down (I'm on AWS). Fighting said outages is often made harder is that the providers themselves just don't admit to anything being wrong, everything's green on the dashboard yet 4 out of 5 requests are timing out.
rajman187 1y ago
> You could cut your MongoDB costs by 100% by not using it ;) Came here to say exactly this
winrid 1y ago
At FastComments we have Mongo deployed across three continents and four regions on dedicated servers with full disk encryption, across two major providers just incase. It was setup by one person. Replication lag is usually under 300ms.
raxxorraxor 1y ago
I don't buy it. It really depends on your service, but I don't believe the reliability story. All large providers have had outages and I do host services on a single server that didn't have an outage in a few years. Depends on the service and its complexity. More complexity means more outages. In most instances a focus on easy recoverability is more productive than preemptive "reliability". As I have said, depends on your service. And prices get premium very fast if you have either a lot of traffic or low traffic but larger file interchange. And you have more work to do if you use the cloud, because it uses non-standard interfaces. Today a well maintained server is a few clicks away. Even for managed servers you have maintenance and configuration. Plus, your provider probably changes the service quite often. I had to accommodate beanstalk while my application was just running on its own, free of maintenance needs.

goastler 1y ago

> you're going to have multiple outages us: 0, aws: 1. Looking good so far ;) > AND incur more cross-internet costs hetzner have no bandwidth traffic limit (only speed) on the machine, we can go nuts. I understand you point wrt the cloud, but I spend as much time debugging/building a cloud deployment (atlas :eyes: ) as I do a self-hosted solution. Aws gives you all the tools to build a super reliable data store, but many people just chuck something on us-east-1 and go. There's you single point of failure. Given we're constructing a many-node decentralised system, self-hosted actually makes more sense for us because we've already had to become familiar enough to create a many-node system for our primary product. When/if we have a situation where we need high data availability I would strongly consider the cloud, but in the situations where you can deal with a bit of downtime you're massively saving over cloud offerings. We'll post a 6-month and 1-year follow-up to update the scoreboard above

MobileVet 1y ago
Thanks for sharing the story and committing to a 6-month and 1 year follow up. We will definitely be interested to hear further how it went over time. In the mean time, I am curious where the time was spent debugging and building Atlas deployments? It certainly isn't the cheapest option, but it has been quite a '1 click' solution for us.
runako 1y ago
> many people just chuck something on us-east-1 and go Even dropping something on a single EC2 node in us-east-1 (or at Google Cloud) is going to be more reliable over time than a single dedicated machine elsewhere. This is because they run with a layer that will e.g. live migrate your running apps in case of hardware failures. The failure modes of dedicated are quite different than those of the modern hyperscaler clouds.
- chubot 1y ago
  It's not an apples-to-apples comparison, because EC2 and Google Cloud have ephemeral disk - persistent disk is an add-on, which is implemented with a complex and frequently changing distributed storage system On the other hand, a Hetzner machine I just rented came with Linux software RAID enabled (md devices in the kernel) --- I'm not aware of any comparisons, but I'd like to see see some It's not straightforward, and it's not obvious the cloud is more reliable The cloud introduces many other single points of failure, by virtue of being more complex e.g. human administration failure, with the Unisuper incident https://news.ycombinator.com/item?id=40366867 https://news.ycombinator.com/item?id=40366867 https://arstechnica.com/gadgets/2024/05/google-cloud-accidentally-nukes-customer-account-causes-two-weeks-of-downtime/ https://arstechnica.com/gadgets/2024/05/google-cloud-acciden... - “Unprecedented” Google Cloud event wipes out customer account and its backups Of course, dedicated hardware could have a similar type of failure, but I think the simplicity means there is less variety in the errors. e.g. A distributed system is one in which the failure of a computer you didn't even know existed can render your own computer unusable - Leslie Lamport
  - travisgriggs 1y ago

Glamklo 1y ago

[dead]

mnutt 1y ago

> we never had any issues, because we didn't depend on calling AWS APIs to continue operating. Things already running continue to run. I think it was just luck of the draw that the failure happened in this way and not some other way. Even if APIs falling over but EC2 instances remaining up is a slightly more likely failure mode, it means you can't run autoscaling, can't depend on spot instances which in an outage you can lose and can't replace.

0xbadcafebee 1y ago
> it means you can't run autoscaling, can't depend on spot instances which in an outage you can lose and can't replace Yes, this is part of designing for reliability. If you use spot or autoscaling, you can't assume you will have high availability in those components. They're optimizations, like a cache. A cache can disappear, and this can have a destabilizing effect on your architecture if you don't plan for it. This lack of planning is pretty common, unfortunately. Whether it's in a software component or system architecture, people often use a thing without understanding the implications of it. Then when AWS API calls become unavailable, half the internet falls over... because nobody planned for "what happens when the control plane disappears". (This is actually a critical safety consideration in other systems)
- mnutt 1y ago
  Sure, you can only use EC2, not use autoscaling or spot and instead just provision to your highest capacity needs, and not use any other AWS service that relies on dynamo as a dependency. We still take some steps to mitigate control plane issues in what I consider a reasonable AWS setup (attempt to lock ASGs to prevent scale-down) but I place the control plane disappearing on the same level as the entire region going dark, and just run multi-region.

toast0 1y ago

> Intelligent, robust design at one provider (like AWS) is way more resilient, and intra-zone transfer is cheaper than going out to the cloud ($0.02/GB vs $0.08/GB). If traffic cost is relevant (which it is for a lot of use cases), Hetzner's price of $1.20/TB ($0.0012 / GB) for internet traffic [1] is an order of magnitude less than what AWS charges between AWS locations in the same metro. If you host only at providers with reasonable bandwidth charges, most likely all of your bandwidth will be billed at less than what AWS charges for inter-zone traffic. That's obscene. As far as I can tell, clouds are balancing their budgets on the back of traffic charges, but nothing else feels under cost either. > For example, during the AWS outage, my company was in us-east-1, and we never had any issues, because we didn't depend on calling AWS APIs to continue operating. Things already running continue to run. This doesn't always work out. During the GCP outage, my service was running fine, but other similar services were having trouble, so we attracted more usage, which we would have scaled up for, except that the GCP outage prevented that. Cloud makes it very expensive to run scaled beyond current needs and promises that scale out will be available to do just in time... [1] https://docs.hetzner.com/robot/general/traffic/ https://docs.hetzner.com/robot/general/traffic/

precommunicator 1y ago
keep in mind, for dedicated servers, traffic is free and unlimited - see the page you've linked
- toast0 1y ago
  Not if you're running at 10G...

dspillett 1y ago

> You're using a single server in a single datacenter. This is a common problem with “bare metal saved us $000/mo” articles. Bare metal is cheaper than cloud by any measure, but the comparisons given tend to be misleadingly exaggerated as they don't compare like-for-like in terms of redundancy and support, and after considering those factors it can be a much closer result (sometimes down as far as familiarity and personal preference being more significant). Of course unless you are paying extra for multi-region redundancy things like the recent us-east-1 outage will kill you, and that single point of failure might not really matter if there are several others throughout your systems anyway, as is sometimes the case.

PenguinCoder 1y ago
Premature optimization. Not every single service needs or require 5 nines.
- withinboredom 1y ago
  and each additional nine increases complexity geometrically.
- dpkirchner 1y ago
  It's true, but I'm woken up more frequently if there are fewer 9s, which is unpleasant. It's worth the extra cost to me.
- chasd00 1y ago
  you have to look at all the factors, a simple server in a simple datacenter can be very very stable. When we were all doing bare metal servers back in the day server uptimes measured in years wasn't that rare.
- hdgvhicv 1y ago
  Hence you can use AWS to host them.
- bastawhiz 1y ago
  What does that mean, though? If I'm storing data on a NAS, and I keep backups on a tape, a simple hardware failure that causes zero downtime on S3 might take what, hours to recover? Days? If my database server dies and I need to boot a new one, how long will that take? If I'm on RDS, maybe five minutes. If it's bare metal and I need to install software and load my data into it, perhaps an hour or more. Being able to recover from failure isn't a premature optimization. "The site is down and customers are angry" is an inevitability. If you can't handle failure modes in a timely manner, you aren't handling failure modes. That's not an optimization, that's table stakes. It's not about five nines, it's about four nines or even three nines.

koakuma-chan 1y ago

> You could cut your MongoDB costs by 100% by not using it ;) I cut my Mongo DB costs by 100% by piping my data to /dev/null.

philote 1y ago
https://github.com/dcramer/mangodb https://github.com/dcramer/mangodb
diffuse_l 1y ago
At least it's ACID compliant

canucktrash669 1y ago

At some point our cross-AZ traffic for Elasticsearch replication at AWS was more expensive than what we'd pay to host the whole cluster replicated across multiple baremetal Hetzner servers. Could we have done better with more sensible configs? Was it silly to cluster ES cross-AZ? Maybe. Point is that if you don't police every single detail of your platform at AWS/GCP and the like, their made-up charges will bleed your startup and grease their stock price.

canucktrash669 1y ago
turns out cross AZ is recommended for ES. perhaps our data team was rewritting the indices too often. but it was an internal requirement. so I think the data schema could have been more efficient to append deltas instead of reindexing all. but none of that will inflate your bill significantly at Hetzner. of course it will at AWS as that's how they incentivise clients to optimize and reduce their impact. and that's how you cut your runway by 3-6 months in compute heavy startups

lxe 1y ago

I think you underestimate how reduction in complexity can increase reliability. becoming a sysadmin for a single inexpensive server instance carries almost the same operational burden as operating an unavoidably very complicated cluster using a cloud provider.

gervwyk 1y ago
not if you are using Atlas. Its as simple as it can be with way more functionality you can ever admin in yourself. As others have said unless the scale of the data is the issue, if your switching because of cost, perhaps you should be going back to your business model instead.
hdgvhicv 1y ago
Nowhere near the same. Admining a few servers is far easier than a mix of AWS cloud services, especially when they are either metal as a service or plain VMs.

navigate8310 1y ago

Usually AWS is pretty good at hiding all the reliability and robustness that goes onto into making a customer's managed service. Customers are not made aware what it takes.

chasd00 1y ago
An interesting experiment would be doing the equivalent at the scale of the median saas company. Setup mongodb (or any database) so that you have geographically distributed nodes with replication+whatever else and maintain the same SLA as one of the big hyperscalers. Blog about how long did it take to setup, how hard is it to maintain, and how much are the ongoing costs. My hunch is a setup on the scale of the median saas company is way more simple and cost effective than you'd think.

port11 1y ago

These types of posts make for excellent karma farming, but this one does present all the issues you've mentioned. Heck, Scaleway has managed Mongo for a bit more money and with redundancy and multi-AZ to boot. Were they trying to go as cheap as possible?

celsoazevedo 1y ago

> You have one server in one datacenter. It doesn't have to be only one server in one datacenter though. It's more work, but you can have replicas ready to go at other Hetzner DCs (they offer bare metal at 3 locations in 2 different countries) or at other cheaper providers like OVH. Two or three $160 servers is still cheaper than what they're paying right now.

whatever1 1y ago

AWS and Azure were down for a full day in the past month. No way I cannot spin up my infra in a full day even if the current datacenter burns to the ground. So we have the same reliability.

hsbauauvhabzb 1y ago
Not if you don’t have hot replicated user data etc, assuming that matters, which it will unless you outsource auth and if you do that you’re back at square 1
- walletdrainer 1y ago
  Just have backups, you could literally sync everything to rsync.net unless you have a ridiculous amount of users

KaiserPro 1y ago

> MongoDB Atlas is deployed to VMs on 2-3 AZs I've not actually seen an AZ go down in isolation, so whilst I agree its technically a less "robust" deployment, in practice its not that much of a difference. > these "we cut bare costs by moving away from the cloud" posts are catnip for HN. But they usually don't make sense. We moved away from atlas because they couldn’t cope with the data growth that we had(4tb is the max per DB). Turns out that its a fuck load cheaper even hosting on amazon (as in 50%). We haven't moved to hertzner because that would be more effort than we really want to expend, but its totally doable, with not that much extra work. > more maintenance tasks (installing, upgrading, patching, troubleshooting, getting on-call, etc) with lower reliability and fewer services, isn't an advantage. Depends right, firstly its not that much of an overhead, and if it saves you significant cash, then it increases your run rate.

korkybuchek 1y ago
> I've not actually seen an AZ go down in isolation Counterpoint: I have. Maybe not completely down, but degraded, or out of capacity on an instance type, or some other silly issue that caused an AZ drain. It happens.
- dvfjsdhgfv 1y ago
  While I agree, I remember we once had cross-region replication for some product but when AWS was down the service was down anyway because of some dependency. Things were working fine during our DR exercises, but when the actual failure arrived, cross-region turned out useless.

hsbauauvhabzb 1y ago

> For example, during the AWS outage, my company was in us-east-1, and we never had any issues, because we didn't depend on calling AWS APIs to continue operating. Things already running continue to run. Naïve. If the network infrastructure is down, your computer goes down, it just happens that the functionality that went down you didn’t rely on. You could not rely on any functions at all by turning the server off, too.

tekno45 1y ago
yeah loadbalancers definitely removed instances during that outage. Watched it happen.

deleted 1y ago

[deleted]

torginus 1y ago

How often do the AZs matter? - I feel like there's a major global outage on every cloud provider of choice, at least every other year, yet I don't remember any outage where only a single AZ went down (I'm on AWS). Fighting said outages is often made harder is that the providers themselves just don't admit to anything being wrong, everything's green on the dashboard yet 4 out of 5 requests are timing out.

rajman187 1y ago

> You could cut your MongoDB costs by 100% by not using it ;) Came here to say exactly this

winrid 1y ago

At FastComments we have Mongo deployed across three continents and four regions on dedicated servers with full disk encryption, across two major providers just incase. It was setup by one person. Replication lag is usually under 300ms.

raxxorraxor 1y ago

I don't buy it. It really depends on your service, but I don't believe the reliability story. All large providers have had outages and I do host services on a single server that didn't have an outage in a few years. Depends on the service and its complexity. More complexity means more outages. In most instances a focus on easy recoverability is more productive than preemptive "reliability". As I have said, depends on your service. And prices get premium very fast if you have either a lot of traffic or low traffic but larger file interchange. And you have more work to do if you use the cloud, because it uses non-standard interfaces. Today a well maintained server is a few clicks away. Even for managed servers you have maintenance and configuration. Plus, your provider probably changes the service quite often. I had to accommodate beanstalk while my application was just running on its own, free of maintenance needs.