5 ms·

Moved from Google Cloud -> Digital Ocean -> OVH. Running our own stuff on high powered servers is very easy and less trouble than you think. Sorted out the dep

Moved from Google Cloud -> Digital Ocean -> OVH. Running our own stuff on high powered servers is very easy and less trouble than you think. Sorted out the deploy with a "git push" and build container(s) meant we could just "Set it and forget it". We have a bit under a terabyte of Postgresql data. Any cloud is prohibitively expensive. I think some people think that the cloud is as good a sliced bread. It does not really save any developer time. But it's slower and more expensive than your own server by a huge margin. And I can always do my own stuff on my own iron. Really, I can't see a compelling reason to be in the cloud for the majority of mid-level workloads like ours.

dx034 5y ago
I also found that running a PostgreSQL database is really simple. Especially if most of your workload is read only, a few dedicated servers at several providers with a PostgreSQL cluster can deliver 100% uptime and more than enough performance for pretty much any use case. At the same time, this will still be cheaper than one managed database at any cloud provider. I've been running a PostgreSQL cluster with significant usage for a few years now, never had more than a few seconds downtime and I spend next to no time maintaining the database servers (apart from patching). If most requests are read only, clusters are so easy to do in Postgres. And even if one of the providers banned my account, I'd just promote a server at another provider to master and could still continue without downtime. I recently calculated what a switch to a cloud provider would cost, and it was at least 10x of what I pay now, for less performance and with lock-in effects. But I understand that there are business cases and industries where outsourcing makes sense.
- tomatowurst 5y ago
  can you share more details, because im in teh process of doing the same since having a few terabytes of postgresql / dynamodb is stupid expensive. for a lot big organizations its a matter of accountability. if they say AWS went down vs our dedicated servers went down, it matters a lot for insurance, clients. what i dont get are 4 man startups paying thousands to AWS ... because everybody does it.
  - dx034 5y ago
    As I said, if most queries are read-only it's really simple. Streaming replication works very well out of the box, just make sure you keep enough WAL segments on master so that slaves can catch up after some downtime. I have a 1-1 relationship between application servers and databases. The application queries replication delay and marks itself as unhealthy and reports an error if the delay is too high. You can also do that via postgres (max_replication_delay), but I found this way to allow for more graceful failovers. With streaming replication, servers are completely identical, so you can easily provision a new server. Failover is done by just one command on a slave. I don't have automatic failover as I only needed to use that once in several years (and that was on purpose), I'd rather accept downtime than having an unwanted failover. With that setup you can always failover and can scale read operations really well. There are solutions for postgres if you need more complicated setups, but I never looked into them. If you're in Europe, it's really cheap to get a dedicated machine from Hetzner with a few TB of NVMe. Just pay the extra money for 10gbit link, otherwise replication will take forever. But there are also some decent providers in the US, it's just more expensive. But with Hetzner, a two machine setup will be <$500 per month for really beefy servers. I'd just be careful with using block storage, I often found that to be a bottleneck with database servers. Local storage is almost always much faster. But in the end it depends on your use case. In the end, your database will usually go down because of a bug in the application or some misconfiguration. Both can happen on any service. It's really so rare these days to lose a server without notice. And Postgres is really stable, I've never seen it crash.
Xenixo 5y ago
Maintainability is much easier on a well working cloud setup from people who have potentially less knowledge. One company had 6 servers and used AWS snapshot for backup + managed MySQL. Backup and recovery of that db is possible by more people in the team as if it would run as non managed service.
lolinder 5y ago
> Really, I can't see a compelling reason to be in the cloud for the majority of mid-level workloads like ours. I work on a very small team. We have a few developers who double as ops. None of us are or want to be sysadmins. For our case, Amazon's ECS is a massive time and money saver. I spent a week or two a few years ago getting all of our services containerized. Since then we have not had a single full production outage (they were distressingly common before), and our sysadmin work has consisted exclusively of changing versions in Dockerfiles from time to time. Yes, most of the problems we had before could have been solved my a competent sysadmin, but that's precisely the point—hiring a good sysadmin is way more expensive for us than paying a bit extra to Amazon and just telling them "please run these containers with this config."
- TrickardRixx 5y ago
  Very similar experience here. I work on a two person "DevOps" team. Without AWS ECS we would have to have a much higher headcount. I get to spend most of my time solving real problems for the engineers on the product team rather than sysadmin work.
  - dijit 5y ago
    What are “real problems” for the engineers or product team?
    - TrickardRixx 5y ago
      Things like automating manual workflows, building small infrastructure debugging tools, or providing infrastructure consultation to an engineer trying to decouple two parts of a legacy code base.
- kissgyorgy 5y ago
  > None of us are or want to be sysadmins. It's such a huge misconception that by using a cloud provider you can avoid having "sysadmins" or don't need that kind of skills. You still need those, no matter which cloud and which service you use.

dx034 5y ago

I also found that running a PostgreSQL database is really simple. Especially if most of your workload is read only, a few dedicated servers at several providers with a PostgreSQL cluster can deliver 100% uptime and more than enough performance for pretty much any use case. At the same time, this will still be cheaper than one managed database at any cloud provider. I've been running a PostgreSQL cluster with significant usage for a few years now, never had more than a few seconds downtime and I spend next to no time maintaining the database servers (apart from patching). If most requests are read only, clusters are so easy to do in Postgres. And even if one of the providers banned my account, I'd just promote a server at another provider to master and could still continue without downtime. I recently calculated what a switch to a cloud provider would cost, and it was at least 10x of what I pay now, for less performance and with lock-in effects. But I understand that there are business cases and industries where outsourcing makes sense.

tomatowurst 5y ago
can you share more details, because im in teh process of doing the same since having a few terabytes of postgresql / dynamodb is stupid expensive. for a lot big organizations its a matter of accountability. if they say AWS went down vs our dedicated servers went down, it matters a lot for insurance, clients. what i dont get are 4 man startups paying thousands to AWS ... because everybody does it.
- dx034 5y ago
  As I said, if most queries are read-only it's really simple. Streaming replication works very well out of the box, just make sure you keep enough WAL segments on master so that slaves can catch up after some downtime. I have a 1-1 relationship between application servers and databases. The application queries replication delay and marks itself as unhealthy and reports an error if the delay is too high. You can also do that via postgres (max_replication_delay), but I found this way to allow for more graceful failovers. With streaming replication, servers are completely identical, so you can easily provision a new server. Failover is done by just one command on a slave. I don't have automatic failover as I only needed to use that once in several years (and that was on purpose), I'd rather accept downtime than having an unwanted failover. With that setup you can always failover and can scale read operations really well. There are solutions for postgres if you need more complicated setups, but I never looked into them. If you're in Europe, it's really cheap to get a dedicated machine from Hetzner with a few TB of NVMe. Just pay the extra money for 10gbit link, otherwise replication will take forever. But there are also some decent providers in the US, it's just more expensive. But with Hetzner, a two machine setup will be <$500 per month for really beefy servers. I'd just be careful with using block storage, I often found that to be a bottleneck with database servers. Local storage is almost always much faster. But in the end it depends on your use case. In the end, your database will usually go down because of a bug in the application or some misconfiguration. Both can happen on any service. It's really so rare these days to lose a server without notice. And Postgres is really stable, I've never seen it crash.

Xenixo 5y ago

Maintainability is much easier on a well working cloud setup from people who have potentially less knowledge. One company had 6 servers and used AWS snapshot for backup + managed MySQL. Backup and recovery of that db is possible by more people in the team as if it would run as non managed service.

lolinder 5y ago

> Really, I can't see a compelling reason to be in the cloud for the majority of mid-level workloads like ours. I work on a very small team. We have a few developers who double as ops. None of us are or want to be sysadmins. For our case, Amazon's ECS is a massive time and money saver. I spent a week or two a few years ago getting all of our services containerized. Since then we have not had a single full production outage (they were distressingly common before), and our sysadmin work has consisted exclusively of changing versions in Dockerfiles from time to time. Yes, most of the problems we had before could have been solved my a competent sysadmin, but that's precisely the point—hiring a good sysadmin is way more expensive for us than paying a bit extra to Amazon and just telling them "please run these containers with this config."

TrickardRixx 5y ago
Very similar experience here. I work on a two person "DevOps" team. Without AWS ECS we would have to have a much higher headcount. I get to spend most of my time solving real problems for the engineers on the product team rather than sysadmin work.
- dijit 5y ago
  What are “real problems” for the engineers or product team?
  - TrickardRixx 5y ago
    Things like automating manual workflows, building small infrastructure debugging tools, or providing infrastructure consultation to an engineer trying to decouple two parts of a legacy code base.
kissgyorgy 5y ago
> None of us are or want to be sysadmins. It's such a huge misconception that by using a cloud provider you can avoid having "sysadmins" or don't need that kind of skills. You still need those, no matter which cloud and which service you use.