6 ms·

> Use managed services for as long as possible Big agree here. Yes, you can save stupid money by handrolling postgres on an extremely beefy Hetzner server, or

> Use managed services for as long as possible Big agree here. Yes, you can save stupid money by handrolling postgres on an extremely beefy Hetzner server, or you can pay someone else and keep building your product: https://onlineornot.com/self-hosting-vs-managed-services-deciding-how-host-your-database#benefits-of-managed-services https://onlineornot.com/self-hosting-vs-managed-services-dec... This isn't to say, "don't bother learning how to do it yourself", but more "learn to pick your battles".

jordiburgos 3y ago
Which one do you recommend? Those get quite expensive.
- rozenmd 3y ago
  I personally use RDS
rldjbpin 3y ago
having no skin in the game, I feel this is a very interesting view, especially when you are offering a managed service building on top of another. go through enough levels of dependencies, and someone who could do a few layers of the onion in-house can sweep in with competitive pricing/offerings a la sherlocking.
Radim 3y ago
It may be a generational thing, a matter of familiarity with computing and computers. For someone who's lived through the 80s, "handrolling postgres" doesn't sound nearly as scary as you imagine. I expect the cost/benefit analysis of "handrolling postgres on a beefy Hetzner server" vs "navigating the menus and options of AWS services" would be different for different teams.
- tacker2000 3y ago
  Yea i also dont see the point of having everything “managed”. RDS is crazy expensive compared to self hosting and if i have the DB on prem its much faster as well. And the admin overhead is not so big to be honest if you are using just one DB. If you are Google scale of course things will change, but I think 80% of loads dont need any managed AWS stuff, replications, multiple nodes, kubernetes, etc… just periodic backups and it runs fine. But people nowadays just like throwing money around I guess, instead of trying to set it up for themselves.
- Ambix 3y ago
  Yep, I'd always prefer freedom and power of hosting my own PG instance upon a some robust VM offering to guiggling with clumsy AWS menus.
- PaulRobinson 3y ago
  If you're managing AWS infra through the web application, you're definitely not doing things as per AWS-prescribed best practice.
- erhaetherth 3y ago
  I've been on unmanaged MySQL for ~8 years now. Considered switching to managed but I'm not seeing any performance or stability issues, so I guess I'll just keep this train going until it craps out on me, then restore a backup onto a managed service, say sorry for the downtime, and that'll be that.
  - whynotmaybe 3y ago
    Do you know how long the downtime might be? Have you tested your backup recently? Gitlab had a long downtime because the backup was huge and on the other side of the country. The backup server was on a low speed network. https://www.arcserve.com/blog/lessons-learned-gitlabs-massive-backup-failure https://www.arcserve.com/blog/lessons-learned-gitlabs-massiv... How much money would you lose if you were down for one week? How many customer would you lose? How much credibility would you lose? For my peace of mind, I can't afford a spof when I know one lingering.
    - tacker2000 3y ago
      I get what your saying here, but its again the comparison with Github and extremely large sites thats the problem. Most of us dont run google/fb/github scale sites and the backup will probably fit on an external HDD and in some cases would be even downloadable over S3 in an hour.
      - whynotmaybe 3y ago
        That's what I can't be comfortable with : "would". How long does it take to try it? A day? Well then try it, either it'll work flawlessly on the first try, either you'll learn that the backup you have doesn't include logins, password and the security configuration that goes with it. Or that the dump you took lost some data because it wasn't in the right encoding. Or the tape drive you're using need specific drivers that aren't available on the web anymore because the company website's closed. ... This is a work of fiction. Any similarity to actual events might be purely coincidental...
- raverbashing 3y ago
  I agree And more to the point, learning how to "handroll" Postgres could be beneficial. You could have learned about options for limiting the amount of memory, etc Sure, managed is easier and use it when you can afford it easily. But before that, it's better to see how things are going (mem usage, disk usage, bottlenecks, etc)
- speedgoose 3y ago
  I'm looking at the ansible playbooks to setup my favourite beefy baremetal Hetzner server (128GB ram, Ryzen 9 5950X 16-Core, 450Gb fast NVME SSD, 3.5TB x2 NVME SSDs, 155€/month): - Install Debian 11 while booted in rescue mode. - Setup the root file system encryption using cryptsetup and dropbear (to enter the key during the boot through SSH). Involves chroot and some fun commands. - Setup ZFS encrypted mirror filesystem for the two additional SSDs. - OpenSSH hardening and Teleport installation. - Kubernetes installation (K3S) - Connecting kubernetes to my Argo CD instance or an existing Kubernetes cluster. And then through GitOps: - Installation of openebs-zfspv - Installation of kube-prometheus-stack helm chart - Installation of (many) postgresql instances and other craps I have been playing with Linux servers for 20 years and I find this fun and rewarding. But I do understand people saying that baremetal Hetzner is not for everyone. Especially if you start to have requirements such as "data must be encrypted at rest".
  - riku_iki 3y ago
    and you absolutely can't be sure your saas vendor is doing encryption and hardening.
  - riku_iki 3y ago
    > - Installation of (many) postgresql instances and other craps and you missing part about fault tolerance and fall back which is most complicated.
    - porsager 3y ago
      pg_auto_failover has your back - https://github.com/hapostgres/pg_auto_failover https://github.com/hapostgres/pg_auto_failover
  - Ambix 3y ago
    OMG you do not need this bloatload for just PG hosting. Just harden SSH, harden PG configs and voila :)
  - gog 3y ago
    If you skip Kubernetes the setup is not that complicated.
- SanderNL 3y ago
  I don't think "modern" stacks are sane enough to "handroll" anymore. Sure, you can do it, but look at the poster in this thread that details the setup of a debian server. Kubernetes, "Argo CD", zero-trust, the sheer amount of "management" is off the chart. "Installation of kube-prometheus-stack helm chart".. "Installation of openebs-zfspv".. It's not postgres that's the problem here.
  - marginalia_nu 3y ago
    A lot of "modern" stacks is just complexity for the sake of complexity. Google is doing it so clearly our 5 man startup will face the same scaling problems, or something like that. Many of the problems these tools solve are problems that wouldn't exist building things the old fashioned way. If you stick relatively close to the metal, operating this stuff is pretty easy. However it's notable that a very valid reason to prefer managed services as a SaaS is to cover your ass if things go wrong. Your SLA violation is their SLA violation.
    - maccard 3y ago
      The complex stacks are insane to operate yourself but very simple to operate if you use a managed offering instead, and they do provide genuine value. I can set up a new golang app on ECS with a load balancer and database, with a CI/CD pipeline, with 0 downtime updates in about 30 minutes. Most of that time is waiting for AWS to give me a load balancer. Our work applications have been running with this setup for over 2 years and the only thing Ivs done with infra in that time is adjusted instance sizes and bumped a MySql version.
      - marginalia_nu 3y ago
        To be fair, I can set up a new service on bare metal in minutes too, mostly because I don't need to set up everything from scratch. I don't really need to set up a database or load balancer or anything like that because it already exists on the server. Just create a new database schema, new systemd service, new nginx rule.
drawkbox 3y ago
Yes especially data, simplicity is key as well. Where possible go with simple but abstracted cloud storage, cloud tables and then a cloud db that is managed. We use Azure mostly right now but our storage system works across Azure storage, Amazon S3, Google Cloud and others. For tables, Azure Tables mainly. For database with filtering/paging better performant and ACID compliant cloud db, CosmosDB currently which is a dream with the differing apis (SQL, Mongo, Cassandra, Tables style). The more you can avoid vendor or dev-lockin the better so simple formats/messaging/routing and abstracted specifics/implementations. When you store data in storage or a cloud db the scaling is "infinite" and you can also snapshot or backup to anther one, you never worry about data. The front ends and APIs are mostly repos pushed to app/web services and everything else in data storage. Super simple and anywhere you need some special service that can be serverless or a dedicated setup, like maybe a RDBMS, chat server, network server or WebRTC/socket endpoint that interacts with the simple side. These managed as well if possible, though not always. Additionally, build cheap and horizontal scaling on web/real-time servers. Vertical scaling and sharding is for suckers. Side note: CosmosDB is like a combination of NoSQL, document databases and GraphQL and it is ACID compliant and you can do REST or SQL, it can even wrap MongoDB and Cassandra and make them ACID compliant. It really feels like the best way. Not many have all that and ACID compliant. Not even Amazon Redshift has that, DynamoDB does if specified. Google Firestore does if specified. I used to be big on RDMBS Oracle then MSSQL then PostgreSQL and those are great for backing/reporting etc but CosmosDB combines all the power of RDMBS, NoSQL, document databases, and ACID compliant with little worry about scale. It is vendor lockin to Azure, which you can route around with platform abstraction, but currently can't be beat. As you got that clean API layer you could change later but best way is limited/clean and if possible, non breaking change API layers/signatures.
- mamcx 3y ago
  > Where possible go with simple but abstracted cloud storage This is the total opposite of simplicity. Simple is PG and backup setup. DONE. Maybe you mean "comfort"?
  - drawkbox 3y ago
    Managed and no need to backup with snapshots. PG still has manage concerns with size, logs, access, scale etc. Nothing you have to do with cloud storage. I do love me some PostgreSQL but use it less and less except for reporting or heavily filtered needs. Horizontal over vertical for comfort and simplicity.
mejutoco 3y ago
Or you could use sqlite until you need postgres. I have to admit I reach for postgres immediately when in many cases sqlite would have served me just as well.
- ghiculescu 3y ago
  I’m super intrigued by SQLite these days. 11 years ago you used Postgres because that’s what Heroku told you to use.
- kristiandupont 3y ago
  SQLite seems to be gaining popularity with even larger projects which is surprising to me. As I see it, the big value prop of SQLite is that it runs in-process which, for a webapp, is almost nil? Other than that, it's not like queries are any simpler and the "simple" type system is, in my opinion, not a feature. I get that some might disagree with that. Is there some other reason why you would prefer it?
  - rozenmd 3y ago
    Check out Cloudflare D1
    - kristiandupont 3y ago
      It looks neat but I am still not sure what the appeal is exactly. Is it cost reduction?
      - rozenmd 3y ago
        Cost reduction and simplicity - costs nothing if no one is using it
  - mejutoco 3y ago
    I agree with you about the worse type system not being a feature. Also missing the json features of postgres is inconvenient. Only one file to backup or deploy is the biggest advantage of sqlite IMO.

jordiburgos 3y ago

Which one do you recommend? Those get quite expensive.

rozenmd 3y ago
I personally use RDS

rldjbpin 3y ago

having no skin in the game, I feel this is a very interesting view, especially when you are offering a managed service building on top of another. go through enough levels of dependencies, and someone who could do a few layers of the onion in-house can sweep in with competitive pricing/offerings a la sherlocking.

Radim 3y ago

It may be a generational thing, a matter of familiarity with computing and computers. For someone who's lived through the 80s, "handrolling postgres" doesn't sound nearly as scary as you imagine. I expect the cost/benefit analysis of "handrolling postgres on a beefy Hetzner server" vs "navigating the menus and options of AWS services" would be different for different teams.

tacker2000 3y ago
Yea i also dont see the point of having everything “managed”. RDS is crazy expensive compared to self hosting and if i have the DB on prem its much faster as well. And the admin overhead is not so big to be honest if you are using just one DB. If you are Google scale of course things will change, but I think 80% of loads dont need any managed AWS stuff, replications, multiple nodes, kubernetes, etc… just periodic backups and it runs fine. But people nowadays just like throwing money around I guess, instead of trying to set it up for themselves.
Ambix 3y ago
Yep, I'd always prefer freedom and power of hosting my own PG instance upon a some robust VM offering to guiggling with clumsy AWS menus.
PaulRobinson 3y ago
If you're managing AWS infra through the web application, you're definitely not doing things as per AWS-prescribed best practice.
erhaetherth 3y ago
I've been on unmanaged MySQL for ~8 years now. Considered switching to managed but I'm not seeing any performance or stability issues, so I guess I'll just keep this train going until it craps out on me, then restore a backup onto a managed service, say sorry for the downtime, and that'll be that.
- whynotmaybe 3y ago
  Do you know how long the downtime might be? Have you tested your backup recently? Gitlab had a long downtime because the backup was huge and on the other side of the country. The backup server was on a low speed network. https://www.arcserve.com/blog/lessons-learned-gitlabs-massive-backup-failure https://www.arcserve.com/blog/lessons-learned-gitlabs-massiv... How much money would you lose if you were down for one week? How many customer would you lose? How much credibility would you lose? For my peace of mind, I can't afford a spof when I know one lingering.
  - tacker2000 3y ago
    I get what your saying here, but its again the comparison with Github and extremely large sites thats the problem. Most of us dont run google/fb/github scale sites and the backup will probably fit on an external HDD and in some cases would be even downloadable over S3 in an hour.
    - whynotmaybe 3y ago
      That's what I can't be comfortable with : "would". How long does it take to try it? A day? Well then try it, either it'll work flawlessly on the first try, either you'll learn that the backup you have doesn't include logins, password and the security configuration that goes with it. Or that the dump you took lost some data because it wasn't in the right encoding. Or the tape drive you're using need specific drivers that aren't available on the web anymore because the company website's closed. ... This is a work of fiction. Any similarity to actual events might be purely coincidental...
raverbashing 3y ago
I agree And more to the point, learning how to "handroll" Postgres could be beneficial. You could have learned about options for limiting the amount of memory, etc Sure, managed is easier and use it when you can afford it easily. But before that, it's better to see how things are going (mem usage, disk usage, bottlenecks, etc)
speedgoose 3y ago
I'm looking at the ansible playbooks to setup my favourite beefy baremetal Hetzner server (128GB ram, Ryzen 9 5950X 16-Core, 450Gb fast NVME SSD, 3.5TB x2 NVME SSDs, 155€/month): - Install Debian 11 while booted in rescue mode. - Setup the root file system encryption using cryptsetup and dropbear (to enter the key during the boot through SSH). Involves chroot and some fun commands. - Setup ZFS encrypted mirror filesystem for the two additional SSDs. - OpenSSH hardening and Teleport installation. - Kubernetes installation (K3S) - Connecting kubernetes to my Argo CD instance or an existing Kubernetes cluster. And then through GitOps: - Installation of openebs-zfspv - Installation of kube-prometheus-stack helm chart - Installation of (many) postgresql instances and other craps I have been playing with Linux servers for 20 years and I find this fun and rewarding. But I do understand people saying that baremetal Hetzner is not for everyone. Especially if you start to have requirements such as "data must be encrypted at rest".
- riku_iki 3y ago
  and you absolutely can't be sure your saas vendor is doing encryption and hardening.
- riku_iki 3y ago
  > - Installation of (many) postgresql instances and other craps and you missing part about fault tolerance and fall back which is most complicated.
  - porsager 3y ago
    pg_auto_failover has your back - https://github.com/hapostgres/pg_auto_failover https://github.com/hapostgres/pg_auto_failover
- Ambix 3y ago
  OMG you do not need this bloatload for just PG hosting. Just harden SSH, harden PG configs and voila :)
- gog 3y ago
  If you skip Kubernetes the setup is not that complicated.
SanderNL 3y ago
I don't think "modern" stacks are sane enough to "handroll" anymore. Sure, you can do it, but look at the poster in this thread that details the setup of a debian server. Kubernetes, "Argo CD", zero-trust, the sheer amount of "management" is off the chart. "Installation of kube-prometheus-stack helm chart".. "Installation of openebs-zfspv".. It's not postgres that's the problem here.
- marginalia_nu 3y ago
  A lot of "modern" stacks is just complexity for the sake of complexity. Google is doing it so clearly our 5 man startup will face the same scaling problems, or something like that. Many of the problems these tools solve are problems that wouldn't exist building things the old fashioned way. If you stick relatively close to the metal, operating this stuff is pretty easy. However it's notable that a very valid reason to prefer managed services as a SaaS is to cover your ass if things go wrong. Your SLA violation is their SLA violation.
  - maccard 3y ago
    The complex stacks are insane to operate yourself but very simple to operate if you use a managed offering instead, and they do provide genuine value. I can set up a new golang app on ECS with a load balancer and database, with a CI/CD pipeline, with 0 downtime updates in about 30 minutes. Most of that time is waiting for AWS to give me a load balancer. Our work applications have been running with this setup for over 2 years and the only thing Ivs done with infra in that time is adjusted instance sizes and bumped a MySql version.
    - marginalia_nu 3y ago
      To be fair, I can set up a new service on bare metal in minutes too, mostly because I don't need to set up everything from scratch. I don't really need to set up a database or load balancer or anything like that because it already exists on the server. Just create a new database schema, new systemd service, new nginx rule.

drawkbox 3y ago

Yes especially data, simplicity is key as well. Where possible go with simple but abstracted cloud storage, cloud tables and then a cloud db that is managed. We use Azure mostly right now but our storage system works across Azure storage, Amazon S3, Google Cloud and others. For tables, Azure Tables mainly. For database with filtering/paging better performant and ACID compliant cloud db, CosmosDB currently which is a dream with the differing apis (SQL, Mongo, Cassandra, Tables style). The more you can avoid vendor or dev-lockin the better so simple formats/messaging/routing and abstracted specifics/implementations. When you store data in storage or a cloud db the scaling is "infinite" and you can also snapshot or backup to anther one, you never worry about data. The front ends and APIs are mostly repos pushed to app/web services and everything else in data storage. Super simple and anywhere you need some special service that can be serverless or a dedicated setup, like maybe a RDBMS, chat server, network server or WebRTC/socket endpoint that interacts with the simple side. These managed as well if possible, though not always. Additionally, build cheap and horizontal scaling on web/real-time servers. Vertical scaling and sharding is for suckers. Side note: CosmosDB is like a combination of NoSQL, document databases and GraphQL and it is ACID compliant and you can do REST or SQL, it can even wrap MongoDB and Cassandra and make them ACID compliant. It really feels like the best way. Not many have all that and ACID compliant. Not even Amazon Redshift has that, DynamoDB does if specified. Google Firestore does if specified. I used to be big on RDMBS Oracle then MSSQL then PostgreSQL and those are great for backing/reporting etc but CosmosDB combines all the power of RDMBS, NoSQL, document databases, and ACID compliant with little worry about scale. It is vendor lockin to Azure, which you can route around with platform abstraction, but currently can't be beat. As you got that clean API layer you could change later but best way is limited/clean and if possible, non breaking change API layers/signatures.

mamcx 3y ago
> Where possible go with simple but abstracted cloud storage This is the total opposite of simplicity. Simple is PG and backup setup. DONE. Maybe you mean "comfort"?
- drawkbox 3y ago
  Managed and no need to backup with snapshots. PG still has manage concerns with size, logs, access, scale etc. Nothing you have to do with cloud storage. I do love me some PostgreSQL but use it less and less except for reporting or heavily filtered needs. Horizontal over vertical for comfort and simplicity.

mejutoco 3y ago

Or you could use sqlite until you need postgres. I have to admit I reach for postgres immediately when in many cases sqlite would have served me just as well.

ghiculescu 3y ago
I’m super intrigued by SQLite these days. 11 years ago you used Postgres because that’s what Heroku told you to use.
kristiandupont 3y ago
SQLite seems to be gaining popularity with even larger projects which is surprising to me. As I see it, the big value prop of SQLite is that it runs in-process which, for a webapp, is almost nil? Other than that, it's not like queries are any simpler and the "simple" type system is, in my opinion, not a feature. I get that some might disagree with that. Is there some other reason why you would prefer it?
- rozenmd 3y ago
  Check out Cloudflare D1
  - kristiandupont 3y ago
    It looks neat but I am still not sure what the appeal is exactly. Is it cost reduction?
    - rozenmd 3y ago
      Cost reduction and simplicity - costs nothing if no one is using it
- mejutoco 3y ago
  I agree with you about the worse type system not being a feature. Also missing the json features of postgres is inconvenient. Only one file to backup or deploy is the biggest advantage of sqlite IMO.