6 ms·

"A while" is underselling it. As long as you have people who are half-decent with SQL, "just put Postgres on a big db server" will get you to 50 million row tab

"A while" is underselling it. As long as you have people who are half-decent with SQL, "just put Postgres on a big db server" will get you to 50 million row tables before you have to start thinking about even hiring a real DBA.

jedberg 3y ago
True. "While" for some large value of time. And if you set up the auto-vacuumer correctly from the start, you can go even further!
- turtles3 3y ago
  Unpopular opinion: if you're skimping on ops/DBA resources (as you may need to do in a startup), then MySQL is a better default. By all means use postgres if your use case demands it, but personally I find the ops story for MySQL takes less engineering overhead.
  - lenkite 3y ago
    I agree with this "unpopular opinion". Worked with both MySQL and postgres based mid-scale apps of several thousands of users. Postgres is so deeply lauded here at HN yet requires two more magnitudes of operations work to keep it up and running. Vacuuming sucks hard.
  - avereveard 3y ago
    That's not wrong, but postgres allows for cramming a lot of functionality in the db, and it's fast at following storage trend, i.e. with pgvector
    - turtles3 3y ago
      That is a very specific use case, and might only be a small subset of your actual data. If you don't have these specific requirements (eg. CRUD apps), you can save yourself a lot of unnecessary headaches by defaulting to MySQL. My main point is attempting to counter the narrative popular on HN that postgres should be an automatic default. For sure there are many aspects in which postgres is superior, I absolutely do not debate that, especially when it comes to developer experience. But there is much more to it than that when it comes to delivering business value. That's where ops and DBA concerns start to matter, and IMO MySQL is so far ahead in this regard that it outweighs all the other hideous warts of working with it, when you consider the bigger picture of the business as a whole.
  - savrajsingh 3y ago
    Yes! Vanilla MySQL and a good ORM (peewee, anyone?) gets you extremely far.
  - PedroBatista 3y ago
    Yes, and most successful companies who started in the last ~20 years started ( and many continue ) with a monolith and a MySQL database. Only the mega-cap ones started to pursue other options mostly due to their type of business and bucket-loads of "free" VC money with explicit orders to burn it and get "unicorn" status - which involves hiring thousands of developers in record time and the whole thing turns into a zoo. Which is an organizational problem, mostly not a tech one. Other than the ones we pretend are the whole Universe, there are thousands and thousands of medium to big companies with billions of revenue who started their product with a monolith and a MySQL database and many still do just that.
baq 3y ago
50M is something that can easily be done with just a dev who understands that sql is more than select, insert and update with the manual, google and chatgpt. You can get really damn far with a fat postgres box.
- hnpxr 3y ago
  It's not even funny how many software engineers just don't know that, SQL is crazy fast and performant, if you have basic understanding about it - I was once refactoring (or, rather, getting rid of) a microservice that was just a JSON blob storage on top of Postgres, without having any schema for blobs, with 100s of 1000s of them, no indices, and main complaint was - it's slow.
  - renegade-otter 3y ago
    Ignorance of database basics is something I allude to in the article, but it's a larger topic that I might vent about later.
- dalbasal 3y ago
  The problem with these discussions is that you can always get a lot out of any given architecture, structure, db approach or whatnot. Can. There's just a lot of daylight between "can" and "likely will." Ultimately, everything has its limitations and tradeoffs. If we respect them, it's generally a smooth ride. Problem is that we rarely do... within a company (startup or otherwise) under real conditions. There's also a dynamic where we build until the point where something stops us. Tech debt, complexity, over-engineering, under-engineering, feature bloat or antagonism between early decisions and current goals. There's a self-regulating aspect to this. If architecture is spot on, perfect for the task at hand we can move faster to reach the point where it no longer is.
  - baq 3y ago
    You're right of course. My point is that postgres is, compared to almost everything else, easy to get from "can" to "likely will" with just somebody with a brain, a manual and google. In absolute terms it of course depends, but the point is relative.
jiggawatts 3y ago
It’s so cute that you think 50M rows is big. Your phone can handle many times that, and update it tens of thousands of times per second.
- dagw 3y ago
  The problem isn't storing or inserting 50M rows, it querying 50M rows in non trivial ways. And the difference in performance between doing that 'right' and 'wrong' is orders of magnitude.
  - omegabravo 3y ago
    doing it on your phone only has one person querying it. Scale that to several thousand and it might appear different.
    - jiggawatts 3y ago
      I've scaled 300M rows in just one of many similarly sized tables to 1M users... in 2007... on a single box with spinning rust in it. Heck, my laptop could handle 100x the production load. It amazes me that my comment (while admittedly flippant) got voted down. It really is true that your phone can update a 50M row table about 10K times per second! That people are incredulous of this is in itself a stunning admission that developers these days don't have the faintest idea what computers can or cannot actually do. Just run the numbers: 50M rows with a generous 1 KB per row is 50 GB. My iPhone has 1TB of flash storage that has a random access latency of something like 50 microseconds, which equates to 200K IOPS. An ordinary NVMe laptop SSD can now do 2M. Writing even 10K random locations every second is well within mobile device capability, with 50% headroom to "scale". At 1 KB per row, this is just 10 MB/s, which is hilariously low compared to the device peak throughput of easily a few GB/s.
      - sethammons 3y ago
        I believe you are downvoted due to the "how cute" part, not the scale part. You could have delivered useful info without a put down
n0us 3y ago
> As long as you have people who are half-decent with SQL :/

jedberg 3y ago

True. "While" for some large value of time. And if you set up the auto-vacuumer correctly from the start, you can go even further!

turtles3 3y ago
Unpopular opinion: if you're skimping on ops/DBA resources (as you may need to do in a startup), then MySQL is a better default. By all means use postgres if your use case demands it, but personally I find the ops story for MySQL takes less engineering overhead.
- lenkite 3y ago
  I agree with this "unpopular opinion". Worked with both MySQL and postgres based mid-scale apps of several thousands of users. Postgres is so deeply lauded here at HN yet requires two more magnitudes of operations work to keep it up and running. Vacuuming sucks hard.
- avereveard 3y ago
  That's not wrong, but postgres allows for cramming a lot of functionality in the db, and it's fast at following storage trend, i.e. with pgvector
  - turtles3 3y ago
    That is a very specific use case, and might only be a small subset of your actual data. If you don't have these specific requirements (eg. CRUD apps), you can save yourself a lot of unnecessary headaches by defaulting to MySQL. My main point is attempting to counter the narrative popular on HN that postgres should be an automatic default. For sure there are many aspects in which postgres is superior, I absolutely do not debate that, especially when it comes to developer experience. But there is much more to it than that when it comes to delivering business value. That's where ops and DBA concerns start to matter, and IMO MySQL is so far ahead in this regard that it outweighs all the other hideous warts of working with it, when you consider the bigger picture of the business as a whole.
- savrajsingh 3y ago
  Yes! Vanilla MySQL and a good ORM (peewee, anyone?) gets you extremely far.
- PedroBatista 3y ago
  Yes, and most successful companies who started in the last ~20 years started ( and many continue ) with a monolith and a MySQL database. Only the mega-cap ones started to pursue other options mostly due to their type of business and bucket-loads of "free" VC money with explicit orders to burn it and get "unicorn" status - which involves hiring thousands of developers in record time and the whole thing turns into a zoo. Which is an organizational problem, mostly not a tech one. Other than the ones we pretend are the whole Universe, there are thousands and thousands of medium to big companies with billions of revenue who started their product with a monolith and a MySQL database and many still do just that.

baq 3y ago

50M is something that can easily be done with just a dev who understands that sql is more than select, insert and update with the manual, google and chatgpt. You can get really damn far with a fat postgres box.

hnpxr 3y ago
It's not even funny how many software engineers just don't know that, SQL is crazy fast and performant, if you have basic understanding about it - I was once refactoring (or, rather, getting rid of) a microservice that was just a JSON blob storage on top of Postgres, without having any schema for blobs, with 100s of 1000s of them, no indices, and main complaint was - it's slow.
- renegade-otter 3y ago
  Ignorance of database basics is something I allude to in the article, but it's a larger topic that I might vent about later.
dalbasal 3y ago
The problem with these discussions is that you can always get a lot out of any given architecture, structure, db approach or whatnot. Can. There's just a lot of daylight between "can" and "likely will." Ultimately, everything has its limitations and tradeoffs. If we respect them, it's generally a smooth ride. Problem is that we rarely do... within a company (startup or otherwise) under real conditions. There's also a dynamic where we build until the point where something stops us. Tech debt, complexity, over-engineering, under-engineering, feature bloat or antagonism between early decisions and current goals. There's a self-regulating aspect to this. If architecture is spot on, perfect for the task at hand we can move faster to reach the point where it no longer is.
- baq 3y ago
  You're right of course. My point is that postgres is, compared to almost everything else, easy to get from "can" to "likely will" with just somebody with a brain, a manual and google. In absolute terms it of course depends, but the point is relative.

jiggawatts 3y ago

It’s so cute that you think 50M rows is big. Your phone can handle many times that, and update it tens of thousands of times per second.

dagw 3y ago
The problem isn't storing or inserting 50M rows, it querying 50M rows in non trivial ways. And the difference in performance between doing that 'right' and 'wrong' is orders of magnitude.
- omegabravo 3y ago
  doing it on your phone only has one person querying it. Scale that to several thousand and it might appear different.
  - jiggawatts 3y ago
    I've scaled 300M rows in just one of many similarly sized tables to 1M users... in 2007... on a single box with spinning rust in it. Heck, my laptop could handle 100x the production load. It amazes me that my comment (while admittedly flippant) got voted down. It really is true that your phone can update a 50M row table about 10K times per second! That people are incredulous of this is in itself a stunning admission that developers these days don't have the faintest idea what computers can or cannot actually do. Just run the numbers: 50M rows with a generous 1 KB per row is 50 GB. My iPhone has 1TB of flash storage that has a random access latency of something like 50 microseconds, which equates to 200K IOPS. An ordinary NVMe laptop SSD can now do 2M. Writing even 10K random locations every second is well within mobile device capability, with 50% headroom to "scale". At 1 KB per row, this is just 10 MB/s, which is hilariously low compared to the device peak throughput of easily a few GB/s.
    - sethammons 3y ago
      I believe you are downvoted due to the "how cute" part, not the scale part. You could have delivered useful info without a put down

n0us 3y ago

> As long as you have people who are half-decent with SQL :/