6 ms·

And replication for postgres is not easy? Sharding in postgres requires some plproxy/pgbouncer knowledge - but they don't cause as many problems in the future a

by eddd 12y ago

And replication for postgres is not easy? Sharding in postgres requires some plproxy/pgbouncer knowledge - but they don't cause as many problems in the future as mongodb. Me personally, when it comes to databases i go for the reliable first - then i can play with scalability.

remon 12y ago
That almost certainly implies you're not working on anything that requires horizontal scalability. That's a luxury not everyone can afford. Scalability, in almost all cases that actually require it, is the first priority. It's like saying it's more important that the car drives reliably rather than everyone that needs to travel fitting into the car in the first place.
- davidw 12y ago
  I think there are probably way too many people out there doing scaling before they're doing "getting things right". It's premature optimization that causes a lot of problems. Naturally, there are plenty of people who do need to scale a lot, too, but presumably by the time they need to, they have a good idea of the problem and exactly how best to scale it.
  - remon 12y ago
    That may be generally true but those two things are not mutually exclusive. Scalability doesn't come at the expense of getting things right, it just might come at the expense of certain features which applies to most distributed system problems.
    - CHY872 12y ago
      Vertical scaling can take you a surprisingly long way
  - craigching 12y ago
    Choosing MongoDB isn't always about scaling immediately. It can be about ease of use and easy to configure replication. See my post here [1] here about our use case. [1] -- https://news.ycombinator.com/item?id=8657458 https://news.ycombinator.com/item?id=8657458
- CHY872 12y ago
  The point at which you need to scale horizontally is (for most applications) when you hit data scale in terabytes. When this is the case, you probably have enough money to hire someone who knows what they're doing with databases in order to set up a migration properly - you don't really need the ease of use configuration features. SQL databases give data durability and transactions, integrity constraints, joins etc, which makes writing software a simpler experience (with a smaller chance of corrupting your data etc) which is generally a nice property to have when you're a small company that's moving fast and breaking things.
  - nemothekid 12y ago
    There are ton of IoT/data analytics startups with data scale in the terabytes that don't have enough money/resources to hire dedicated systems admin.
- eddd 12y ago
  I currently work with postgres cluster with 16 shards on 4 machines. Overall database size is bigger than a few terabytes. Horizontal scaling with postgres is possible and it works more reliable than mongos. What's more, the exact data in mongo would require a few(dozens) times more disk space - so maybe scaling in mongo is easier, but the way i see it, mongo tries to solve problems that it actually creates.
craigching 12y ago
eknkc has a point. Let me explain our needs and why we chose MongoDB over PostgreSQL (and I have been a huge fan of PGQL since before the MySQL/PGSQL flame wars). Replication is not very easy to setup in PGSQL. Sure, I can do it, but our product is an on-prem product that our users are responsible for configuring and maintaining. Having them configure PGSQL replication would be something they're not willing to do. MongoDB's configuration of replica sets is quite trivial compared to PGSQLs. I bring this up every time someone says "PGSQL > MongoDB", even on PGSQL threads where PGSQL devs participate and they admit it's not quite there yet, but that they're working on it. And all that said, it is getting demonstrably better to configure replication in PGSQL, so it's going to get there some day and I can't wait.
- crdoconnor 12y ago
  >I bring this up every time someone says "PGSQL > MongoDB", even on PGSQL threads where PGSQL devs participate and they admit it's not quite there yet Do you have a link to such a thread where pgsql devs admit its inferiority?
  - craigching 12y ago
    Here's a link to my thread. I don't know that I'd say "inferior" just "harder to configure." Please bear that in mind, that's all I'm saying here. https://news.ycombinator.com/item?id=6693027 https://news.ycombinator.com/item?id=6693027
- eddd 12y ago
  If your users have problems with configuring replication for something give them a tool to do it, i advise some template engine i simple python script. Once you write data to mongo, it is not your data - don't forget it.
ohyesyodo 12y ago
Tell me how to set up a PG cluster where nodes can come and go as they like (for example without the need to reseed databases when they come online) in less than 30 minutes.
- eddd 12y ago
  If you want to scale your system horizontally in 30 mins, you probably don't need to do it. It is fun in tutorials, but when you are dealing with tera(peta)bytes of data, you probably need to be more cautious.
  - ohyesyodo 12y ago
    This is just an incorrect assumption in my opinion. Say you are running on Microsoft Azure for example. There are no uptime SLA unless you run your service on at least two machines, because machines needs to reboot, be upgraded etc. Also, if you take a look at perf when running PG on for example Azure or EC2, you will realize that IO is pretty slow but nodes are cheap. So you want to scale out early. Running stuff on a single machine sounds like a perfect single point of failure to me. The actual size of the data does not affect wheter single point of failures are acceptable in a business. I've seen so many people recommend PGSQL, saying its very simple, but when actually asked about how to set up a simple cluster which fulfills the absolutely basic requirements, then everyone just responds somethin similar to what you wrote. I find it very annoying to be honest.

by eddd 12y ago

remon 12y ago

That almost certainly implies you're not working on anything that requires horizontal scalability. That's a luxury not everyone can afford. Scalability, in almost all cases that actually require it, is the first priority. It's like saying it's more important that the car drives reliably rather than everyone that needs to travel fitting into the car in the first place.

davidw 12y ago
I think there are probably way too many people out there doing scaling before they're doing "getting things right". It's premature optimization that causes a lot of problems. Naturally, there are plenty of people who do need to scale a lot, too, but presumably by the time they need to, they have a good idea of the problem and exactly how best to scale it.
- remon 12y ago
  That may be generally true but those two things are not mutually exclusive. Scalability doesn't come at the expense of getting things right, it just might come at the expense of certain features which applies to most distributed system problems.
  - CHY872 12y ago
    Vertical scaling can take you a surprisingly long way
- craigching 12y ago
  Choosing MongoDB isn't always about scaling immediately. It can be about ease of use and easy to configure replication. See my post here [1] here about our use case. [1] -- https://news.ycombinator.com/item?id=8657458 https://news.ycombinator.com/item?id=8657458
CHY872 12y ago
The point at which you need to scale horizontally is (for most applications) when you hit data scale in terabytes. When this is the case, you probably have enough money to hire someone who knows what they're doing with databases in order to set up a migration properly - you don't really need the ease of use configuration features. SQL databases give data durability and transactions, integrity constraints, joins etc, which makes writing software a simpler experience (with a smaller chance of corrupting your data etc) which is generally a nice property to have when you're a small company that's moving fast and breaking things.
- nemothekid 12y ago
  There are ton of IoT/data analytics startups with data scale in the terabytes that don't have enough money/resources to hire dedicated systems admin.
eddd 12y ago
I currently work with postgres cluster with 16 shards on 4 machines. Overall database size is bigger than a few terabytes. Horizontal scaling with postgres is possible and it works more reliable than mongos. What's more, the exact data in mongo would require a few(dozens) times more disk space - so maybe scaling in mongo is easier, but the way i see it, mongo tries to solve problems that it actually creates.

craigching 12y ago

eknkc has a point. Let me explain our needs and why we chose MongoDB over PostgreSQL (and I have been a huge fan of PGQL since before the MySQL/PGSQL flame wars). Replication is not very easy to setup in PGSQL. Sure, I can do it, but our product is an on-prem product that our users are responsible for configuring and maintaining. Having them configure PGSQL replication would be something they're not willing to do. MongoDB's configuration of replica sets is quite trivial compared to PGSQLs. I bring this up every time someone says "PGSQL > MongoDB", even on PGSQL threads where PGSQL devs participate and they admit it's not quite there yet, but that they're working on it. And all that said, it is getting demonstrably better to configure replication in PGSQL, so it's going to get there some day and I can't wait.

crdoconnor 12y ago
>I bring this up every time someone says "PGSQL > MongoDB", even on PGSQL threads where PGSQL devs participate and they admit it's not quite there yet Do you have a link to such a thread where pgsql devs admit its inferiority?
- craigching 12y ago
  Here's a link to my thread. I don't know that I'd say "inferior" just "harder to configure." Please bear that in mind, that's all I'm saying here. https://news.ycombinator.com/item?id=6693027 https://news.ycombinator.com/item?id=6693027
eddd 12y ago
If your users have problems with configuring replication for something give them a tool to do it, i advise some template engine i simple python script. Once you write data to mongo, it is not your data - don't forget it.

ohyesyodo 12y ago

Tell me how to set up a PG cluster where nodes can come and go as they like (for example without the need to reseed databases when they come online) in less than 30 minutes.

eddd 12y ago
If you want to scale your system horizontally in 30 mins, you probably don't need to do it. It is fun in tutorials, but when you are dealing with tera(peta)bytes of data, you probably need to be more cautious.
- ohyesyodo 12y ago
  This is just an incorrect assumption in my opinion. Say you are running on Microsoft Azure for example. There are no uptime SLA unless you run your service on at least two machines, because machines needs to reboot, be upgraded etc. Also, if you take a look at perf when running PG on for example Azure or EC2, you will realize that IO is pretty slow but nodes are cheap. So you want to scale out early. Running stuff on a single machine sounds like a perfect single point of failure to me. The actual size of the data does not affect wheter single point of failures are acceptable in a business. I've seen so many people recommend PGSQL, saying its very simple, but when actually asked about how to set up a simple cluster which fulfills the absolutely basic requirements, then everyone just responds somethin similar to what you wrote. I find it very annoying to be honest.