6 ms·

Indeed — it's great to see a vendor (team, in this case) that doesn't try to downplay a Jepsen result, and instead fixes the issues. However, there is one more

by jwr 6y ago

Indeed — it's great to see a vendor (team, in this case) that doesn't try to downplay a Jepsen result, and instead fixes the issues. However, there is one more takeaway here. I've heard too many times "just use Postgres", repeated as an unthinking mantra. But there are no obvious solutions in the complex world of databases. And this isn't even a multi-node scenario!

Scarbutt 6y ago
"Just use Postgres" may have become meme but for good reason and is well grounded IMO. Many immature databases with not much wide use are better avoided though, we manage to break datomic three times during development, the first two bugs were fixed in a week, the third took a month, which they called in their changelog "Fix: Prevent a rare scenario where retracting non-existent entities could prevent future transactions from succeeding" so yeah, we went back to "just use postgres", who wants to go through the nightmare of hitting those bugs in production and who knows how many more?scary situation.
pnathan 6y ago
"Use PostGres until you have an engineering - data driven rationale not to" is my standard answer for non-blob data storage when a project starts. Why? because when `n` is small (tables, rows, connections), postgres works well enough, and if `n` should ever become large, we'll have interest in funding the work to do a migration, if that's appropriate - and we'll be able to evaluate the system at scale with real data.
- jmalicki 6y ago
  postgres works great for `n` in ranges in excess of (10k, 10B, 1k) at least, which not many would consider small. The only real justification I've heard is postgres' story on replication still lags mysql, and the write amplification bit which they've done a lot of work on for pg13 (see [0]) Even for many very large companies I've seen, noSQL databases are mostly used as caches that can afford to be delayed or a bit out of sync, not as the database of record. [0] https://eng.uber.com/postgres-to-mysql-migration/ https://eng.uber.com/postgres-to-mysql-migration/
  - pnathan 6y ago
    I mean, you can go a loooong way on Postgres. Things that probably are going to start causing issues: - the story of replication - ultra-high volume write contention - high volume timeseries data streams For #3 there, I was consulting on a project a few years ago where there was about 1 gigabyte of timeseries data in Postgres/RDS. Some ad hoc queries were really running into slowness - COTS RDS isn't great at speed. I want to say 1 billion rows, but that seems a off. Anyway, we were talking data sharding, etc. Didn't go anywhere, because the project was canceled, etc. But we could have done a nice job if we wanted to. Point being, there are design limits on postgres, particularly regarding distributed systems. Which is fine. I just have this allergy to designing for scale on Very Small Project. Also worth noting that the anti-monolith dogma plays out here, a few well built monoliths probably are better for a SQL database than "one bajillion microservices".
    - jmalicki 6y ago
      I'm totally with you that those three are reasonable limits (hell, I named two in my rebuttal)... Just that scale in itself is not the issue, it has highly specific issues. And for things like high update time series databases, those in my experience are usually metrics databases that say Prometheus serves well, and pg extensions like timescaledb do a stellar job for when you truly need ACID semantics.
    - jmalicki 6y ago
      Also most webscale use of nosql databases I've seen basically boils down to "eventually consistent materialized views"
arcticfox 6y ago
> there is one more takeaway here I don't think the "just use Postgres" mantra takes any hits at all from this. (If anything, I feel better about it). I've used maybe a dozen (?) databases/stores over the years - graph databases, NoSQL databases, KV stores, the most boring old databases, the sexiest new databases - and my general approach is now to just use Postgres unless it really, really doesn't fit. Works great for me.
- jwr 6y ago
  All the answers to my post are missing the point. I'm happy Postgres works for you. It works for me, too, in a number of setups. But one should never accept advice like "just use Postgres" without thinking and careful consideration. As the Jepsen article above shows.
  - Carpetsmoker 6y ago
    I have rarely seen people give "just use PostgreSQL" as advice, but rather "just use PostgreSQL unless you have a compelling reason not to". There's a pretty big difference between the two.
    - Thaxll 6y ago
      "just use PostgreSQL unless you have a compelling reason not to" is just as a bad advice tbh. At work we did not even consider it because no HA support / availability. What do you do exactly when your master is down, your entire backend just stop working? PG is not a silver bulet especially not when you're looking for good managed / HA solution. There is no such thing as "use x.y.z unless" this statement does not make any sense.
      - speedgoose 6y ago
        The classic high availability setup for postgresql is to have standby instances, usually one, ready to become the new primary if the primary goes down. https://www.postgresql.org/docs/current/high-availability.html https://www.postgresql.org/docs/current/high-availability.ht... I used CouchDB in a project because the high availability was much better on the paper, but I think postgresql would have been fine after all. Good managed postgresql options with high availability are in many public clouds.
jeffdavis 6y ago
"I've heard too many times 'just use Postgres', repeated as an unthinking mantra." You use "unthinking" pejoratively, but being able to skip past some decisions without over-analyzing is really important. If you are an 8-person startup, you don't have time for 3 of the people to spend weeks discussing and experimenting with alternatives for every decision. Databases are really important, but people make tons of important decisions based on little information. If you have little information other than "I need to pick a database", then Postgres is a pretty good choice -- a sensible default, if you will. Everyone wants to be the default, of course, so you need some criteria to choose one. It could be a big and interesting discussion, but there are relatively few contenders. If it's not free, it's probably out. If it's not SQL, it's probably out. If it's not widely used already, it's probably out. If it's tied to any specific environment/platform (e.g. Hadoop), it's probably out (or there will be a default within that specific platform). By "out", I don't mean that it's unreasonable to choose these options, just that they would not make a sensible default. It would be weird if your default was an expensive, niche, no-SQL option that runs inside of Hadoop. So what's left? Postgres, MySQL, and MariaDB. SQL Server and MongoDB, while not meeting all of the criteria, have a case for being a default choice in some circles, as well. Apparently, out of those options, many on HN have had a good experience with Postgres, so they suggest it as default. But if you have additional information about your needs that leads you to another choice, go for it.

by jwr 6y ago

Scarbutt 6y ago

"Just use Postgres" may have become meme but for good reason and is well grounded IMO. Many immature databases with not much wide use are better avoided though, we manage to break datomic three times during development, the first two bugs were fixed in a week, the third took a month, which they called in their changelog "Fix: Prevent a rare scenario where retracting non-existent entities could prevent future transactions from succeeding" so yeah, we went back to "just use postgres", who wants to go through the nightmare of hitting those bugs in production and who knows how many more?scary situation.

pnathan 6y ago

"Use PostGres until you have an engineering - data driven rationale not to" is my standard answer for non-blob data storage when a project starts. Why? because when `n` is small (tables, rows, connections), postgres works well enough, and if `n` should ever become large, we'll have interest in funding the work to do a migration, if that's appropriate - and we'll be able to evaluate the system at scale with real data.

jmalicki 6y ago
postgres works great for `n` in ranges in excess of (10k, 10B, 1k) at least, which not many would consider small. The only real justification I've heard is postgres' story on replication still lags mysql, and the write amplification bit which they've done a lot of work on for pg13 (see [0]) Even for many very large companies I've seen, noSQL databases are mostly used as caches that can afford to be delayed or a bit out of sync, not as the database of record. [0] https://eng.uber.com/postgres-to-mysql-migration/ https://eng.uber.com/postgres-to-mysql-migration/
- pnathan 6y ago
  I mean, you can go a loooong way on Postgres. Things that probably are going to start causing issues: - the story of replication - ultra-high volume write contention - high volume timeseries data streams For #3 there, I was consulting on a project a few years ago where there was about 1 gigabyte of timeseries data in Postgres/RDS. Some ad hoc queries were really running into slowness - COTS RDS isn't great at speed. I want to say 1 billion rows, but that seems a off. Anyway, we were talking data sharding, etc. Didn't go anywhere, because the project was canceled, etc. But we could have done a nice job if we wanted to. Point being, there are design limits on postgres, particularly regarding distributed systems. Which is fine. I just have this allergy to designing for scale on Very Small Project. Also worth noting that the anti-monolith dogma plays out here, a few well built monoliths probably are better for a SQL database than "one bajillion microservices".
  - jmalicki 6y ago
    I'm totally with you that those three are reasonable limits (hell, I named two in my rebuttal)... Just that scale in itself is not the issue, it has highly specific issues. And for things like high update time series databases, those in my experience are usually metrics databases that say Prometheus serves well, and pg extensions like timescaledb do a stellar job for when you truly need ACID semantics.
  - jmalicki 6y ago
    Also most webscale use of nosql databases I've seen basically boils down to "eventually consistent materialized views"

arcticfox 6y ago

> there is one more takeaway here I don't think the "just use Postgres" mantra takes any hits at all from this. (If anything, I feel better about it). I've used maybe a dozen (?) databases/stores over the years - graph databases, NoSQL databases, KV stores, the most boring old databases, the sexiest new databases - and my general approach is now to just use Postgres unless it really, really doesn't fit. Works great for me.

jwr 6y ago
All the answers to my post are missing the point. I'm happy Postgres works for you. It works for me, too, in a number of setups. But one should never accept advice like "just use Postgres" without thinking and careful consideration. As the Jepsen article above shows.
- Carpetsmoker 6y ago
  I have rarely seen people give "just use PostgreSQL" as advice, but rather "just use PostgreSQL unless you have a compelling reason not to". There's a pretty big difference between the two.
  - Thaxll 6y ago
    "just use PostgreSQL unless you have a compelling reason not to" is just as a bad advice tbh. At work we did not even consider it because no HA support / availability. What do you do exactly when your master is down, your entire backend just stop working? PG is not a silver bulet especially not when you're looking for good managed / HA solution. There is no such thing as "use x.y.z unless" this statement does not make any sense.
    - speedgoose 6y ago
      The classic high availability setup for postgresql is to have standby instances, usually one, ready to become the new primary if the primary goes down. https://www.postgresql.org/docs/current/high-availability.html https://www.postgresql.org/docs/current/high-availability.ht... I used CouchDB in a project because the high availability was much better on the paper, but I think postgresql would have been fine after all. Good managed postgresql options with high availability are in many public clouds.

jeffdavis 6y ago

"I've heard too many times 'just use Postgres', repeated as an unthinking mantra." You use "unthinking" pejoratively, but being able to skip past some decisions without over-analyzing is really important. If you are an 8-person startup, you don't have time for 3 of the people to spend weeks discussing and experimenting with alternatives for every decision. Databases are really important, but people make tons of important decisions based on little information. If you have little information other than "I need to pick a database", then Postgres is a pretty good choice -- a sensible default, if you will. Everyone wants to be the default, of course, so you need some criteria to choose one. It could be a big and interesting discussion, but there are relatively few contenders. If it's not free, it's probably out. If it's not SQL, it's probably out. If it's not widely used already, it's probably out. If it's tied to any specific environment/platform (e.g. Hadoop), it's probably out (or there will be a default within that specific platform). By "out", I don't mean that it's unreasonable to choose these options, just that they would not make a sensible default. It would be weird if your default was an expensive, niche, no-SQL option that runs inside of Hadoop. So what's left? Postgres, MySQL, and MariaDB. SQL Server and MongoDB, while not meeting all of the criteria, have a case for being a default choice in some circles, as well. Apparently, out of those options, many on HN have had a good experience with Postgres, so they suggest it as default. But if you have additional information about your needs that leads you to another choice, go for it.