9 ms·

Google Cloud SQL now supports PostgreSQL 13

171 points by k-rus 6y ago

merb 6y ago
still no wal_level = logical, so sadly.
- ithkuil 6y ago
  TIL, thanks!
- comboy 6y ago
  May I ask why do you care about that setting? edit: I thought they support "replica" but not "logical" thus the question
  - jpgvm 6y ago
    It's required for logical replication which is an important feature for downtimeless major version upgrades and more interesting replication topologies, change data capture, etc.
  - craigkerstiens 6y ago
    Without logical replication you're effectively locked in the same way people talk about Oracle lock-in. If you have any sizable amount of data you'd have to do a dump/restore to get it out which would be a change of data size operation. For 100 GB of data you're looking at 1-3 hrs depending, for 1 TB you'd be looking at a day probably. Logical replication allows you to create replication slots and send it to other places. It does allow for more interesting things as well. One really common use case (not even migrating off) is change data capture out of Postgres into Kafka leveraging Debezium. In older versions of Postgres you could use wal2json for this, now more more recent versions with logical replication supported the pgoutput plugin is great here. Upgrades are a thing, but in place upgrades can be fully captured without logical replication. The short is though they are limiting a lot of what Postgres can do by not supporting it.
    - gunnarmorling 6y ago
      It's unclear to me why they don't support this. RDS has had it since years, Azure Postgres added support for Debezium recently (we helped a bit with that), Heroku has it, but Postgres on GCP is missing support for logical replication and thus Debezium. I'm subscribed to the feature request (https://issuetracker.google.com/issues/70756171 https://issuetracker.google.com/issues/70756171), it's upvoted by users weekly, if not more often. Disclaimer: I'm the lead of Debezium
  - merb 6y ago
    mostly because it' makes a lot of stuff easier. like database upgrades (no downtime), especially when google gives zero fucks in implementing pg_upgrade support. also everything that @craigkerstiens said. (shamlessly point to his text)
  - edoceo 6y ago
    My favourite feature of logical is to push some data from master to partial replicas for closer geo-proximty on read queries.
- thinkingemote 6y ago
  +1 it's a show stopper for me with a 10tb db....
stevencorona 6y ago
Can you upgrade from a previous version without a full export and import?
- k-rus 6y ago
  Looking to the documentation it seems that a full export and import are needed. The docs still are focused on 12.
  - NeckBeardPrince 6y ago
    I wish my org used AWS more and more I have to deal with GCP. With AWS this would be a simple click and Apply.
sz4kerto 6y ago
Please Google, implement binary replication from an external primary. There's no way we can move to managed SQL because it'd take weeks to import the SQL dump.
- rafaelturk 6y ago
  IMO, event this is ever supported, you still should migrate using a Dump, or proper synchronization service
  - ithkuil 6y ago
    Why?
  - jdc 6y ago
    Why is that?
  - craigkerstiens 6y ago
    Whether you should use replication for upgrade is a bit more of a loaded question. But dump/restore should definitely NOT be the mechanism for upgrades. Postgres has support for in place upgrades with pg_upgrade for some time now. This is the mechanism used on RDS, and the same mechanism we'll use on Crunchy Bridge, and what Heroku Postgres uses as well. I'm not sure that Azure supports in place upgrade yet, and GCP does not per their docs. A dump and restore is simply not viable for a database of any size. 100GB datababase which is not at all in the "large" territory would be down for at least an hour if not longer. Pg_upgrade is generally the right shape of thing for this problem. One could debate whether replication is a better approach for even reducing that down time. (Pg_upgrade is on the order of minutes, it is not a size of data operation but rather a size of catalog operation). But that dump/restore is acceptable and the best option isn't really the case these days.
    - gkop 6y ago
      Did parent ninja edit? Currently, it doesn’t mention upgrading, so your response looks like a non sequitur.
erickj 6y ago
Only this week did I receive the final Google Play Music notification that the service is permanently disabled. Just before that Google failed to properly implement the payment settings to continue my Youtube Premium subscription at my previous price in the bundled GPM monthly packaging that I received. Thus, I lost access to both GPM and YT premium in single day. I will never purchase another Google product.
fizixer 6y ago
Can't you spin up your own software in a docker instance on the cloud? what does official support mean here? New to cloud.
- kevincox 6y ago
  This is talking about the managed database service. There is nothing stopping you from renting a VM and running whatever you want.
- SpicyLemonZest 6y ago
  You can. This is for Google's fully managed SQL service, "fully managed" meaning that you don't have to directly manage Docker instances or VMs yourself.
  - AndrewDucker 6y ago
    But you do have to specify CPU and memory and pay for those for having the database running - even if you're not calling it. What I'd like is to pay for what I use when calls are being made. Providing an SQL interface as a service, rather than actually running a whole personally copy of a database.
    - dewey 6y ago
      Maybe that's more of a BigQuery use case then where you pay for usage?
      - hendiatris 6y ago
        Unfortunately it’s not a relational database, so no constraints or any of the other features.
    - phonon 6y ago
      Something like https://aws.amazon.com/blogs/aws/amazon-aurora-postgresql-serverless-now-generally-available/ https://aws.amazon.com/blogs/aws/amazon-aurora-postgresql-se... ?
hellcow 6y ago
I'm already evaluating my company's move off GCP. We were using their managed MySQL instance. Without any communication, they pushed a silent update which broke our year-long security system that updated MySQL _priv tables in line with specific permissions and roles. There is no workaround. When I reported the bug, it was labeled "Working as intended," again with no communication from Google as to why this breaking change was made, or even confirmation that it was something that Google changed at all. We spent days trying to figure out why we could reproduce it in some databases but not others, including diving into the MySQL source code. I don't have experience with their Postgres product, but pushing breaking changes to a database with no notice or communication (either before or after doing it) simply isn't acceptable.
- jey 6y ago
  It seems like that usage isn't supported by upstream MySQL either. Quoting from https://dev.mysql.com/doc/refman/8.0/en/grant-tables.html https://dev.mysql.com/doc/refman/8.0/en/grant-tables.html: > Direct modification of grant tables using statements such as INSERT, UPDATE, or DELETE is discouraged and done at your own risk. The server is free to ignore rows that become malformed as a result of such modifications.
  - hellcow 6y ago
    Discouraged, but not broken--except in GCP after a silent update. GCP now blocks changes from being made to those tables entirely, breaking with MySQL's behavior, as well as GCP's own earlier behavior. We have battle-tested tooling that makes this work brilliantly in all other environments. Updating the _priv tables directly is the only way to modify privileges within a transaction in MySQL.
    - ericpauley 6y ago
      > Updating the _priv tables directly is the only way to modify privileges within a transaction in MySQL. Out of genuine curiosity, why do you want to do that? Is the transactional consistency of the applied permissions critical?
ccleve 6y ago
Meanwhile, AWS Aurora is still on Postgres 11. Postgres 12 is more than a year old. I've been using Aurora because it's managed, I don't have to worry about backups, it's faster and cheaper and easier to scale, etc. But Postgres 12 has some really important features and performance improvements, and we really need them for my app. I've got to wonder why I'm paying for a managed service when they can't even manage to do a major version upgrade in a year.
- ropiku 6y ago
  There's a big difference between AWS Aurora and AWS RDS PG. Both are managed services. Amazon supports PG 12 and 13 is in beta. Aurora is Amazon's fork of Postgres which has different storage and replication so will always be behind.
  - slifin 6y ago
    A year doesn't sound too bad I think MySQL 8 has been generally available since april 2018 from Oracle Honestly I don't know if they're even working on MySQL 8 for Aurora MySQL
- NeckBeardPrince 6y ago
  That isn't an apples to apples comparison.
- redis_mlc 6y ago
  > Postgres 12 is more than a year old. This is not a reasonable expectation for managed database software. For example, I wouldn't put production data on anything younger than 6 months. If you need bleeding edge versions, you should manage it yourself. Source: DBA.
- anarazel 6y ago
  That's the price of a heavily patched postgres fork. It's really expensive to maintain them. Far from the first time such forks fell behind quickly (e.g. redshift, greenplum).
- tekno45 6y ago
hn_throwaway_99 6y ago
I'm really glad to see this. GCP was really slow to add point-in-time-recovery support to Postgres, it went live just a couple months ago, so I'm really impressed to see Postgres 13 support just a month and a half after it was released.
LaserToy 6y ago
Google Cloud Sql is a shit product. Ask me why.
- LaserToy 6y ago
  Wow, is it google team that downvoting? Ok, here are some: if you resize your instance you will get a downtime. There is no support for CDC. You pay for multi master but failover is painfully slow If you launch more than one using terraform all except one will fail and will stuck in an unrecoverable state They broke something last week so our proxies lost connectivity
  - tpetry 6y ago
    The author is trolling with a bold statement (and attack on someone) without _any_ stated reasons. This is something which is not tolerated here.
    - LaserToy 6y ago
      I’m the author, I’m not trolling and I gave examples. CloudSql is a misleading name and a bad product.
      - jsmeaton 6y ago
        It’s the “ask me why” that got you the downvotes. If you had of just stated your reasons without waiting for engagement you probably would have been upvoted.
      - simonebrunozzi 6y ago
        You gave examples after being downvoted. Your first comment was not in line with HN guidelines. Your second comment is much better because it gives reasons. I would have downvoted your first (and I do not work for GCP, mind you); I am now inclined not to just because you posted the reasons in your second comment.
molf 6y ago
I love Cloud SQL for PostgreSQL. But upgrading still is a giant pain. I sincerely wish there was a way to schedule an automatic upgrade to newer major versions.
ngrilly 6y ago
That's great! But when can we have zero downtime maintenance?
say_it_as_it_is 6y ago
If you're going to pay for someone to manage Postgres for you, why not pay an official supporter/contributor to the project? I don't see Google investing in features nor community. Are you simply reaching for whatever seems more convenient?
- jasonvorhe 6y ago
  Yes. Of course. What other reason would there be? If you're on GCP already, of course you'd opt to use their managed Postgres service, otherwhise you'd have to worry about egress traffic, transport encryption, IAM, etc by yourself (or pay someone else do it for you, of course, making it more difficult to calculate your infra costs, having a 2nd support contract, etc) without much benefit. How do you know that Google isn't supporting Postgres in any way, e.g. by supplying upstream patches, etc? The same goes for AWS, Azure, Heroku.
  - DoctorOW 6y ago
    >How do you know that Google isn't supporting Postgres in any way, e.g. by supplying upstream patches, etc? The same goes for AWS, Azure, Heroku. There's a list of contributors and what organization they're from here <https://www.postgresql.org/community/contributors/ https://www.postgresql.org/community/contributors/>. You won't see "Azure" listed but that's because it's considered part of Microsoft. Of the ones you listed, Azure is the only one that is considered a major contributor though you'll see there are companies that specialize in managed Postgres specifically.
- Can_Not 6y ago
  Where can I get a small pg instance (competitive to GCP'S ~$7/month offering) that is ran by an official supporter/contributor?

171 points by k-rus 6y ago

merb 6y ago

still no wal_level = logical, so sadly.

ithkuil 6y ago
TIL, thanks!
comboy 6y ago
May I ask why do you care about that setting? edit: I thought they support "replica" but not "logical" thus the question
- jpgvm 6y ago
  It's required for logical replication which is an important feature for downtimeless major version upgrades and more interesting replication topologies, change data capture, etc.
- craigkerstiens 6y ago
  Without logical replication you're effectively locked in the same way people talk about Oracle lock-in. If you have any sizable amount of data you'd have to do a dump/restore to get it out which would be a change of data size operation. For 100 GB of data you're looking at 1-3 hrs depending, for 1 TB you'd be looking at a day probably. Logical replication allows you to create replication slots and send it to other places. It does allow for more interesting things as well. One really common use case (not even migrating off) is change data capture out of Postgres into Kafka leveraging Debezium. In older versions of Postgres you could use wal2json for this, now more more recent versions with logical replication supported the pgoutput plugin is great here. Upgrades are a thing, but in place upgrades can be fully captured without logical replication. The short is though they are limiting a lot of what Postgres can do by not supporting it.
  - gunnarmorling 6y ago
    It's unclear to me why they don't support this. RDS has had it since years, Azure Postgres added support for Debezium recently (we helped a bit with that), Heroku has it, but Postgres on GCP is missing support for logical replication and thus Debezium. I'm subscribed to the feature request (https://issuetracker.google.com/issues/70756171 https://issuetracker.google.com/issues/70756171), it's upvoted by users weekly, if not more often. Disclaimer: I'm the lead of Debezium
- merb 6y ago
  mostly because it' makes a lot of stuff easier. like database upgrades (no downtime), especially when google gives zero fucks in implementing pg_upgrade support. also everything that @craigkerstiens said. (shamlessly point to his text)
- edoceo 6y ago
  My favourite feature of logical is to push some data from master to partial replicas for closer geo-proximty on read queries.
thinkingemote 6y ago
+1 it's a show stopper for me with a 10tb db....

stevencorona 6y ago

Can you upgrade from a previous version without a full export and import?

k-rus 6y ago
Looking to the documentation it seems that a full export and import are needed. The docs still are focused on 12.
- NeckBeardPrince 6y ago
  I wish my org used AWS more and more I have to deal with GCP. With AWS this would be a simple click and Apply.

sz4kerto 6y ago

Please Google, implement binary replication from an external primary. There's no way we can move to managed SQL because it'd take weeks to import the SQL dump.

rafaelturk 6y ago
IMO, event this is ever supported, you still should migrate using a Dump, or proper synchronization service
- ithkuil 6y ago
  Why?
- jdc 6y ago
  Why is that?
- craigkerstiens 6y ago
  Whether you should use replication for upgrade is a bit more of a loaded question. But dump/restore should definitely NOT be the mechanism for upgrades. Postgres has support for in place upgrades with pg_upgrade for some time now. This is the mechanism used on RDS, and the same mechanism we'll use on Crunchy Bridge, and what Heroku Postgres uses as well. I'm not sure that Azure supports in place upgrade yet, and GCP does not per their docs. A dump and restore is simply not viable for a database of any size. 100GB datababase which is not at all in the "large" territory would be down for at least an hour if not longer. Pg_upgrade is generally the right shape of thing for this problem. One could debate whether replication is a better approach for even reducing that down time. (Pg_upgrade is on the order of minutes, it is not a size of data operation but rather a size of catalog operation). But that dump/restore is acceptable and the best option isn't really the case these days.
  - gkop 6y ago
    Did parent ninja edit? Currently, it doesn’t mention upgrading, so your response looks like a non sequitur.

erickj 6y ago

Only this week did I receive the final Google Play Music notification that the service is permanently disabled. Just before that Google failed to properly implement the payment settings to continue my Youtube Premium subscription at my previous price in the bundled GPM monthly packaging that I received. Thus, I lost access to both GPM and YT premium in single day. I will never purchase another Google product.

fizixer 6y ago

Can't you spin up your own software in a docker instance on the cloud? what does official support mean here? New to cloud.

kevincox 6y ago
This is talking about the managed database service. There is nothing stopping you from renting a VM and running whatever you want.
SpicyLemonZest 6y ago
You can. This is for Google's fully managed SQL service, "fully managed" meaning that you don't have to directly manage Docker instances or VMs yourself.
- AndrewDucker 6y ago
  But you do have to specify CPU and memory and pay for those for having the database running - even if you're not calling it. What I'd like is to pay for what I use when calls are being made. Providing an SQL interface as a service, rather than actually running a whole personally copy of a database.
  - dewey 6y ago
    Maybe that's more of a BigQuery use case then where you pay for usage?
    - hendiatris 6y ago
      Unfortunately it’s not a relational database, so no constraints or any of the other features.
  - phonon 6y ago
    Something like https://aws.amazon.com/blogs/aws/amazon-aurora-postgresql-serverless-now-generally-available/ https://aws.amazon.com/blogs/aws/amazon-aurora-postgresql-se... ?

hellcow 6y ago

I'm already evaluating my company's move off GCP. We were using their managed MySQL instance. Without any communication, they pushed a silent update which broke our year-long security system that updated MySQL _priv tables in line with specific permissions and roles. There is no workaround. When I reported the bug, it was labeled "Working as intended," again with no communication from Google as to why this breaking change was made, or even confirmation that it was something that Google changed at all. We spent days trying to figure out why we could reproduce it in some databases but not others, including diving into the MySQL source code. I don't have experience with their Postgres product, but pushing breaking changes to a database with no notice or communication (either before or after doing it) simply isn't acceptable.

jey 6y ago
It seems like that usage isn't supported by upstream MySQL either. Quoting from https://dev.mysql.com/doc/refman/8.0/en/grant-tables.html https://dev.mysql.com/doc/refman/8.0/en/grant-tables.html: > Direct modification of grant tables using statements such as INSERT, UPDATE, or DELETE is discouraged and done at your own risk. The server is free to ignore rows that become malformed as a result of such modifications.
- hellcow 6y ago
  Discouraged, but not broken--except in GCP after a silent update. GCP now blocks changes from being made to those tables entirely, breaking with MySQL's behavior, as well as GCP's own earlier behavior. We have battle-tested tooling that makes this work brilliantly in all other environments. Updating the _priv tables directly is the only way to modify privileges within a transaction in MySQL.
  - ericpauley 6y ago
    > Updating the _priv tables directly is the only way to modify privileges within a transaction in MySQL. Out of genuine curiosity, why do you want to do that? Is the transactional consistency of the applied permissions critical?

ccleve 6y ago

Meanwhile, AWS Aurora is still on Postgres 11. Postgres 12 is more than a year old. I've been using Aurora because it's managed, I don't have to worry about backups, it's faster and cheaper and easier to scale, etc. But Postgres 12 has some really important features and performance improvements, and we really need them for my app. I've got to wonder why I'm paying for a managed service when they can't even manage to do a major version upgrade in a year.

ropiku 6y ago
There's a big difference between AWS Aurora and AWS RDS PG. Both are managed services. Amazon supports PG 12 and 13 is in beta. Aurora is Amazon's fork of Postgres which has different storage and replication so will always be behind.
- slifin 6y ago
  A year doesn't sound too bad I think MySQL 8 has been generally available since april 2018 from Oracle Honestly I don't know if they're even working on MySQL 8 for Aurora MySQL
NeckBeardPrince 6y ago
That isn't an apples to apples comparison.
redis_mlc 6y ago
> Postgres 12 is more than a year old. This is not a reasonable expectation for managed database software. For example, I wouldn't put production data on anything younger than 6 months. If you need bleeding edge versions, you should manage it yourself. Source: DBA.
anarazel 6y ago
That's the price of a heavily patched postgres fork. It's really expensive to maintain them. Far from the first time such forks fell behind quickly (e.g. redshift, greenplum).
tekno45 6y ago

hn_throwaway_99 6y ago

I'm really glad to see this. GCP was really slow to add point-in-time-recovery support to Postgres, it went live just a couple months ago, so I'm really impressed to see Postgres 13 support just a month and a half after it was released.

LaserToy 6y ago

Google Cloud Sql is a shit product. Ask me why.

LaserToy 6y ago
Wow, is it google team that downvoting? Ok, here are some: if you resize your instance you will get a downtime. There is no support for CDC. You pay for multi master but failover is painfully slow If you launch more than one using terraform all except one will fail and will stuck in an unrecoverable state They broke something last week so our proxies lost connectivity
- tpetry 6y ago
  The author is trolling with a bold statement (and attack on someone) without _any_ stated reasons. This is something which is not tolerated here.
  - LaserToy 6y ago
    I’m the author, I’m not trolling and I gave examples. CloudSql is a misleading name and a bad product.
    - jsmeaton 6y ago
      It’s the “ask me why” that got you the downvotes. If you had of just stated your reasons without waiting for engagement you probably would have been upvoted.
    - simonebrunozzi 6y ago
      You gave examples after being downvoted. Your first comment was not in line with HN guidelines. Your second comment is much better because it gives reasons. I would have downvoted your first (and I do not work for GCP, mind you); I am now inclined not to just because you posted the reasons in your second comment.

molf 6y ago

I love Cloud SQL for PostgreSQL. But upgrading still is a giant pain. I sincerely wish there was a way to schedule an automatic upgrade to newer major versions.

ngrilly 6y ago

That's great! But when can we have zero downtime maintenance?

say_it_as_it_is 6y ago

If you're going to pay for someone to manage Postgres for you, why not pay an official supporter/contributor to the project? I don't see Google investing in features nor community. Are you simply reaching for whatever seems more convenient?

jasonvorhe 6y ago
Yes. Of course. What other reason would there be? If you're on GCP already, of course you'd opt to use their managed Postgres service, otherwhise you'd have to worry about egress traffic, transport encryption, IAM, etc by yourself (or pay someone else do it for you, of course, making it more difficult to calculate your infra costs, having a 2nd support contract, etc) without much benefit. How do you know that Google isn't supporting Postgres in any way, e.g. by supplying upstream patches, etc? The same goes for AWS, Azure, Heroku.
- DoctorOW 6y ago
  >How do you know that Google isn't supporting Postgres in any way, e.g. by supplying upstream patches, etc? The same goes for AWS, Azure, Heroku. There's a list of contributors and what organization they're from here <https://www.postgresql.org/community/contributors/ https://www.postgresql.org/community/contributors/>. You won't see "Azure" listed but that's because it's considered part of Microsoft. Of the ones you listed, Azure is the only one that is considered a major contributor though you'll see there are companies that specialize in managed Postgres specifically.
Can_Not 6y ago
Where can I get a small pg instance (competitive to GCP'S ~$7/month offering) that is ran by an official supporter/contributor?