11 ms·
Cloud SQL for PostgreSQL now generally available
- elvinyung 8y agoPostgres wire-compatible Spanner when? :P
- Crash0v3rid3 8y agoWondering what kind of use case you would have for this compatibility?
- elvinyung 8y agoI know that it's definitely not going to be 100% the same (especially since Spanner doesn't even support SQL DML right now), but I think a drop-in replacement into a managed autoscaling database is a really nice alternative to manual sharding. Right now basically the options are Aurora, Citus, and running CockroachDB yourself.
- renchap 8y agoFinally! Nice to see more PG support. Any news about PostgreSQL 10 support?
- briffle 8y agoI asked that yesterday in a meeting with my gcloud rep, they said all work has gone into getting to GA, and once that is done, look for them to start doing things like updating, more new features, etc.
- deleted 8y ago[deleted]
- lobster_johnson 8y agoHas anyone used the beta and got any feeling for how maintenance downtime impacts things? A bit nervous about how you can only set a "maintenance window" and not be able to plan ahead for disruption; as far as I can tell, they won't even tell you ahead of time. The HA seems really solid (zero-lag "regional disks"), but it's still a bit disconcerting.
- manigandham 8y agoThere is disruption yes. It's usually short however we always see retries in our logs for a few minutes. Our app doesn't need perfect uptime though and we haven't tried the HA setup.
- ciguy 8y agoUnfortunately last time I used CloudSQL for MySQL it was incredibly unstable. They would take down our master AND standby at the same time for maintenance. When we filed a ticket they just said it was a known bug with no plans to fix. A major client of mine migrated to AWS because of this and other issues.
- tinix 8y agoI've been thinking about moving us to Google's Cloud Platform. What I found in regards to maintenance here: https://cloud.google.com/compute/docs/regions-zones/#maintenance https://cloud.google.com/compute/docs/regions-zones/#mainten... states that they do live migrations without any down time. Can anyone elaborate? Is this only for Compute Engine? In that case, if one can run postgres on a Compute Engine instance, why not do that instead? Surely, if one can setup a highly available postgres cluster, Google can do updates without affecting uptime??? To be fair, we wouldn't use GCP for anything but virtual servers and storage replication... I have no desire to tie us to Google's infrastructure any more than necessary. Were your master and standby in the same availability zone? Can't you set diff maintenance windows? WTF? https://cloud.google.com/sql/faq#maintenancerestart https://cloud.google.com/sql/faq#maintenancerestart According to the link above, you can taper your upgrade windows, it looks like.
- manigandham 8y agoWe actually just migrated off yesterday... we went with an independent vendor like Aiven instead of the clouds because they move too slowly and don't have enough features.
- gingerlime 8y agoLike aiven or aiven? :) I’m not aware of too much competition in this space. We’re happy with aiven so far, but they’re quite unknown / under the radar it seems...
- manigandham 8y agoThey're the best we've found as well, but there are lots of managed database providers of all kinds. https://www.compose.com https://www.compose.com is another big one.
- gingerlime 8y agoWe’re using compose for redis, but we might switch it to aiven as well, despite aiven having some missing functionality... For Gcloud and especially in Europe, I think these two are the only options that I’m aware of (at least for PG and redis)
- manigandham 8y agoIf you want managed Redis, there’s nothing better than RedisLabs
- gingerlime 8y agoYeah, I was practically begging them to offer service on GCloud in europe ... :) but they don't cover it...
- flamingcow 8y ago[I'm the Cloud SQL TL] What features would've kept you on Cloud SQL?
- brightball 8y agoWill we be able to use the Citus extension? Been itching to get Citus running on GCP.
- jadbox 8y agoDoes any major cloud provider support it?
- kdv 8y agoMy understanding is people are able to run and manage Citus themselves on GCP (not Cloud SQL), but Citus Cloud (the managed solution) is only available in AWS.
- manigandham 8y agoThey do not support the Citus extension currently and custom extension loading is not allowed. You would have to run your own Citus install using VMs.
- eloff 8y agoI'm looking around for more details on these "regional" disks that replicate between two zones at the block level. Is that just a fancy term for os level mirrored disks using the cloud persistent disks?
- laixer 8y agoThere's a previous blog post[1] about HA replication. It uses block-level replication managed by PD infrastructure. [1] https://cloudplatform.googleblog.com/2017/11/Cloud-SQL-for-PostgreSQL-adds-high-availability-and-replication.html https://cloudplatform.googleblog.com/2017/11/Cloud-SQL-for-P...
- eloff 8y agoI read that, but it doesn't answer the question. What are these regional disks, and can they be used directly?
- lobster_johnson 8y agoA regional disk is a logical disk that is synchronously replicated at the block level across exactly two zones within the same region [1]. Since the disks are always identical, with no replication lag, the HA control plane can seamless fail over the whole database to a new master that plugs into the same disk. It's all in the article. Regional disks aren't publicly available yet, but they are in alpha [2]. Like normal persistent disks, everything is backed by Google's internal Colossus system [3]. [1] https://cloudplatform.googleblog.com/2017/11/Cloud-SQL-for-PostgreSQL-adds-high-availability-and-replication.html https://cloudplatform.googleblog.com/2017/11/Cloud-SQL-for-P... [2] https://cloud.google.com/sdk/gcloud/reference/alpha/compute/disks/create https://cloud.google.com/sdk/gcloud/reference/alpha/compute/... [3] https://cloud.google.com/files/storage_architecture_and_challenges.pdf https://cloud.google.com/files/storage_architecture_and_chal...
- nh2 8y agoBlock device based replication for Postgres seems a bit unconventional given that Postgres has native synchronous replication support with WAL streaming. Intuition tells me that you might get better performance if you let the DB itself do the replication but I can't really justify that without real review of what happens. The postgres docs (https://www.postgresql.org/docs/10/static/different-replication-solutions.html https://www.postgresql.org/docs/10/static/different-replicat...) say that the WAL solution has no "Master server overhead" in contrast to the File System Replication solution, but it's not explained and I'm not sure what is meant by that. I guess with a block device based solution, recovery takes longer, because failover entails you have to actually mount the block device (as no 2 machines can mount it rw at the same time), and then start the DB (or in a more basic implementation, just boot the entire second machine as part of failover), while with WAL streaming both postgres instances would already be running? Wo failover would be faster with WAL streaming? I would be great if somebody from GCP could elaborate what the tradeoffs here are, how long failover takes, and whether we can expect similar performance and behaviour as with WAL shipping.
- TeeWEE 8y agoHow expensive is this compared to, for example AWS RDS?
- michaelanckaert 8y agoThe cost is virtually the same. A db.t2.small instance should compare to a db-pg-g1-small instance. Pricing is around 90$ on AWS vs 93$ on GCP. I’m an AWS consultant so I could have messed up on the GCP instance type.
- theptip 8y agoGreat news! This has been a long time coming (over a year since the beta started), looking forward to migrating my MySQL instances over.
- zawerf 8y agoHow do you deal with custom extensions (for example plv8)? I started off with heroku and they don't support the same subset: https://cloud.google.com/sql/docs/postgres/extensions https://cloud.google.com/sql/docs/postgres/extensions https://devcenter.heroku.com/articles/heroku-postgres-extensions-postgis-full-text-search https://devcenter.heroku.com/articles/heroku-postgres-extens...
- Goopplesoft 8y agohttps://cloud.google.com/sql/docs/postgres/extensions https://cloud.google.com/sql/docs/postgres/extensions Looks like a preset list of extensions. I'd assume custom extensions would be very difficult to support in managed postgres.
- sitkack 8y agoWhy is that? Systems are built for extension, to not allow it deprives one of essential qualities.
- contrahax 8y agoThey don't have a great story around extensions - the ones they do have are unsupported/buggy. For example, PostGIS is missing "ST_FromGeoJSON" because it was compiled with the wrong flag - and has been this way for over a year despite hundreds of user complaints.
- aserafini 8y agoThis is what I expect from Google. They are not customer-centric and no-one is really prioritising or cares about customer feedback.
- numbsafari 8y agoStill working for AMZN?
- 8y ago
- kerng 8y agoGreat to see all three major cloud providers offering this now. Should be good for pricing!
- pritambarhate 8y agoIs point in time recovery available now in GA? I had checked it a couple of weeks ago for the beta service and it was not available. I think for a managed DB hosting point in time recovery is a critical feature.
- flamingcow 8y ago[I'm the Cloud SQL TL] No, it isn't. We agree with you that it's an important feature for managed databases, and we're working to get it right. We decoupled it from this launch to get PostgreSQL to GA faster.
- gleenn 8y agoWhat's a TL?
- gauravphoenix 8y agotech lead?
- brandmeyer 8y ago(I am also a Google employee, totally unrelated to this product). Yes, "TL" is Google jargon for Tech Lead.
- tacticus 8y agoUnrelated to pg but could you badger the spanner team to make a mini spanner product :) Related to postgres. We have many many concurrent connections but a load satisfied by an n1-standard-4 atm do you recommend a connection pooler or something to help us get down to the 100 to 200 connections we need to be at to use cloudsql?
- renchap 8y agoConnection pooling is recommended for any non-trivial PG deployment. I can recommend pgbouncer, worked flawlessly for us.
- kdv 8y agoDoes anyone have insight or experience using this in production? We're currently running PostgreSQL 10 w/ pg_partman on our own hardware but looking at a several options for cloud migration. Unfortunately, Citus Cloud on GCP doesn't appear to be an option (yet?) - Google Cloud SQL (PostgreSQL) - Citus Cloud (AWS Only) - Citus (managed ourselves) on GCP
- alexnewman 8y agocitus isn't simply postgresql. It has many tradeoffs
- kdv 8y agoWe get that, but I left off (for brevity) the reasons why Citus would be a net positive for our use case.
- ivanr 8y agoI would appreciate if you could list some of those differences and tradeoffs, for some of us who are interested in Citus but haven't yet had time to look at it in more detail. Thanks!
- manigandham 8y agoWhat do you mean? It's just an extension to PostgreSQL and runs anywhere that allows you to load it.
- manigandham 8y agoWe did, but now use https://aiven.io https://aiven.io Highly recommended if you want a fast and featured managed db service.
- misterbowfinger 8y agowhat's the benefit of using them vs. AWS or GCP directly?
- foobarbazetc 8y agoDo these use the standard persistent disks (capped at 240mb/s)?
- litewulf 8y ago[I am a Cloud SQL-er] Yes, you can use both standard and SSD persistent disks. If you create a larger instance with more vCPUs and a big enough disk, you can achieve greater than 240mb/s, see the docs: https://cloud.google.com/compute/docs/disks/performance#ssd-pd-performance https://cloud.google.com/compute/docs/disks/performance#ssd-...
- optimusclimb 8y agoStarted down the path of using this when it was in beta, but had to abort when we saw there was no option to connect to it from Python App Engine Standard. Now that it's GA...it looks like that hasn't changed. Is the classic, Python, App Engine standard becoming a second class citizen? Or was there some reason why this wasn't considered GA worthy for Postgres? Trying to understand if going forward Google is trying to push everyone to the flexible environment or not - as I would have really expected connectivity between these two products.
- theptip 8y agoLast I checked it wasn't possible to whitelist internal IPs (e.g. Kubernetes nodes or VM instances) to access Cloud SQL instances at all -- the options are either to use the non-standard cloud SQL proxy sidecar app, or allow connections from all endpoints (public or private). This seems like a major omission, and AWS has had this for ages.
- tarr11 8y agoFrom the docs: https://cloud.google.com/sql/docs/postgres/connect-external-app#appaccessIP https://cloud.google.com/sql/docs/postgres/connect-external-... > You can grant any application access to a Cloud SQL instance by authorizing the IP addresses that the application uses to connect. > You can not specify a private network (for example, 10.x.x.x) as an authorized network. > PostgreSQL instances support only IPv4 addresses. They are automatically configured with a static IP address.
- theptip 8y agoAh, misremembered exactly what the issue was -- you're right, individual endpoints can be whitelisted. Internal networks cannot, which is what I (or anyone else using GKE) would need, since node IPs are ephemeral. I believe the same issue would apply to VM instances that are not pets, (in auto-scaling groups for example), since I'm not aware of being able to auto-assign static IPs there either.
- jordic 8y ago
- cellis 8y agoI tried using this, but couldn't get the root certs to work. So I went to AWS RDS instead. A pity, it's much cheaper ( especially in beta pricing )
- litewulf 8y ago[Cloud SQL person, SSL+connections is my jam] When you say you couldn't get the root certs to work... what do you mean? Cloud SQL automatically generates server certificates, and we offer UI+API for creating additional client certificates. The two should not share a root CA.
- Lunatic666 8y agoLast time I was at Google for a workshop (If you have the chance to visit Google, do it. The food alone is worth it), they didn’t seem to push CloudSQL a lot, because they wanted to guide people more in the spanner direction. Without a solid RDS counterpart however, I don’t think bigger companies will consider moving from AWS. Happy to see they changed their mind and continue to expand their SQL services. The competition by Google put a lot of pressure on AWS who seemed to be gotten a bit lazy. Google was ahead of the game with their global load balancers and network speed and quality. Now AWS countered with their 5th series C5/M5, which solve the bandwidth problem of the smaller C and M instances
- sirn 8y agoSince there seems to be some Googler who work on Cloud SQL here, I wonder is there any chance Cloud SQL will be available in asia-southeast1 soon? It's the only region (I believe) where Cloud SQL isn't available at all, and one of the main reasons we can't fully migrated from AWS to GCP just yet.
- litewulf 8y ago[I am a Cloud SQL-er, and worked on region expansion... among other things] Cloud SQL has only been available in regions with at least three zones (since we believe that is the minimum to make sure we can maintain HA in the event of a single zone failure). asia-southeast1 currently only has two zones, when a third zone is launched, Cloud SQL will become available in that region.
- sirn 8y agoThank you. That makes sense and totally understandable. Looking forward to when there's third zone in asia-southeast1. :-)