6 ms·

Wow, the title of this post is very calm compared to what is actually happening. CloudSQL Postgres is running with a misconfigured OS OOM killer, crashes Postm

by aeyes 5y ago

Wow, the title of this post is very calm compared to what is actually happening. CloudSQL Postgres is running with a misconfigured OS OOM killer, crashes Postmaster randomly even if memory use is below instance spec. GCP closes this bug report as "Won't fix". This is a priority 1 issue. Seeing a wontfix for this has completely destroyed my trust of their judgement. The bug report states that they have been in contact with support since February. Unbelievable attitude towards fixing production critical problems of their platform affecting all customers.

breakingcups 5y ago
It exposes a very problematic communication pattern. The engineering team doesn't respond to the support team (accidentally or deliberately). The support team then just decides to close the issue instead of prodding the engineering team for an actual response (even if it's just "Yeah, we're not fixing it"). Now the issue is just in limbo and the only one who feels the pain is the customer.
- perlgeek 5y ago
  Another "fun" interaction pattern: User reports a bug (or a feature request), several others subscribe to and/or vote for this to be solved, and then a service rep closes the issue because there wasn't any recent activity. I've observed with with Atlassian where I wanted to report a Jira bug, but found that it had already been opened some years before, more than a hundred people had subscribed, bug was still closed as "no activity, must not be relevant". I just found the exact same bug reported for Jira Cloud (I had observed it in the on-prem version): https://jira.atlassian.com/browse/JSWCLOUD-8865 https://jira.atlassian.com/browse/JSWCLOUD-8865 and it was closed there for the very same reason. I didn't leave a comment because the original report described the issue perfectly, and adding a "me too" comment is just noise in the bug tracker. Guess I'll be noise in future :-(
  - jaredsohn 5y ago
    I don't think a 'me too' on this bug would be noise right now. Last activity was in 2019; having someone state the problem is still real in 2021 could impact if it gets fixed.
    - breakingcups 5y ago
      Not for Atlassian. I'm subscribed to several serious issues, ranging from usability and compatibility to causing actual data loss since about 2015. Each week I'll get "me too"s and "+1"s in my mailbox, but nothing from Atlassian. Thankfully, they've deprecated their on-premise products, leading us to finally find a better alternative.
slimsag 5y ago
This doesn't really surprise me. We use CloudSQL at my work (Sourcegraph) and have run into all sorts of weird issues actually putting it into production, e.g. segmentation faults when turning on Query Insights (which, lol, is supposed to give insight into why your DB might be behaving poorly.) For the most part it works okay and is fine, but there have definitely been a fair number of quirks.. https://issuetracker.google.com/u/2/savedsearches/559773?pli=1&q=(componentid:187202%2B)%20OR%20(componentid:187065%2B%20customfield82940:%22Cloud%20SQL%22)%20segmentation%20fault https://issuetracker.google.com/u/2/savedsearches/559773?pli...
yashap 5y ago
So many GCP products are surprisingly terrible. Certainly not all of them, some are really good, like GKE, Cloud Storage and Cloud Load Balancer. But Cloud SQL is pretty weak, and products like Cloud Logging, Cloud Metrics and Cloud Tracing are legitimately terrible. Cloud NAT is pretty sketchy too, and can lead to a lot of egress issues if not configured perfectly. My current workplace uses GCP, my last workplace used AWS, and personally I’ve found AWS to have much higher average quality. At my current workplace we’ve stopped using Cloud SQL, and moved our Postgres usage to Aiven (with VPC peering). Aiven seem to do a much better job operating Postgres than GCP do.
- RNCTX 5y ago
  Not really surprising if you consider their likely motivations. Google isn't in the business of selling things to end users, they're in the business of selling ads. The only thing GCP gives them (outside of getting wall streeters off their backs a few years ago when everyone and their brother was starting a cloud service) is a credit to their own infrastructure cost by selling excess to random joes. Therefore I'm not surprised that AWS continues to be the defacto, they do sell things to end users. I'm not surprised that Azure is growing quickly, either, since MS also sells things to end users and they needed a way to transition their on-premise stuff to the wires.
  - yashap 5y ago
    I mean, GCP is a decent source of revenue for them. i.e. last quarter: * Alphabet did ~$55 billion in revenue overall last quarter, ~$4 billion was from "Cloud", which is GCP + Workspace (I don't think they disclose how much is GCP alone?). Although, for now it's a money loser for them, they had operating losses of ~$1 billion for Cloud, but the operating losses are shrinking over time, it'll become profitable eventually * In contrast, Amazon did ~$108 billion in revenue overall last quarter, and ~$13.5 billion was from AWS. Although unlike GCP, AWS is highly profitable, ~$4 billion in operating income for the quarter, which is almost half of Amazon's total operating income But AWS isn't THAT much higher a percentage of Amazon's revenue than GCP is of Alphabet's revenue. And in terms of COSTS, AWS is actually spending less, relative to their overall revenue (Amazon spending ~$9.5 billion of $108 billion total revenue on AWS, Google spending ~$5 billion of $55 billion total revenue on "Cloud"). AWS has been around longer than GCP, and they've certainly spent more absolute dollars, so it makes sense it's further ahead and more polished. Yeah, AWS is more used to selling things to end users than Google, they may have a better culture for quality there, but Google invests heavily in GCP, and it's a pretty significant revenue stream for them. I'm guessing their motivations are similar, both see Cloud offerings as a big revenue stream first and foremost.
arcticfox 5y ago
I migrated off CloudSQL even when they tried to pay me to use it (startup credits). It's not worth risking your business with GCP. Sad, but that's what I've learned... I'd consider Aiven if I were still on GCP and looking for a solid managed Postgres provider. As it is, I'm now on DigitalOcean and fairly happy with their managed Postgres offering, but there are a few rough edges so I'm actually still looking at Aiven even though everything else I have is on DO...
Sytten 5y ago
I concur with the other comments, Cloud SQL is a very mediocre service at best. Lot's of weird issues and the engineering team doesn't seem to care. We also had the segfault due to query insight. Just the fact that you can't upgrade your database version without creating a new instance and restoring a backup is just bad. I also suggest aiven as an alternative that works very well and cost is reasonable.
- ris 5y ago
  > I also suggest aiven as an alternative that works very well and cost is reasonable. Seconded. Responsive support too.

by aeyes 5y ago

breakingcups 5y ago

It exposes a very problematic communication pattern. The engineering team doesn't respond to the support team (accidentally or deliberately). The support team then just decides to close the issue instead of prodding the engineering team for an actual response (even if it's just "Yeah, we're not fixing it"). Now the issue is just in limbo and the only one who feels the pain is the customer.

perlgeek 5y ago
Another "fun" interaction pattern: User reports a bug (or a feature request), several others subscribe to and/or vote for this to be solved, and then a service rep closes the issue because there wasn't any recent activity. I've observed with with Atlassian where I wanted to report a Jira bug, but found that it had already been opened some years before, more than a hundred people had subscribed, bug was still closed as "no activity, must not be relevant". I just found the exact same bug reported for Jira Cloud (I had observed it in the on-prem version): https://jira.atlassian.com/browse/JSWCLOUD-8865 https://jira.atlassian.com/browse/JSWCLOUD-8865 and it was closed there for the very same reason. I didn't leave a comment because the original report described the issue perfectly, and adding a "me too" comment is just noise in the bug tracker. Guess I'll be noise in future :-(
- jaredsohn 5y ago
  I don't think a 'me too' on this bug would be noise right now. Last activity was in 2019; having someone state the problem is still real in 2021 could impact if it gets fixed.
  - breakingcups 5y ago
    Not for Atlassian. I'm subscribed to several serious issues, ranging from usability and compatibility to causing actual data loss since about 2015. Each week I'll get "me too"s and "+1"s in my mailbox, but nothing from Atlassian. Thankfully, they've deprecated their on-premise products, leading us to finally find a better alternative.

slimsag 5y ago

This doesn't really surprise me. We use CloudSQL at my work (Sourcegraph) and have run into all sorts of weird issues actually putting it into production, e.g. segmentation faults when turning on Query Insights (which, lol, is supposed to give insight into why your DB might be behaving poorly.) For the most part it works okay and is fine, but there have definitely been a fair number of quirks.. https://issuetracker.google.com/u/2/savedsearches/559773?pli=1&q=(componentid:187202%2B)%20OR%20(componentid:187065%2B%20customfield82940:%22Cloud%20SQL%22)%20segmentation%20fault https://issuetracker.google.com/u/2/savedsearches/559773?pli...

yashap 5y ago

So many GCP products are surprisingly terrible. Certainly not all of them, some are really good, like GKE, Cloud Storage and Cloud Load Balancer. But Cloud SQL is pretty weak, and products like Cloud Logging, Cloud Metrics and Cloud Tracing are legitimately terrible. Cloud NAT is pretty sketchy too, and can lead to a lot of egress issues if not configured perfectly. My current workplace uses GCP, my last workplace used AWS, and personally I’ve found AWS to have much higher average quality. At my current workplace we’ve stopped using Cloud SQL, and moved our Postgres usage to Aiven (with VPC peering). Aiven seem to do a much better job operating Postgres than GCP do.

RNCTX 5y ago
Not really surprising if you consider their likely motivations. Google isn't in the business of selling things to end users, they're in the business of selling ads. The only thing GCP gives them (outside of getting wall streeters off their backs a few years ago when everyone and their brother was starting a cloud service) is a credit to their own infrastructure cost by selling excess to random joes. Therefore I'm not surprised that AWS continues to be the defacto, they do sell things to end users. I'm not surprised that Azure is growing quickly, either, since MS also sells things to end users and they needed a way to transition their on-premise stuff to the wires.
- yashap 5y ago
  I mean, GCP is a decent source of revenue for them. i.e. last quarter: * Alphabet did ~$55 billion in revenue overall last quarter, ~$4 billion was from "Cloud", which is GCP + Workspace (I don't think they disclose how much is GCP alone?). Although, for now it's a money loser for them, they had operating losses of ~$1 billion for Cloud, but the operating losses are shrinking over time, it'll become profitable eventually * In contrast, Amazon did ~$108 billion in revenue overall last quarter, and ~$13.5 billion was from AWS. Although unlike GCP, AWS is highly profitable, ~$4 billion in operating income for the quarter, which is almost half of Amazon's total operating income But AWS isn't THAT much higher a percentage of Amazon's revenue than GCP is of Alphabet's revenue. And in terms of COSTS, AWS is actually spending less, relative to their overall revenue (Amazon spending ~$9.5 billion of $108 billion total revenue on AWS, Google spending ~$5 billion of $55 billion total revenue on "Cloud"). AWS has been around longer than GCP, and they've certainly spent more absolute dollars, so it makes sense it's further ahead and more polished. Yeah, AWS is more used to selling things to end users than Google, they may have a better culture for quality there, but Google invests heavily in GCP, and it's a pretty significant revenue stream for them. I'm guessing their motivations are similar, both see Cloud offerings as a big revenue stream first and foremost.

arcticfox 5y ago

I migrated off CloudSQL even when they tried to pay me to use it (startup credits). It's not worth risking your business with GCP. Sad, but that's what I've learned... I'd consider Aiven if I were still on GCP and looking for a solid managed Postgres provider. As it is, I'm now on DigitalOcean and fairly happy with their managed Postgres offering, but there are a few rough edges so I'm actually still looking at Aiven even though everything else I have is on DO...

Sytten 5y ago

I concur with the other comments, Cloud SQL is a very mediocre service at best. Lot's of weird issues and the engineering team doesn't seem to care. We also had the segfault due to query insight. Just the fact that you can't upgrade your database version without creating a new instance and restoring a backup is just bad. I also suggest aiven as an alternative that works very well and cost is reasonable.

ris 5y ago
> I also suggest aiven as an alternative that works very well and cost is reasonable. Seconded. Responsive support too.