8 ms·

I've had the same experience with GCP account reps. They always go missing and someone new emails us about how they are taking over 6 months later. Every call w

by cyral 6y ago

I've had the same experience with GCP account reps. They always go missing and someone new emails us about how they are taking over 6 months later. Every call we have had with them has not resulted in anything meaningful. Our biggest issue is how their "highly available" Cloud SQL goes down every couple months for maintenance, not how we can use BigQuery.

byteofbits 6y ago
We’re currently migrating to Spanner for a variety of reasons - but the mandatory downtime on their Postgres CloudSQL offering will be the part I miss the least. It’s insane that even with all of their HA and failover turned on they take the whole cluster down for as long as they like every few months!
- ethbr0 6y ago
  One thing that customer obsession at an entire-organization-depth level does is encourage broad customer use awareness. To an engineer, things are things, because of how they architect and build them. To an engineer who understands a customer, things are things and all the things people actually use them for. Big difference. It also makes "Well, that customer is using it wrong" less of an exceptable engineering dodge.
  - throwaway894345 6y ago
    Surely product makes these decisions, not engineers, right? I agree that customer empathy is important, but I don't think we can conclude that the engineering team (rather than the product team) is the source of the deficiency?
    - majormajor 6y ago
      > Surely product makes these decisions, not engineers, right? I agree that customer empathy is important, but I don't think we can conclude that the engineering team (rather than the product team) is the source of the deficiency? I haven't worked inside AWS or GCP, but I've never seen product get everything they want, especially around maintenance/downtime. If "less downtime" is on the roadmap but engineering is constantly pushing back "that'll be really really hard and take a long time and they're just using it wrong anyway," I can't imagine it getting done as quickly as at a place where the engineering team was also focused on customer satisfaction.
      - throwaway894345 6y ago
        > that'll be really really hard and take a long time It probably is hard and intensive. Engineering shouldn't lie and promise that it will be easier. Product has to take that engineering estimate and determine whether to work uptime or some sexy feature (and sexy features usually win because of perverse incentives). Moreover, I have a hard time believing this for a couple reasons: first of all, I've scarcely met engineers who were opposed to improving product reliability, maintainability, etc. The portrait of Google engineers arguing that database services fundamentally shouldn't be HA (and customers are "using it wrong" for wanting HA DBs) is particularly incredulous. Secondly, I've never heard of an organization where engineering held political power over product decisions, but I have worked in several places where product dictated engineering solutions. Businesses trust product more readily than engineering because the things that engineering is always petitioning for are abstract and "costly" (deferring some immediate profit for reduced costs in the long run) while the things product wants are usually tangible and profitable.
        rutthenut 6y ago
        Sounds familiar. Product and sales areas of the business look at immediate revenues - and hopefully profit - but are always more keen to do that at the cost of increasing technical debt within the engineering side of the business. Unless sales see a real impact to technical debt, they will always choose the short-term approach.
        throwaway894345 6y ago
        Agreed, and I don't think that's even necessarily the worst thing. It just means that engineering and product/sales have to have a conversation, mutual trust, and a shared vision that extends beyond the next quarter. These are hard things to cultivate, however.
        ethbr0 6y ago
        Most of the places I've seen this go bad were because one side was incentivized to not care about the other. If product has PM or financial incentives that reward without regard to technical debt... If engineering has PM or financial incentives that reward without regard to user experience... If you want people to cooperate, set up shared, team-based incentives!
    - lmeyerov 6y ago
      It's a full-company culture problem where ~everyone has personal responsibility. Behavioral properties are end-to-end, like speed, security, customer success, uptime, etc. Everyone and everything on the hotpath for that has to be on board, as just one violator means nobody is achieving it. Operationally, part of achieving that means company norms, such as when collaborating, planning, executing, etc. It's 100% fine not to provide any of these, and is even reasonable given it's hard to change one's DNA end-to-end... but better to be an explicit decision. Likewise, if starting a company... a lot easier to pick & ingrain these habits ahead of time.
      - jiggawatts 6y ago
        I tried explaining this to some Azure product teams, and they gave me a blank stare in return. Sure, you can have zone-redundant Azure SQL Databases, but not Azure App Service! You can have zonal Azure App Service, but with a different network model than the default, so it's a breaking change for some apps. It's as if nobody has actually sat down at Microsoft to build something similar to what their customers are building. It's all just tech demos, "get your blog hosted in 5 easy steps", and crap like that. Nobody at Microsoft has ever built anything of substance with the majority of their own platform. It was the same story with Windows Presentation Foundation (WPF). It was hot garbage when it was first released, but Microsoft kept telling all their partners to prefer it over the legacy GDI+. People tried and failed to write GUIs in it. When after many years Microsoft finally tried to convert Visual Studio to WPF -- the first time they had used their own framework for one of their own apps -- magically the core issues were fixed and WPF become viable for real apps.
        mathattack 6y ago
        My experience with MSFT is that the account reps are clueless by design. Even the sales assistants with fancy technical titles are short of clues. Easier for them to oversell when they don’t know what’s really going on underneath. We have to go 3 or 4 deep into their product org to find someone who will fess up to product issues that we are already aware of.
        6y ago
- ransom1538 6y ago
  WAIT. Be careful. That is a super expensive product with a high likelyhood of lockin. It doesn't support all SQL features. Also! I run hundreds of GCP databases and never ran into: "but the mandatory downtime on their Postgres CloudSQL", maybe it is only Postgres?
- adamparsons 6y ago
  I had no idea this was a thing, is there any justification at all for this behavior? Sounds like a massive dealbreaker to me.
- renewiltord 6y ago
  I use Aurora just for this reason. RDS is just a great product. Doesn't even need the cloud sql proxy equivalent since you can just install the authenticator into the database.
  - paulryanrogers 6y ago
    Aren't RDS and Aurora two different things?
    - renewiltord 6y ago
      Aurora is one of the DB things you can launch in RDS.
      - paulryanrogers 6y ago
        Thanks for clarifying. I'd associated RDS with the non-Aurora offerings of Mysql and Postgres. Judging by upstream version compatibility it appears Aurora is more heavily forked than their non-Aurora siblings.
        dragonwriter 6y ago
        I don't think Aurora MySQL/Postgres are just forks, I think they are a completely custom datastore behind a MySQL or Postgres-compatible interface (which probably uses a lot of non-engine code from the open-source base database.)
        paulryanrogers 6y ago
        Regardless it looks like anyone choosing Aurora should not hold their breath for Mysql 8 or Postgres 10+ compatibility. Seems like only one major version bump has happened since they launched the first one (Mysql 5.6-to-5.7). Which is fine. It can just be a little confusing as they drift and the caveats grow.
        6y ago
Axsuul 6y ago
Same here. I have sent my GCP rep 5-10 follow up emails by now and still no response. It really feels like I have no one to talk to over there and I'm spending thousands per month.
- milesward 6y ago
  Hit me, more than happy to make fun of the salient GCP rep for you in a way they really won't like :)
  - trulyme 6y ago
    Curious what your approach would be? That said, this is not some poor rep's problem. This is a problem of Google culture and one that will be very difficult to fix. They simple don't care much about their customers and never needed to.
    - milesward 6y ago
      I’ve seen both! Plenty of culture issues, or tools, and sometimes someone’s just checked out...
  - Clewza313 6y ago
    Bit of a surprising comment coming from the CTO of a large Google partner? https://sada.com/about/leadership-team/miles-ward/ https://sada.com/about/leadership-team/miles-ward/
    - wbl 6y ago
      What do you think partners are for if not this?
    - milesward 6y ago
      I think you’ll find me surprising :) It’s surfacing and chasing down root cause of these misses that make things better for all customers, and that’s my goal :)
ciguy 6y ago
I helped a major user of GCP migrate off the platform to AWS for this exact reason. Totally insane that they still do this when AWS has had a rock solid offering in the form of RDS for like 10 years now.
- markdog12 6y ago
  Does RDS not do this to their customers? This alone would make me seriously consider taking my business away from GCP.
  - ciguy 6y ago
    Absolutely not. I'm sure it's happened but I have never personally seen an RDS instance go down, in over 10 years consulting and building in AWS. Google Cloud SQL was going down for multiple minutes every month for one of my clients, and their support just said there was nothing they could do about it. That cost Google cloud at least a million dollars in revenue over the next few years as this was a fast growing startup.
    - plasma 6y ago
      I’ve been a long time Azure user, their SQL Azure product was very glitchy and had constant outages (“just add retries!”), it’s gotten better in the last year or so, but it was totally unacceptable.
      - jiggawatts 6y ago
        AFAIK that's a "feature", not a bug. They throttle concurrent requests and return a different error code compared to the on-premises SQL Server. I just watched a load test hit that issue, the developers cranked up the traffic until the response times were measured in minutes. At that point Azure SQL started throwing errors related to too many concurrent queries. Compared to just locking up or timing out, it's probably better. See: https://docs.microsoft.com/en-us/azure/azure-sql/database/resource-limits-logical-server#sessions-and-workers-requests https://docs.microsoft.com/en-us/azure/azure-sql/database/re... And: https://docs.microsoft.com/en-us/azure/azure-sql/database/resource-limits-vcore-elastic-pools#general-purpose-service-tier-generation-4-compute-platform-part-1 https://docs.microsoft.com/en-us/azure/azure-sql/database/re... E.g.: the 4-CPU pool has only 210 Max concurrent workers per pool, but if you think about it, that's over 50 queries running at once per CPU core, which is nuts!
markdog12 6y ago
> "highly available" Cloud SQL goes down every couple months for maintenance Yep, drives me up the wall. We put a lot of work in making sure we're zero-downtime, pay for "Highly Available", and every couple of months I get the dreaded email. No explanation, nothing. Then we have to do the whole downtime dance. If anyone from Cloud SQL is reading this, please, please stop doing this.

by cyral 6y ago

byteofbits 6y ago

We’re currently migrating to Spanner for a variety of reasons - but the mandatory downtime on their Postgres CloudSQL offering will be the part I miss the least. It’s insane that even with all of their HA and failover turned on they take the whole cluster down for as long as they like every few months!

ethbr0 6y ago
One thing that customer obsession at an entire-organization-depth level does is encourage broad customer use awareness. To an engineer, things are things, because of how they architect and build them. To an engineer who understands a customer, things are things and all the things people actually use them for. Big difference. It also makes "Well, that customer is using it wrong" less of an exceptable engineering dodge.
- throwaway894345 6y ago
  Surely product makes these decisions, not engineers, right? I agree that customer empathy is important, but I don't think we can conclude that the engineering team (rather than the product team) is the source of the deficiency?
  - majormajor 6y ago
    > Surely product makes these decisions, not engineers, right? I agree that customer empathy is important, but I don't think we can conclude that the engineering team (rather than the product team) is the source of the deficiency? I haven't worked inside AWS or GCP, but I've never seen product get everything they want, especially around maintenance/downtime. If "less downtime" is on the roadmap but engineering is constantly pushing back "that'll be really really hard and take a long time and they're just using it wrong anyway," I can't imagine it getting done as quickly as at a place where the engineering team was also focused on customer satisfaction.
    - throwaway894345 6y ago
      > that'll be really really hard and take a long time It probably is hard and intensive. Engineering shouldn't lie and promise that it will be easier. Product has to take that engineering estimate and determine whether to work uptime or some sexy feature (and sexy features usually win because of perverse incentives). Moreover, I have a hard time believing this for a couple reasons: first of all, I've scarcely met engineers who were opposed to improving product reliability, maintainability, etc. The portrait of Google engineers arguing that database services fundamentally shouldn't be HA (and customers are "using it wrong" for wanting HA DBs) is particularly incredulous. Secondly, I've never heard of an organization where engineering held political power over product decisions, but I have worked in several places where product dictated engineering solutions. Businesses trust product more readily than engineering because the things that engineering is always petitioning for are abstract and "costly" (deferring some immediate profit for reduced costs in the long run) while the things product wants are usually tangible and profitable.
      - rutthenut 6y ago
        Sounds familiar. Product and sales areas of the business look at immediate revenues - and hopefully profit - but are always more keen to do that at the cost of increasing technical debt within the engineering side of the business. Unless sales see a real impact to technical debt, they will always choose the short-term approach.
        throwaway894345 6y ago
        Agreed, and I don't think that's even necessarily the worst thing. It just means that engineering and product/sales have to have a conversation, mutual trust, and a shared vision that extends beyond the next quarter. These are hard things to cultivate, however.
        ethbr0 6y ago
        Most of the places I've seen this go bad were because one side was incentivized to not care about the other. If product has PM or financial incentives that reward without regard to technical debt... If engineering has PM or financial incentives that reward without regard to user experience... If you want people to cooperate, set up shared, team-based incentives!
  - lmeyerov 6y ago
    It's a full-company culture problem where ~everyone has personal responsibility. Behavioral properties are end-to-end, like speed, security, customer success, uptime, etc. Everyone and everything on the hotpath for that has to be on board, as just one violator means nobody is achieving it. Operationally, part of achieving that means company norms, such as when collaborating, planning, executing, etc. It's 100% fine not to provide any of these, and is even reasonable given it's hard to change one's DNA end-to-end... but better to be an explicit decision. Likewise, if starting a company... a lot easier to pick & ingrain these habits ahead of time.
    - jiggawatts 6y ago
      I tried explaining this to some Azure product teams, and they gave me a blank stare in return. Sure, you can have zone-redundant Azure SQL Databases, but not Azure App Service! You can have zonal Azure App Service, but with a different network model than the default, so it's a breaking change for some apps. It's as if nobody has actually sat down at Microsoft to build something similar to what their customers are building. It's all just tech demos, "get your blog hosted in 5 easy steps", and crap like that. Nobody at Microsoft has ever built anything of substance with the majority of their own platform. It was the same story with Windows Presentation Foundation (WPF). It was hot garbage when it was first released, but Microsoft kept telling all their partners to prefer it over the legacy GDI+. People tried and failed to write GUIs in it. When after many years Microsoft finally tried to convert Visual Studio to WPF -- the first time they had used their own framework for one of their own apps -- magically the core issues were fixed and WPF become viable for real apps.
      - mathattack 6y ago
        My experience with MSFT is that the account reps are clueless by design. Even the sales assistants with fancy technical titles are short of clues. Easier for them to oversell when they don’t know what’s really going on underneath. We have to go 3 or 4 deep into their product org to find someone who will fess up to product issues that we are already aware of.
        6y ago
ransom1538 6y ago
WAIT. Be careful. That is a super expensive product with a high likelyhood of lockin. It doesn't support all SQL features. Also! I run hundreds of GCP databases and never ran into: "but the mandatory downtime on their Postgres CloudSQL", maybe it is only Postgres?
adamparsons 6y ago
I had no idea this was a thing, is there any justification at all for this behavior? Sounds like a massive dealbreaker to me.
renewiltord 6y ago
I use Aurora just for this reason. RDS is just a great product. Doesn't even need the cloud sql proxy equivalent since you can just install the authenticator into the database.
- paulryanrogers 6y ago
  Aren't RDS and Aurora two different things?
  - renewiltord 6y ago
    Aurora is one of the DB things you can launch in RDS.
    - paulryanrogers 6y ago
      Thanks for clarifying. I'd associated RDS with the non-Aurora offerings of Mysql and Postgres. Judging by upstream version compatibility it appears Aurora is more heavily forked than their non-Aurora siblings.
      - dragonwriter 6y ago
        I don't think Aurora MySQL/Postgres are just forks, I think they are a completely custom datastore behind a MySQL or Postgres-compatible interface (which probably uses a lot of non-engine code from the open-source base database.)
        paulryanrogers 6y ago
        Regardless it looks like anyone choosing Aurora should not hold their breath for Mysql 8 or Postgres 10+ compatibility. Seems like only one major version bump has happened since they launched the first one (Mysql 5.6-to-5.7). Which is fine. It can just be a little confusing as they drift and the caveats grow.
        6y ago

Axsuul 6y ago

Same here. I have sent my GCP rep 5-10 follow up emails by now and still no response. It really feels like I have no one to talk to over there and I'm spending thousands per month.

milesward 6y ago
Hit me, more than happy to make fun of the salient GCP rep for you in a way they really won't like :)
- trulyme 6y ago
  Curious what your approach would be? That said, this is not some poor rep's problem. This is a problem of Google culture and one that will be very difficult to fix. They simple don't care much about their customers and never needed to.
  - milesward 6y ago
    I’ve seen both! Plenty of culture issues, or tools, and sometimes someone’s just checked out...
- Clewza313 6y ago
  Bit of a surprising comment coming from the CTO of a large Google partner? https://sada.com/about/leadership-team/miles-ward/ https://sada.com/about/leadership-team/miles-ward/
  - wbl 6y ago
    What do you think partners are for if not this?
  - milesward 6y ago
    I think you’ll find me surprising :) It’s surfacing and chasing down root cause of these misses that make things better for all customers, and that’s my goal :)

ciguy 6y ago

I helped a major user of GCP migrate off the platform to AWS for this exact reason. Totally insane that they still do this when AWS has had a rock solid offering in the form of RDS for like 10 years now.

markdog12 6y ago
Does RDS not do this to their customers? This alone would make me seriously consider taking my business away from GCP.
- ciguy 6y ago
  Absolutely not. I'm sure it's happened but I have never personally seen an RDS instance go down, in over 10 years consulting and building in AWS. Google Cloud SQL was going down for multiple minutes every month for one of my clients, and their support just said there was nothing they could do about it. That cost Google cloud at least a million dollars in revenue over the next few years as this was a fast growing startup.
  - plasma 6y ago
    I’ve been a long time Azure user, their SQL Azure product was very glitchy and had constant outages (“just add retries!”), it’s gotten better in the last year or so, but it was totally unacceptable.
    - jiggawatts 6y ago
      AFAIK that's a "feature", not a bug. They throttle concurrent requests and return a different error code compared to the on-premises SQL Server. I just watched a load test hit that issue, the developers cranked up the traffic until the response times were measured in minutes. At that point Azure SQL started throwing errors related to too many concurrent queries. Compared to just locking up or timing out, it's probably better. See: https://docs.microsoft.com/en-us/azure/azure-sql/database/resource-limits-logical-server#sessions-and-workers-requests https://docs.microsoft.com/en-us/azure/azure-sql/database/re... And: https://docs.microsoft.com/en-us/azure/azure-sql/database/resource-limits-vcore-elastic-pools#general-purpose-service-tier-generation-4-compute-platform-part-1 https://docs.microsoft.com/en-us/azure/azure-sql/database/re... E.g.: the 4-CPU pool has only 210 Max concurrent workers per pool, but if you think about it, that's over 50 queries running at once per CPU core, which is nuts!

markdog12 6y ago

> "highly available" Cloud SQL goes down every couple months for maintenance Yep, drives me up the wall. We put a lot of work in making sure we're zero-downtime, pay for "Highly Available", and every couple of months I get the dreaded email. No explanation, nothing. Then we have to do the whole downtime dance. If anyone from Cloud SQL is reading this, please, please stop doing this.