17 ms·
GCP automatically lowered our quota, caused an incident, and refused to upgrade
- pawelduda 3y agoWonder what does GCP side have to say on this
- redman25 3y agoWe had a similar thing happen with our company. Google had given a bigquery concurrent query limit extension to 300 queries. In January they removed the extension back down to 100 concurrent queries because they were introducing “query queues” that should cover the difference. Unfortunately for us, query queues only affect spikes in concurrent queries and not overall throughput. We’ve been struggling with bigquery support ever since.
- newhouseb 3y agoQuota management is indeed nonsensical. We serve large cash assistance programs that have stampedes of people applying all at once where we needed decent geocoding to determine eligibility. We were exceeding the default 50qps quota. I asked nicely (to double it, I think) and explained our use case and they said... no. So... we left and switched to Smarty and haven't looked back since. We spend tens of thousands on geocoding annually. Really mind-boggling behavior from GCP.
- betaby 3y agoWhat is 'Smarty'? Never heard about that company/product.
- virtualpain 3y agoI too was confused, all Google search result was PHP templating library. So I re-read the comment and added "smarty geocoding" to display correct result.
- newhouseb 3y agoYeah, they used to be called SmartyStreets and then dropped the Streets. Not the greatest branding call.
- kyrra 3y agohttps://www.smarty.com/ https://www.smarty.com/
- jtchang 3y agoWhy don't I hear about these incidents with AWS? Is it just less common or different business takes on quotas and rate limits?
- balls187 3y agoAWS is great. Gotta check out FBA/Amazin Marketplace to find all the nasty on Amazon.
- elankart 3y agoIn what world is AWS great? A world where they don’t have a clean way to group resources and clean them up? Their own twisted identity products in AWS that doesn’t integrate outside their cloud or their other products. A world where they leave lingering resources when you cleanup something. AWS has to much hype riding behind it. It’s not enterprise class and looks very incohesive.
- ripper1138 3y agoIs this comment from 2015? AWS just had a $20B quarter.
- balls187 3y ago> In what world is AWS great? Mine. And Gartner’s. I use GCP and AWS, and AWS outclasses GCP handily.
- whoknowswhat11 3y agoThe point about lack of projects on AWS I agree with if that’s what you are talking a about. That said AWS is absolutely a cloud leader. We use azure and AWS now and there continue to be reasons AWS does well despite cost and quirks
- rhtgrg 3y ago> Is it just less common or different business takes on quotas and rate limits? Neither, you're probably not seeking out such stories (there are plenty, even on HN). Google tends to get more negative attention on this particular forum, which might play a role, but the larger component is probably just chance.
- Animats 3y agoGoogle giveth, and Google taketh away. Blessed be the name of Google.
- civilitty 3y agoTo our queries, it lights the way. In its Matrix, we are but a doodle. Blessed be the name of Google, come what may.
- invalidname 3y agoFrom my past experience Google is by far the worst when it comes to support. By far. Even if you pay for their gold support you're lucky if you reach an engineer and even then they blame you for the problem with no "proof" or direction. Most unhelpful ever... See: https://medium.com/hackernoon/why-and-how-we-left-app-engine-after-it-almost-destroyed-us-40ac2fc0b1a8 https://medium.com/hackernoon/why-and-how-we-left-app-engine... AWS were surprisingly helpful by comparison and even pro-active. Not a fan of theirs overall but much better than Google. This is a deep cultural problem with Google that also expresses itself in mobile development and everywhere: https://dev.to/codenameone/google-play-kafkaesque-experience-mp3 https://dev.to/codenameone/google-play-kafkaesque-experience... They aren't the cheapest and their service is terrible. I can't think of a good reason to use GCP.
- showdeddd 3y agoThis medium article seems silly, just add custom metrics to your app for what was fetched from cache vs DB. And label the metric by route/query/pattern. To control costs, don't tick the metric for every single request, instead accumulate locally and post to metrics API every X minutes.
- tyingq 3y ago> accumulate locally Is that an option in AppEngine? The memcache docs seem to indicate the free tier has undocumented eviction policies.
- showdeddd 3y agoYou have to implement that with your own code but it isn't much more than a dict/map and a timestamp for last update.
- tyingq 3y agoSame-ish problem, though. You wouldn't know for sure the instance will run again...your dict/map data can be dropped. I don't see any sort of instance timeout callback where you could guard against that.
- showdeddd 3y agoThis article seems silly, just add custom metrics to your app for what was fetched from cache vs DB. And label the metric by route/query/pattern. To control costs, don't tick the metric for every single request, instead accumulate locally and post to metrics API every X minutes.
- parpfish 3y agoI’ve recently had an issue where GCP keeps shutting down my e2.micro because they think I’m crypto mining. I have no idea what I’m doing to trigger it, and I don’t blame them for not sharing their secret anti fraud heuristics, but it’s a freakin e2.micro. They think I’m mining on that?!
- missingdays 3y agoWhy would they care if you mine crypto? You pay for resource usage, you use resources
- professorsnep 3y agoGCE allows one e2.micro instance in their free tier
- herdrick 3y agoProviders often just ban all crypto mining. I think it might be because of pressure from authorities fighting money laundering.
- Sebguer 3y agoNo, it's because it's an extraordinarily common fraud vector, and causes noisy neighbor problem because despite GP's remark about 'paying for resource usage', cloud providers don't actually anticipate anyone using 100% of CPU 24/7/365. And the folks who really are crypto mining are never going to pay their monthly invoice. Source: Worked at a cloud infra provider that struggled deeply with this problem.
- jmaker 3y agoSo you pay expecting full utilization but are expected to utilize on a fair-share basis only? And if I run my Monte Carlo simulation, which can take a week to complete, my instance is flagged for alleged crypto mining?
- profwalkstr 3y agoIt feels as if Google is intentionally trying to sabotage its own cloud business
- exabrial 3y agoOnce Again: Do not use Google for anything. Given they treat the consumer side of their business like cattle (Gmail, maps, etc) (,and despite the fact that an email address is an essential part of one's daily life: access to banking, investments, communication, cell phone, employment opportunities), they operate with absolutely zero support. What makes you think they'll treat your small business any differently? You will be thrown under the bus the second you make them lift a finger to help you.
- chlorion 3y agoGoogle services work perfectly for me. I think the users of this website just really like to repeat anything that fits the "Google bad" narrative, and if you don't have interactions with people outside of the bubble, it seems like Google services are on fire and practically unusable, but in reality it's fine for 99% of people.
- Demmme 3y agoJust an anecdotal counter point: very happy with gcp. Best network from all and coherent modern ui. Not the usability hell like azure... (You know when clicking on a often used resource on the start page which let's you jump directly to it but doesn't allow you to jump a level up of all the other resources of the same type which totally works fine when you navigate to it the normal way... Or the huge hassle and complexity of resource groups for f everything...) But you know the tweet not even states what quota was reduced.
- textninja 3y ago[flagged]
- seanhunter 3y agoI think they may be being downvoted not because their opinion is different but because usability of the web UI is possibly the least important attribute in choosing a cloud provider for a lot of people given the use of tools such as terraform.
- bluepizza 3y agoOP is being downvoted because his anecdotal defense of GCP due to nice UI is irrelevant to the link posted. And probably to cloud computing in general.
- jiggawatts 3y agoA resource group is literally just a folder — a name — and is the best feature of Azure. For comparison, any large AWS account is always a total mess. Just an endless list of randomly named things that are totally unrelated to each other. If that is his criticism, he deserves the down votes. I bet his desktop has a hundred icons on it strewn randomly where half of them are named “file.txt” and “document.doc”
- deathanatos 3y agoWell, part of his criticism is that the UI is woefully inconsistent with regards to "upwards" navigation. If you navigate to a resource via the resource group, you can navigate back "up" to the parent RG. If you nagivate to the resource from the home screen, IIRC, you cannot. That's the criticism. Yes, RG's are a pretty killer feature, and trying to understand the organization of resources in GCP is hard by comparison, and an utter nightmare in AWS. I'm not sure why he's knocking that. (And … GCP requires projects, which seem equivalent to the complaint against RGs.) … that said … there are so many other things utterly and horrifically wrong with Azure that I wouldn't put them on a pedestal for resource groups. As much as I do like RGs. (Also, "A resource group is literally just a folder" sigh, no, because they're not hierarchical. Azure goofed hard there.)
- endisneigh 3y agoTweet has basically no details
- Havoc 3y agoThat's presumably how they felt too...
- yashg 3y agoI keep getting badgered by GCP channel partners to move over to GCP. One guy once said if I can show him our AWS usage he can give me exact configuration in GCP and cost would be 20-25% less. I was intrigued so I shared. He came back with a price estimate higher than our AWS bill! I haven't entertained them ever since.
- neilv 3y agoThey lost a sale, but they got competitive data. Maybe that's how that salesperson or their org thinks?
- nordsieck 3y ago> Maybe that's how that salesperson or their org thinks? Nah - it's just a numbers game. Sometimes they'll be wrong; that's a dead end. Sometimes they'll be right. And some of those times, they'll make a sale.
- neilv 3y agoGiven how hard it is to do enterprise sales as a startup vendor, it's disappointing to think of some salesperson, repping a big brand, just doing a "numbers game" drive-by spray, and seeing what they hit.
- yashg 3y agoCloud platforms are very much commoditized with all of them offering same core products at similar price points. Just different fancy names. Also shifting a cloud provider completely is not worth the small savings in most cases. So these days they don't pitch you to move entirely, they ask you to consider their platform for any upcoming projects. One more trick used by channel partners it to move the billing under their name instead of billing directly with the platform. They promise to give a % of the commission they will receive from the platform back to customer. So they promise savings that way. They also promise to optimise your cloud infra, shut down unused instances, downsize some services that are not fully utilised and getting reserved instances for ones which are not yet on reserve. And for some companies it does end up with significant cost savings without actually switching the provider.
- uoaei 3y agoIt is interesting to note the differences in apparent defensibility of companies such as Google and Reddit changing terms of use and behaviors of platforms. According to the general sentiment on HN, Google is being mean to developers and shouldn't get away with things like this, but Reddit merely has a rug and developers are silly to build things on top of that rug when it can be pulled away at any time. I wonder why there's a difference here.
- anon84873628 3y agoWell, GCP is literally selling a "platform" to developers. Whereas Reddit is a consumer website?
- uoaei 3y agoSo?
- mvdtnz 3y agoQuota for what?
- Spasnof 3y agoWarning this is rando internet detective work so YMMV JustJake at twitter would likely know the actual answer to this question. It was mentioned that the service was artifact registery[1]. They were 5000/10min and approved at 18k/min [2] The ui appears to show a limit of 100 but not what of [3]. So reading the docs on artifact registery quota docs the only things that stands out is the "1 repository creation or deletion operation every 2 seconds."[4] so doing the math that is a rate limit of 30 per minute or 300 per 10 min so aroudn the ballpark of what they are. Looking at my own project this matches up and I can add request new quotas[4] but it needs to go through a human on googles end to approve. Anyhoo if they were approved for 18k/min that is nearly 600x the normal quota. So they were likely (pure speculation) caught up in some kind of dragnet of quotas that were well beyond the norm. Skimming the site[6] it appears to be a rapid cloud based ide / deployment thing and I can't imagine how they are creating so many repos unless for each customer build they pass along to gcp's artifact registery 1-1. I use artifactory repo as well for managing docker images but mostly just make new tags off a handful of repos and am not typically creating them at such scale. Sucks that they had the quota lowered without warning but "if" they were operating at the speculated quota that is flying pretty close to the sun. [1] https://twitter.com/JustJake/status/1667660212902453248 https://twitter.com/JustJake/status/1667660212902453248 [2] https://twitter.com/JustJake/status/1667492928758095872 https://twitter.com/JustJake/status/1667492928758095872 [3] https://twitter.com/JustJake/status/1667478906591666176 https://twitter.com/JustJake/status/1667478906591666176 [4] https://cloud.google.com/artifact-registry/quotas#project-quota https://cloud.google.com/artifact-registry/quotas#project-qu... [5] https://console.cloud.google.com/apis/api/artifactregistry.googleapis.com/quotas https://console.cloud.google.com/apis/api/artifactregistry.g... [6] https://railway.app/ https://railway.app/
- holografix 3y agoA friend works at GCP and has told me this time and time again: if you don’t buy support from GCP _you have no support_.
- azmodeus 3y agoEven if you buy support it is not comparable to AWS. I suffered GCP Dataflow support in the past it was: 1. slow 2. condescending 3. not helpful Really don't recommend GCP for long term use. If you get some free credits it's good to use it to train some ML models but AWS is much more customer friendly.
- zeroclick 3y agoBased on the comments in the thread, it seems that even if you do buy support, you also have no support.
- brianwawok 3y agoBuy support and generally disagree.
- willtemperley 3y agoI had to close my GCP account because I was being spammed with "Action Required" emails every day for a non-existent tax admin issue. Bug reports were immediately closed and getting through to a human seems impossible.
- nathants 3y agothe purpose of not aws is two: - put price/quality pressure on aws - fill weird niches made by regulatory capture and other nonsense not using aws is like not using linux. there are valid reasons, but you don’t want any of them.
- aftbit 3y agoWell unless you just want to run your own hardware. That can be orders of magnitude cheaper (business trajectory changing stuff, think offering a free plan vs not) for the right kind of workloads.
- nathants 3y agooptionality. having aws doesn’t mean using it for everything. for most uses cases, netflix model seems like the right one. control plane on cloud, data plane on not cloud. at a minimum you probably want to backup some high value data in s3 unless you have a more durable store somewhere.
- cshokie 3y agoOr your business competes with Amazon and you don’t want to pay a competitor in order to compete with them.
- crescentfresh 3y agoAnecdotally, I know a paying GCP customer that recently got quota blocked on the number of global static IPs they are allowed to create. We could not figure out how to contact someone to increase it, and nobody knew why the quota was set so low (4). We just kept running into automated systems that gave circular information on how to request an increase. We kept getting links to sign up for a free tier account.
- ktta 3y agoAre you by chance referring to anycast IPs? Having a limit of 4 for static IPs sounds so incredulous
- elguyosupremo 3y agoSearch for "all quotas" in the GCP console, from there you can view all your quotas and request increases.
- crescentfresh 3y agoRight. Unfortunately their right to request increases was removed. The exact wording: “Based on your service usage history you are not eligible for quota increase at this time” and then dead ends trying to reach someone. They had billing accounts set up for all their projects. They were happy to hand over money but had no ability to.
- deleted 3y ago[deleted]
- klodolph 3y agoIn my limited experience, quota is one of the pain points of GCP. It's where the cloud "leaks".
- aledalgrande 3y agoI've never used GCP for anything but demos, but on the other side AWS support is worth the (high) price: I always had my issues solved, quickly, and sometimes I was even able to put in feature requests that were shipped a few weeks later.
- ineedasername 3y agoHilariously reminiscent of Kafka: Client: we were under the limit GCP: No, you exceeded the limit Client: Then what is the limit? GCP: I don’t know
- robin_reala 3y agoThere was a follow-up Tweet that “explains” the problem: https://twitter.com/JustJake/status/1667492928758095872 https://twitter.com/JustJake/status/1667492928758095872 From the call with Google just now: Them: "You exceeded the rate limit" Us: "We did 5000/10min. The quota was approved at 18k/min" Them: "That's not the rate limit" Us: "What's the rate limit" Them: "Not sure have to check with that team" So, the quota is completely cosmetic...
- kaliszad 3y agoOne thing I don't understand is how in 2023 people use GCP with the bad level of IPv6 support they have. Other Google services usually support IPv6 quite well. Only GCP, to some degree Chromecast, probably Nest and the DHCPv6 problem of Android are some well known exceptions. Google also tend to put algorithms first and people and their business second so that is why I would be very particular about using GCP for anything serious. We do use Google Workspace and for what we do with it, it is mostly ok, however rather expensive.