8 ms·
Trying to get a bunch of GPUs on Google Cloud
- jandrese 4y agoHis comment about not being able to pre-load an account with enough money to cover the usage to avoid the noob filter seems like it should be easy to fix, but I'm not sure it would solve the "whoops, I just accidentally wasted 6 figures, can I have it back?" problem Google presumably implemented this policy to fight. Sure Google won't necessarily be on the hook since they already have the money, but at the same time it's hard for customer service to go tell someone like that to pound sand. They clearly have money to spend even if they managed to waste a ton of it this time. Maybe a flag you could set in your account that says "I know what I'm doing and I promise not to ask for refunds for my mistakes." Obviously if the money is lost due to a Google screwup then yeah, I'm going to want that back, but if I forgot to turn off a few thousand max GPU instances over the weekend then that's on me. Maybe combine that flag with a prepayment system? And if you blow out your account then it halts your instances until you top it off, hopefully with a bunch of text messages and emails to the admin.
- bombcar 4y agoAWS famously lets you spend billions overnight, so it can't be that big of a problem. I suspect it's mostly Google being Google and horribly at customer service, even when the customers are waving money at them.
- llaolleh 4y agoGoogle customer service is atrocious. Their ability to solve customer support issues is effectively a toss up. Please do better Google.
- nszceta 4y agoYou shouldn't need to ask Google for mercy. When a company gives you bad service you should be free to leave for a competitor. If you are not able to you need to ask yourself why you are willing to put up with such abuse?
- minhazm 4y agoI think it's largely about capacity. Even if you have the money they don't want you to randomly spin up up a few hundred GPU instances for 6 hours because it will end up causing capacity issues for all of the other people trying to also allocate those instances. GPU's are in crazy high demand and I doubt even GCP, Azure, or AWS can get their hands on enough of them. Contrary to what cloud providers may have you believe you can't actually spin up hundreds of instances unless you already have thousands of instances or you've talked to them and have already told them you're going to spend millions of dollars with them and got the quota pre-approved.
- willcipriano 4y ago> Contrary to what cloud providers may have you believe you can't actually spin up hundreds of instances I've had three software engineer jobs, learned this lesson on the first one and watched others get tripped up by it in the other two. Nobody ever believes me when they say "We will just spin up a instance for every file in the directory" or similar and I respond with "No you won't, they aren't going to give you a thousand instances". This is a lesson that can only be learned first hand.
- fragmede 4y agoFor GPUs, yeah. If you're talking compute though, you just keep requesting your limits be raised until you're happy. You'll probably hit your ability to pay before you hit actual AWS limits. Talk to your AWS TAM if you have questions on what your spend has to be before they'll give you that kind of access. If you don't have a TAM then you might be too small.
- deleted 4y ago[deleted]
- nunez 4y agos/might/are/
- Aeolun 4y ago
- badlucklottery 4y ago> I'm not sure it would solve the "whoops, I just accidentally wasted 6 figures, can I have it back?" problem Google presumably implemented this policy to fight. Or the "with enough GPUs I can quickly launder a large pile of cash I scammed off the elderly/businesses/etc. into crypto currency before this account gets frozen" problem.
- codedokode 4y agoThere are other ways to launder money, for example casino, gift cards or betting on sports.
- xxpor 4y agoI suspect Google isn't worried about the financial aspects of fraud so much as they're worried about fraud using extremely limited capacity. His point about prepaying makes sense from the customer side, but I'd be willing to bet google doesn't want to deal with having negative balances in accounts that are backed by real cash (as opposed to credits). Maybe they already allow that if you overpay? Just guessing here.
- thespacecow 4y agoWe had a project almost 2 years ago where we needed to run g3.8xlarge instances on AWS in "serverless" mode to process videos using photogrammetry in real time. Going to AWS offices and asking for the solution resulted in "well, seems like you need Lambdas-on-steroids. We won't provide that service but feel free to build it by yourself". So we built it and the moment we ran the crash test (we tried to go up to 1000 simultanious instances), it turned out eu-west-2 had just 48 instances available. We followed up and AWS confirmed that we just capped their region capacity for those instances. They didn't raise any quotas and suggested we use other regions to bypass the cap.
- Aeolun 4y agoAWS always suggests you use a variety of instance types/availability zones to fulfill your capacity needs. I can imagine they don’t keep a few thousand really expensive instances around (in every region/availability zone).
- KptMarchewa 4y agoThey don't have that many GPUs to just enable everyone to use it.
- throwawaaarrgh 4y agoBut if you literally drive a briefcase full of cash up to their data center, they should let you use some GPUs. I don't think that's a ridiculous suggestion. The problem is 1) they won't just let you pay them and 2) even if you could, they still won't just let you use some GPUs.
- bvhvhfcgxfdf 4y agoThe more I read about Google the more I am convinced that the only reason it proliferated because it's an ad company. Google would've gone bankrupt in its preliminary phase if it was a company that added any real value.
- mkl95 4y agoThe more I read about Google the more I am convinced it cannot be treated as a single organization, but as many organizations operating under the same name - some of them brilliant, some of them utterly clueless.
- joemi 4y agoBack when they were just a search engine, they added HUGE value. Search was really terrible before them, and when using google for the first time after using all the other search engines, it really felt like actual magic.
- kalleboo 4y agoGoogle Maps also felt like magic when it came out and you could just drag to scroll vs the slow click-click-click of Mapquest. Similarly everyone was convinced Gmail was an April Fool's joke when they gave you 1 GB of storage when everyone else just gave you 3 MB.
- nmfisher 4y agoWhat’s also amusing is that their support for advertisers is just as bad. The reality is that companies will tolerate just about anything as long as you have eyeballs to sell.
- type_Ben_struct 4y agoI think that’s an unfair statement. There are good and bad parts of Google, but to say they add no value is a stretch. Can you really say that across YouTube, Search, Gmail, Drive, Cloud, Open Source Contributions (Kubernetes, Tensorflow, etc) and so on they don’t add any real value anywhere at all?
- gary_0 4y agoImagine the problems startups have who don't have a Big Name at the helm; Google/Amazon/Apple can ignore or impede them indefinitely. People in this thread are bringing up fraud and customer service issues, but these filters should not be something reputable corporations (and famous pioneers like Carmack) have to worry about. Something is very wrong indeed. Google-sized megacorps should not be able to impede innovation and economic activity like this. We're all losing because of it.
- denkmoon 4y agoGoogle doesn't have to rent their GPUs to anyone if they don't want to. Besides, their incompetence allows companies like Lambda to fill the niche, and probably do a better job of it than Google do.
- gary_0 4y agoGoogle can use their immense scale and resources to buy up GPUs so other companies can't. Apple buys up all of TSMC's cutting-edge production capacity for its own chips, preventing competition. The market has become centralized and monopolized and thus cannot be relied on to route around failures.
- deleted 4y ago[deleted]
- Teever 4y ago> Google doesn't have to rent their GPUs to anyone if they don't want to. You're just restating the current status quo as if that some how refutes the notion that businesses can be regulated, and that businesses of this size should be regulated in a manner that promotes competition.
- scombridae 4y agoGoogle should not be able to impede economic activity Bad producers can only impede as much activity as stupid consumers give them.
- jiggawatts 4y agoDigital Ocean was just obliterating legitimate businesses when their GPU or CPU usage appeared to look like crypto mining. Since they only accept credit, this is exposing Digital Ocean to the risk of handing out cash-equivalents for potentially bad credit. This "forces" them to be aggressive against abuse. Unfortunately, heuristics aren't perfect, so they also occasionally execute a hapless business by accident. John C made the same comment that I did. Other than outright crime (child porn, etc...), there ought not be any limits on instance usage in clouds, provided that customers pay cash. Heck, let them crypto mine if they wish, as long as they pay cash! Who cares? It's just computation.
- fragmede 4y agoBecause smart money was on the GPU market eventually coming back down to Earth due to a crash in the crypto market. Which it mostly has. What no one could reasonably predict is when (market can stay irrational and all that). Cloud providers buy obscene amounts of hardware for obscene amounts of money, and if their capacity predictions are off, they lose tons of money. Disallowing crypto mining on GPUs then just makes sense. You just never know when those customers are going to evaporate leaving you high and dry with a GPU farm that people aren't paying for.
- brunoTbear 4y agoCash doesn't really exist in the payments stack for an online business. Credit card chargebacks can roll in months after a compute instance is closed down and companies like Google have little recourse here. "Just take cash" isn't an option.
- paranoidrobot 4y agoI know AWS does, or did, take wire transfers for payment. Those are much harder to claw back, particularly if you have a signed agreement from the customer saying "We want to pre-pay for $50k in compute resources. We understand that once the pre-paid amount is used, it cannot be refunded." I'm sure GCP and any other legitimate cloud provider could also take wire transfers for sizable payments, without issue.
- tveita 4y agoIt's a pain. I recently had a compute job that would take a couple of days to run locally. "This is exactly what the cloud is for," I thought and went to Google Cloud where I already had an account. First try I'm allowed to spec up a beefy server with AVX-512 and click "create", but it never actually starts. Turns out the default CPU limits are tiny - I need to apply to bump two separate limits and wait for that to go through. Second try I'm actually able to spin up a couple of servers. I set the compute job running and go for lunch. An hour later both are terminated and I get a message about "violation of our Acceptable Use Policy". There is a link that says crypto mining is forbidden on free instances. I send a reply informing them that a) these are not free instances, and b) I am not mining crypto. I get a reply back saying more or less "Yes it says free instances but why would you expect anything we write in the documentation to be correct. Also you might think you weren't mining crypto but actually your VM was probably compromised because no one would buy compute instances and actually run compute jobs on them. We're not giving you back access to your instances, maybe delete them and create new ones lol." I actually find it an easier product to get started with than AWS, but man they really really don't want your money. When you are actively trying to prevent your customers from scaling up resources you are failing at being a cloud.
- dilyevsky 4y agoImo these tiny quotas are specifically put in bc a lot of people (especially on this site) always complain how they managed to rack a huge bill by accident. You can’t have it both ways
- zerr 4y ago@JohnCarmack Can you please ask Google when do they plan to provide merchant registration support for leftover countries on Play Store? Thank you. https://support.google.com/googleplay/android-developer/answer/9306917 https://support.google.com/googleplay/android-developer/answ...
- andrewstuart 4y agoForget cloud GPUs. It makes much more sense from every perspective to buy them and put them in your bedroom. Cloud GPUs are slow, VERY expensive and not even available. You shouldn't need the superstar power of being John Carmack to get access to computing resources. The promise of the cloud is for the most part, sssnake oil.
- xxpor 4y agoHow could a cloud GPU that's a specific model be any different in speed from the same thing in your bedroom, if you could get nvidia to even sell you one in the first place?
- MasterScrat 4y agoThey're probably alluding to the large number of GPU models available to retail customers which are not allowed in datacenters.
- xxpor 4y agoThen they're completely missing the point of the cloud GPUs. If you're not running something that needs NCCL et al, yeah, go run something under your desk.
- MasterScrat 4y ago4090s support NCCL and are not allowed in datacenters
- andrewstuart 4y agoCloud GPUs aren't better.
- andrewstuart 4y agoThe consumer GPUs are much faster, cheaper and more available than most of the cloud GPUs.
- Havoc 4y agoI’d probably try a provider like tensordock first for cloud gpus. Reckon smaller providers are also more likely to have someone that picks up the phone
- merb 4y agothe first time you go over 1500 €/USD per month you basically get a account manager on google cloud nowadays which resolves this issue for you