9 ms·
Google will let companies run Gemini models in their own data centers
- rwmj 2y agoA bit thin on detail, but will this require confidential VMs with encrypted GPUs? (And I wonder how long before someone cracks SEV-SNP and TDX and pirate copies escape into the wild.)
- unsnap_biceps 2y agoThe number of folks that have the hardware at home to run it is going to be very low and the risk of companies for leaking it is gonna make it unlikely IMHO.
- notpushkin 2y agoI think home users would be the least of their concerns.
- RadiozRadioz 2y agoIt only takes one company to leak it
- franga2000 2y agoOr one company to get hacked and the hackers leak it
- spacebanana7 2y agoRealistically the only people able to run models of this size are large enterprises. Those enterprises won’t take the risk of being sued for using a model without proper permission.
- nxobject 2y agoI don't know – if there's still dumb money being thrown towards AI in non-tech and non-privacy-heavy industries, especially ones traditionally targeted by ransomware, there'll always be a chance of datasets getting leaked. I'm thinking retail and consumer product-oriented companies. (There's always non-Western governments without strong security orgs, too.)
- blackoil 2y agoNations.
- FilosofumRex 2y agoor large government sponsored entities like Mossad. Air gapping won't protect against spying. Good luck trying to sue them
- BiteCode_dev 2y agoThey can get "hacked" and wooops.
- deleted 2y ago[deleted]
- vasco 2y agoAt the pace models improve, the advantage of going the dark route shouldn't really hold for long, unless I'm missing something.
- miohtama 2y agoAccess to proprietary training data: Search, YouTube, Google Books might give some moat.
- maxloh 2y agoWe have Common Crawl, which is also scraped web data for training LLMs, provided for free by a non-profit.
- UltraSane 2y agoThe Common Crawl is going to become increasingly contaminated with LLM output and training data that is more likely to have less LLM output will become more valuable.
- kouteiheika 2y agoI see this misconception all the time. Filtering out LLM slop is not much different than filtering out human slop. If anything, LLM generated output is of higher quality that a lot of human written text you'd randomly find on the internet. It's no coincidence that state-of-art LLMs increasingly use more and more synthetic data generated by LLMs themselves. So, no, just because training data was produced by a human doesn't make it inherently more valuable; the only thing that matters is the quality of the data, and the Internet is full of garbage which you need to filter out one way or another.
- SXX 2y agoProblem with filtering is that LLMs can generate few orders of magnitude more slop than humans.
- bjackman 2y ago> I wonder how long before someone cracks SEV-SNP https://bughunters.google.com/blog/5424842357473280/zen-and-the-art-of-microcode-hacking https://bughunters.google.com/blog/5424842357473280/zen-and-...
- NoahZuniga 2y agoI'd expect watermarked model weights plus a lot of liability to distinctivise leaking the model.
- blitzar 2y agoIs The Gavin Belson Signature Edition Box is needed to run these?
- unixhero 2y agoNo, Jack Barkers' revamped version is needed.
- FirmwareBurner 2y agoI miss that show. Too bad it ended right before the AI hype.
- radicalbyte 2y agoFingers crossed that it'll do an Arrested Development.
- candiddevmike 2y agoRichard has a student with an idea involving AI and joins his company as an advisor but can't keep his opinions to himself. Ends up ruining the company because everything he touches turns to shit.
- hinkley 2y agoAh, I’d forgotten how much I wanted to strangle Richard by the end of that show. He kind of lived long enough to become the villain.
- FirmwareBurner 2y agoThat was the point of his character arc. All SV Big tech Bros turn into the people they sought to destroy. Chasing money and growth.
- fonsai 2y agoWasn't the last season mainly about AI?
- fxtentacle 2y agoI believe these are pure word tricks to suggest privacy without actually delivering it. As context, you need to remember that Google deleted their "Don't Be Evil" motto and became a defense contractor. The customer will most likely receive a black box owned and set up by Google. That means they have no way of knowing if the system inside is phoning home or being remote controlled by an US government agency, or not. You can then say that the model is hosted in your own data center, which might make some people feel good, but using it with personal information is still a violation of the GDPR. If Google, however, would make these boxes fully offline capable and I was also allowed to wipe all hard disks myself before returning it, that would convince me of their good intentions.
- greggsy 2y agoYou’re making a lot of assumptions there. It’s trivial to monitor traffic patterns from modern appliances, even if it’s encrypted. Also, companies have been sharing data with cloud security organisations for years now. There a robust means of assessing the risk. License agreements are a very real thing.
- fxtentacle 2y agoI don't fully disagree, but the only reason why this product is noteworthy is precisely because companies don't trust cloud providers with their data anymore. And while you might be able to prevent data exfiltration by monitoring the traffic patterns, you probably can't prevent sabotage that way.
- sitkack 2y agoI would have more problems with it even being on the network before we start talking about exfiltration.
- greggsy 2y agoI’m confused and not sure what you mean. Are you implying that Google will sell a product that is designed to ‘sabotage’ their own customer’s business? The legal and reputational damage far outweigh the value of stolen information. Or do you mean that it could be a vector of attack? That can happen with literally any piece of software, hardware, or appliance you install in or out of your datacentre.
- holografix 2y agoThis is obvious government contract baiting. Kudos though they might actually move some Google Distributed Cloud this way
- aduffy 2y agoI don’t think so. To my knowledge GCP has no approval for classified networks, which is by far the hardest part. Contrast with Azure OpenAI has been approved to run on government networks for over a year now. This feels like a play for companies in highly regulated industries, GCP has a notable list of biopharma customers.
- deleted 2y ago[deleted]
- ZeroCool2u 2y agoFedRAMP High is the mark you really want to hit for the US Government and GCP's service coverage is surprisingly broad in that realm.
- skybrian 2y agoFrom Google's blog post: > Our GDC air-gapped product, which is now authorized for US Government Secret and Top Secret missions, and on which Gemini is available, provides the highest levels of security and compliance.
- Maxious 2y ago>Today at Google Cloud Next, we're thrilled to announce another significant milestone for Google Public Sector: the authorization of Google Distributed Cloud Hosted (GDC Hosted) to host Top Secret and Secret missions for the U.S. Intelligence Community, and Top Secret missions for the Department of Defense (DoD). https://cloud.google.com/blog/topics/public-sector/google-public-sector-achieves-top-secret-and-secret-cloud-authorization https://cloud.google.com/blog/topics/public-sector/google-pu...
- aduffy 2y ago
- deleted 2y ago[deleted]
- miohtama 2y agoIs Gemini tied/benefitting from Google TPU hardware? Because you need hardware in the data center to run this, and I feel it is somewhat specialised.
- disgruntledphd2 2y agoThe Google blogpost notes that it's a partnership with Nvidia, so using cuda rather than TPUs apparently.
- stingraycharles 2y agoMakes complete sense, as NVidia has a lot more experience building these types of appliances.
- ddingus 2y agoSome one said it could also mean Google hardware has some advantage they would rather stay inside the G-silo.
- MortyWaves 2y agoGoogle abandoned Coral in true Google style.
- drdirk 2y agoGemini models are written in Jax which through the XLA compiler can be compiled either to TPU or GPU hardware. Performance may differ but Google (and Nvidia) are very interested in having good performance on both platforms.
- cavisne 2y agoThe raw computation is just a bunch of matrix multiplications in a row, most of the algorithmic complexity/secret stuff would be around scaling & efficiency. For training the model the HW is much more important as you need to scale up to as many chips as possible without being bottlenecked by the network. This would just be inference, and it doesn't need to be very efficient as its for on prem usage not selling API access. So you could strip out any efficiency secrets, and it would probably look like a bigger Gemma (their open source model). I wonder if they would/could try and strip out stuff like whatever tricks they use for long context + video support (both of which they are a bit ahead of everyone else on).
- amelius 2y agoIt's still an advertising company you're doing business with. I mean, would you buy cookies from a brand that is known for producing rodenticides?
- crazygringo 2y ago...yes? I mean, the company that makes Raid also makes Saran Wrap and Ziploc bags. Corporate conglomerates can do lots of things. The entire Google Cloud org is funded by regular customers paying money, not advertising.
- righthand 2y ago> The entire Google Cloud org is funded by regular customers paying money That can’t be true, how did they bootstrap it? How do they pay for R & D for their half baked offerings?
- wiseowise 2y agoDidn’t they start it for themselves and then saw an opportunity to make it a business? In a sense, yes, it was bootstrapped by ads and now pays for itself.
- surajrmal 2y agoWould you prefer VCs to have fronted the money to bootstrap it? How is it relevant today if ads are no longer enabling their financial viability? Ads largely finance Google's consumer offerings, not their enterprise offerings. Most enterprise Google customers understand the difference.
- tfsh 2y agoI don't think you're wanting to converse in good faith, but on the off chance this is a question - yes, GCP was revenue losing for a number of years, but since Q1 2023 they've been profitable. It takes money to bootstrap anything - obviously - this is the case for the vast majority of companies and their offerings, especially so for one which requires vast amounts of compute resources, SREs, legal, etc.
- aussieguy1234 2y agoWhat is the risk that some hacker could exfliltrate the weights?
- dist-epoch 2y agoVery low if they use confidential VMs (CPU rooted encryption). Just like the Xbox uses and remains unhacked 10 years later.
- onlyrealcuzzo 2y agoWhat would be the benefit of hacking the Xbox? What would you get?
- cavisne 2y agoSeems pretty high, this is an air gapped product so at some point the employees of whatever government they are giving it to would need to SSH into the VM's to load new weights etc. Lots of ways to make it tricky/watermark the weights though.
- roro_7 2y ago[dead]
- djoldman 2y agoGoogle announcement: https://cloud.google.com/blog/products/ai-machine-learning/run-gemini-and-ai-on-prem-with-google-distributed-cloud https://cloud.google.com/blog/products/ai-machine-learning/r...
- mkl 2y agoInteresting that the hardware is NVidia Blackwell, not Google TPUs. That means Google will likely have an energy efficiency and cost advantage, and keep their proprietary hardware out of other people's reach.
- bitexploder 2y agoPart of the reason for this is likely due to customers preference to have CUDA available which TPUs do not support. TPU is superior for many use cases but customers like the portability of targeting CUDA
- alienthrowaway 2y agoWhat are the pros of using CUDA-enabled devices for inference?
- bitexploder 2y agoMy limited understanding is that CUDA wins on smaller batches and jobs but TPU wins on larger jobs. It is just easier to use and better at typical small workloads. At some point for bigger ML loads and inference TPU starts making sense.
- j5r5myk 2y agoWhich use cases are TPUs superior for?
- riwsky 2y agoRunning Gemini models, for one.
- throwaway48476 2y agoLike you can with deep seek? Or will it be more complicated and expensive. I don't know who would actually want that.
- yoavm 2y agoAbsolutely many would, especially those with deep pockets. The biggest concern I'm hearing from companies adopting AI, for basically any use case, is data leaving their network. Especially (but not only) in the EU.
- throwaway48476 2y agoDeepseek is just the model weights. Nothing about it requires network access.
- surajrmal 2y agoFolks who would prefer to run deepseek are not in the end customer for this product. Deepseek doesn't provide a service contract.
- nsriv 2y agoThis might be a great way for them to strengthen their model through federated learning. https://federated.withgoogle.com/ https://federated.withgoogle.com/
- ein0p 2y agoThe whole point of deploying such things on-prem is air-gapping it from Google and its "learning".
- fc417fc802 2y agoThat's the point of the privacy scheme. It would only be able to learn things common to multiple clients. Private data wouldn't make it through the noise.
- ein0p 2y agoThat is beside the point. No lawyer will allow any "learning" on private data. Even without legal counsel, companies would be stupid to do it without compensation.
- fc417fc802 2y agoHow is it beside the point? Legal might initially advise against cloud storage but if you come back to them with one that's encrypted on the client side that likely changes things. Compensation could be of the form "we get a cheaper rate from Google" or even "this is the only form in which the service is offered" or perhaps "we aren't big enough to qualify for the fully airgapped offering".
- ein0p 2y agoThe point is, for this to work, your system has to be connected to the internet. And if that's the case, there's no material difference between hosting it on prem and hosting it on GCP (where Google can promise you that it won't exfiltrate data). That's if you trust Google. And if you don't (and you shouldn't - it's an ad / mass surveillance company first and foremost), your only option for sensitive data is to self host and air gap.
- jaggs 2y agoNot a cat's chance in hell that any eu organisation will rush to this offer right now. Or maybe ever in fact.
- geodel 2y agoHuh, how does that matter? Maybe OVH AI is fine for EU.
- tziki 2y agoI don't understand how Google is willing to do this but won't sell TPUs to other days centers. It should be obvious from Nvidia's market cap that they're missing a huge opportunity.
- dehrmann 2y agoThe only reasons I can think of is they see them as their secret sauce, they don't want to support them for customers long-term, or they don't have the foundry capacity.
- deleted 2y ago[deleted]
- paxys 2y agoIt's definitely #3. The GPUs have to first satisfy Google's own computing needs, and only then can they start selling them to others. Given how much training and inference the company is doing and how much demand there is internally it's very unlikely they are able to manufacture loads of extras, especially not profitably.
- azemetre 2y agoThis is a trillion dollar corporation. Would Google seriously have trouble raising the funds to build a chip fab? This seems like something they could do if they actually want to but I’d guess that would take actual leadership when they appear to have none. Especially in today’s political climate, building this in a purple state would ensure longevity too. The Trump admin would probably let them break ground immediately if they had the plans and I doubt democratic leadership would disagree either. So what gives?
- deleted 2y ago[deleted]
- motoxpro 2y agoTPU's are vertically integrated. If I was an investor and Google said they are going to now compete with Nvidia and TSMC I would take that as a sign they the leadership has completely lost the ability to see what their core competency is. Investing 100-200+ billion into fabs just to be on an equal playing field, is not it. Would be a poor allocation of capital. Especially since, as they build up capacity for their own jobs, they get to see the excess to customers.
- _cs2017_ 2y agoCurious if this was forced on Google Cloud by Sundar, or was it something that Google Cloud as an org wanted to do? At first glance, it seems Google Cloud might lose some revenue from customers who can now deploy Gemini in-house. On the other hand, it's not a complete loss, since presumably Google Cloud is still involved in providing some underlying tech? Not to mention, some customers would never consider using off-premises setup anyway.
- wmf 2y agoI assume Google Distributed Cloud is part of the larger Cloud org so they get the revenue either way. The on-prem version may even cost more.
- ein0p 2y agoWhat a sudden change of heart. Thank you, DeepSeek!
- bushbaba 2y agoReminds me of the Google search appliance.
- jonhohle 2y agoThat’s the first thing I thought of as well. I had to integrate one into our custom CMS early in my career. I vaguely remember explaining to management that I was not responsible for the order or quality of search results and tweaking queries (now prompts?) with hints to restrict searches to certain paths. It was such an opaque device, but provided better results than MySQL did at the time.
- rr808 2y agoI work for a bank and we're banned from using AI because of the privacy problem. I'd think there is a big market for a product like this.
- replete 2y agoGoogle AI rackmount appliance coming soon?