5 ms·
When you say its hard to beat Google and their TPU clusters, do you mean Azure has inferior capability? Isnt OpenAI limited by what they can do on Azure?
by anaganisk 4y ago
When you say its hard to beat Google and their TPU clusters, do you mean Azure has inferior capability? Isnt OpenAI limited by what they can do on Azure?
- albertzeyer 4y agoI don't have numbers, so I don't really know, but yes, I would assume Google (incl DeepMind) has more computing power than Microsoft + OpenAI.
- dubcanada 4y agoWhy? Both are huge super computer users, and both have similar setups. Both own a search engine, both run ads. I don't really see any reason to suggest Microsoft is less capable than Google. Unless I am missing something?
- adammarples 4y agoGoogle has been at the forefront AI for years, decades maybe. Do azure even have tpus? It's reasonable to assume Google have close to the best ML compute resources out of probably everyone in the world.
- visarga 4y agoWhere do you think GPT-3 was trained and currently runs? In Azure. It doesn't need TPUs.
- adammarples 4y agoGpt3 is an impressive model, I think it took 512 GPUs 10 days to train though. This alone doesn't mean that Microsoft has more computing power than Google
- hot_gril 4y agoIt does mean that you don't need the most computing power to train a successful model.
- albertzeyer 4y agoMy assumption is based on this: For a long time, Microsoft's core business was Windows and (offline) Office. You don't need a super cluster for that. Google on the other side always had search as a service, available to the public, which needs more compute. So historically, Google needed more compute than Microsoft. Then, from experience, as I do research on neural networks, read papers in the field, Google/DeepMind were always more active in this field than Microsoft, so again, historically, they needed more computing power because of this. Microsoft has catched up, but still, I think Google/DeepMind is more active in the field. Then, Google has TPUs. I think this gives them computing power much more cost effectively than when you need to buy Nvidia GPUs. This is maybe the biggest argument. But in any case, as you say, both companies are big enough to easily train sth like ChatGPT, or also sth like GPT4, however big it might be. Their computing power is almost infinite (for today's standards), so it doesn't really matter who has more.
- rrdharan 4y agoHistorically Google far outspent Microsoft on their “super computers”, though in recent years MS has been closing the gap. https://www.platformonomics.com/2022/02/follow-the-capex-cloud-table-stakes-2021-retrospective/ https://www.platformonomics.com/2022/02/follow-the-capex-clo...
- hot_gril 4y agoGoogle supposedly hires more of the ML experts and custom hardware experts. Whenever someone says that GCP is inferior to the competition, it's usually followed by "except in ML."
- summerlight 4y agoI'm also interested in Azure's FPGA instance capacity which is a comparable counterpart to Google's TPU AFAIK. From what I know, Google has a large number of TPU units (not sure about the exact numbers, but I would assume this would exceed 5 digits?) and also has the luxury of sharing TPU instances with products that operate one of the largest models on the planet, namely search and ads. So Google has some advantages on flexibility of resource allocation if it really wants to prioritize AI stuffs.