5 ms·
Release of Fugaku-LLM – a large language model trained on supercomputer Fugaku
- koito17 2y agoDoes anyone know how this model compares to GPT-4 for Japanese output? Taking a look at GPT-4o today, the Japanese example in the landing page feels unnatural and represents a huge regression from the quality I would expect of GPT-4. With that said, both GPT-4 and GPT-4o seem to do a good job at understanding the semantics of prompts written in Japanese. I would like to see how this model compares, given that it seems like it's trained with more Japanese data (but that may not necessarily be useful if all they did was scrape affiliate blogs)
- rgrieselhuber 2y agoI’ve gotten very good results with Japanese output in GPT-4 but it takes a little work.
- wahnfrieden 2y agoYou haven’t tried the recent Japanese specialized gpt4 variant? Hope it’s updated for gpt4o.
- glandium 2y agoThe Japanese specialized GPT-4 variant is not generally available, per https://openai.com/index/introducing-openai-japan/ https://openai.com/index/introducing-openai-japan/.
- alexey-salmin 2y ago> GPUs (7) are the common choice of hardware for training large language models. However, there is a global shortage of GPUs due to the large investment from many countries to train LLMs. Under such circumstances, it is important to show that large language models can be trained using Fugaku, which uses CPUs instead of GPUs. The CPUs used in Fugaku are Japanese CPUs manufactured by Fujitsu, and play an important role in terms of revitalizing Japanese semiconductor technology.
- LeoPanthera 2y agoARM CPUs, specifically. Fairly unusual in the Top500 list.
- apsec112 2y agoThis honestly feels kind of silly. Back-of-the-envelope, this training run (13B model on 380B tokens) could be done in two months on a single 8x H100 node using off-the-shelf software, at a cost of around $35K from a cloud compute provider. They don't seem to list training time, but this cluster appears to use ~3.5 MW of power, so it's going to burn something like $500/hr just in electricity costs.
- LeoPanthera 2y agoBut the whole point of this was to show that it could be done on CPUs during the worldwide shortage of GPUs. Doing it in the CPU was the whole point.
- fnordpiglet 2y agoIt could be done using abacuses too. A global shortage of GPUs is probably better addressed by producing more GPUs and making techniques more efficient.
- slt2021 2y agoI would love to see human abacus calculating gradients like in 3 body problem https://i.imgur.com/pdz7xGh.jpeg https://i.imgur.com/pdz7xGh.jpeg
- fnordpiglet 2y agoYes best scene on tv ever.
- Eisenstein 2y agoYou are right. I assume that you have a fab ready to go and are licensed to produce CUDA capable GPUs?
- fnordpiglet 2y agoI’m certain they’re being built.
- NKosmatos 2y agoFor those curious about Fugaku, it’s currently the 4th fastest supercomputer in the TOP500 list: https://www.top500.org/system/179807/ https://www.top500.org/system/179807/
- Thaxll 2y agoSo every day we're going to get post about new x.y.z LLM?
- delichon 2y agoThese are bubbling to the top due to high interest. When the interest subsides so will the posts. Are we fascinated by the wrong things?
- Delmololo 2y agoYes! Impressive advancements right?
- Art9681 2y agoIt's almost as if LLMs have become an integral part of a modern development workflow with or without your disapproval! There would be less friction without it.
- sveinek 2y agoThere is probably a simple answer to this question, but why isn't it possible to use a decentralized architecture like in crypto mining to train models?
- CaptainOfCoit 2y agohttps://scholar.google.com/scholar?q=distributed+training https://scholar.google.com/scholar?q=distributed+training
- LeoPanthera 2y agoWouldn't every single participant need a copy of the entire training set?
- samus 2y agoThat's the next big problem. And there need to be mechanisms to ensure that the network is not poisoned with undesirable input.
- andai 2y agoCan you copy a neural network, train each copy on a different part of the dataset, and merge them back together somehow?
- bilbo0s 2y agoThere are a lot of issues with federated learning. Really depends on your problem, but in practice, the answer is usually "no".
- mirekrusin 2y agoNo. Training is offset relative to starting point. If you distribute it from same point you'll have bunch of unrelated offsets. It has to be serial - output state of one training is input state of the next. If you could do it, we'd already have SETI like networks for AI.
- 2y ago
- Foobar8568 2y agoIt's based on a Fujitsu custom ARM, with 32GB HBM2 and 512b SIMD. There have been a few discussions over the years. I am not sure how such chips could replace GPU if there is a shortage of them as it's highly customized and fairly expensive to get the hand of them. https://www.fujitsu.com/global/about/resources/publications/technicalreview/2020-03/article03.html https://www.fujitsu.com/global/about/resources/publications/...