7 ms·
Why is Japan still investing in custom floating point accelerators?
- numpad0 1y agoIt's unfortunate that they don't sell them on open markets. There are few of these accelerators that could threaten NVIDIA monopoly if prices(and manufacturing costs!) were right.
- WithinReason 1y agoThe hardware is the easy part of accelerating NN training. Nvidia's software and infrastructure is so well designed and established that no competitor can threaten them even if they give away the hardware for free.
- saagarjha 1y agoI don't know about well designed but it's definitely established.
- CoastalCoder 1y agoCould you elaborate? I've only done a little work on CUDA, but I was pretty impressed with it and with their NSys tools. I'm curious what you wish was different.
- saagarjha 1y agoI actually really hate CUDA's programming model and feel like it's too low-level to actually get any productive work done. I don't really blame Nvidia because they basically invented the programmable GPU and it wouldn't be fair to have them also come up with the perfect programming model right out of the gate but at this point it's pretty clear that having independent threads work on their own programs makes no sense. High performance code requires scheduling across multiple threads in a way that is completely different if you are coming from CPUs. Of course, one might mention that GPUs are nothing like CPUs–but the programming model works super hard to try to hide this. So it's not really well designed in my book. I actually quite like the compilers that people are designing these days to write block-level code, because I feel like it better represents the work people want to do and then you pick which way you want it lowered. As for Nsight (Systems), it is…ok, I guess? It's fine for games and stuff I guess but for HPC or AI it doesn't really surface the information that you would want. People who are running their GPUs really hard know they have kernels running all the time and what the performance characteristics of them are. Nsight Compute is the thing that tells you that but it's kind of a mediocre profiler (some of this may be limitations of hardware performance counters) and to use it effectively you basically have to read a bunch of blog posts by people instead of official documentation. Despite not having used it much, my impression was that Nvidia's "moat" was that they have good networking libraries, that they are pretty good (relatively) and making sure all their tools work, and they have had consistent investment on this for a decade.
- electroglyph 1y agoi mean, it could be worse... it could be Vulkan
- jandrewrogers 1y agoGPUs are a type of barrel processor, which are optimized for workloads without cache locality. As a fundamental principle, they replace the CPU cache with latency hiding behavior. Consequently, you can't use algorithms and data structures designed for CPUs, since most of those assume the existence of a CPU cache. Some things are very cheap on a barrel processor that are very expensive on a CPU and vice versa, which changes the way you think about optimization. The wide vectors on GPUs are somewhat irrelevant. Scalar barrel processors exist and have the same issues. A scalar barrel processor feels deceptively CPU-like and will happily compile and run normal CPU code. The performance will nonetheless be poor unless the C++ code is designed to be a good fit for the nature of a barrel processor, code which will look weird and non-idiomatic to someone who has only written code for CPUs. There is no way to hide that a barrel processor is not a CPU even though they superficially have a lot of CPU-like properties. A barrel processor is extremely efficient once you learn to write code for them and exceptionally well-suited to HPC since they are not latency-sensitive. However, most people never learn how to write proper code for barrel processors. Ironically, barrel processor style code architecture is easy to translate into highly optimized CPU code, just not the reverse.
- glitchc 1y agoI wanted to upvote you originally, but I'm afraid this is not correct. A GPU is not a barrel processor. In a barrel processor a single context is switched between multiple threads after each instruction. A barrel processor design has a singular instruction pipeline and a singular cache across all threads. In a GPU, due to the independence of the execution units, those threads will execute those instructions concurrently on all cores, as long as a program-based instruction dependency between threads is not introduced. It's true parallelism. Furthermore, each execution unit embeds its own instruction scheduler, it's own pipeline and its own L1 cache (see [1] for NVidia's architecture). [1] https://docs.nvidia.com/deeplearning/performance/dl-performance-gpu-background/index.html#gpu-arch https://docs.nvidia.com/deeplearning/performance/dl-performa...
- WithinReason 1y agoWho has better software than Nvidia for NN training? Meaning the least amount of friction getting a new network to train.
- saagarjha 1y agoJust because their tools are the best doesn't mean they are designed well.
- jacquesm 1y agoI've used DSPs, custom boards with compute hardware (FPGA image processing), and various kinds of GPUs. I would have a very hard time trying to point to ways in which the NVIDIA toolkit could be compared to what's out there and not come away with a massive sense of relief. For the most part 'it just works', the models are generic enough that you can actually get pretty close to the TDP on your own workloads with custom software and yet specific enough that you'll find stuff that makes your work easier most of the time. I really can't complain, now, FPGAs, however... And if there ever is a company that comes out and improves substantially on this I'll be happy for sure but if you asked me off the bat what they should improve I honestly wouldn't know, especially not taking into account that this was an incremental effort over ~2 decades and that originated in an industry that has nothing to do with the main use case today and some detours into unrelated industries besides (crypto, for instance). From fluid dynamics, FEA, crypto, gaming, genetics, AI and many others with a single generic architecture and delivering very good performance is no mean feat. I'd love to hear in what way you would improve on their toolset.
- programjames 1y agoNot the guy you replied to, but here are some improvements that feel obvious: 1. Memory indexing. It's a pain to avoid banking conflicts, and implement cooperative loading on transposed matrices. To improve this, (1) pop up a warning when banking conflicts are detected, (2) make cooperative loading solved by the compiler. It wouldn't be too hard to have a second form of indexing memory_{idx} that the compiler solves a linear programming problem for to maximize throughput (do you spend more thread cycles cooperative loading, or are banking conflicts fine because you have other things to work on?) 2. Why is there no warning when shared memory is unspecified? It isn't hard to check if you're accessing an index that might not have been assigned a value. The compiler should pop out a warning and assign it to 0.0, or maybe even just throw an error. 3. Timing - doesn't exist. Pretty much the gold standard is to run your kernel 10_000 times in a loop and subtract the time from before and after the loop. This isn't terribly important, I'm just getting flashbacks to before I learned `timeit` was a thing in Python.
- KeplerBoy 1y agoIt's not all about NNs and AI. Take a look at the Top500, a lot of people are doing classical HPC work on Nvidia GPUs, which are increasingly not designed for this. Unfortunately the HPC market is just a lot smaller than the AI bubble.
- rwmj 1y agoIf the hardware isn't available at all, we'll never find out if the software moat could be overcome.
- DrNosferatu 1y ago> if they give away the hardware for free. Seriously doubt that: free hardware (or 10s of bucks) would galvanize the community and achieve huge support - look at the Raspberry Pi project original prices and the consequences.
- deleted 1y ago[deleted]
- DrNosferatu 1y agoIn fact, if any such thing would happen, I would wager Nvidia stock would tank massively. Say, release has extensions to a RISC-V design.
- DrNosferatu 1y ago*as :D
- londons_explore 1y agoThe math of NN training isn't complex at all. Designing the software stack to make a new pytorch backend is very doable with the budgets these AI companies have. I suspect that whenever you look like you're making good progress on this front, nvidia gives you a lot of chips for free on condition you shelve the effort though! The latest example being Tesla, who were designing their own hardware and software stack for NN training, then suspiciously got huge numbers of H100's ahead of other clients and cancelled the dojo effort.
- AlotOfReading 1y agoI doubt that's what happened. They had designs that were massively expensive to fab/package, had much worse performance than the latest Nvidia hardware, and still needed massive amounts of custom in-house development. To combat all of these issues, they were fighting with Nvidia (and losing) for access to leading edge nodes, which kept going up in price. Their personnel costs kept rising as the company became more politicized, people left to join other companies (e.g. densityai), and they became embroiled in the salary wars to replace them. My suspicion is that Musk told them to just buy Nvidia instead of waiting around for years of slow iteration to get something competitive. The custom silicon I was involved with experienced similar issues. It was too expensive and slow to try competing with Nvidia, and no one could stomach the costs to do so.
- deleted 1y ago[deleted]
- nromiun 1y agoI don't know why you are getting downvoted. This is 100% true. It's not like you can take any random data and train it into a NN. You have to transform the data, you have to write the low level GPU kernels which will actually run fast on that particular GPU, you also have to get the output and transform that as well. All of this is hard and very much impossible to create from scratch. If people use PyTorch on a Nvidia GPU they are running layers and layers of code written by those that know how to write fast kernels for GPUs. In some cases they use assembly as well. Nvidia stuck to one stack and wrote all their high level libraries on it, while their competitors switched from old APIs to new ones and never made anything close to CUDA.
- woooooo 1y agoBecause in the context of LLM transformers, you really just need matrix multiplication to be hyper-optimized, it's 90-99% (citation needed) of the FLOPs. Get some normalization and activation functions in and you're good to go. It's not a massive software ecosystem. CUDA and CUBLAS being capable of a bunch of other things is really cool, and would take a long time to catch up with, but getting the bare minimum to run LLMs on any platform with a bunch of GDDR7 channels and cores at a reasonable price would have people writing torch/ggml backends within weeks.
- nromiun 1y agoHave you tried to write a kernel for basic matrix multiplication? Because I have and I can assure you it is very hard to get 50% of maximum FLOPs, let alone 90%. It is nothing like CPUs where you write a * b in C and get 99% of the performance by the compiler. Here is an example of how hard it is: https://siboehm.com/articles/22/CUDA-MMM https://siboehm.com/articles/22/CUDA-MMM And this is just basic matrix mult. If you add activation functions it will slow down even more. There is nothing easy about GPU programming, if you care about performance. CUDA gives you all that optimization on a plate.
- woooooo 1y agoWell, CUDA gives you a whole programming language where you have to figure out the optimization for your particular card's cache size and bus width. I'm saying the API surface of what to offer for LLMs is pretty small. Yeah, optimizing it is hard but it's "one really smart person works for a few weeks" hard, and most of the tiling techniques are public. Speaking of which, thanks for that blog post, off to read it now.
- pclmulqdq 1y agoThey do sell these on the open market. You just have to be in the market for an entire cluster. The minimum order quantity for Pezy is several racks.
- numpad0 1y agoI thought they're more like "wire us our share of METI grant, we'll forward it to TSMC". Besides they wouldn't be going anywhere if that was chasing away 100% of customers. Another one of these I still sometimes think about is NEC VectorEngine - they had 5 TFLOPS FP32 with 48GB of HBM2 totaling 1.5TB/s bandwidth at $10k in 2020. That was within a digit or two against NVIDIA at basically the same price. But they then didn't capitalize on it, just kept delivering to national institutes in ritualistic manners. I do have basic conceptual understanding of these grant businesses and have vague intuitions as to how bureaucracy wants substantial capital investments and report files without commercial capitalizations, with emphasis on the last part, as it would disrupt internal politics inside government agencies and also creates unfair government competitive pressure against civilian sectors, but at some point it starts looking like cash campfires. I don't know exactly how slow are M4 Mac Studios relative to NVIDIA Tesla clusters normalized for VRAM, but they're considered comparable regardless just because they run LLMs at 10-20 tok/s. So it's just, unfortunate, that these accelerators of basically same nature as M-series CPUs are built, kept on idle, and then recycled. The one that is in my mind as "no way these brochure figures are real" is PFN MN-Core - though it looks like they might be doing an LLM specific variant in the future. Hopefully they retail them.
- tho23i23434234 1y agoI've come to wonder if this just because the culture of Japan itself has been so "crabby". It's just too inward looking these days - probably why technical innovations in Japan don't get shaped to meet the worlds needs, but gets sold as if it were a luxury artisan-product (ala "swiss-made" stuff). The trouble with people who criticize Japan (incl. Japanese) is that they think this is because of "old people & culture" - but actually, no, the "old" Japanese (in the 1900s-1980s) seemed to have been extraordinarily curious about the world, and also very clever in marketing things. The issue is most definitely "modern", but ofc. saying that is verboten in the dogma of liberalism.
- numpad0 1y agoYou might be onto something. Anecdotally, I just came across some armchair economist lamenting lack of aptitude among Japanese startups for foreign currency acquisition, and I could only agree - I can't recall many Japanese startups or corporate business expansions primarily focusing on foreign sales. The mental model is always to get rich quick within Japan and/or South Asia and retire. "The world" outside is treated like tiny separate bonus rooms. Thinking about Japanese economy in general and how it hadn't grown in 30-40 years: 40 years is technically two generations, but life in Japan hadn't deteriorated meaningfully during that period. Substantial socio-political improvements were made, university entrance had rose somewhat absurdly high, some new infrastructures were built, convenience store sandwich prices hasn't doubled, overtimes and harassments at workplaces are way more strictly scrutinized. There's the problem of employment ice age, but it's not as bad as the collapse of the Soviet Union; not at "Vladimir Putin was laid off KGB and drove taxi to make ends meet" levels, only "PhDs drove trucks". So "Japan still using FAX" narratives only partially make sense. Overall, it does feel that there is something strange is going on in this country, something like effective isolationism. I guess my point is... it's unfortunate that these efforts and costs spent goes to waste, and we don't know why it's only built to be recycled.
- pezezin 1y ago> The trouble with people who criticize Japan (incl. Japanese) is that they think this is because of "old people & culture" - but actually, no, the "old" Japanese (in the 1900s-1980s) seemed to have been extraordinarily curious about the world, and also very clever in marketing things. The issue is most definitely "modern", but ofc. saying that is verboten in the dogma of liberalism. I have been living in Japan for the past 7 years, and in my experience all generations are guilty. I work for a big international research project, so I have met many old Japanese professors and high-level researchers. They all lament how their generation wanted to go abroad, see the world, and change things, whereas their new graduates don't even want to learn English, just stay in their little Japanese bubble and do what they are told to do. But for every one of those outgoing old Japanese people, you meet 10 who are dead set in their ways and don't want any change, and with the population getting older, the country has become stagnant.
- aredox 1y agoIs there a know secondhand marketplace for retired supercomputer hardware?
- Maxious 1y agoYou can pickup previously state of the art supercomputers at auction. https://news.ycombinator.com/item?id=40197277 https://news.ycombinator.com/item?id=40197277
- thiago_fm 1y agoGreat article documenting PEZY. It's incredible how close they are from NVidia despite being a very small team. To me, this looks like a win. Governments are there to finance projects like this that enable the country to have certain skillsets that wouldn't exist otherwise because of other countries having better solutions in the global market.
- eru 1y agoGovernments are terrible at picking winners.
- actionfromafar 1y agoEveryone is, and what survives, survives. But what governments often can do, is break local optimums clustering around the quarter economy and take moonshot chances and find paths otherwise never taken. Hopefully one of these paths are great. The difficult thing becomes deciding when to pull the plug. Is ITER a good thing or not? (Results wise, it is, but for the money? Who can tell really.)
- eru 1y agoHave a look at the history of the share prices and the profits of eg Amazon and Tesla, and tell me again that the market only looks at the quarter.
- eru 1y ago> Everyone is, and what survives, survives. Well, at least VCs don't burn through my tax money, whilst they are failing at picking winners.
- elzbardico 1y agoThere wouldn't be a Silicon Valley without the DARPA and NASA.
- p_l 1y agoOr just plain military procurement, even before ARPA existed.
- kragen 1y agoFascinating. https://en.m.wikipedia.org/wiki/Single_program,_multiple_data https://en.m.wikipedia.org/wiki/Single_program,_multiple_dat... explains the relation to SIMT.
- sylware 1y agoLast time I heard about that it was for "super computers": nearly or even faster than the alternatives with a massive energy consumption advantage.
- pclmulqdq 1y agoPezy and the other Japanese native chips are first and foremost about HPC. The world may have picked up AI in the last 2 years, but the Japanese chipmakers are still thinking primarily about HPC, with AI as just one HPC workload. These Pezy chips are also made for large clusters. There is a whole system design around the chips that wasn't presented here. The Pezy-SC2, for instance, was built around liquid immersion cooling. I am not sure you could ever buy an air-cooled version.
- VladVladikoff 1y ago>liquid immersion cooling Is the whole board submersed in liquid? Or just the processor?
- neom 1y agohttps://www.wikiwand.com/en/articles/Gyoukou https://www.wikiwand.com/en/articles/Gyoukou "Each immersion tank can contain 16 Bricks. A Brick consists of a backplane board, 32 PEZY-SC2 modules, 4 Intel Xeon D host processors, and 4 InfiniBand EDR cards. Modules inside a Brick are connected by hierarchical PCI Express fabric switches, and the Bricks are interconnected by InfiniBand."
- numpad0 1y agoI remember some offhand remarks on that. Apparently the rooms for these systems had cheap ladles hanged somewhere and engineers would have fun scooping out water puddles collecting on top of enclosures full of fluorinert coolants. That's tanks full of PFAS in layman's terms...
- kragen 1y agoBut, importantly, nontoxic PFAS.
- renewiltord 1y agoFunny site. Seems to be a reskin of the Wikipedia article https://en.wikipedia.org/wiki/Gyoukou https://en.wikipedia.org/wiki/Gyoukou
- ghaff 1y agoIt may also be worth noting that Japan has a pretty long history of marching to their own drummer in computing. They either created their own architectures or adopted others after pretty much everyone had moved on.
- curtisszmania 1y ago[dead]
- johnklos 1y agoWhen you're building your own CPUs, why be beholden to US companies for GPUs? This makes perfect sense. GPUs are great if your workload can use them, but not so great for more general tasks. These are more appropriate to more traditional supercomputing tasks, as in they're not optimized for lower precision AI stuff, like NVIDIA GPUs are.
- Aissen 1y agoBecause the LLM craze has rendered last-gen Tensor accelerators from NVIDIA (& others) useless for all those FP64 HPC workloads. From the article: > The Hopper H200 is 47.9 gigaflops per watt at FP64 (33.5 teraflops divided by 700 watts), and the Blackwell B200 is rated at 33.3 gigaflops per watt (40 teraflops divided by 1,200 watts). The Blackwell B300 has FP64 severely deprecated at 1.25 teraflops and burns 1,400 watts, which is 0.89 gigaflops per watt. (The B300 is really aimed at low precision AI inference.)
- kbolino 1y agoDo cards with intentionally handicapped FP64 actually use anywhere near their TDP when doing FP64? It's my understanding that FP64 performance is limited at the hardware level--whether by fusing off the extra circuits, or omitting them from the die entirely--in order to prevent aftermarket unlocks. So I would be quite surprised if the card could draw that much power when it's intentionally using only a small fraction of the silicon.
- Aissen 1y agoIt's really to save die space for other functions, AFAIU there is no fusing to lock the features or anything like this.
- kbolino 1y agoI'm finding conflicting info on this. It seems to be down to the specific GPU/core/microarchitecture. In some cases, the "missing" FP64 units do physically exist on the dies, but have been disabled--likely some of them were defective in manufacturing anyway--and this disabling can't be undone with custom firmware AFAIK (though I believe modern nVidia cards will only load nVidia-signed firmware anyway). Then, there are also dies that don't include the "missing" FP64 units at all, and so there's nothing to disable (though manufacturing defects may still lead to other components getting disabled for market segmentation and improved yields). This also seems to be changing over time; having lots of FP64 units and disabling them on consumer cards seems to have been more common in the past. Nevertheless, my point is more that if FP64 performance is poor on purpose, then you're probably not using anywhere near the card's TDP to do FP64 calculations, so FLOPS/watt(TDP) is misleading.
- andrepd 1y agoI wonder how much progress (if any) is being done on floating point formats other than IEEE floats; on serious adoption in hardware in particular. Stuff like posits [1] for instance look very promising. [1] https://posithub.org/docs/posit_standard-2.pdf https://posithub.org/docs/posit_standard-2.pdf
- adgjlsfhk1 1y agothe problem with posits is that they aren't enough better to be worth a switch. switching the industry over would cost billions in software rewrites and there are benefits, but they are fairly marginal.
- hedgehog 1y agoFor deep learning workloads the software for posits isn't the issue, it's doing anything that's not NVIDIA if you want to do it as a standalone product. For NVIDIA it's likely the penalty of not being able to share logic with standard size IEEE floats. If adopting posits allowed significantly smaller data types then NVIDIA would likely have adopted already.
- andrepd 1y agoBut 8-bit posits are actually a very nice alternative for deep learning, especially when using the quire for dot products!
- hedgehog 1y agoI don't disagree, it's just that the advantage hasn't yet been shown to be big enough to justify dedicating die area in a mainstream chip. There's potential there, if I were designing an accelerator today I would look hard at posits and variations of blocked representations especially around four bits. A few years back I got to have coffee with John Gustafson which was pretty neat and got me more excited about the idea.
- nibles_and_bits 1y ago
- retube 1y agoWhat is an "accelerator" in this context?
- markstock 1y agoSomething doesn't add up here. The listed peak fp64 performance assumes one fp64 operation per clock per thread, yet there's very little description of how each PE performs 8 flops per cycle, only "threads are paired up such that one can take over processing when another one stalls...", classic latency-hiding. So the performance figures must assume that each PE has either an 8-wide SIMD unit (and 16-wide for fp32) or 8 separately schedulable execution units, neither of which seem likely given the supposed simplicity of the core (or 4 FMA EUs). Am I missing something?
- nxobject 1y agoInteresting that they’re investing in standard _AI_ toolchains, rather than standard HPC toolchains, even though I imagine Japanese supercomputing has more demand for the latter.
- Avlin67 1y agoyou can get 8TFlops of fp64 on xeon 6980P which is 6K€ now
- monocasa 1y agoAdding to what everyone else has said, Japan is known to be a threshold nuclear state (from a weapons perspective). They explicitly stay around just weeks away from being able to perform a nuclear weapons test, and they are commonly referred to being "a screwdriver's turn" away from having a nuclear weapon. They have massive government investment in not only maintaining that status, but also doing so on a completely domestic supply chain as much as possible. Therefore they have the same need for supercomputers that the US national labs do (perhaps more so, since they're even more reliant on simulation), and heavily prefer locally sourced pieces of that critical infrastructure. I wouldn't be surprised if an incredibly large part of the local push for Rapidus is to pull them off of TSMC and the supply chain risk for their nuclear program in case the whole China/Taiwan thing comes to a head.
- runlaszlorun 1y agoOh that's very interesting. Unfortunately for all of us many countries are revisiting nuclear ambitions it seems, but I can see how that makes sense from the Japanese perspective, given an environment with more aggression in general and a lot much reliable US as an ally.
- hungmung 1y agoFunny because I met quite a few Japanese that liked Trump leading up to his first term. They thought he was really going to fuck with China.
- Jabbles 1y ago> They explicitly stay around just weeks away from being able to perform a nuclear weapons test Do you have a citation for "weeks away"? Wikipedia only says "within one year": https://en.wikipedia.org/wiki/Japanese_nuclear_weapons_program#De_facto_nuclear_state https://en.wikipedia.org/wiki/Japanese_nuclear_weapons_progr...
- skhr0680 1y agoThere’s no citation, because why would there be? But: if you consider the amount of nuclear generating capacity has (4th in the world, more than Russia), and its advanced space program, “within one year” probably means closer to “weeks or months” than “three hundred and sixty four days”.