6 ms·
SiFive Rolls Out RISC-V Cores Aimed at Generative AI and ML
- throwaway_5753 3y agoMore of a press release than anything else. Could one actually train something interesting on a processor like this? I assume you'd need a pretty custom toolchain. Hard to imagine where these processors would have a competitive advantage (but would love to hear if anyone has encountered something like this in the wild)
- LukeB42 3y ago>you'd need a pretty custom toolchain Just port/extend pytorch to make it easy to utilize fused ops.
- elwypea 3y agoEasier said than done. Even with Google level resources, TPU support for pytorch is patchy (https://arxiv.org/abs/2309.07181 https://arxiv.org/abs/2309.07181). Device abstraction is not great, assumes CUDA in unexpected places.
- omneity 3y agoPytorch XLA is such a pain to use. And once you go TPU you need the same energy to switch back, so you can’t quickly test out how it performs on your problem.
- lumost 3y agoOne of the big reasons custom hardware solutions struggle. IMO - you’d have better luck as a hardware vendor implementing an LLM toolchain and bypassing a general purpose DL framework. At the very least you should be able to post impressive results with this approach rather than a half baked pytorch port.
- elwypea 3y agoThat might be good enough to get a hardware startup acquired, but not good enough to get major sales. Users want pytorch and negligible switching cost between chips. Bigger problem for startups trying to muscle in on LLMs is that there isn't much room for improvement on existing solutions to do something radically different.
- lumost 3y ago>Bigger problem for startups trying to muscle in on LLMs is that there isn't much room for improvement on existing solutions to do something radically different. aye - unless you are able to notch a 10x cost/performance improvement. The migration overhead will just make it not worth it to switch.
- omneity 3y agoI feel like that would make it harder for a vendor to keep up with the industry. Say you took all the effort in the world to build your custom LLM toolchain to train a Llama on custom hardware. And then suddenly someone comes up with LoRA. You didn't even finish porting it to your toolkit then someone comes up with GPTQ. Can't keep up with a custom toolchain imo. It's like a forked linux kernel. Eventually you're gonna have to upstream if you're serious about it, which is what AMD is actively doing with pytorch for ROCm (masquerading it as CUDA for compatibility).
- ronsor 3y agoI disagree. llama.cpp[0] is a good counterpoint to this, since it uses a custom ML framework created from scratch. Despite not having the developer team of a large company, it still keeps up with many of the advancements in LLMs. [0] https://github.com/ggerganov/llama.cpp https://github.com/ggerganov/llama.cpp
- elwypea 3y agollama.cpp is not necessary for creating lots of demand for the chip it was originally written for (Apple M1), whereas new hardware vendors need to demonstrate they can plugin to existing tools to generate enough demand to ship in volume.
- adolph 3y agoTypically Google resources go to TF before PyTorch, no?
- elwypea 3y agoEven after prioritising tensorflow, keras, jax etc., they can still afford to have a very large team working on torch_xla and still hedge their bets with a separate team on torch_mlir.
- visarga 3y agoThe Groq AI chip startup has solved this problem. They don't use hand written kernels at all, instead they use a compiler, and they have the top speed in the world on LLaMA2-70B, 240tokens/s. https://www.youtube.com/@GroqInc/videos https://www.youtube.com/@GroqInc/videos Other interesting Groq tidbits - their models are deterministic, the whole system up to thousands of chips runs in sync on the same clock, memory access and network are directly controlled without any caches or intermediaries so they also run deterministically. That speeds up communication and allows automatic synchronisation across thousands of chips running as one single large chip. The compiler does all the orchestration/optimisation. They can predict the exact performance of an architecture from compile time. What makes Groq different is that they started from the compiler, and only later designed the hardware.
- elwypea 3y agoWhat is the pass rate on torchbench? This gives a more realistic measure of how good a vendor's pytorch support is. All the big chip startups have their own pytorch compiler that works on the examples they write themselves. From what I've seen of Groq it doesn't appear to be any different. The problem is that pytorch is incredibly permissive in what it lets users do. torch.compile is itself very new and far from optimal.
- itsdevin 3y ago[flagged]
- itsdevin 3y ago[flagged]
- sanxiyn 3y agoNote that Google planned to use SiFive X280 for TPU, although I don't know the current status. https://www.semianalysis.com/p/sifive-powers-google-tpu-nasa-tenstorrent https://www.semianalysis.com/p/sifive-powers-google-tpu-nasa...
- snvzz 3y agoThere were talks covering it as a success story in last years' RISC-V Summit.
- knorker 3y agoOooh, I'm also excited about Vector 1.0 in silicon, on those cores.
- snvzz 3y agoLook into SG2038[0] and Kendrite K230[1]. Some k230 are shipping this month to developers. An explosion of Vector 1.0 chips and boards is imminent. 0. https://twitter.com/sophgotech/status/1713852202970935334 https://twitter.com/sophgotech/status/1713852202970935334 1. https://www.canaan.io/product/k230 https://www.canaan.io/product/k230
- Avlin67 3y agoputting AI everywhere to get more money from investors
- baq 3y agoexactly right, buzzword-driven capital acquisition play
- jug 3y agoYup, there's definitely a window of opportunity here before the market has matured and the hype is still on. I honestly blame myself for being too lazy to capitalize on my basic AI skillz and get paid $$$ for basic advice/guidance to organizations and municipalities here. I know some that are cashing in right now.
- loudmax 3y agoYes, there's a lot of hype and buzzwords. Certainly companies are slapping the "AI" label onto their products and jumping on the bandwagon in order to gin up interest. But there's a real technological change going on. Machines are capable of performing tasks that were nearly unthinkable two years ago. There's real progress beneath the hype. AI workloads are going to be dramatically more significant in the next decade. A shift like this opens up real opportunities for smaller players to capture enormous market share. If SiFive manages to develop the best AI hardware, they could become bigger than Apple. Or they could fumble and go out of business. Either of these outcomes is genuinely possible when this kind of technological shift happens.
- giantrobot 3y ago> But there's a real technological change going on. Machines are capable of performing tasks that were nearly unthinkable two years ago. There's real progress beneath the hype. Like what? What's unthinkable tasks are so useful thanks to AI?
- i67vw3 3y agoMid-journey is a good example.
- Andrew018 3y ago[dead]
- snvzz 3y agoMildly related, SOPHGO's pre-announced[0] a new chip based on SiFive's P670+X280. These are CPUs with Vector 1.0. Details will follow in the actual announcement, tomorrow. 0. https://twitter.com/sophgotech/status/1713852202970935334 https://twitter.com/sophgotech/status/1713852202970935334
- j16sdiz 3y agoIMO, there are too many extension for RISC-V. They need to consolidate some of them.
- snvzz 3y agoLook into the profiles spec, and the platforms spec while at it.
- RetroTechie 3y agoWhy? Extensions that are not implemented in silicon: who cares. For extension that are implemented, better if there's a standard. As for the # of them: just look what's been bolted onto x86 over the years, and their counterparts in other ISAs like Arm. Consolidation will happen in the market, over time. Some combinations of extensions will be common, some rare. And due to the modularity, not a big issue for software.
- jorvi 3y agoIsn’t that one of the problems of X86? Extension that get used just barely enough that you have to include them, but not enough that they should actually warrant transistors.
- bavell 3y agoUnavoidable tradeoff? Fundamental tension between modularity and efficiency.
- nickik 3y agoUnless you want to make application specific processors that's just a reality. We are gone include things like crypto even if the computer never uses any crypto. The thing is, for some things you can get 100x improvement. Even if not used often its gone make a difference. But for them most part, things that are part of RISC-V RVA23 profile make sense to be on a standard PC or Server. Asked differently what isn't really necessary that is in RISC-V? One could make arguments for a simpler vector extensions for example. And a few other tiny things you can argue about. Overall RISC-V RVA23 is a reasonable and feature complete set for a modern processor. x86 on the other-hand has quite a few things that are not really needed if you redesigned it.
- gigatexal 3y agoHow performant are SiFive boards for general computing?
- unethical_ban 3y agoWhat do my HN colleagues think about the assertion by some in the US congress that RISC-V is a clever conspiracy by China to dodge sanctions and get clever American universities to do their research for free? While I think "information wants to be free" and that stopping CPU collab is like trying to stop encryption in the 90s, I can't help but wonder that RISC-V neuters attempts to lock China out of military AI advancements. Then again, don't they have x86 factories there? If China has a factory, doesn't the government pretty brazenly learn trade secrets from it? --- edit to Nickik, since HN has throttled me for more than an hour. Gotta get around the system somehow, since the goddamned rules are so opaque. I'll move this to a comment when I am in the site's good graces again. --- Highlight where I suggested any of that. I even said that actually attempting to curtail open development, even if desired, is futile. I was only wondering if there is validity in suspecting that China will explicitly finance/encourage RISC-V development in order to dodge EU/US sanctions of processor technology. So all your sarcasm was for nothing.
- cmrdporcupine 3y agoRISC-V does not have its origin in China. It's not a "clever conspiracy" at all. Are Chinese companies jumping on board this existing effort to avoid sanctions? Almost certainly. Other than that, don't care much what US congressmen think, since I'm not American, but attempting to close the door on RISC-V will not only be harmful to the broad industry (not just China) but will also be ineffective.
- nickik 3y agoWhy don't we just no longer have opensource. We can also shut down the wikipedia. Lets also shut down google. Otherwise China might use these things. Why not just nuke Bay Area, NY and LA. Then we can make sure China can't profit from the people there. All universities should be closed, those people might produce research that China can read.
- phendrenad2 3y agoThis all comes down to the US wanting to have both freedom of speech, but also prevent US citizens from talking to Chinese citizens. Because what if the American invents a better mousetrap and tells the Chinese person? We can't let the communists win the rodentiacide race!
- azinman2 3y agoFor claiming it’s a powerhouse in training, you really gotta post some benchmarks against Nvidia chips. Anyone know how it compares (drivers aside)?
- monocasa 3y ago> Google will utilize the base scalar and vector of X280 in a way that allows push/pop vector instructions. What are push/pop vector instructions? Is that kind of like the old school OpenGL 1 era matrix stacks? Or literally just traditional stack style access between memory and a vector register file? Or something else entirely?