4 ms·
Question is, how much of an improvement has it gotten to over a GPU or ASIC?
by 1-6 3y ago
Question is, how much of an improvement has it gotten to over a GPU or ASIC?
- dartos 3y agoNothing in software will ever beat an equivalent ASIC.
- postalrat 3y agoSure there is. Software is easy to change.
- dartos 3y agoBy “beat” I meant in performance. Obviously you can’t change an asic
- fragmede 3y agoan asic is fixed function, so it'll never be able to boot my pc and then be the CPU, even though an asic beats the pants off anything else computing Sha hashes for Bitcoin mining.
- dartos 3y agoBy “beat” I meant performance. Obviously an ASIC is not a general purpose machine like a cpu.
- fulafel 3y agoMost ASICs are cost or power optimizations.
- dartos 3y agoExactly. They’re much faster for their specific tasks and thus are more power efficient and potentially cost efficient
- fulafel 3y agoNo. Eg of the hardware discussed on the article, the Raspberry Pi uses an ASIC that's slow, cheap and low power vs the Intel or AMD chips. In some cases ASICs are faster than general purpouse CPUs, but usually not.
- LtdJorge 3y agoIs the LLM running on an ASIC for the Pi here? I dout it.
- fulafel 3y agoThe ARM cores and the VideoCore parts are all in the ASIC, it's a SoC type ASIC.
- LtdJorge 3y agoHmm, yeah it's a SoC, but not an ASIC. Maybe you mean APU? ASICs are circuits thst can only do one thing, CPU cores are definitely not that. Edit: an example ASIC the Pi has is the video encoder/decoder, with JPEG also supported. I think it's embedded in the GPU.
- fulafel 3y agoASIC just means it's an application specific IC (= chip), meaning it's fabbed for that specific product (like in this case the Raspberry Pi). A functional block like a JPEG codec contained therein is not an ASIC. Quoth wikipedia: "Modern ASICs often include entire microprocessors, memory blocks including ROM, RAM, EEPROM, flash memory and other large building blocks. Such an ASIC is often termed a SoC (system-on-chip)." What you're describing in the JPEG codec might be termed a fixed function IP block in semiconductor design terminology.
- dartos 3y agoThat’s weird that Wikipedia says there’s no real distinction between SoCs and ASICS. Colloquially, I’d never call anything with a programmable processor an ASIC
- yjftsjthsd-h 3y agoI think that should be phrased more like "what fraction of GPU speed can this reach?", because it'll always be less than 1x.
- deleted 3y ago[deleted]
- gpapilion 3y agoSo... I was struggling with this for a while. I would says anywhere from 2x to an order of magnitude faster with a GPU. (I've been looking at a lot of GPU benchmarks lately, and they are REALLY hard to compare since they are all so specific) I do think long term there gets to be more hope for CPUs here with inference largely because memory bandwidth becomes more important than the gpu. You can see this with reports of the MI-300 series outperforming h100, largely because it has more memory bandwidth. MCR dimms give you close to 2x the exiting memory bw in intel cpus, and when coupled with AMX you may be able to exceed v100 and might touch a100 performance levels. HBM and the general GPU architecture gives it a huge memory advantage, especially with the chip to chip interface. Even adding HBM to a CPU, you are likely to find the CPU is unable to use the memory bw effectively unless it was specifically designed to use it. Then you'd still likely have limited performance with things like UPI being a really ugly bottleneck between CPUs.
- imtringued 3y agoIf someone releases DDR5 or DDR6 based PIM, then most of the memory bandwidth advantage of GPUs evaporates overnight. I expect CPUs to be king at inference in the future.
- gpapilion 3y agoBut then you'll get GDDR6 delivered via HBM5 or whatever. I don't think CPUs will ever really keep up with the memory bandwidth, because for most applications it doesn't matter. MCR DIMM is like 1/2 the memory bandwidth that is possible with HBM4, plus it requires you to buy something like 2TB of memory. It might get there, but I'd keep my money on hbm and gpus.
- baq 3y agoFrom the article, passage about the 14900k: > For example, when I run my spam.sh shell script, it only takes 420 milliseconds, which is 7x faster than my Raspberry Pi 5. That's right, when it comes to small workloads, this chip is able to finish before CUDA even gets started. So… it depends :)
- jchw 3y agoI think I understand what you are thinking. You may be fixing "than other ways of running them" to the end of the title, but it's actually "than it was on CPU before now".