10 ms·

Ask HN: What is an A.I. chip and how does it work?

• HarHarVeryFunny 3y ago
  Modern AI/ML is increasingly about neural nets (deep learning), whose performance is based on floating point math - mostly matrix multiplication and multiply-and-add operations. These neural nets are increasingly massive, e.g. GPT-3 has 175 billion parameters, meaning that each pass thru the net (each word generated) is going to involve in excess of 175B floating point multiplications! When you're multiplying two large matrices together (or other similar operations) there are thousands of individual multiply operations that need to be performed, and they can be done in parallel since these are all independent (one result doesn't depend on the other). So, to train/run these ML/AI models as fast as possible requires the ability to perform massive numbers of floating point operations in parallel, but a desktop CPU only has a limited capacity to do that, since they are designed as general purpose devices, not just for math. A modern CPU has multiple "cores" (individual processors than can run in parallel), but only a small number ~10, and not all of these can do floating point since it has specialized FPU units to do that, typically less in number than the number of cores. This is where GPU/TPU/etc "AI/ML" chips come in, and what makes them special. They are designed specifically for this job - to do massive numbers of floating point multiplications in parallel. A GPU of course can run games too, but it turns out the requirements for real-time graphics are very similar - a massive amount of parallelism. In contrast to the CPUs ~10 cores, GPUs have thousands of cores (e.g. NVIDIA GTX 4070 has 5,888) running in parallel, and these are all floating-point capable. This results in the ability to do huge numbers of floating point operations per second (FLOPS), e.g. the GTX 4070 can do 30 TFLOPS (Tera-FLOPS) - i.e. 30,000,000,000,000 floating point multiplications per second !! This brings us to the second specialization of these GPU/TPU chips - since they can do these ridiculous number of FLOPS, they need to be fed data at an equally ridiculous rate to keep them busy, so they need massive memory bandwidth - way more than the CPU needs to be kept busy. The normal RAM in a desktop computer is too slow for this, and is in any case in the wrong place - on the motherboard, where it can only be accessed across the PCI bus which is again way too slow to keep up. GPU's solve this memory speed problem by having a specially designed memory architecture and lots of very fast RAM co-located very close to the GPU chip. For example, that GTX 4070 has 12GB of RAM and can move data from it into its processing cores at a speed (memory bandwidth) of 1TB/sec !! The exact designs of the various chips differ a bit (and a lot is proprietary), but they are all designed to provided these two capabilities - massive floating point parallelism, and massive memory bandwidth to feed it. If you want to get into this in detail, best place to start would be to look into low level CUDA programming for NVIDIAs cards. CUDA is the lowest level API that NVIDIA provide to program their GPUs.
  • unnouinceput 3y ago
    A few finer points: 1 - It's RTX 4070, not GTX 4070 2 - the 30 TFLOPS you mention are at the very top when overclocked, they go for 22 normally. 3 - Also those are single precision TFLOPS, as in 32 bit. What really matter nowadays is double precision. And in double precision a 4070 is 0.35 TFLOPS (or 350 GFLOPS). 2 orders of magnitude lower, still impressive though
    • HarHarVeryFunny 3y ago
      For neural nets it's actually the opposite - half-precision bfloat16 is enough. You need large range, but not much accuracy. Yes, the exact numbers are going to vary, but just giving a data point to indicate the magnitude of the numbers. If you want to quibble there's CPU SIMD too.
      • unnouinceput 3y ago
        For gaming do matter those double precision. And we were talking about a certain GPU, which is used for gaming, not AI. Hence why the AI chips exists in the first place - dedicated hardware for dedicated tasks (or ASIC for short)
        • HarHarVeryFunny 3y ago
          The NVIDIA cards are all dual-use for gaming and compute/ML. Some features like the RTX 4070's Tensor Cores (incl. bfloat16) are there primarily for ML, and other features like ray tracing are there for gaming.
          • unnouinceput 3y ago
            The NVIDIA cards are for mining crypto-coins too, and they successfully did that for years, before being made obsolete in that area by ASICs. Now it's time for the same thing in AI/ML too, hence why AI chips are being developed, they are the ASICs for this domain. That's the big picture. In 2 to 3 years none is going to use NVIDIA gaming cards for AI/ML anymore, no matter how many GFLOPS future 5000/6000 series are going to offer. They will be for gaming only. End of story.
• ftxbro 3y ago
  This isn't a technical deep dive, but here's a simplified explanation. It's a matrix multiplication (https://en.wikipedia.org/wiki/Matrix_multiplication https://en.wikipedia.org/wiki/Matrix_multiplication) accelerator chip. Matrix multiplication is important for a few reasons. 1) it's the slow part of a lot of AI algorithms, 2) it's a 'high intensity' algorithm (naively n^3 computation vs. n^2 data), 3) it's easily parallelizable, and 4) it's conceptually and computationally simple. The situation is kind of analogous to FPUs (floating point math co-processor units) when they were first introduced before they were integrated into computers. There's more to it than this, but that's the basic idea.
• fragmede 3y ago
  Starting from https://cloud.google.com/tpu/docs/system-architecture-tpu-vm https://cloud.google.com/tpu/docs/system-architecture-tpu-vm what are you looking for?
  • rramadass 3y ago
    Yes, this is what i am looking for. System Architecture of these chips with detailed Functional Units and how they are used by the AI algorithm Instruction/Data streams.
• neximo64 3y ago
  Short story, CPU's can do calculations, they can do them one at a time. Think of something like 1+1, = 2. If you had 1 million equations like these, CPU's will generally do them one at a go, i.e the first one, then the second, etc. GPUs were optimised to draw, so were able to do dozens of these at a go. So these can be used for AI/ML in both gradient descent and inference (forward passes). Because you can do many at a go, in parallel, they speed things up dramatically. Geoff Hinton experimented with GPUs exploiting their ability to do this, but they aren't actually optimised to do that. It just turned out that it is the best way available to do it at the time, and still currently. AI chips, are optimised to do either inference or gradient descent. They are not good at drawing like GPUs are. They are optimised for machine learning and joining other AI chips together so you can have massive networks of chips that can parallel compute. One other class of chips that has not yet shown up are ASICs that mimic the transformers architectures for even more speed - though it changes too much at the moment for it to be useful. Also because of the mechanics of scale manufacturing: GPUs are currently cheaper per flop of compute as the aggregate of scale is shared with graphical uses. Though with time if there is enough scale AI chips should end up cheaper
  • strohwueste 3y ago
    Do you have any sources on those informations? I find it really hard to find stuff for what you describe. Also do you know about the detail of producing those Asics? Are they CMOS or flash (in-memory-compute?)
    • thrtythreeforty 3y ago
      All current AI accelerators (that aren't a research project) are ordinary CMOS. Google published some papers about TPUv3. You should read them if you want to know more about the architecture of these kinds of chips.
• neom 3y ago
  Veritasium did a pretty good video on some of them: https://www.youtube.com/watch?v=GVsUOuSjvcg https://www.youtube.com/watch?v=GVsUOuSjvcg
  • atgctg 3y ago
    That video is about analog computers
    • neom 3y ago
      The video is about analog chips for ML/NNs. His profile of this company was particularly interesting: https://mythic.ai/ https://mythic.ai/ OP asked about chips for AI.
      • formerly_proven 3y ago
        Do companies pay him or does he do these ads for free?
        • cinntaile 3y ago
          He gets paid. It says so at the start of the video, but I guess that could depend on where you live.
• binarymax 3y ago
  I’d start with CUDA, because knowing what a chip does won’t click until you see how it can be programmed to do massive parallel computation and matmul. I read the first book in this list about 10 years ago, and though it’s pretty old the concepts are solid. https://developer.nvidia.com/cuda-books-archive https://developer.nvidia.com/cuda-books-archive
  • dogma1138 3y ago
    CUDA abstracts most of the parallelism, the magic of CUDA is it gave developers a C/C++ API or language if you will that doesn’t really requires them to think about that they can continue writing their problems as they did when programming for mostly single core single threaded CPUs back in the day and CUDA takes care of the rest. Even “manual” CUDA optimizations deal more with concurrency and data residency than parallelism and even those are usually limited to following the compute guide for your specific hardware and feature set and the driver does the majority of the heavy lifting.
• nologic01 3y ago
  It may help your digging and search if you have in mind what those chips really try to do: Accelerate numerical linear algebra calculations. If you are familiar with linear algebra these specialized chips literally etch silicon so as to perform vector (and more general multi-array or tensor) computations faster than a general purpose CPU. They do that by loading and operating a whole set of numbers (a chunk of a vector or a matrix) simultaneously (whereas the CPU would operate mostly serially - one at a time). The advantage is (in a nutshell) that you can get a significant speedup. How much depends on the problem and how big a chunk you can process simultaneously but it can be a significant factor. There are disadvantages that people ignore in the current AI hype: * The speedup in a one-off gain, the death of Moore's law is equally dead for "AI chips" and CPU's * It is extremely specialized and fine-tuned software you need to develop and run and it only applies to the above linear algebra problems. * In the past such specialized numerical algebra hardware was the domain of HPC (high performance computing). Many a supercomputer vendor went bankrupt in the past because the cost versus market size was not there.
  • deleted 3y ago
    [deleted]
  • benreesman 3y ago
    While a given generation of accelerator can only target model architectures that are comparatively proven out, and there’s a lag time, it’s measured in years not decades. I remember when NVIDIA didn’t have hardware for ReLU. The fact of the matter on Moore’s law is that we’ve got transistors, but not TDP to burn and have for years. These stupid big L3 caches are just: “fuck it, I’ve got die to burn”. This is an old story, things migrate in and out of the “CPU”, but the current outlook is that we’ll be targeting specialized hardware more rather than less for the foreseeable future.
    • microtonal 3y ago
      I remember when NVIDIA didn’t have hardware for ReLU. Could you elaborate? ReLU is max(x,0) and CUDA had fmaxf since CUDA 1.0 (2007).
• fulafel 3y ago
  Here's one from Google (paper link at the end): https://cloud.google.com/blog/topics/systems/tpu-v4-enables-performance-energy-and-co2e-efficiency-gains https://cloud.google.com/blog/topics/systems/tpu-v4-enables-...
• phh 3y ago
  There are very different architectures in the wild. Some are simply standard GPUs (maybe with additional support for bf16/float16) (Rockchip RK1808 has one like that). You give it a list of instructions pretty much like a CPU (except massively parallel), and it'll execute it. BTW when I say standard GPU, I'm not saying "kinda like GPU, but really literally GPU architecture. Linux mainline support for Amlogic A311D2's NPU is 10 lines, and it's declaring a no-output vivante GPU. Some are just hardware pipeline to compute 2d/3d convolutions + activation function (Rockchip RK3588 has one like that). You give it the memory address + dimensions of the input matrix, the memory address + dimensions of the weights, memory address + dimensions of the output, which activation function you want (there are only like 4 supported), then you tell it RUN, you wait for a bit, and you have the result at the output memory address. (I took Rockchip example to show that even in one microcosm it can change a lot) And then you can imagine any architecture in-between. AFAIK they all work with some RAM as input and some RAM as output, but some may have its own RAM, some may share RAM with the system, some might have mixed usage (RK3588 has some SRAM, and when you ask it to compute the convolution, you can tell it either to write to SRAM or system RAM) It's possible that there are some components that are border line between ISP (Image Signal Porcessing) and a NPU, where the input is the direct camera stream, but my guess is that they do some very small processing on the camera stream, then dump it to RAM, then do all the heavy work from RAM to RAM. I think that Pixel 4-5 had something like that.
  • grozzle 3y ago
    The RK3588 seems like a beast for performance per dollar, I have one running desktop Linux for emulators and games, but I haven't used the NPU yet, because I haven't taken the time to figure out how to get OpenCV to talk to it. Is Rockchip's stuff "good", as NPUs go? I'm thinking of buying another 3588 SoC for my robotics hobby - you seem like you'd know if that's a decent idea or not.
    • reesul 3y ago
      Yeah it’s performance vs cost is honestly nuts. 6 TOPS is a pretty solid NPU, but I don’t know what their software is like. Programming those accelerators is often difficult, especially if you’re a small time customer. Curious if anyone can weigh in on their SW usability. A quick search for their user level tools showed examples/documentation in Chinese(?)
• mongol 3y ago
  What would be an affordable/ cheap way to get hands on with this type of hardware? Right now I have zero knowledge.
  • ajb117 3y ago
    I'm pretty sure you can't buy TPUs, but people usually buy GPUs instead. If you're building a personal rig, these days, you can get an Nvidia RTX 3090 for about $720 USD on ebay used, which is pretty cheap for 24GB VRAM. There's also the A6000 with 48GB VRAM but that'll cost about $5000 on Amazon. Of course, there's new cards that are faster with more VRAM like the 4090 and RTX 6000, but they're also more expensive. Of course, this is all pretty expensive still. If your models are small enough you can get away with even older GPUs with less VRAM like a GTX 1080 Ti. And then of course there's services like Google Collab and vast.ai where you can rent a TPU or GPU in the cloud. I'd check out Tim Dettmers' guide for buying GPUs: https://timdettmers.com/2023/01/30/which-gpu-for-deep-learning/ https://timdettmers.com/2023/01/30/which-gpu-for-deep-learni...
    • jmrm 3y ago
      AFAIK Google Coral is an inexpensive TPU you can buy right now: https://coral.ai/products/accelerator/ https://coral.ai/products/accelerator/
      • worldsayshi 3y ago
        Very cool! Although seems to have no memory to speak of so many use cases like LLM goes away because of that I guess?
        • muxamilian 3y ago
          It has 8 MB of memory. It also supports live streaming the neural network to the chip, although that is slower than when it is cached in the memory.
      • kmeisthax 3y ago
• tomek32 3y ago
  You might find this talk interesting, The AI Chip Revolution with Andrew Feldman of Cerebras, https://youtu.be/JjQkNzzPm_Q https://youtu.be/JjQkNzzPm_Q It's the founder of a new AI chip company and they talk a bit on the differences
• elseless 3y ago
  H.T. Kung’s 1982 paper on systolic arrays is the genesis of what are now called TPUs: http://www.eecs.harvard.edu/~htk/publication/1982-kung-why-systolic-architecture.pdf http://www.eecs.harvard.edu/~htk/publication/1982-kung-why-s...
• sharph 3y ago
  Great video from asianometry explaining AI chips (GPGPUS, general purpose GPUs) roots in GPUs (graphics processing units) -- how did we get here and what do these chips do? https://www.youtube.com/watch?v=GuV-HyslPxk https://www.youtube.com/watch?v=GuV-HyslPxk
• psychphysic 3y ago
  Google's TPU which they sell via Coral is just a systolic array of multiply-accumulates arranged in a grid. Here's a decent overview from the horse's mouth. https://cloud.google.com/blog/products/ai-machine-learning/an-in-depth-look-at-googles-first-tensor-processing-unit-tpu https://cloud.google.com/blog/products/ai-machine-learning/a... It's called a systolic array because the data moves through it in waves similar to what an engineer imagines the heart looks like :)
• nottorp 3y ago
  An "AI" chip is marketing. But as other posts say, "linear algebra coprocessor" doesn't roll of the tongue as well. Incidentally there used to be a proper "AI" chip. The original perceptron was intended to be implemented in hardware. But general purpose chips evolved much faster. https://en.wikipedia.org/wiki/Perceptron https://en.wikipedia.org/wiki/Perceptron
• arroz 3y ago
  AI chips is just regular chips that do AI stuff faster So dedicated hardware to do math stuff
• zoogeny 3y ago
  There is a YouTube channel TechTechPotato [1] that has a podcast on AI hardware called "The AI Hardware Show". Pretty small and it gives you a view on how niche this market is - but if you want the 10k foot view from young budding tech journalists then I think this fits the bill. Some random examples of video titles from the last 6 months of the channel: * A Deep Dive into IBM's New Machine Learning Chip * Does my PC actually use Machine Learning? * Intel's Next-Gen 2023 Max CPU and Max GPU * A Deep Dive into Avant, the new chip from Lattice Semiconductor (White Paper Video) * The AI Hardware Show 2023, Episode 1: TPU, A100, AIU, BR100, MI250X I think the podcasters background is actually in HPC (High Performance Computing), i.e. super computers. But that overlaps just enough with AI hardware that he saw an opportunity to capitalize on the new AI hype. 1. https://www.youtube.com/c/TechTechPotato https://www.youtube.com/c/TechTechPotato
  • rramadass 3y ago
    Nice, looks like a good starting point to survey the field.
• imakwana 3y ago
  Relevant: Stanford Online course - CS217 Hardware acceleration for machine learning https://online.stanford.edu/courses/cs217-hardware-accelerators-machine-learning https://online.stanford.edu/courses/cs217-hardware-accelerat... Course website with lecture notes: https://cs217.stanford.edu/ https://cs217.stanford.edu/ Reading list: https://cs217.stanford.edu/readings https://cs217.stanford.edu/readings
  • rramadass 3y ago
    Excellent!
• pulkas 3y ago
  It is what ASIC for bitcoin. A new era for AI models.
• joyeuse6701 3y ago
  Interestingly, this might be well answered by the LLMs built on the technology you’re interested in.
• nl 3y ago
  There's a lot of information here about chips which are mostly built for training neural networks. It's worth noting there are very widely deployed chips primarily built for inference (running the network) especially on mobile phones. Depending on the device and manufacturer sometimes this is implemented as part of the CPU itself, but functionally it's the same idea. The Apple Neural Engine is a good example of this. This is separate to the GPU which is also on the CPU. Further information is here: https://machinelearning.apple.com/research/neural-engine-transformers https://machinelearning.apple.com/research/neural-engine-tra... The Google Tensor CPU used in the pixel has a similar coprocessor called the EdgeTPU.
• sremani 3y ago
  I want to latch on this question a bit -- which company out there is primed to bring us a CUDA competitor. AMD has failed, so any wise words from the people in the industry?
• anon291 3y ago
  I've worked in this space for the past five years. The chips are essentially highly parallel processors. There's no unifying architecture. You have the graph-based / hpc-simulator chips like Cerebras, Graphcore, etc which are basically a stick-as-many-cores-as-possible situation with a high-speed networking fabric. You have the 'tensor' cores like Groq where the chip operates as a whole and is just well suited for tensor processing (parallelizable, high-speed memory, etc). At the end of the day, it's matrix multiplication acceleration mostly, and then IO optimization. Literally most of the optimization has nothing to do with compute. We can compute faster than we can ingest.
  • rramadass 3y ago
    >The chips are essentially highly parallel processors Right. AFAIK we already were doing SIMD, Vector Processing, VLIW etc. to speed up parallel processing for numerical calculations in AI/ML. What then is the reason for the explosion of these different categories of chips? Are they just glorified ASICs designed for their specific domains or FPGAs programmed accordingly? If so what is their architecture i.e. what are their functional units and how do they work together?
    • anon291 3y ago
      Realistically, a good AI chip will have provisions for high throughput IO (the most important thing, and the key differentiator) and the actual processing really doesn't matter because with enough engineering effort you'll be able to saturate the chip. GPUs have a high speed output in the form of an HDMI link. However, there is no high speed input. Reads/writes to/from the GPU are slow. The Cerebras wafer chip for example has 8-16 FPGA driven IO chips that directly read from TCP/IP onto the chip and off again in parallel. Each FPGA connects to its own ethernet port. So you can get the data on/off the chip as fast as possible. That's it really. As for the processing engines. They're usually just standard cores with a high speed interconnect and maybe some matrix multiplication optimizations. Some, like Groq, have a high speed fabric with specialized processors at various locations.
• ttul 3y ago
  On top of what others have said here about TPUs and their kin, you can make things really scream by taping out an ASIC for a specific frozen neural network (i.e. including the weights and parameters). If you never have to change the network - for instance to do image segmentation or object recognition - then you can’t get any more efficient than a custom silicon design that bakes in the weights as transistors.
• visarga 3y ago
  Trying to make a list of AI accelerator chip families, anything missing? - GPU (Graphics Processing Unit) - TPU (Tensor Processing Unit): ASIC designed for TensorFlow - IPU (Intelligence Processing Unit): Graphcore - HPU (Habana Processing Unit): Intel Habana Labs' Gaudi and Goya AI - NPU (Neural Processing Unit): Huawei, Samsung, Microsoft Brainwave - VPU (Vision Processing Unit): Intel Movidius - DPU (Data Processing Unit): NVIDIA data center infrastructure processing unit - Amazon's Inferentia: Amazon's accelerator chip focused on low cost - Cerebras Wafer Scale Engine (WSE) - SambaNova Systems DataScale - Groq Tensor Streaming Processor (TSP)
  • rramadass 3y ago
    Nice and very much in line with what i am looking for ! If you don't mind, could you add links to authoritative wiki pages/whitepapers/articles for each of the above? I think it will give us a good starting point to start our study/research from.
• sethgoodluck 3y ago
  Remember crypto miners (ASICS). Exact same thing but built for the math around AI work instead of Blockchain work.
• m3kw9 3y ago
  AI chip is basically a chip that calculates matrix’s better than general purpose CPUs
• gojo_dog 3y ago
  There's a five part series outlining AI accelerator chips that came out last year. Starting with Introduction, Motivation, Technical Foundations, and Existing Solutions: https://medium.com/@adi.fu7/ai-accelerators-part-i-intro-822c2cdb4ca4 https://medium.com/@adi.fu7/ai-accelerators-part-i-intro-822...
  • rramadass 3y ago
    Excellent; just what i was looking for! Please do submit this as a top-level HN submission so that it gets greater visibility.
• turbojerry 3y ago
  The NVIDIA Deep Learning Accelerator (NVDLA) is a free and open architecture that promotes a standard way to design deep learning inference accelerators. http://nvdla.org/ http://nvdla.org/