8 ms·
Render a neural network into CUDA/HIP code
- samstave 3y agoELI5 what this means? I am losing my bibliography, etymology and vocabulary with every single AI advancement article. Where learn AI vocab, please? - I nee an FN AI teacher to just give me daily updates on AI and verbiage, models, etc... Hey AI - if you're so smart, build a podcast that teaches me about yourself and how to be a better meat parent whom made you.///\\\\
- bagels 3y agoWhat is an FN AI?
- samstave 3y agoA 'fUCKIn' ai'
- dragonwriter 3y agoAn autonomous Belgian firearm.
- nerpderp82 3y agoSoon. :(
- skirmish 3y agoStarting with a trained PyTorch model, it builds optimized C++ binaries for running inference (not training) on NVidia and AMD GPUs. Various optimizations mentioned a lot, so presumably models run faster than with just running them via regular PyTorch.
- stevenwliao 3y agoHow much faster is it?
- pumanoir 3y agoDepends on the model and GPU. Here is an example of almost 2x on a 3060 for StableDiffusion: https://www.youtube.com/watch?v=_6BsUijOWoM https://www.youtube.com/watch?v=_6BsUijOWoM
- prsutherland 3y agoI'm curious why that is called "rendering" rather than "compiling". Is the code boiler plate and just a change in the NN's representation?
- iaw 3y agoVery much not an expert here but what I understand is that most deep learning frameworks (PyTorch, Tensorflow, etc.) have some overhead associated with them just being on the graphics card. This takes PyTorch code and removes the overhead by translating the network into a "native" language for the card (CUDA for NVIDIA). What I'm not sure is what "HIP" is in this context. The way I'm reading this is it's the difference between running code in an interpreter vs. on the bare metal (for the GPU)
- entropicdrifter 3y agoHIP is AMD's open-source re-implementation of CUDA libraries.
- viewtransform 3y agoHIP is AMD's contribution to the open source community to overcome Nvidia's CUDA software moat You write in HIP C++ language and run them on either NVIDIA or AMD platforms. This way you get cross-platform code and are not stuck with Nvidia. Use HIPify tool to automatically convert existing sources from CUDA to HIP. It's been around for many years - but the fact that so many people still don't know about it - speaks for the sad state of AMD communication. https://docs.amd.com/bundle/HIP-Programming-Guide-v5.3/page/Introduction_to_HIP_Programming_Guide.html https://docs.amd.com/bundle/HIP-Programming-Guide-v5.3/page/...
- my123 3y agoHIP is a pathetic CUDA API clone. Gratuitous renames don't do it any good and are more representative of NIH rather than anything else (sadly). They should have shipped a proper header set instead of hipify.
- deleted 3y ago[deleted]
- femto113 3y agoIt doesn't really help understand what they are, but for completeness CUDA is an acronym for "Compute Unified Device Architecture" while HIP is "Heterogeneous-compute Interface for Portability"
- homarp 3y agoCUDA: NVIDIA GPU 'framework' HIP: AMD GPU 'framework' This takes Neural Network defined in python and convert them to C++ code calling CUDA / HIP for maximum inference speed
- iaw 3y agoAnyone seen details on if this can handle splitting a model across GPUs?
- sosodev 3y agoThe latency improvements are impressive but the ability to run models beyond their typical memory limitations is way cooler.
- born-jre 3y agoat first glance i thought may be its like tinygrad. but looks has many ops than tiny grad but most maps to underlying hardware provided ops? i wonder how well tinygrad's apporach will work out, ops fusion sounds easy, just walk a graph, pattern match it and lower to hardware provided ops? Anyway if anyone wants to understand the philosophy behind tinygrad, this file is great start https://github.com/geohot/tinygrad/blob/master/docs/abstractions.py https://github.com/geohot/tinygrad/blob/master/docs/abstract...
- bguberfain 3y agoIt reminds me Theano
- antinucleon 3y agoAITemplate's original designer is here. We quit Meta in January and start HippoML (https://hippoml.com/ https://hippoml.com/). We just disclosed our new engine's performance on LLM: https://blog.hippoml.com/large-language-model-inference-from-datacenter-to-edge-ed2f94da4a81 https://blog.hippoml.com/large-language-model-inference-from... On Apple M2 Max our new engine encode/decode is 13.8X/2.4X faster than llama.cpp
- ralfd 3y agoWhat is your planned business model here?
- antinucleon 3y agoWe will disclose more details very soon.
- brucethemoose2 3y agoVery interesting. Is 8bit/4bit support in the works? Will it work with bitsandbytes out of the box? Speedy inference is great, but in practice many users are running the biggest ~4-bit LLM that will fit into their RAM/VRAM pool these days. This is why llama.cpp is so good, its (AFAIK) the only implementation that will split a 4 bit quantized model so easily.
- antinucleon 3y agoYes. We support >= 1bit <= 16bit models out of box for various of models.
- mhh__ 3y agoReally doesn't surprise me that much. Llama.cpp seems like an OK first passs but I assume there is loads of time left on the table in terms of graph optimizations optimizing for the memory hierarchy properly.
- brucethemoose2 3y ago
- brucethemoose2 3y agoI just ran a 512x512 Stable Diffusion benchmark with this yesterday Pytorch Eager Mode with some optimizations: ~6it/s Pytorch Inductor (torch.compile with dynamic=True): ~7it/s AITemplate: ~9it/s All of them support changing settings and such, albeit with some work in progress bugs/caveats. That is 512x512 on a 2060, so I would expect the gains to be bigger on newer GPUs with more overhead to take advantage of.
- maxilevi 3y agoDid you try TensorRT?
- brucethemoose2 3y agoNot yet. TRT diffusion has been an enormous pain in the past, so I have kinda avoided it, but Nvidia just recently contributed an img2img pipline in HF diffusers.
- Havoc 3y agoInteresting that AMD GPUs seem to be 1st class citizens here. Consumer class gear is much cheaper per unit of VRAM by the looks of it
- rerx 3y agoThe thing with AMD is that the selection of GPUs that can actually run your GPGPU code is typically much more limited then with Nvidia, so things work on far fewer consumer GPUd. Here it's limited to "CDNA2 (MI-210/250) GPUs". Those are priced comparatively to Nvidia's A100, 10k+ per card.
- brucethemoose2 3y agoAlso here are some other interesting projects in the ML compilation space: - Apache TVM (mlc-llm is a good demo) - Hidet (a torch.compile backend) - Alibaba BladeDISC - Nvidia TensorRT (a classic, but much less of a nightmare to install now) - Torch MLIR (SHARK has some demos/implementations)
- jahewson 3y agoAnd of course, Chris Lattner’s Modular AI https://www.modular.com/ https://www.modular.com/
- philipturner 3y agoYes. I look forward to a future where we don't have to wrestle between software and hardware. AI just runs, whether on CPU, GPU, TPU, QPU, or whatever architecture you want to design. Time to move on and focus on more important problems, like solving climate change.
- hintymad 3y agoI like this humility: "AITemplate is co-created by Meta engineers: Bing Xu, Ying Zhang, Hao Lu, Yang Chen, and Terry Chen, with major contributions coming from more talented engineers."
- throwaway14356 3y agoaliens
- cypress66 3y agoI don't see any comparisons with torch.compile. Kind of unfair to compare it to eager mode.
- pavelstoev 3y agoWould like to see updated benchmarks supporting the claimed performance gains in this project. Currently showing torch 1.12 which is rather weak compared to the latest 2.0 and torch.compile()
- philipturner 3y agoantinucleon lol LLaMA is a memory-bound AI model, where the dominant factor in execution time is how fast the processor transfers weights from RAM to registers. LLaMA.cpp uses a misaligned memory pattern that's painful to the RAM I/O interface and requires many CPU instructions to re-align. It also has access to 1/2 the bandwidth of the GPU on Apple's highest-end chips, making GPU *theoretically* 2x faster without algorithm changes. Funny how you announced it the exact day after I open-sourced a high-bandwidth 4-bit GEMV kernel for LLaMA. If anyone wants to see how to achieve 180+ GB/s on the M1/M2 Max, you can reproduce the methodology here: https://github.com/ggerganov/llama.cpp/pull/1642#issuecomment-1574138217 https://github.com/ggerganov/llama.cpp/pull/1642#issuecommen... I later explained the technique to write very optimized kernels for a specific quantization format. *It's very unlikely my code was copied verbatim*, but it was probably used as inspiration and/or a reference. I also disclosed a high-precision means of measuring bandwidth utilization, which is critical for designing such GPU kernels.
- philipturner 3y agoTo give more credit, I know your company was working on your own optimizations, which are prior art. It's possible that you made a 180 GB/s shader on your own (quite slow compared to my 319 GB/s). Or that the 319 GB/s was used, but the self-attention bottleneck was non-negligible. However, for whatever reason, when Greg Gerganov started work on the Metal backend, you made a product announcement almost a day later. That seems like a non-coincidence and there must be some logical explanation.