lhl

13 ms·

91.
by lhl 2y ago
Although there are some less powerful cards (the Intel A310/A380s are available for ~$100 and would be fine as basic display adapters w/ good video engines) if you go lower end, modern iGPUs are generally good enough for anything
92.
by lhl 2y ago
Having spent a fair amount of time over the past few years as someone working on top of both Nvidia and AMD toolchains (and having spent decades working up and down the stack from mcus, mobile device, to webscale stacks), I have an alternat
93.
by lhl 2y ago
This GPU has a 192-bit memory bus. At 32-bit GDDR bus width (well w/ 6, 2x16b data channels per chip), that means you have 6 channels. With regular GDDR6, the largest produced size is 16Gb (2GB), so 12GB is what you get. You could doub
94.
by lhl 2y ago
Recently I did some testing of the IPEX-LLM llama.cpp backend on LNL's Xe2: https://www.reddit.com/r/LocalLLaMA/comments/1gheslj/testing... Based on scaling by XMX/engine clock napkin math, the
95.
by lhl 2y ago
Since it looks like from the announcement, the model hasn't changed much, here's the Zamba 1 paper for reference: https://arxiv.org/pdf/2405.16712 Zamba 1 has a single shared attention block that is applied e
96.
by lhl 2y ago
You should have ROCm/HIP support on the iGPU as well, be sure to compile llama.cpp w/ the LLAMA_HIP_UMA=1 flag. If you take a look at https://github.com/amd/RyzenAI-SW you can see there's a fair amount o
97.
by lhl 2y ago
I think fazkan was confused about which repo you were talking about. For the llm-tracker doc, that's something I maintain. It's based on stuff I test but if you want to submit a PR or issue w/ info in a way that I can verify
98.
by lhl 2y ago
Here's the big files in my /opt/rocm/lib which is most of it: 4.8G hipblaslt 1.6G libdevice_conv_operations.a 2.0G libdevice_gemm_operations.a 1.4G libMIOpen.so.1.0.60200 1.1G librocblas.so.4.2.60200 1.6G l
99.
by lhl 2y ago
For inference, if you have a supported card (or probably architecture if you are on Linux and can use HSA_OVERRIDE_GFX_VERSION), then you can probably run anything with (upstream) PyTorch and transformers. Also, compiling llama.cpp is has
100.
by lhl 2y ago
BTW, I got a chance to read through the model card and there's a section that shows their SD gains: https://huggingface.co/amd/AMD-Llama-135m#speculative-decodi... - 1.75x-2.80x on MI250 - 2.83x-2.98x on NPU - 3.5
101.
by lhl 2y ago
Runpod.io rents the next-gen MI300X's for $4/hr, although since they also rent H100's for $3/hr (that are easier to work with/faster for training) it might be more of a novelty.
102.
by lhl 2y ago
There's actually seems to be a bunch of stuff now: * https://github.com/amd/RyzenAI-SW - has a list of demos and how to use it directly (including apparently w/ PyTorch and LLMs) * https://github.c
103.
by lhl 2y ago
Orders of magnitude seems a bit ambitious. The implementation from the DeepMind paper achieved a 2-2.5X https://arxiv.org/pdf/2302.01318 and most of the tests I've seen [1][2] have been similar, but there are dif
104.
by lhl 2y ago
When was this profitability report, because the cost per token generation has dropped significantly. When GPT4 was launched last year, the API cost was about $36/M blended tokens, but you can now get GPT4o tokens for about $4.4/M
105.
by lhl 2y ago
Yeah, those numbers are correct as of their testing (in June) although people who are really interested should check out the linked repo and do their own runs as software/optimizations have continued to change a lot and the RDNA3 side
106.
by lhl 2y ago
While Vulkan can be a good fallback, for LLM inference at least, the performance difference is not as insignificant as you believe. I just ran a test on the latest pull just to make sure this is still the case on llama.cpp HEAD, but text ge
107.
by lhl 2y ago
A couple months ago I did some testing on some consumer cards. [1] I think you should be able to use torchtune or axolotl without anything besides installing the ROCm version of PyTorch. [1] https://wandb.ai/augmxnt/tra
108.
by lhl 2y ago
While your project is neat and I'd like to see how the performance compares, for LLM training, PyTorch, including torch.compile works completely OOTB on AMD. All you have to do is pip install the ROCm version of PyTorch (or run the doc
109.
by lhl 2y ago
Yes, for single user multiturn kvcache reuse could help a lot. vLLM has support for this via Automatic Prefix Caching (APC) so you’d be able to take advantage of this w/ Strix Halo now. llama.cpp has had a “prompt-cache” option but whe
110.
by lhl 2y ago
For those interested, a few months ago someone posted benchmarks with their MBP 14 w/ an M3 Max [1] (128GB, 40CU, theoretical: 28.4 FP16 TFLOPS, 400GB/s MBW) The results for Llama 2 70B Q4_0 (39GB) was 8.5 tok/s for text gene
111.
by lhl 2y ago
I totally feel you on this, and have spent a lot of time yak-shaving similar stuff, especially Linux laptops. IMO one of the best reasons to prefer to buy/use a laptop from a Linux-first laptop company (Framework, Tuxedo, Slimbook, Sta
112.
by lhl 2y ago
These are plugs with "flat attenuation" - the Etymotics custom-molded Musicians Earplugs are the classic ones (you go to an audiologist to make a mold) - they actually have a variety of attenuators, including a 25dB one (for drumm
113.
by lhl 2y ago
For mobile coverage and speed, I've found Japan (I've been from Ishigaki to Shiretoko in the past few years) to be pretty world class (assuming your device supports the proper bands - for anyone visiting, refer to https:/&#x
114.
by lhl 2y ago
I don't know why anyone would think a meh performing iGPU would encourage local LLM adoption at all? A 7B local model is already not going to match frontier models for many use cases - if you don't care about using a local model
115.
by lhl 2y ago
From Notebookcheck's benchmarks it looks like the Radeon 890M is punching at about a GeForce 1650 Mobile's performance: https://www.notebookcheck.net/Computer-Games-on-Laptop-Graph... Based on https://w
116.
by lhl 2y ago
Just as a point of reference, this is what a 65W power-limited 7940HS (Radeon 790M) with 64GB of DDR5-5600 looks like w/ a 7B Q4_K_M model atm w/ llama.cpp. While it's not amazing, at 240 t/s prefill, it means that at 4K
117.
by lhl 2y ago
One issue is that even if you are using GTT (dynamically allocated memory), this is still limited as a percentage of total RAM. Eg, currently on my 7940HS, I have 64GB of memory, 8GB of dedicated VRAM (GART), and then a limit of 28GB of GTT
118.
by lhl 2y ago
I did ICL a few years ago, had a good experience. I did a fair amount of research. One interesting thing to note is that the FDA is often several years behind for both ICL and Laser tech (eg, it looks like the EVO/EVO+ (V4c, V5) lenses
119.
by lhl 2y ago
There was a mini-uproar when GPT-4o-mini (an obviously "dumber" model) outscored claude-3.5-sonnet on Chatbot Arena, so much so that LMSYS released a subset of the battles: https://huggingface.co/spaces/lmsys&
120.
by lhl 2y ago
Yes, answers were distilled from a much stronger model. On the one hand, you can argue that this is exactly what the LMSYS, WildBench etc datasets are for (to improve performance/alignment on real-world use cases), but on the other han

More ›