4 ms·
The winner will be one of two approaches: 1) Getting great performance using regular DRAM - system memory. 2) Bringing the compute to the RAM chips - DRAM is ac
by phkahler 2y ago
The winner will be one of two approaches: 1) Getting great performance using regular DRAM - system memory. 2) Bringing the compute to the RAM chips - DRAM is accessed 64Kb per row (or more?) and at ~10ns per read you can use small/slow ALUs along the row to do MAC operations. Not sure how you program that though.
Current "at home" inference tends to be limited by how much RAM your graphics card has, but system RAM scales better.
- rfoo 2y agoThe winner, unfortunately, will be on cloud inference.
- eth0up 2y agoI'll probably get stoned for asking here, but... since you seem knowledgeable on the subject: I just got llama3.1-8b (standard and instruct). However, I cannot do anything with it on my current hardware. Can you recommend the best AI model that I: 1) can self host 2) run on 16GB ram with no dedicated graphics card and an old intel i5 3) use on Debian without installing a bunch of exo-repo mystery code? Any recommendation, directly or semi related would be appreciated - I'm doing my 'research' but haven't made much progress nor had any questions answered.
- arcanemachiner 2y agoSetting up Ollama via Docker was the easiest way for me to get up and running. Not 100% sure if it fits your constraints, but highly recommended.
- programd 2y agoAnother option is to download and compile llama.cpp and you should be able to run quantized models at an acceptable speed. https://github.com/ggerganov/llama.cpp https://github.com/ggerganov/llama.cpp Also, if you can spend the $60 and buy another 32GB of RAM, this will allow you to run the 30GB models quite nicely.
- eth0up 2y agoUnfortunately motherboard is capped at 16Gb ram
- smokel 2y agoRunning LLMs on that kind of hardware will be very slow (expect responses with only a few words per second, which is probably pretty annoying). LM Studio [1] makes it very easy to run models locally and play with them. Llama 3.1 will only run in quantized form with 16GB RAM, and that cripples it quite badly, in my opinion. You may try Phi-3 Mini, which has only 3.8B weights and can still do fun things. [1] https://lmstudio.ai/ https://lmstudio.ai/
- eth0up 2y agoMuch appreciated. Thanks for this!
- deleted 2y ago[deleted]
- wkat4242 2y agoI don't find llama3.1 noticeably worse on 8 bit integer quantised than the original fp16 to be honest. It's also a lot faster. Of course even then you're not going to reach the whole 128k context window on 16GB but if you don't need that it works great.
- mikewarot 2y agoCompletely eliminating the separation between RAM and compute is how FPGAs are so fast, they do most of the computing as a series of Look Up Tables (LUTs), and optimize for latency and utilization with fancy switching fabrics. The downside of the switching fabrics is that optimizing a design to fit an FPGA can sometimes take days.
- ein0p 2y ago+1. For inference especially compute is abundant and basically free in terms of energy. Almost all of the energy is spent on memory movement. The logical solution is to not move unaggregated data.