lhl

19 ms·

151.
by lhl 2y ago
Just spotted this link. Just to clarify, I (not the original poster, although everyone's welcome to share this link, it's a public doc) maintain this list (and the rest of the sheet) manually. While I keep the foundation models th
152.
by lhl 3y ago
llama.cpp probably won't be getting Jamba support anytime soon: https://github.com/ggerganov/llama.cpp/issues/6372#issuecomm... There is an MLX Mamba implementation, but nothing for Jamba either: https:
153.
by lhl 3y ago
For anyone that doesn't need the highest performance and where efficiency is super important, I can recommend the current generation of mini PCs/NUCs that run mobile chips. Last summer I picked up Ryzen 7 7940HS-based mini PC that
154.
by lhl 3y ago
While the G series (basically mobile chips) do idle a lot lower, here's a chart that shows that Ryzen chips idle at about the same (a few watts less actually, but negligible) than their Intel high-end desktop counterparts: https:&#x2F
155.
by lhl 3y ago
There have been updates on core parking ("Game mode") in Windows so it's probably fine, but I think for max perf, people are still using Process Lasso (I don't use a mixed X3D chip myself, so I haven't paid super cl
156.
by lhl 3y ago
Largely yes, but the 7800X3D ($370->$300 on sale, 120W TDP) still beats the 14900KS since the extra L3 cache is more important than clock for most games: https://www.techpowerup.com/review/intel-core-i9-14900ks/
157.
by lhl 3y ago
Claude 3 is very capable, but it is (likely) a 1T class model, not something that can be run on the edge, while 7B class models can already be run on phones and can be easily fine-tuned to do specialized work that can perform comparably to
158.
by lhl 3y ago
I’d highly recommend anyone considering trying one out to give it a pin. I’ve used a CGM multiple times besides being a great feedback mechanism, it also can give you really interesting insights. Here’s one interesting tidbit showing that h
159.
by lhl 3y ago
Someone got ZLUDA running llama.cpp a while back (search the ZLUDA/llama.cpp issues). If I recall, it ran about half the speed of the existing ROCm implementation. I tried ROCm on my iGPU last year and you do get a bit of a benefit for
160.
by lhl 3y ago
I have a similar system. When idle, nvtop reports about 11-12W/4090. When doing bs=1 inference, the default PL for my cards is 450W each, although I've found that I can keep 97% of prefill and 99%+ of inference performance lowerin
161.
by lhl 3y ago
You can run a Q4 quant of a 70B model in about 40GB of RAM (+context). You're single user (batch size 1, bs=1) inference speed will be basically memory bottlenecked, so on a dual channel dedicated box you'd expect somewhere about
162.
by lhl 3y ago
Just an FYI, Mixtral is a Sparse Mixture of Experts that has 47B parameters for memory costs (but 13B active parameters per token). For those interested in reading more about how it works: https://arxiv.org/pdf/2401.040
163.
by lhl 3y ago
Well, you probably read a inaccurate headline about it. The project is called ZLUDA https://github.com/vosen/ZLUDA and it had a recent public update because of the opposite - AMD decide not to continue sponsoring wor
164.
by lhl 3y ago
If you are running a local LLM already (which no one in the "general public is") then the bar is really not that much higher for fine-tuning (either for an individual or community member to do). And you don't need any additio
165.
by lhl 3y ago
It is trivial to fine tune any model (whether a base model or an aligned model) to your preferred output preferences as long as you have access to the model weights .
166.
by lhl 3y ago
here are some GGUFs https://huggingface.co/brittlewis12/Snorkel-Mistral-PairRM-D...
167.
by lhl 3y ago
A new 7B model, Snorkel-Mistral-PairRM-DPO, using a similar self-rewarding pipeline was just released: * Announcement: https://twitter.com/billyuchenlin/status/1749975138307825933 * Model Card: https://
168.
by lhl 3y ago
I haven't but I know there are writeups I've seen online that should be easy to find of people using SD w/ APUs.
169.
by lhl 3y ago
For macOS and Linux, Ollama is probably the easiest way to try Mixtral (and a large number of models) locally. LM Studio is also nice and available for Mac, Windows, and Linux. As these models can be quite large and memory intensive, if you
170.
by lhl 3y ago
You can make a privacy request for OpenAI to not train on your data here: https://privacy.openai.com/ Alternatively, you could also use your own UI/API token (API calls aren't trained on). Chatbot UI just got a ma
171.
by lhl 3y ago
ChatGPT's web search is interminably slow and I've added to my custom prompt to not do web searches unless explicitly asked. However, I'd give Perplexity.ai a try - I've found it to be incredibly fast and useful (funn
172.
by lhl 3y ago
My understanding is that Mistral uses a regular 4K RoPE that is "extends" the window size with SWA. This is based on looking at the results of Nous Research's Yarn-Mistral extension: https://huggingface.co/Nou
173.
by lhl 3y ago
Yes, I think "predecessor" is a more apt term than competitor. AGP was introduced in 1997 and was pretty ubiquitous for almost a decade. While the first PCIe graphics cards started coming out around 2004, AGP cards continued to be
174.
by lhl 3y ago
I've given Aider and Mentat a go multiple times and for existing projects I've found those tools to easily make a mess of my code base (especially larger projects). Checkpoints aren't so useful if you have to keep rolling bac
175.
by lhl 3y ago
I've been really enjoying Perplexity as well. It's a much better Internet/search focused experience than ChatGPT, Bing, or Bard. For anyone interested, until the new year (~20 more hours?) there's a code for 2mo free P
176.
by lhl 3y ago
Most LoRAs are less effective for facts since changes largely shift attention (Q and K, not V layers) and only touch a tiny percentage of weights at that, however full fine-tunes on models are pretty effective for introducing new facts (y
177.
by lhl 3y ago
Claimed in the linked article is about a 2X. The linked post has more details (as does the Model Card).
178.
by lhl 3y ago
Sure, I think their quant format is pretty basic, something similar to bnb q4 - my plan will to be scripting a framework for testing, so should do that as well since the omniquant implementation is in mlc-llm anyways.
179.
by lhl 3y ago
It's been on my list to do a proper shootout of all the various new quant formats floating around (my list here: https://llm-tracker.info/books/llms/page/quantization-overvi... ) but a lot of them don&#x2
180.
by lhl 3y ago
Recently I've been using GCP to train a model, some notes: * Like @leetharris, credits were pulled/not distributed, what was promised had to be cajoled out with most of it being sent to some weird SaaS product that we'll neve

More ›