lhl

25 ms·

241.
by lhl 3y ago
I made some notes recently on the various code helpers/editors available, both SaaS/services-based and ones that are more BYOB: https://llm-tracker.info/books/llms/page/code-assistants
242.
by lhl 3y ago
> 2. GPT-4 does not seem to have improved on real-world coding tasks since March, so it's unclear where any purported HumanEval gains could've come from Once Markdown formatting is accounted for, the June model improves answers
243.
by lhl 3y ago
LocalAI https://localai.io/ and LMStudio https://lmstudio.ai/ both have fairly complete OpenAI compatibility layers. llama-cpp-python has a FastAPI server as well: https://github.com/abetlen
244.
by lhl 3y ago
It is indeed a fact that OpenAI's Terms of Use do state that you can't use their service to develop competing models: Section 2.c.iii - https://openai.com/policies/terms-of-use Now of course, the terms are n
245.
by lhl 3y ago
Kagi https://kagi.com/ is the paid search I've tried. It's was decent, but not a noticeable enough improvement to win me over as a customer. OTOH, I am a paying ChatGPT Plus user and there are some days where I fe
246.
by lhl 3y ago
I dunno, I gave up on Wirecutter for anything I actually cared about almost 10 years ago - I found that for anything that I had actually done even basic research on, the Wirecutter was always wrong (and not just like sort of wrong, but like
247.
by lhl 3y ago
Yes, just use the "translate" instead of the "transcribe" option.
248.
by lhl 3y ago
Probably, although you could more easily use WhisperX and get the same results twice as fast and without any additional scripting.
249.
by lhl 3y ago
I gave it a spin a little bit ago. Per usual, install docs didn't quite work OOTB, here's how I got it working: https://llm-tracker.info/books/howto-guides/page/speech-to-t... One limitation that se
250.
by lhl 3y ago
Also recently released is "CoDeF: Content Deformation Fields for Temporally Consistent Video Processing" https://qiuyu96.github.io/CoDeF/ "We present the content deformation field (CoDeF) as a new type o
251.
by lhl 3y ago
Onsen are great (being out in a starry night in a rotenburo (outdoor bath) with a light snowfall is a must-experience IMO), but a lot of what you're paying for will be the traditional ryokan (traditional Japanese inn) experience and ka
252.
by lhl 3y ago
I saw that show, and as an onsen fan loved it, but recently I stumbled upon a pretty interesting article on how Roman baths were... not all that great from a health/hygiene perspective: https://archive.is/Y1un4
253.
by lhl 3y ago
I've harped on this on the past, the "official" hardware support list is tiny: https://rocm.docs.amd.com/en/latest/release/gpu_os_support.h... But, it's worth noting there's different
254.
by lhl 3y ago
For not much more (US$200) you can find lots of P40s, which is a generation newer and will give you double the memory bandwidth and FP32. That being said, used 3090s are going for about $600 now and are much better bang/buck and easier
255.
by lhl 3y ago
You can use a calculator: https://www.unitsconverters.com/en/Mt/S-To-Gb/S/Utu-6007-376... - enter in 5600 MT/s and get 44.8 GB/s x 2 channels = 89.6 GB/s I have a 7940HS on my desk now and
256.
by lhl 3y ago
Basically, the most aggressive quants for a larger sized model is always better than the unquantized model of a smaller model: https://github.com/ggerganov/llama.cpp/pull/1684 Tim Dettmers did a bunch of rese
257.
by lhl 3y ago
Just the other day someone published ARC comparison results for different quants as well as the code for the harness that they used to easily run lm-eval against quants to your heart's content: https://www.reddit.com/r&
258.
by lhl 3y ago
I think it's pretty useful to share concrete examples like this. I basically always have a ChatGPT4 CI tab open these days when I'm developing these days - it's a usually much faster/better go-to than Google or SO for lo
259.
by lhl 3y ago
Those are very different questions... If you want to simply run inference or do QLoRA fine tunes of "the best, largest open source models" eg the llama2-70b models, you can do so with 2 x RTX 3090 24GB (~$600 used), so for about $
260.
by lhl 3y ago
For more standardized speed benchmarking, I'd recommend benchmarking with something like `-c 128 -n 1920 --ignore-eos` (and skipping the `-p` entirely). The number that you care most about would be the "eval time" tokens&#x2F
261.
by lhl 3y ago
On Windows, llama.cpp has OpenCL support (CLBlast) and MLC LLM ( https://mlc.ai/mlc-llm/docs/ ) has Vulkan acceleration. On Linux, ExLlama and MLC LLM have native ROCm support, and there is a HIPified fork of llama.
262.
by lhl 3y ago
WizardCoder-15B (an evol-instruct starcoder fine-tune) is probably the best performing open model atm: https://github.com/nlpxucan/WizardLM/tree/main/WizardCoder
263.
by lhl 3y ago
There are more evals than you can shake a stick at, but none of them are comprehensive or without their own issues. If you're looking for a lists of lists, I keep one here: https://llm-tracker.info/books/evals&#x2F
264.
by lhl 3y ago
That specific attack requires adversarial inputs crafted crafted against gradients so only works against open models (requires known model weights). There are dangers that include leaking PII from your current context, but also worse if y
265.
by lhl 3y ago
While there's some contamination, it's not like the community isn't aware of it. For example, here's this discussion: https://huggingface.co/sahil2801/replit-code-instruct-glaive... This was the Hum
266.
by lhl 3y ago
Note, that just because you're hosting itself won't mean you're securing it properly... Here's a just published injection attack that only works on open source models (public model weights): https://twitter.co
267.
by lhl 3y ago
You will want to look at HumanEval ( https://github.com/abacaj/code-eval ) and Eval+ ( https://github.com/my-other-github-account/llm-humaneval-ben... ) results for coding. While Llama2 is an improvem
268.
by lhl 3y ago
It depends on the eval, but I think it's fair to say that it's close. Here is the AGI Eval results organized into a table w/ averages (also I put in the new Hermes LLama2 13B model as well: https://docs.google.com&
269.
by lhl 3y ago
This was just posted a few hours ago and when I tried it they were neck and neck (for me, LLama 2 won by a 1 question, but it was close): https://llmboxing.com/ It looks like the eval is open sourced so you could easily bui
270.
by lhl 3y ago
The more relevant question is which GPU is the OP using? The only officially ROCm supported GPU available for retail purchase is the RDNA2-based Radeon Pro W6800. [1] In practice it probably means that gfx1030 (Navi 21) GPUs should work (

More ›