lhl

32 ms·

331.
by lhl 3y ago
I watched Huang's Computex presentation and was pretty impressed by the hardware they're launching, but it's worth noting some caveats. While it was announced to be in "full production," according to the blog post,
332.
by lhl 3y ago
The more VRAM the better if you'd like to run larger LLMs. Old Nvidia P40 (Pascal 24GB) cards are easily available for $200 or less and would be easy/cheap to play. Here's a recent writeup on the LLM performance you can expec
333.
by lhl 3y ago
Maybe in text, but we won't be running out of multi-modal training data (images, audio, video, sensor data, etc) any time soon.
334.
by lhl 3y ago
No need to make assumptions, the pachinko market has been continuously shrinking for decades: https://www.statista.com/statistics/1171592/japan-market-siz...
335.
by lhl 3y ago
Namespace collisions are inevitable, especially w/ how fast-moving the LLM space is right now, just wanted to point out that besides this "Open-Llama" project (which looks really interesting, and well documented in the Github
336.
by lhl 3y ago
There's a method (speed running glitch) to get the master sword w/o the heart requirement. Do a search for "botw get the master sword early" and you should have lots of step by steps if you ever want to do it.
337.
by lhl 3y ago
It does have the same system so I guess you should return it/cancel your order. Like BOTW, even from the very beginning of the game, you never run out of weapons/always have more weapons than slots, so for anyone else, I'd re
338.
by lhl 3y ago
GPT4 had no problem with this. Which model are you running? https://sharegpt.com/c/edmPzmP
339.
by lhl 3y ago
Here is Jason Wei's response "Common arguments regarding emergent abilities": https://www.jasonwei.net/blog/common-arguments-regarding-eme... Wei was the lead author of the original "Emergent Abilit
340.
by lhl 3y ago
https://www.reddit.com/r/Oobabooga/comments/126dejd/comment/...
341.
by lhl 3y ago
I've seen reports of people being able to run LLMs at decent speeds on old Nvidia P40s. These are 24GB Pascal GPUs and can be bought for as low as $100 (although more commonly $200) on eBay.
342.
by lhl 3y ago
llama-65b on a 4-bit quantize sizes down to about 39GB - you can run that on a 48GB A6000 (~$4.5K) or on 2 x 24GB 3090s (~$1500 used). llama-30b (33b really but who's counting) quantizes down to 19GB (17GB w/ some optimization), s
343.
by lhl 3y ago
I think from the perspective of a Google researcher/engineer, it must be alarming to see the crazy explosion going on w/ LLM development. We've gone from just one or two weirdos implementing papers (eg https://gith
344.
by lhl 3y ago
> Paradoxically, the one clear winner in all of this is Meta. Because the leaked model was theirs, they have effectively garnered an entire planet’s worth of free labor. Since most open source innovation is happening on top of their arch
345.
by lhl 3y ago
Although there are multiple bottlenecks, my understanding (and why at a certain point, throwing more threads doesn't work) is that inference for dense LLMs are largely limited by memory bandwidth. Most desktop computers will have dual
346.
by lhl 4y ago
While I agree we probably aren't getting exponentially increasing parameter counts (GPT4 is by all accounts 1T paramaters and of course, it is significantly better than GPT3) we are still seeing lots of improvements - 3.5 is much bet
347.
by lhl 4y ago
It seems to me, from a copyright perspective, all commercial use of generative AI depends on whether the output is transformative fair use (vs derived work). While the courts will have its say, ultimately whether new rules are carved out or
348.
by lhl 4y ago
While 7000U laptops have yet to be benchmarked, dual-channel DDR5/quad-channel LPDDR5 systems top out at about 60GB/s. (The M1/M2 by comparison is a 100GB/s, and doubles for Pro, Ultra, and Max up to 800GB/s). As a
349.
by lhl 4y ago
Looks like my edit window closed, but my results ended up being very low so there must be something wrong (I've reached out to StabilityAI just in case). It does however seem to roughly match another user's 3B testing: https:&#
350.
by lhl 4y ago
I'm still on the waitlist for GPT-4 API access. Note, that text-davinci-003 cost about $90 to benchmark at $0.02/1K tokens, so if you're able to use a GPT-4 model (for completion and not just instruction) that'll probabl
351.
by lhl 4y ago
https://arxiv.org/abs/2205.14135 - Section 5 suggests that the biggest limitation is that custom CUDA kernels need to be coded on a per-GPU architecture basis.
352.
by lhl 4y ago
Yeah, although looks like it currently has some issues with coqa: https://github.com/EleutherAI/lm-evaluation-harness/issues/2... There's also the bigscience fork, but I ran into even more problems (alth
353.
by lhl 4y ago
I've been diving in lately and while it's not efficient, the only way to do manage is to create a new conda/mamba environment, or a custom Docker image for all the conflicting packages. For safety and speed, you should prefer
354.
by lhl 4y ago
I think "sweet spot" is going to depend on your task, but here's a good recent paper that may give you some more context on thinking about training and model sizes: https://www.harmdevries.com/post/model-
355.
by lhl 4y ago
FYI, I'm running lm-eval now w/ the tests Bellard uses (lambada_standard, hellaswag, winogrande, piqa,coqa) on the biggest 7B an 40GB A100 atm (non-quantized version, requires 31.4GB) so will be directly comparable to what various
356.
by lhl 4y ago
I started keeping a list btw, there are about 20 completely open (Apache 2.0, BSD, MIT, CC-BY) 1B+ parameter foundational LLMs at the moment: https://docs.google.com/spreadsheets/d/1kT4or6b0Fedd-W_jMwYp...
357.
by lhl 4y ago
I saw this in my feed recently which was an interesting analysis on how code training was added as a fine tune (Codex) on a foundational model (GPT-3): https://yaofu.notion.site/How-does-GPT-Obtain-its-Ability-Tr... I do wo
358.
by lhl 4y ago
Sure, and I was merely clarifying that only last-gen architectures work. While I'm glad that SD works for you, if we are getting into it, I think that having no support on current-gen flagship models does equal broadly terrible softw
359.
by lhl 4y ago
Fabrice Bellard has run a standard set of benchmarks w/ lm-eval on a big chunk of open models here: https://bellard.org/ts_server/ - Flan T5 XXL and GPT-NeoX 20B both outperform Pythia 12B on average (LLaMA 13B+ t
360.
by lhl 4y ago
Yes, you can run inference at decent speeds on CPU with llama.cpp. A token is about 0.75/words, so you can see lots of people getting 4-8 words/s on their CPUs: https://github.com/ggerganov/llama.cpp/issu

More ›