Taek

19 ms·

301.
by Taek 3y ago
It's called a backronym :)
302.
by Taek 3y ago
Maybe we will finally get a robust ecosystem of micropayments on the web
303.
by Taek 3y ago
There's also research showing that the perplexity reduction is less at higher parameter counts. E.g. a 65b parameter model barely has any impact at all when reducing from 16bit to 4bit
304.
by Taek 3y ago
I feel like there should be a basic 'curriculum' that gets passed to all foundational LLMs that teaches them the basics of language. Maybe 100 million files where the first 10 million are all first grade reading level, the second
305.
by Taek 3y ago
I've successfully gotten at-home models ( https://huggingface.co/NousResearch/GPT4-x-Vicuna-13b-fp16 ) to go through my messages and pull out key todo items. For example, reminding me to message my father about trav
306.
by Taek 3y ago
Without performance penalties? If the model is larger than the vram you have to constantly be pulling data from disk/ram right?
307.
by Taek 3y ago
Its probably only the 8k model that runs on 2
308.
by Taek 3y ago
Didn't some OpenAI engineer state that GPT4 runs on 2xH100? At 4 bit quantization, that gives an upper bound of 320B params, realistic upper bound probably more like 250B
309.
by Taek 3y ago
75B tokens is not really enough data to make an intelligent model. Llama was trained on over 1 trillion tokens. And yes, training on top of LLaMA could introduce a lot of unexpected behavior, but that's just where the State-of-the-Art
310.
by Taek 3y ago
The main moat here is VRAM, not raw compute power.
311.
by Taek 3y ago
The biggest llama model has near 100% fidelity (its like 99.3%) at 4 bit quantization, which allows it to fit on any 40GB or 48GB GPU, which you can get for $3500. Or at about a 10x speed reduction you can run it on 128 GB of RAM for only a
312.
by Taek 3y ago
How is this different from what RedPajamas is doing? Also, most people don't mind running LLaMA 7B at home so much because of enforceability, but a lot of commercial businesses would love to run a 65b parameter model if possible and ca
313.
by Taek 3y ago
I have it running locally using the oobabooga webui, setup was moderately annoying but I'm definitely no python expert and I didn't have too much trouble.
314.
by Taek 3y ago
This reads a lot like "The future of the Internet is niche (fax machines), not generalized)
315.
by Taek 3y ago
There are good reasons to believe that this will be the new state of the art by a comfortable margin. Hard to know until we can actually play with it.
316.
by Taek 3y ago
For anyone who doesn't know, DeepFloyd is a StableDiffusion style image model that more or less replaced CLIP with a full LLM (11b params). The result is that it is much better at responding to more complex prompts. In theory, it is al
317.
by Taek 3y ago
Teenagers are mostly into screwing around and pushing boundaries, and myai is an absolute killjoy
318.
by Taek 4y ago
One big benefit of LLMs is that they should be able to parse the HTML page and detect dark patterns, and then re-write the UI to fix them.
319.
by Taek 4y ago
Nobody knows where to find 10 trillion tokens of good data. Publicly available / data without a license seems to cap at around 1.5 trillion tokens total. The internet isn't as big as you thought! (Or at least, all the good stuff i
320.
by Taek 4y ago
Well, based on all the data we have available now it seems like you don't get much benefit yet from going above 200 billion.
321.
by Taek 4y ago
They used training data with restricted licenses to produce the fine-tuned models, it's not a decision that stability was able to make.
322.
by Taek 4y ago
NC is just for the fine tuned model
323.
by Taek 4y ago
That may be, but the weights you can download today were trained on 800B
324.
by Taek 4y ago
Devs confirmed that the small ones use 800B, 1.5T is for the large ones
325.
by Taek 4y ago
I think you underestimate how intelligent LLMs are. If the training data only explains a certain concept in French, the LLM will nonetheless be able to tell you about that concept in any other language it has proficiency in. There's cl
326.
by Taek 4y ago
Are you working at all with Stability, Eleuther, or LAION? There have been some rumors that they are doing something similar to this and I'm wondering if this is a duplicated effort. Either way, huge fan, it would be awesome to have a
327.
by Taek 4y ago
It's also relatively cheap to make your own llama-30 weights, the real value of OpenAssistant is in the training data, and all of that data has been made available. The OpenAssistant effort gets an A+ for open source contributions.
328.
by Taek 4y ago
It means LLMs might self-improve beyond the point where we can comprehend how intelligent they are. An LLM with 10x the capabilities of all human brains combined is indistinguishable to a human with one that has 100x the capabilities of all
329.
by Taek 4y ago
I think part of the issue is that consumers have never needed more than 12 GB of VRAM before. Game developers just don't have requirements that high. Crypto mining also doesn't require that much. Now that there's clear demand
330.
by Taek 4y ago
Definitely disappointed to see how paternalistic the open model is, was hoping it'd be more free-flowing and less corporate. However, all of the data is open, and you can actually prompt these models to revise the training data to remo

More ›