valine

13 ms·

121.
by valine 3y ago
It’ll come down to who invents AGI. If it were me I’d open source it. I’m sure a lot of people would do the same.
122.
by valine 3y ago
Ask ourselves? Who is us? Your comment doesn’t apply if everyone has access to their own local AGI. Right now training is expensive and inference is cheap. There’s no reason to believe that inference cost will suddenly balloon when we cros
123.
by valine 3y ago
I’ve been studying and tinkering with open weight LLMs since the original llama weights leaked. I’ve very recently become convinced that the true data and compute requirements needed to fine tune and produce an “unsafe” model are orders of
124.
by valine 3y ago
Makes me wonder what will happen as new, more efficient methods of training are discovered. Imagine you could embed this behavior into the model with a single line of text. Creating a malicious model would be far easier, but it would also b
125.
by valine 3y ago
Embeddings at lower layers aren’t going to be looking very far beyond nearby or adjacent embeddings as they refine their meaning. For a number like 3.14, the tokens 3 and 14 are important to each other, but entirely unimportant to understan
126.
by valine 3y ago
The method they use is surprisingly simple. They claim GPTs can’t effectively generate beyond the context window because our models overfit on positional encodings. The fix is literally to cap the positional encodings at inference time. It
127.
by valine 3y ago
> It is an incredibly common refrain amongst experts in the field that it is a compression of the dataset This was a common idea three years ago. No one in the field seriously believes this today.
128.
by valine 3y ago
You really think we’re gonna be able to solve hallucination before this regurgitation problem? Please.
129.
by valine 3y ago
This might be a controversial take, but I just can’t imagine being emotionally invested in this whole copyright drama. What’s the use case here NYT is trying to prevent? A user spending hours to force the model to regurgitate articles that
130.
by valine 3y ago
I see the source of your confusion. LLMs are not actually zips of the training dataset.
131.
by valine 3y ago
The idea that reading a piece of text constitutes copyright infringement is ridiculous. Copyright isn’t some infectious thing. Reading copyrighted text doesn’t give the copyright holder a claim to the future creative work of the reader. You
132.
by valine 3y ago
When I read your comment I trained my own mental model on your words. How is that any different? When a human reads words they apply a sophisticated theory of mind to contextualize the writing and the mental state of the author. If anything
133.
by valine 3y ago
The bandwidth limitations mean it won’t be useful in populated areas. This is for when you’re out camping and there are five people on the network per square mile.
134.
by valine 3y ago
GM is not equipped to build software, and that’s okay. What’s not okay is that GM lacks the basic self awareness to recognize their own limitations. The schadenfreude is strong with this one.
135.
by valine 3y ago
CCS1 is still dominate in North America. CCS2 only got traction in Europe.
136.
by valine 3y ago
When people talk about standardizing the charge port they’re talking about the shape of the port not the protocol. Tesla has supported the CCS protocol for years now. The Tesla port is vastly superior to the CCS type 1 port. They should get
137.
by valine 3y ago
We have laws that prevent people being subjected to brain surgery against their will. The credit score concept is ridiculous. The real battle will be with law enforcement who get a warrant to look at your brain in an MRI.
138.
by valine 3y ago
Uh huh, try GPT4 and report back. It’s a generational leap above copilot. I use copilot to auto complete one liners and GPT4 to generate whole methods.
139.
by valine 3y ago
The way I write code was fundamentally altered in the last year by GPT4 and copilot. Try having GPT4 write your code, you won’t be so certain about the future of programming afterward I guarantee it.
140.
by valine 3y ago
Think of it more like wood fiber reenforced epoxy. It will behave more like epoxy for most applications.
141.
by valine 3y ago
That’s cool. A couple hours on a single GPU or like 8x a100s?
142.
by valine 3y ago
How much data do you need for UNA? Is a typical fine tuning dataset needed or can you get away with less than that?
143.
by valine 3y ago
Holy crap that demo is misleading. Thanks for the link.
144.
by valine 3y ago
More like 10 years ago, that’s when Microsoft dropped the ball.
145.
by valine 3y ago
That’s more down to dumb luck partnering with OpenAI. You have a point with the cloud computing. I’d hesitate to call it a revolution though. I’d never willingly use a microsoft cloud product if I wasn’t forced to by my employer.
146.
by valine 3y ago
Sounds about right to me. They missed the entire smartphone revolution because they were too slow to adapt their OS (literally their main product) to run on mobile devices.
147.
by valine 3y ago
How do you think the cup demo works? Lots of still images?
148.
by valine 3y ago
It’s not live, but it’s in the realm of outputs I would expect from a GPT trained on video embeddings. Implying they’ve solved single token latency, however, is very distasteful.
149.
by valine 3y ago
It’ll be around 17GB + context. For longer contexts you’ll add a couple GBs.
150.
by valine 3y ago
I don’t think there’s many three body problems in the real world where compute is the limiting factor. If you’re projecting so far out into the future that simulation speed is a problem, you’re initial measurement error will have compounded

More ›