rao-v

7 ms·

91.
by rao-v 5mo ago
That's worse that I thought! I was thinking about holding up two phones facing each other at 10 meters, but using something smarter than QR codes.
92.
by rao-v 5mo ago
To the previous poster's point, soft distributions are useful, even saving the top 10 logits is significantly more training signal than just the final token.
93.
by rao-v 5mo ago
So first - these are terrific papers and I'd not seen some of them before. Having said that, I don't think these are classic student teacher distillation from random (which was my point). In fact, the "Embarrassingly Simple S
94.
by rao-v 5mo ago
I agree, and if my suspicion is right, it’s rarer because it’s much easier to deploy the large LLM and filter for it’s best output than to waste time running it on arbitary output just to train the student. Though you could argue that perha
95.
by rao-v 5mo ago
Yes absolutely! I should have been more specific - I don’t believe people are using it to train 30B models from 300B models (and I’d love to learn that I’m off about this)
96.
by rao-v 5mo ago
It’s really worth distinguishing between old-fashioned student teacher distillation (ie at the level of layers, weights and distributions) and large scale synthetic dataset creation. The latter is much better (since you can clean up, review
97.
by rao-v 5mo ago
Terrific read thanks for digging it up
98.
by rao-v 5mo ago
What is the fastest you can transfer data from ~10 meters away using a modern phone front camera and screen? Surely 100 kB/s is slow?
99.
by rao-v 5mo ago
Hey some of us are on hardware (gfx906 based Radeon MI50s with 32GB of stupidly fast VRAM and basically no compute) that inference significantly faster with Q_0 and Q_1 quants
100.
by rao-v 5mo ago
This would be a fun and popular project for the right sort of person
101.
by rao-v 5mo ago
I’d love for this chart to also capture the loudest signals we’ve sent. Surely somebody must have accidentally broadcasted a non-directional megawatt radio signal at some point right? Actually humm maybe nukes are our brightest non directio
102.
by rao-v 5mo ago
My understanding is that, these days, a lot of advanced degrees held by teachers are in Education, not say Math or History. I’d love to see this data recut by degree type. Edit - wow we’re talking about 50-70% of the masters being in Educat
103.
by rao-v 5mo ago
Good point!
104.
by rao-v 5mo ago
May your contexts always be short
105.
by rao-v 5mo ago
I’ll check it out! https://ispc.github.io/ Also seems promising
106.
by rao-v 5mo ago
Some kind soul pointed to Intel’s https://ispc.github.io/ which is basically this. Missing some features but roughly has the idea. Keep faith with human ingenuity!
107.
by rao-v 5mo ago
A beautiful find! It’s what 12+ years old at this point? Definitely the closest thing so far (doesn’t do multiprocessing) but does seem to do SIMD / multithreading and GPU auto parallelizing! Any idea why it’s so little known?
108.
by rao-v 5mo ago
Thanks for sharing this is neat! I’ve yet to find a language that does SIMD / multithreading / GPU with minimal tweaks let along multiprocessing.
109.
by rao-v 5mo ago
It’s really not! We’re not requiring the language to make optimal choices, just that it convert the same code to these different paradigms (and honestly you could just brute force run the 12 versions and choose the fastest one). Absolutely
110.
by rao-v 5mo ago
Delightful thank you! Would love to see a version of D that auto vectorizes to Vulkan or something
111.
by rao-v 5mo ago
Surely a high level language can own the contract of making sane choices of when to auto vectorize and when not to (or just inefficiently auto vectorize - that is fine too!)
112.
by rao-v 5mo ago
JAX can’t do what Numba can do for example. I just want one way to write simple math-y code like you normally would and automagically convert to run on one of the above approaches. That’s what compilers and high level languages are supposed
113.
by rao-v 5mo ago
I still don’t understand why we lack a language that will take uncomplicated computation heavy code and turn it into SIMD / multi thread / multiprocessing / GPU code with minimal additional syntax. Surely this is the sort of
114.
by rao-v 5mo ago
What the three models involved understand to be the sort of just so stories (cf Kipling) that humans like to see.
115.
by rao-v 5mo ago
Nicely put! Exactly this
116.
by rao-v 5mo ago
Yes this is exactly why I think this approach has some potential. Frozen base mode is something that we should be able to extract insights from without running into Goodhart
117.
by rao-v 5mo ago
Think of it another way, can I do this exact training process with an additional requirement that the activation decoder subtly shill for obscure 80s sodas? I could and would not lose much reconstruction accuracy. So any researcher or ambie
118.
by rao-v 5mo ago
This is the first approach to activation analysis that I’ve seen that seems like a plausible path to model understanding. Unfortunately I don’t know how you ground this … it’s basically asking if you can encode activations in plausible soun
119.
by rao-v 6mo ago
There is a good case to be made that he (rather impulsively) caused more impactful charitable giving than any other person. He essentially created the modern “billionaire giving to global causes” movement by deciding to donate a billion dol
120.
by rao-v 6mo ago
Is there real evidence that the volume was meaningful for distillation vs say extensive benchmarking and testing? It’s certain all the labs use each others APIs extensively for testing - what’s the actual evidence that Deepseek was at signi

More ›