kolinko

24 ms·

511.
by kolinko 3y ago
Thanks! The last full month was all about making sure all the research is as replicable and as trustworthy as I can do it. The original implementation was extremely inefficient, and even once I had the Metal/GPU mmul op working fast, I
512.
by kolinko 3y ago
Thanks for checking it out! I also can’t wait to have more tests done. Also, I chose Apple Silicon because it was easier for me to develop on. It is possible that the algorithm would also have good performance on other architectures.
513.
by kolinko 3y ago
Ah, I get the question now. If I understand correctly CSR, it stores indexes and values as a list. I store them also as a list - that's why the comparison is there. The difference is that with CSR you store say 15% of the values from a
514.
by kolinko 3y ago
Haha thanks! :) As far as I understand I had to implement the memory reads and some other things the opposite way to what is considered a proper approach. Would love to have that code reviewed by someone who actually knows stuff about Metal
515.
by kolinko 3y ago
That was the claim with a lot of the software in the past, but open source won in many places in the end.
516.
by kolinko 3y ago
The algorithm still uses all the weights, just not all the time - just skips the weights when they are not important given an input vector. Also, approximation methods, as a field, are not new and they have shown their use. Having said all
517.
by kolinko 3y ago
Ah yes, forgot to make the repo public. Thanks a ton for pointing it out and writing the comment - I'd miss it if you didn't point it out on HN.
518.
by kolinko 3y ago
Haha yes :) Publish it, Kacper! The project is a nerdsnipe for math geeks, because there are multiple small things that beg to be proven / described by math there. For example - what's the tradeoff between the number of bits we lo
519.
by kolinko 3y ago
oh, thanks, I fixed it. No idea what I meant there originally. There are still a few typos on the page, I'll be fixing them tomorrow - it's midnight now, and my mental batteries are slowly drying out :)
520.
by kolinko 3y ago
Nah, sadly this most likely will not work with Q1/Q1.5 implementations - initially I was playing with monte carlo approximations (before I arrived at bucketMul), and the convergence was very slow for binary/ternary networks. Or, i
521.
by kolinko 3y ago
Yes! That would be awesome. Especially since there are ~32*6 independent effort settings for every single token. I tested the most basic implementation, with a flat effort setting for all the muls, but I bet the results could be pushed even
522.
by kolinko 3y ago
Oh, nice to finally bump into someone who has experience with CSR! bucketMul has few uncoalesced reads, and it uses a different data structure than the regular CSR - it's decribed here: https://kolinko.github.io/effort&
523.
by kolinko 3y ago
Hahah thanks, although I was hoping for a flame to get adrenaline flowing to push through the night :D I also hope there will be an extra knob - or more like knobs, because effort can be regulated smoothly layer by layer, token by token, ma
524.
by kolinko 3y ago
The whole inference is slow, but it's matrix multiplications that count. They work reliably on all the Macbooks that I tested - at 50% effort it's the same speed as the state of the art matrix multiplications, at 25% they are twic
525.
by kolinko 3y ago
Nah, that's not it, I measure the CPU & GPU work separately, and 15ms happens between the kernel invocations. It also happens when I don't print out text. Thanks for the idea though! I treat it as the first community contribut
526.
by kolinko 3y ago
In this case though, the algorithm should be just as useful to closed models as to open models. There is nothing there that is optimised specifically for Mistral - aside from the hard-coded dimensions in multiple places in the code :D Havin
527.
by kolinko 3y ago
It originally started as a fork to Recmo’s cria pure numpy llama impl :) https://github.com/recmo/cria Took a whole night to compute a few tokens, but I used it to do the first tests. Also, my friend pasted the paper t
528.
by kolinko 3y ago
Per VRAM - not much netter, because it still uses all the weights, just not all the time. I mean - it can also load less weights, but quality seems to degrade quick after offloading more than 20-30% weights. In other words - this algorithm
529.
by kolinko 3y ago
I’d say that this expands on MoE really - MoE chooses dynamically which groups of weights may be needed, but it’s whole matrixes. Here it’s ingle weights. Also, this works on top of MoE beautifully - most of the development and testing was
530.
by kolinko 3y ago
Haha halfwit! I’m waiting for such a fork. As for predicting the weights - not necessarily so. It seems most weights are being used, just not all the time. Kind of like that saying that humans are using just 5% of their brain - perhaps they
531.
by kolinko 3y ago
It’s explained in detail here: https://kolinko.github.io/effort/equations.html Long story short - I kind of sort them and pick only the top % that would give a highest result. One part is choosing them though - I think
532.
by kolinko 3y ago
hard to say for now, I’m curious as well, but I used simpler tests so far because of the implementation issues - most test suites are geared towards testing models and not model implementation. I didn’t want to wait any longer with the rele
533.
by kolinko 3y ago
Hahaha :)
534.
by kolinko 3y ago
Similar theme, but they skip whole neurons, in my case it’s down to a level of single weights. From my experiment, skipping whole neurons (so whole rows/columns of matrixes) didn’t allow for such good results. In my case 30-50% neurons
535.
by kolinko 3y ago
Can you rephrase the question? Not sure I get it.
536.
by kolinko 3y ago
Yes! I hope, if it’s proven, that it will be implemented into the main inference engines. 40% effort is only a bit faster than a full base multiplication, but I hope both the speed and the accuracy could be improved further.
537.
by kolinko 3y ago
Yeah, that is the point more or less - it dynamically chise the weights layer per layer depending on the internal state. A bit technical explaination here. https://kolinko.github.io/effort/equations.html
538.
by kolinko 3y ago
Hahah, thanks! It was a marathon to get develop this, and I'm glad it reached the front page. The name was proposed by chatgpt :) It claims it doesn't recognise this approach - so there is a chance it's really a new thing. I
539.
by kolinko 3y ago
Here to answer questions! A friend of mine has published the original link on HN before me, so I hope a double post won't be an issue :)
540.
by kolinko 3y ago
I was just planning to do a show HN on this and you were ahead! The author here, open for comments :)

More ›