cgel

4 ms·

1.
Brumby-14B-Base: The Strongest Attention-Free Base Model(manifestai.com)
7 points by cgel 11mo ago|1 comments
2.
by cgel 11mo ago
We have trained a completely attention-free LLM whose performance is competitive with state-of-the-art models. This model, which we call Brumby-14B-Base, has a familiar Transformer-style architecture, except it uses power retention layers i
3.
by cgel 1y ago
Thanks for posting! I'm an author of the work. Here to answer any questions.
4.
by cgel 2y ago
Yes. That is mostly the idea. But calling the state of a linear transformer KV cache is not quite right. A KV cache grows with the sequence length. But the linear transformer state just stores V @ K.T, an object with fixed size.
5.
by cgel 7y ago
Author. There is a misunderstanding in your argument. Your point is about the dataset being sampled from the true distribution. We are happy with that assumption (it's orthogonal to our point). The problem we have is that to apply Baye
6.
by cgel 7y ago
Thanks! We just fixed an issue. I was able to test it on FF for MacOS but it would be very helpful if you could confirm the problem is fixed in your end. Also, I'll admit that messing up Bayes rule in a blog post criticizing Bayesian N
7.
by cgel 7y ago
Imagine you have two random variables A and B, which are 0 with prob 0.5 and 1 with prob 0.5. They just have the property that when A=1 then B is always 0 and vice versa. Thus, when you have seen the value of B, that clearly has changed the
8.
by cgel 7y ago
Author here. Sorry that the typo and the render errors affected you so much. We don't see any rendering issues on our end, if you tell us what browser you are using maybe we can replicate and fix them.
9.
by cgel 7y ago
Author here. The blog post contains a clear explanation of Bayesian Inference and provides careful arguments about its potential benefits and limitations. Further, Bayes rule was typed correctly when it was being used to show that the poste