JohannaAlmeida

3 ms·

1. 1.▲
   by JohannaAlmeida 6mo ago

   The more updated code is here sorry . In another monorepo https://github.com/JohannaWeb/Bastion/tree/main/projects/Aur...
2. 2.▲
   by JohannaAlmeida 6mo ago

   Not sure yet just want to make it more functional rn
3. 3.▲
   by JohannaAlmeida 6mo ago

   For a minimal browser yeah . 8kloc
4. 4.▲
   Show HN: Aurora – a browser engine experiment in Rust

   4 points by JohannaAlmeida 6mo ago|5 comments
5. 5.▲
   by JohannaAlmeida 6mo ago

   Thank you so much . The next thing i want to tackle is the training bottleneck we have right now . That will probably be another HN post when i figure it out.
6. 6.▲
   by JohannaAlmeida 6mo ago

   Yeah RWKV is definitely related in spirit (recurrent state for long context). Here I’m combining local windowed attention with a gated recurrent path + KV cache compression, so it’s more hybrid than fully replacing attention
7. 7.▲
   by JohannaAlmeida 6mo ago

   Yeah auto complete is an amazing use case. I needed a small model that used transformers , could fit on my weak consumer GPU . So i needed to make fundamental arquitecture changes .Do some KV cache tricks. And then prove the new arquitectur
8. 8.▲
   by JohannaAlmeida 6mo ago

   Full attention O(n²): 17.96s / 5.6 tok/s HybridAttention O(n·W + n·D): 0.35s / 286.6 tok/s
9. 9.▲
   Hybrid Attention

   40 points by JohannaAlmeida 6mo ago|9 comments