4 ms·
Yeah, this is a great finetune that kinda flew under the radar. It will indeed take 32K context without hacky context extension (and theoretically should work O
by brucethemoose2 3y ago
Yeah, this is a great finetune that kinda flew under the radar. It will indeed take 32K context without hacky context extension (and theoretically should work OK stretched to 48K).
It still behaves more like an good unaligned LlamaV2 13B finetune.
And it loves to summarize and retrieve facts. In fact, when I give it a huge context and instruct it to storytell, deduct or whatever, sometimes it will summarize the context instead... But the summary is excellent.
- hfjjbf 3y agoIn my experience llama2 13b is fairly worthless in comparison to 70b. I think L70b is the mark to beat, I’m hopeful mistral will take the crown with a smaller model.
- brucethemoose2 3y agoMistral 7B is way better than LlamaV2 13B chat. But some of the 3rd party 13B finetunes/merges are also much better that Meta's chat model.
- hfjjbf 3y agoIf a Mistral 13B beats Llama2 70B it would be a game changer.