3 ms·
> A common feedback we receive for the RWKV multi-lingual approach is: it hurts our English evaluation scores (...) Has anyone quantified that specifically? I'
by viraptor 3y ago
> A common feedback we receive for the RWKV multi-lingual approach is: it hurts our English evaluation scores (...)
Has anyone quantified that specifically? I'd love to read more details since I'd expect the concepts to start mapping between languages at some point. I.e. with enough language fluency I'd expect learning knowledge/reasoning in language to improve the result in another. But I can't find any paper talking about specifically about that.
- marmaduke 3y agoI think the group working on rwkv has yes, even if they don't show the details in this article. (I followed them on their discord channels for quite some time). The interesting take away for me was that training rwkv from zero to intelligible sentences for minority language was more faster (in units of tokens trained!) than other architectures, making it more accessible for cases where large corpus like the Pile don't or can't exist.
- sanxiyn 3y agohttps://arxiv.org/abs/2306.06688 https://arxiv.org/abs/2306.06688 studied this. (Check out figure 3.) In short, yes, multilingual transfer does happen. But importantly, not at 7B. Being English-centric is possibly okay for LLaMA 70B, but it really is not okay for LLaMA 7B.