2 ms·
The most interesting bit is that this is RWKV model, meaning constant size state (no quadratic attention). AFAIK the biggest open weights non-transformer model.
by mmoskal 3y ago
The most interesting bit is that this is RWKV model, meaning constant size state (no quadratic attention). AFAIK the biggest open weights non-transformer model.