5 ms·
Its presumably llama just like Mistral. Everything open source is llama now. Facebook all but standardized the architecture. I dunno about the moe. Is there e
by brucethemoose2 3y ago
Its presumably llama just like Mistral.
Everything open source is llama now. Facebook all but standardized the architecture.
I dunno about the moe. Is there existing transformers code for that part? It kinda looks like there is based on the config.
- jasonjmcghee 3y agoMistral is not llama architecture. https://github.com/mistralai/mistral-src https://github.com/mistralai/mistral-src
- brucethemoose2 3y agoIts basically llama architecture, all but drop in compatible with llama runtimes.
- refulgentis 3y agoBecause it's JSON? :)