3 ms·
This looks worth a try. Great test results, very good example output. No way to know if it’s cherry picked / overtuned without giving it a spin, but it will go
by vessenes 2y ago
This looks worth a try. Great test results, very good example output. No way to know if it’s cherry picked / overtuned without giving it a spin, but it will go on my list. Should fit on an M2 Max at full precision.
- Onavo 2y agoWhat's the size of your M2 Max memory?
- treefry 2y agoLooks like 64GB or more
- SubiculumCode 2y agoHow do you figure out the required memory? The MoE aspect complicates it.
- ProofHouse 2y agoEach model added, no?
- vessenes 2y agoIt does; in this case, though, a 25b f16 model will fit. The paper mentions an A100 80G is sufficient but a 40 is not; M2 Max has up to 192G. That said, MoEs are popular in lower memory devices because you can swap out the experts layers -- their expert layers are like 3-4b parameters, so if you are willing to have a sort of pause on generation where you load up the desired expert, you could do it in a lot less RAM. They pitch the main benefit here as faster generation, it's a lot less matmul to do per token generated.