3 ms·

I'm not even sure they're loading everything into memory for MoE; maybe they can get away with only the relevant experts being paged in.