I'm surprised this isn't getting more attention here. This may well be the future. These large hybrid transformer-SSM models are faster and more efficient but

I'm surprised this isn't getting more attention here. This may well be the future. These large hybrid transformer-SSM models are faster and more efficient but it seems not really by a huge amount: a big deal but Moore's law progression level not breakthrough level. I gave it a tricky maths problem (from AIMO) and it spat out about 25k thinking tokens (my estimate), going in circles, which took many minutes, far longer than a better model, until finally figuring out the right answer. Not sure whether that's evidence that it is or isn't weaker when looking beyond recent context.