6 ms·
Paving the way to efficient architectures: StripedHyena-7B
- mmaunder 3y agoIs the model available or is this just an API/app?
- lelag 3y agoWeights seem available at https://huggingface.co/togethercomputer/StripedHyena-Nous-7B/tree/main https://huggingface.co/togethercomputer/StripedHyena-Nous-7B....
- SparkyMcUnicorn 3y agoAnd here's the base model: https://huggingface.co/togethercomputer/StripedHyena-Hessian-7B https://huggingface.co/togethercomputer/StripedHyena-Hessian... And GH repo: https://github.com/togethercomputer/stripedhyena https://github.com/togethercomputer/stripedhyena
- anon373839 3y agoThis is a seriously impressive model.
- kcorbitt 3y agoFor short context tasks looks like it's slightly stronger than Llama 7B and slightly weaker than Mistral 7B. Really impressive showing for a completely new architecture. I've also heard that it was trained on far fewer tokens than Mistral, so likely still room to grow. Overall incredibly impressive work from the team at Together!
- tempusalaria 3y agoDid they disclose the training compute/token count?
- stellaathena 3y agoNope :(
- bratao 3y agoAnd this uses Hyena, that can be considered a "previous generation" of Mamba. I think that this anwsers the question about the scalability of SSM and the transformer finally found an opponent.
- deleted 3y ago[deleted]
- algo_trader 3y ago> H 7B is always faster than optimized Transformers > (>10%, >20% and >50% end-to-end faster than FlashAttention 50% improvement on very large sequences is great. But it is not yet transformative (pun intended..) Presumably FlashAttention already does lots of the less-than-quadratic improvement
- ttul 3y agoWell, since Together did hire the author of Mamba…
- skerit 3y ago7B models are so exciting. So much is happening with those smaller models.
- minimaxir 3y agoI wish they were a smidge smaller since 7B LLMs just barely run on a 16GB VRAM GPU (like a T4 server GPU) without quantization shenanigans. Fortunately the learnings from finding better 7B models will trickle down, or more will be done with distillation (e.g. Gemini Nano)
- 3abiton 3y agoWhat's wrong with quantization?
- minimaxir 3y agoThere's still a (subjective) generative quality loss, even with recent tricks to minimize it.
- leetharris 3y agoQ8 is generally less than 1% degradation, Q5KM is around 3%. After that is when it starts to really degrade.
- triyambakam 3y agoCan you explain for a noob why?
- ttul 3y agoQuantization means reducing the number of bits used to encode each floating point number constituting a parameter in the model So instead of having billions of possible values per weight, you might have just 255. The model has to have its weights crammed into a much smaller number of possible values, which reduces its ability to produce good outputs.
- firejake308 3y agoDarn, I was hoping the RWKV people had finally obtained reportable results. This is still interesting, though. Maybe we will see more alternatives to transformers soon
- refulgentis 3y agoThere are RWKV numbers in there, so you sort of got your wish sideways. :) Has RWKV not released anything until now? I thought it was an open project that was in use, if sort of by a hipster 1%
- stellaathena 3y agoPaper: https://arxiv.org/abs/2305.13048 https://arxiv.org/abs/2305.13048 Models (v4 are the ones from the paper): https://huggingface.co/RWKV https://huggingface.co/RWKV GitHub: https://github.com/BlinkDL/RWKV-LM https://github.com/BlinkDL/RWKV-LM
- senseiV 3y agoV5 7b is out, close to hyena, gets 1400 t/s on a 3090, while an h100 llama 7b 8bit is 1200 t/s
- stellaathena 3y agoRWKV had a paper accepted at EMNLP and released models which match the performance of equivalent transformers. What else are you looking for?
- senseiV 3y agoThey Do, the latest rwkv v5, matches mamba at 3b scale, and from the benchmarks I see, its similar to hyena
- goalonetwo 3y agoThere seems to be a new model every single day. How do people have time to keep track with everything going on in AI?
- hinkley 3y agoFrom decades of observing at a distance and observing observers at a distance, I think it's safe to say that, like fusion, there are walls that AI run into, not unlike the risers on a staircase, and when we collectively hit one, there's a lot of scuttling back forth. A lot of movement, but no real progress. If that plateau goes on too long, excitement (and funding) dry up and things die down. Then someone figures out how to get past the current plateau, and the whole process repeats. That could be new tech, a new architecture, or it could be old tech that was infeasible and had to wait for Moore's Law. Right now we are on the vertical part of the sawtooth pattern. Everyone hopes this will be the time that takes us to infinity, but the old people are just waiting for people to crash into the new wall.
- goalonetwo 3y agoThanks for putting this so eloquently. That's exactly how I feel as well.
- fvv 3y agoWhy things should dry up when contrary to fusion ai is already usable by millions daily ? Even if prpgress should stall a bit the product or fine-tunes or normal progress will still be super supeful , the "too soon" point has been surpassed
- actionfromafar 3y agoDepends on what you are looking for. I have this hesitation too. What we have and are on track for is useful and cool, but how far will we come in this spurt until we are back at slight incremental gains? Implementation wise in business, we are very early though. It feels like email in 1995, we have barely scratched the surface of what LLMs can mean for business and everyday life.
- icyfox 3y agoCommon wisdom in most industries is to release bad PR announcements on a Friday and good ones towards the start of the week. It's interesting how the advent of twitter communication has shifted the ML ecosystem to publishing work whenever it's ready versus trying to find an optimal weekday. Or maybe they're optimizing for the weekend hackers that will take improvements released late in the week and put them into practice on Saturdays and Sundays.
- refulgentis 3y agoIt is interesting, been thinking about that myself a bit: some random associated thoughts: NeurIPS is next week, & there was OpenAI drama news that went under the radar due to Friday rule.
- alsodumb 3y agoNeurIPS is also starting this Sunday, I'm sure some of these releases are meant to get the word going during the conference.
- deleted 3y ago[deleted]
- mkesper 3y agoPlease stop using non-zero-based y axes. Every comparison chart in this post uses it, overemphasizing the difference between models.