3 ms·
Meta Open-Sources Megalodon LLM for Efficient Long Sequence Modeling
- skilled 2y agoRelated: Megalodon: Efficient LLM Pretraining and Inference with Unlimited Context Length - https://news.ycombinator.com/item?id=40054901 https://news.ycombinator.com/item?id=40054901 - April 2024 (28 comments)
- causalmodels 2y agorepo link: https://github.com/XuezheMax/megalodon https://github.com/XuezheMax/megalodon
- koolala 2y agoyay dinosaurs!
- meindnoch 2y agoMegalodon was not a dinosaur. It was a fish (shark).
- lochiero 2y agoSo old... Another epoch talking about Llama2
- arjvik 2y ago> For what it's worth, RWKV's website on that matter mentions that yes it's bad on recall, but for the vast majority of tasks you can just ask the question before the content, and it'll handle the task just fine.
- amluto 2y agoIf you ask me a question before giving me the material being queried, I do better, too! It’s the difference between an open-book and closed-book test. The Transformer read-then-query model is, IMO, a bit odd. It preprocesses the input and then people want it to be able to answer any question about the input and would also like the query to run in time independent of the input length, and people are sad that the best models take time linear in the input length. No kidding: that’s the algorithmic complexity of even a straightforward, non-ML approach!
- deleted 2y ago[deleted]
- 1024core 2y agoMEGALODON builds on the research team's previous model, MEGA (exponential moving average with gated attention) I like how the team gingerly avoided calling the previous model "MAGA" ... :-D
- codetrotter 2y agoI could give two sh*ts about Mr D.T. but I’m more curious to see the reaction of one Kim Dotcom, aka the Mega dude himself.
- ai_what 2y agoDidn't this happen in April? Edit: Yes it did, https://github.com/XuezheMax/megalodon https://github.com/XuezheMax/megalodon
- deleted 2y ago[deleted]
- markwilliams21 2y ago[dead]