13 ms·
Meta AI Unleashes Megabyte, a Scalable Model Architecture
- crakenzak 4y agoPaper: https://arxiv.org/abs/2305.07185 https://arxiv.org/abs/2305.07185 Wow, seems like Meta AI is so ahead of the curve compared to even Google and OpenAI recently especially with their open sourcing pushes. Great for the research community as a whole!
- joshxyz 4y agogreat positioning for zuck, impressive
- chinaman425 4y ago[dead]
- mturmon 4y agoCommoditizing their complement?
- 1024core 4y agoMeta has no horse in the race (i.e. they don't have a search engine). So, they don't mind throwing random things out. Withholding it won't really make much of a difference for them, as they don't have a way to productionize the tech.
- abwizz 4y agowould not be surprised if they integrated a "suggested conversations" feature based on your chat history and behaviour, where the user just picks sentences from a list and both parties can enjoy a effortless "organic" conversation.
- dumpsterdiver 4y agoWhile I disagree that having a search engine is the only way to have a "horse in the race", I must agree that at this point Meta does not appear to have a horse in the race. Other companies are providing services that are so useful that it makes us think twice about how secure our jobs are. Then there is Meta, who seems to think that the world at large will forget about the terrible motion sickness that their VR products have wrought upon us. I for one will not forget. I'm actually traumatized, and even thinking about putting on VR goggles at this point makes me feel queasy.
- DebtDeflation 4y ago>Meta does not appear to have a horse in the race Their horse seems to be "AI-generated ads". I'm still not sold on the idea though. I can see corporate Marketing departments using AI as a tool to ASSIST with ad copy development, but I'm skeptical that they'll let a third party like Meta generate the ad copy on the fly before pushing to targets. Maybe just tiny parts of it for "personalization".
- wolfd 4y agoI would be _shocked_ if Zuck wasn't thinking 24/7 about how to capitalize on LLMs. I'm sure there are a thousand ideas (maybe even a few good ones?) being thrown around Meta at how to use LLMs to beat Google/Microsoft+OpenAI at the "search buddy" game.
- danielbln 4y agoGood, I couldn't give two craps about the Metaverse, but give me a powerful, useful AI and you have my attention.
- zoogeny 4y agoMeta still has the Portal line of products - which were competitors to the Alexa, Siri and similar Google audio assistant product lines. I just searched and it looks like Meta are currently partnering with Amazon to license Alexa on these devices, but I could imagine they might want to replace it with their own LLM eventually. I am a bit surprised that no one is talking about how these new LLM models will disrupt Alexa, Siri, etc. since that seems to be the most applicable market I can imagine.
- Kilenaitor 4y agoMeta discontinued the Portal line https://www.theverge.com/2022/11/11/23454019/meta-portal-smartwatch-efforts-shutting-down-hardware https://www.theverge.com/2022/11/11/23454019/meta-portal-sma...
- jerpint 4y agoDeepmind was behind this kind of thinking years ago with Perceiver models, but meta is crushing it with high quality publications lately
- jacobn 4y agoMy main argument against the AI doomsayers has so far been that the current scaling laws simply make runaway singularity style scenarios algorithmically impossible (if for each step of improvement you need 10x parameters and 100x training, you quickly run into a brick wall). This is part of why I’m not worried about the current crop of generative AI. I am however both curious and concerned about what the tsunami of talent and $$$ chasing the current trend will achieve. If this n^(4/3) alt transformer compute scaling is real (and there’s been many a pretender, so it’s too early to tell), then that could fundamentally change the overall AI scaling law, substantially lowering the brick wall. And that could be a game changer.
- p-e-w 4y agoI don't buy the idea (with either architecture) that "10x"-type scaling is required for another breakthrough. Think of a human with below average intelligence. Then think of a human genius. Now consider how incredibly similar their brains are, despite the massive performance gap. It's not like one has 10x the number of neurons/synapses/connections etc. of the other. They're both healthy human brains, and you need powerful technology to even distinguish them structurally. Considering this, it seems perfectly possible that a model like GPT-4 is just a hair's breadth away from vastly superhuman performance. It certainly beats the average human at many tasks already. The gap between a moron and a genius is a lot larger than that between GPT-4 and a superhuman.
- anon291 4y ago> It certainly beats the average human at many tasks already This is not the revolution you think it is. ```python > 338383*887282 ... ANSWER ... ``` Yet skynet never came.
- nemothekid 4y ago>Think of a human with below average intelligence. Then think of a human genius. LLMs are not AGI. A human with below average intelligence is still a league above a chimpanzee. A chimpanzee will never be able to read, not because "it's too dumb", but because a chimp's brain lacks the actual hardware for reading. The LLM is the chimpanzee. The gap between an LLM and a "human with below average intelligence" is far more than 10x.
- HellDunkel 4y agoMegabyte sounds tiny.
- motoxpro 4y agoPretty large when things are measured in kilobytes now.
- deleted 4y ago[deleted]
- abwizz 4y agofeels like the 80's again :) double-density floppy gogo!
- ulrikhansen54 4y agowe'll be in the terabyte architectures soon enough : - )
- deleted 4y ago[deleted]
- zxexz 4y agoGreat paper. But wow, I really wish everyone would use more easily searchable names for their projects. In 6 months, there’s a high probability I’ll end up googling/ddging “megabyte model” trying to find this paper again.
- mitthrowaway2 4y agoThe trick is to search HN submissions and filter by the date you remember reading about it. That's how I deal with these unsearchable names.
- seydor 4y agoPatchformers
- machdiamonds 4y agoI've been using myReach as a bookmark manager lately. I've found it to be pretty useful for resurfacing information from links I have saved. I think they're using embeddings to help you track down information across different documents, articles, posts, etc. The big sell is that it's a personalized AI assistant that answers your specific queries based on what data you feed it (e.g.,"what was my electricity bill last month"). However, I'm hesitant about uploading personal data, so I am just using it as a bookmark manager. Their LLM although a little slow, works pretty well. I had this HN link saved about an open-source TTS model. A commenter said they were going to release a model later that week that could be comparable to Elevenlabs. I asked the chatbot on myReach: "What's that TTS model that's looking to rival ElevenLabs?" It surfaced the article and used the specific comment for a response. I'm not sure about the future of these kinds of services from startups. Given their ability to integrate browsers, emails, cloud storage, photos, and more, Google and Microsoft are likely to develop a similar service. With the resources at their disposal, they could probably design something superior and streamline the process of joining. But for now, I think I'll continue using myReach.
- radq 4y agoThis reminds me of the Octuple MIDI tokenization scheme introduced in the MusicBERT paper (https://arxiv.org/pdf/2106.05630.pdf https://arxiv.org/pdf/2106.05630.pdf). Would be interesting to see how much of a performance difference results from using a smaller decoder model (in Megabyte) instead of just doing a softmax (a la Octuple).
- antman 4y agoWhat infrastructure does it require to test?
- mkaic 4y agoGiven that the largest model configurations are only on the order of ~1-2B parameters, I think you could probably run inference on a 3090. Training might be possible too (it's something I plan to try) but will be very slow on most consumer hardware.
- riwsky 4y agoMy LLM is totally real! And she’s the best, way better than GPT-4. But you wouldn’t know her, she goes to another school. In Canada.
- danielbln 4y agoNormally I would agree, especially if it's some Arxiv paper with two dudes talking about mind reading from fmri (you know who you are..). But Meta AI has shown that they are very much capable of creating serious models and releasing them, so I remain optimistic this isn't a smoke screen.
- iamnotsure 4y agoYet another example of words appropriation by a large software company for marketing purposes? https://en.wikipedia.org/wiki/Cultural_appropriation https://en.wikipedia.org/wiki/Cultural_appropriation ?
- andrewstuart 4y agoSearchability matters.
- forrestthewoods 4y agoFirst “Transformer” architecture and now “Megabyte”? I swear they’re trolling us! We’re just a generation or two away from the “computer” model.
- mkaic 4y agoI think you mean the Constantly Online Meta-Programming Universal Task EnactoR model
- dr_dshiv 4y agoWow: “researchers discovered that the Megabyte model's maximum capacity exceeded 1.2M tokens. For comparison, OpenAI's GPT-4 has a limit of 32,000 tokens, while Anthropic's Claude has a limit of 100,000 tokens” In my understanding, this is accomplished through a hierarchical approach of “patches” of tokens.
- DrScientist 4y agoThere is an obvious trend to scaling and performance by making models hierarchical ( not strictly but an element of local learning and global tuned connections ). The obvious next step is to have specialised models loosely connected ( and trained [1] ) as a whole. [1] We've had multiple models connected with text->concept and concept-> image etc but I'm not sure if that connection is trained yet.
- seanhunter 4y agoThat’s sort of what happens in the “toolformer” paper[1] which sets out the basic architecture by which things like chatGPT plugins work. Language models can learn to use “tools” provided by plugins. Those tools might themselves bee the specialized models you are describing (although for some plugins they could be a non-AI tool such as web search or whatever). [1] https://arxiv.org/pdf/2302.04761.pdf https://arxiv.org/pdf/2302.04761.pdf
- mkaic 4y agoUgh. I just spent the past few days reading this exact paper and preparing a detailed presentation on it for my job as an AI researcher, and headlines like this make me roll my eyes very hard. MEGABYTE is indeed a cool new architecture, but it is still very much just a proof of concept at the moment. The paper shows that the model can compete with (but not decimate) vanilla Transformers on the scale of ~1B parameters in long-sequence prediction tasks. They did not "unleash" anything, and scalability to very large parameter counts and datasets still has not been tested. I'm certainly very excited to see where this architecture goes as the community gets ahold of it and starts developing it, but to call it "revolutionary" this early on is disingenuous. I personally have a few experiments I want to run with it, but I put the probability of it being a true GPT-killer at <20%. I would love to be wrong, though!
- nwoli 4y agoI wish you had a link to your blog or twitter in your bio, you have the kind of nuanced tone I’d like to hear more from
- riwsky 4y agothey should consider doing this for a living
- mkaic 3y agoI'll consider doing it for a living when it can pay my rent, which is unlikely to be any time soon :P
- mkaic 4y agoOh, thanks! Though to be honest, my blog does have a good bit of wild speculation on it, as I'm a bit of a hypocrite on that front :) I've added links to both in my bio.
- lordofgibbons 4y agoI'd love to watch a recording of this presentation or write up, if possible!
- brianjking 4y agoThoughts on this approach versus the RNN+Transformers approach that RWKV-LM is taking?
- NanoWar 4y agoMissed opportunity to call it METABYTE!
- otabdeveloper4 4y ago> Megabyte > not Metabyte You had one job...
- dizzydes 4y ago(Potentially dumb) side q: What is Meta's play with their recent AI releases? Google and Microsoft have tons to gain through search and their cloud; Meta's direction I struggle to understand.
- asynchronous 4y agoAdvertisement categorization perhaps? Or automated moderation?
- udkl 4y agoIf your question is 'why is Meta open sourcing their models ?', stratechery had an article on the topic : https://stratechery.com/2023/free-meta-open-sources-another-ai-model-moats-and-open-source-apple-and-meta/ https://stratechery.com/2023/free-meta-open-sources-another-... > Zuckerberg was specifically talking about cloud infrastructure software, but the same point applies to AI capabilities as well: Meta isn’t selling its capabilities; rather, it sells a canvas for users to put whatever content they desire, and to consume the content created by other users. It follows, then, that Meta ought to be fairly agnostic about how and where that content is created; by extension, if Meta were to open source its content creation models, the most obvious place where the content of those models would be published is on Meta platforms. To put it another way, Meta’s entire business is predicated on content being a commodity; making creation into a commodity as well simply provides more grist for the mill.
- dizzydes 4y agoBoom - thanks! For me I still feel they're underplaying it and in some ways diluting their own niche by making content too saturated but it's probably just that I hate social media lol
- peter303 4y agoAstrophysicists use a similar algorithm to build giant particle evolution models of galaxies and cosmology. It would be computationally prohibitive to compute the gravitational or electromagnetic force for every combinatorial pair of particles. So the mass/force are averaged in a hierarchical set of subcube centroids.