18 ms·
Meta Superintelligence Labs' first paper is about RAG
https://arxiv.org/abs/2509.01092 https://arxiv.org/abs/2509.01092
- bigyabai 1y ago> Long awaited first paper from Meta Superintelligence Labs is not a model layer innovation. What does this mean? It means you're reading into it too much and need to be let down, gently, from the hype train.
- nine_k 1y agoA great post, it starts with this: TL;DR • MSI’s first paper, REFRAG, is about a new way to do RAG. • This slightly modified LLM converts most retrieved document chunks into compact, LLM-aligned chunk embeddings that the LLM can consume directly. • A lightweight policy (trained with RL) decides which chunk embeddings should be expanded back into full tokens under a budget; the LLM runs normally on this mixed input. • The net effect is far less KV cache and attention cost, much faster first-byte latency and higher throughput, while preserving perplexity and task accuracy in benchmarks. I wish more long posts followed this model of a scientific paper.
- jongjong 1y agoInteresting. All developers I know who tinkered around with embeddings and vector similarity scoring were instantly hooked. The efficiency of computing the embeddings once and then reusing as many times as needed, comparing the vectors with a cheap <30-line function is extremely appealing. Not to mention the indexing capabilities to make it work at scale. IMO vector embedding is the most important innovation in computing of the last decade. There's something magical about it. These people deserve some kind of prize. The idea that you can reduce almost any intricate concept including whole paragraphs to a fixed-size vector which encapsulates its meaning and proximity to other concepts across a large number of dimensions is pure genius.
- _jayhack_ 1y agoVector embedding is not an invention of the last decade. Featurization in ML goes back to the 60s - even deep learning-based featurization is decades old at a minimum. Like everything else in ML this became much more useful with data and compute scale
- senderista 1y agoYup, when I was at MSFT 20 years ago they were already productizing vector embedding of documents and queries (LSI).
- jongjong 1y agoInteresting. Makes one think.
- senderista 1y agoTo be clear, LSA[1] is simply applied linear algebra, not ML. I'm sure learned embeddings outperform the simple SVD[2] used in LSA. [1] https://en.wikipedia.org/wiki/Latent_semantic_analysis https://en.wikipedia.org/wiki/Latent_semantic_analysis [2] https://en.wikipedia.org/wiki/Singular_value_decomposition https://en.wikipedia.org/wiki/Singular_value_decomposition
- ekidd 1y agoVector embeddings are slightly interesting because they come pre-trained with large amounts of data. But similar ways to reduce huge numbers of dimensions to a much smaller set of "interesting" dimensions have been known for a long time. Examples include principal component analysis/single value decomposition, which was the first big breakthrough in face recognition (in the early 90s), and also used in latent semantic indexing, the Netflix prize, and a large pile of other things. And the underlying technique was invented in 1901. Dimensionality reduction is cool, and vector embedding is definitely an interesting way to do it (at significant computational cost).
- liampulles 1y ago
- mountainriver 1y agoThis was a very obvious next step, I played around with implementing something similar at one point. In general we need to make it simpler for LLMs to take in different forms of embeddings. At least frameworks that simplify it.
- cm2012 1y agoAt first I thought the super intelligence wrote a novel scientific paper
- Imnimo 1y agoI'm curious whether this is work that was specifically begun under the "superintelligence" umbrella, or if it's just that the people who were working on it had been shifted to the Superintelligence team by the time they wrote the paper. I would guess the former?
- lblume 1y agoAnother commenter claims the latter: https://news.ycombinator.com/item?id=45554169 https://news.ycombinator.com/item?id=45554169
- naasking 1y ago> the core insight here is actually: if embeddings are generated by layers within the LLM, it makes no sense to convert them back to natural language, just for another LLM to compress those tokens back to embeddings. Doesn't this tie the two layers together in a way that they can't evolve separately?
- xvector 1y agoWorking in big tech it's pretty wild to see how integral AI has become to our work internally, vs the public perception of it. People are NOT prepared.
- fishmicrowaver 1y agoNot prepared for what? Seems like the rest of the world is desperate to be shown the way to unlock something of value?
- Workaccount2 1y agoI think at this point it's software devs looking for the value unlock. Non-software devs are actually making functional programs for themselves for the first time ever. The value is crazy.
- ceejayoz 1y agoIt’s not the first time ever. People did the same with Access and HyperCard in the 90s.
- fishmicrowaver 1y agoSure but I'm the real world do you think businesses are going to deploy piles of code into production generated this way? No, non technical people will continue to whip up MS PowerApps. AI generated code has no value to many businesses.
- xvector 1y agoThe value of AI is not in generating code. That's just a "nice-to-have." The value of AI is in having a scalable, human-like decision maker that you can plug into anything, anywhere. This has unlocked countless use cases for my team, that we could scarcely imagine a few years ago.
- cbg0 1y ago
- godelski 1y agoIt's kinda funny, Meta has long had some of the best in the field, but left them untapped. I really think if they just took a step back and stop being so metric focused and let their people freely explore then they'd be winning the AI race. But with this new team, I feel like meta mostly hired the people who are really good at gaming the system. The people that care more about the money than the research. A bit of this is true at every major lab. There's tons of untapped potential. But these organizations are very risk adverse. I mean why not continue with the strategy that got us to the point we're at in the first place. Labs used to hire researchers and give them a lot of free reign. But those times ended and AI progress also slowed down. Maybe if you want to get ahead you gotta stop thinking like everyone else Well meta... you can "hold me hostage" for a lot cheaper than those guys. I'm sure this is true for hundreds of passionate ML researchers. I'd take a huge pay cut to have autonomy and resources. I know for a fact there's many working at Mets right now that would do the same. Do maybe if you're going to throw money at the problem, diversify a bit and look back at what made SV what it is today and what made AI take leaps forward
- bboygravity 1y agoAI progress has slowed down?! By what metric? Quite the statement for anybody who follows developments (without excluding xAI).
- bobxmax 1y agoI thought Alex Wang was a very curious choice. There are so many foundational AI labs with interesting CEOs... I get that Wang is remarkable in his own right, but he basically just built MTurk and timed the bubble. Doesn't really scream CEO of AGI to me.
- thereitgoes456 1y agoThe reportings at the time said that he was Mark’s 5th choice or similar. It is fairly clear he would prefer Ilya, Murati, Mark Chen, and perhaps others, but they said no, and Alex Wang was the first one to say yes.
- ipsum2 1y agoThis has nothing to do with superintelligence, it's just the people that were working on the paper prior to the re-org happened to publish after the name change. Though it is notable that contrary to many (on HN and Twitter) that Meta would stop publishing papers and be like other AI labs (e.g. OpenAI). They're continued their rapid pace of releasing papers AND open source models.
- Zacharias030 1y agoShould be the top comment. MSL is not only those few high profile hires.
- ekianjo 1y agoOpen weights models, not open source. And even their weights are under a specific license not as permissive as apache 2.
- drexlspivey 1y agoDoes an “open source” model the way you describe it exist or is it a mythical creature?
- bigcat12345678 1y agohttps://docs.lamini.ai/memory_rag/ https://docs.lamini.ai/memory_rag/ Similar approaches have been tried before already
- pppoe 1y agoI find it absurd that, compared to the past, large companies now have more abundant stock prices and cash than ever before, yet nearly every AI Lab in these companies is facing greater pressure than ever, being asked to generate short-term profits. In the midst of AI's unprecedented boom, the research environment and atmosphere in the industry seem to have worsened compared to the past.
- signatoremo 1y agoIs this Meta’s lab pressured to generate short term profits? Which other under pressure labs are you talking about?
- inquirerGeneral 1y ago[dead]
- sefrost 1y agoIs it because of the "winner takes all" and "lock-in effects" of being the first to market?
- foldl2022 1y agoSo, show me the model weights, please.
- yalogin 1y agoI am not surprised because the culture at meta is not at all, even in the slightest, to focus on science for the sake of it. It’s actively actively purged out of you. The focus is on metrics and how the bottom line is impacted. So this is in line with that
- rhetocj23 1y agoYeah and this problem is near impossible to fix once it has infested into the culture of the firm.
- DangitBobby 1y agoIt's not always a bad thing though, like in this case they looked for a practical win and found one because impractical wins can't make them money.
- georgeburdell 1y agoIt’s not that simple. I worked at a supplier of Meta and they paid us large NREs to fund our exploratory work
- alex1138 1y ago"People are using our service more!" turns out to be a horrible metric when they outright lie to you (x has sent you a message! - when no message exists)
- CShorten 1y agoHere is a video I made diving into the paper, hopefully helpful! https://www.youtube.com/watch?v=Ek0tZootK00 https://www.youtube.com/watch?v=Ek0tZootK00
- deleted 1y ago[deleted]
- nmca 1y agoThis is not work by any of the high profile new hires, in case folks are confused.
- elyobo 1y agoCan we have a more informative, less clickbaity, title?
- dang 1y agoWhat would a more informative, less clickbaity title be? (preferably using representative language from the article)
- smeeger 1y agothere should be a guideline to get rid of clickbait titles. its an epidemic here
- dang 1y agoThere is of course such a guideline: https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html We don't catch every case, but if you're talking about the frontpage, I'm surprised to hear you say "epidemic". What are some recent examples?
- lanyard-textile 1y agoI wouldn’t give much weight to the person that had an opinion about the guidelines without reading them :)
- dang 1y ago
- puttycat 1y agoSeems very incremental and very far from the pompous 'superintelligence' goal.
- antonvs 1y agoIt’s unlikely that the existing LLM architecture will evolve into anything that resembles superintelligence any more than it does already. Which means that modifications to the architecture, and combining it with other components and approaches, are the next likely step. This paper fits that.
- naasking 1y agoA 30 fold improvement seems a tad more than incremental.
- vasco 1y agoI can start brushing my teeth 30 times faster but it won't change my life. This is nice for RAG but it's a very localized improvement. And 30× sounds big but is just an order of magnitude improvement also.
- naasking 1y agoBrushing your teeth is not central to your life, recalling facts correctly is, and a 30 fold improvement in the latter very well could change your life. I'll leave it to you to figure out which is a better analogy to RAG.
- vasco 1y agoJust remember that in this example you don't remember 30x more things, you just remember the same things 30x faster. That is a significant difference.
- btilly 1y agoIf you can collapse "retrieve this complex chunk when it is needed" into a single token, what else can you put into a token? "Send this through the math coprocessor." "Validate against the checklist." "Call out to an agent for X." "Recheck against input stream Y." And so on. Retrieval augmentation is only one of many uses for this. If this winds up with better integration with agents, it is very possible that the whole is more than the sum of its parts.
- koolala 1y agoDid a "superintelligence" lab publish a superintelligence related paper with no results for intelligence? What measured improvements did this proposal make in their LLM's intelligence?
- pbd 1y agohttps://github.com/simulanics/REFRAG https://github.com/simulanics/REFRAG
- singularity2001 1y agosomewhere in my hacker news comment history I presented this very idea
- dangsecondalt 1y ago[dead]
- asim 1y agoThis was inevitable. You can't keep training LLMs and expect that's the answer to the evolution of AI. Yes it'll happen and we'll keep creating new more refined and bigger models but it's like DNA or something like the cortex of the brain. After that you need these systems that essentially "live" for years digesting information and develop a more refined way to process, store and retrieve the information. Compression of RAG was also inevitable. It's like the btree index of a database. The thing is, we're probably one or two iterations away from being good enough on the RAG pipeline and then we'll need to focus more on the other pieces of sensory input that need to be connected and processed at higher throughput. Right now it's not fast or efficient enough. This is where the likes of Google will shine. They are probably two decades ahead of everyone on internal technology and there is some team with the breakthrough but it hasn't seen the light of day yet. What's coming out of DeepMind is really a forced effort in productization and publication of work in a consumable format but internally they are likely way ahead. I don't have as much faith in Meta's efforts despite seeing things like this. Quite frankly those people, the ones doing the work should move to more honourable companies. Not feed crack addiction in the form of Meta's universe.
- smeeger 1y agoexactly. the real focus internally is working on new architectures. there is no other possibility.
- zem 1y agothis was really weird to read: > But RAG is a very real world, practical topic for something as significant as a new lab’s first paper. I would expect exactly the opposite - that a new lab would put out a few random papers that happen to be in areas their researchers were interested in and already working on, and once people had been working together a while and developed some synergy they would maybe come out with something really groundbreaking. do people really view a "first paper" as something deeply significant and weighty? because that just seems like a good way to get bogged down in trying to second guess whether any given paper was good enough to be your all-important debut!
- deleted 1y ago[deleted]
- Al-Khwarizmi 1y agoAs an academic I would expect the same as you, and no, to my knowledge "first paper" is meaningless, at least in academia. Most people's first paper is some small contribution to what their PhD supervisor is doing at the time, where the student tries their best at writing but it ends up so heavily edited that probably 90% of the final text comes from the supervisor :) So typically first papers don't define or represent a researcher. When you start you just don't have the experience to have a great idea and carry it through to a good paper. Of course here we are talking about a lab, not an individual person, but still I haven't heard of first papers being considered special in any way, even for labs.
- schmorptron 1y agoOne thing I don't get about the ever-reoccuring RAG discussions and hype men proclaiming "Rag is dead", is that people seem to be talking about wholly different things? My mental model is that what is called RAG can either be: - a predefined document store / document chunk store where every chunk gets a a vector embedding, and a lookup decides what gets pulled into context as to not have to pull whole classes of document, filling it up - the web search like features in LLM chat interfaces, where they do keyword search, and pull relevant documents into context, but somehow only ephemerally, with the full documents not taking up context in the future of the thread (unsure about this, did I understand it right?) . with the new models with million + tokens of context windows, some where arguing that we can just throw whole books into the context non-ephemerally, but doesnt that significantly reduce the diversity of possible sources we can include at once if we hard commit to everything staying in context forever? I guess it might help with consistency? But is the mechanism with which we decide what to keep in context not still some kind of RAG, just with larger chunks of whole documents instead of only parts? I'd be extatic if someone who really knows their stuff could clear this up for me
- kgeist 1y agoTechnically, RAG is anything that augments generation with external search. However, it often has a narrower meaning: "uses a vector DB." Throwing everything into one large context window is often impractical - it takes much more time to process, and many models struggle to find information accurately if too much is going on in the context window ("lost in the middle"). The "classic" RAG still has its place when you want low latency (or you're limited by VRAM) and the results are already good enough.
- make3 1y agono one is saying rag is dead, you're never going to put the whole Internet in the context of the model, & the more you put the more expensive it is.
- viraptor 1y agoLots of people say rag is dead: https://kagi.com/search?q=rag+is+dead&r=au&sh=g52XEb93vx691I_87H-2VA https://kagi.com/search?q=rag+is+dead&r=au&sh=g52XEb93vx691I...
- armcat 1y agoI couldn't immediately see in their graphs/tables any comparison against simple lexical/statistical based context compression, such as candidate selection of chunks using TF-IDF, word overlap etc. For most of us in the industry we need to find these quick wins that give us equivalent performance to sending huge amount of information to the LLM, while compressing by 10x.
- macleginn 1y agoSo this looks essentially like continuous prompting (see prefix tuning) with RL-driven selection of what to present as tokens and what as continuous inputs (embeddings).
- i5heu 1y agoCan we please get rid of the clickbait titles?
- RataNova 1y agoRefreshing (and slightly unexpected) to see Meta Superintelligence start with something this practical instead of a headline-grabbing new model
- mark_l_watson 1y agoA great idea, bypassing as much conversion as possible between vector space and natural language tokens. Reminds me of a discussion of having AI’s “talk” to each other using vector space. There was an interesting quote “plain old BM25 from 1994 outperforms vector search on recall” and super relevant to what I did yesterday. I am trying to use small local models more often and yesterday I wrote Common Lisp code that uses a large corpus of text and a user query or prompt to construct a fairly concise one-shot prompt with select context from the text corpus. This is RAG, and I used both BM25 and vector embeddings matching. I added the code and an example as a new chapter in my CL book (link directly to new material: https://leanpub.com/lovinglisp/read#leanpub-auto-autocontext-prepare-effective-prompts-with-context-for-llm-queries https://leanpub.com/lovinglisp/read#leanpub-auto-autocontext...) yesterday afternoon. BM25 is fast. This is new code, and I will certainly be experimenting more with it, but as-is it is useful when working with small local LLMs.
- Palmik 1y agoThe observation about the "block-diagonal patterns" in RAG isn't new and has been exploited / explored before: - https://arxiv.org/abs/2410.07590 https://arxiv.org/abs/2410.07590 (literally titled "Block-Attention for Efficient RAG") - https://arxiv.org/abs/2409.15355v3 https://arxiv.org/abs/2409.15355v3 - https://arxiv.org/abs/2212.10947 https://arxiv.org/abs/2212.10947 The REFRAG paper does not cite any of these.
- SknCode 1y agoI am not sure if I understand things correctly. I came to believe the LLMs work with token embeddings. Is then the REFRAG only "something" in front of the LLM, and the decoder is the RL policy which expands only some token chunk embeddings into token embeddings feedable to LLM? Or the REFRAG needs you to 'tune' the LLM to be able to work with both token embeddings and token chunk embeddings?
- mikepalmer 1y agoI hate articles that don't define their acronyms! Lazy? Intentionally exclusive? So that others don't also have to look it up, it's Retrieval-Augmented Generation (RAG). They even say it's "a topic that we didn’t expect"... so... perhaps many people wouldn't have heard of it?
- aurohacker 1y agoFigure 1 in the paper is all about the encoder and how the context and query is packaged and sent to the decoder. I wish it were more complete...