9 ms·
Copy is all you need
- VHRanger 3y agoThis resonates with the current AI skeptic view that language models are a supercharged search engine on the pile of text they're trained on. Also the fact that evaluating language models is difficult, and we tend to end up with models that game the evaluation benchmarks.
- wongarsu 3y agoGood information retrieval is a problem we are trying to solve for thousands of years, so even if that's all LLMs are doing then that's still a great achievement. Of course a more explicit approach like this paper is a really good step in that direction by making it easier to trace information provenance. It might still be nontrivial to answer why the model selected this specific piece of information, and why it was composed in this specific way, but it seems trivial to say where the model got the information from. Which is really all we demand from humans too.
- croes 3y agoWas the training data quality checked? If so then LLMs are search engines for catalogs like Yahoo once was and not a good search engine for SEO optimized click farms. Google search once was great too but then ads and SEO killed it.
- actionfromafar 3y agoEh, the free to use commercial LLMs will surely be spiced with commercials eventually.
- bob1029 3y ago> Good information retrieval is a problem we are trying to solve for thousands of years Quantum computers would have something to say about this, assuming they ever materialize.
- hdior 3y ago[dead]
- opnac 3y agoI wish we could stop with the “X is all you need” papers! The first one was unintuitive and so are the rest.
- mottiden 3y agoI agree. The paper is really interesting, but the title not so much :)
- jillesvangurp 3y agoA bit click baity at least. And without opening it you have no chance to understand what this is about. I know HN has a policy against editorializing but in this case, a brief summary would have been helpful.
- mottiden 3y agoThe paper introduces a new method for text generation, named Copy-Over-Generate (COG), which differs from traditional approaches that generate words from a fixed vocabulary. Instead, COG progressively copies phrases from a massive text collection, aiming to generate coherent text continuations through multiple rounds of phrase retrieval. COG stands on the line of retrieval-augmented text generation research but takes a radical step forward. Unlike previous work that combines retrieval and generation, in COG, retrieval is generation. COG shares some ideas with previous work such as replacing the fixed vocabulary with a nonparametric phrase table. The paper presents experimental results showing the advantages of COG over strong baselines in three experimental settings: standard language modeling (using the WikiText-103 dataset), domain adaptation (using the Law-MT dataset), and an enlarged phrase index (using the En-Wiki dataset). Despite the promising results, the authors acknowledge that there are some flaws in the COG method. For example, COG may copy a phrase that is incoherent with the previously copied phrase, or it may only copy a part of a complete phrase, leading to inaccurate generation results.
- CamperBob2 3y agoBefore long, if these NN refinements continue at their current pace, it's going to become impossible to tell synthetic HN posts from organic ones. Going to get weird.
- naillo 3y agoSeems like a common pattern. State of the art models being well replaced by a information retrieval layer (top 10 results) fed into a much lighter model that does something with that plus the original input. Cool result!
- falcor84 3y agoYeah, that actually sounds amazing to me. If we could limit the LLM to somehow only act as a "reasoning" rather than a "knowledge" layer, such that all the non-trivial domain knowledge has to come from the information retrieval layer, in a fully referenced way, that could potentially "solve" the hallucination problem, no? Even more than that, I wonder if we could then apply something like this to power some sort of "fact provenance" for the web as a whole, e.g. by populating Wikidata with referenced facts (preferably with extensive human QA).
- esjeon 3y agoYeah, and, on top of that, I think this can lead to smaller (and snappier) agent models, because we no longer have to encode every single piece of information into models. As we carve out more and more parameters and input data, AI development will get more accessible, and we'll get more novel applications. (I'm certainly dreaming here.)
- spacemanspiff01 3y ago[dead]
- redox99 3y agoI don't know. ChatGPT and Bing both dramatically deteriorate if you allow them to search the web. And systems that allow you to "talk" to a PDF via top results of vector search being added to the prompt are also pretty underwhelming.
- twic 3y agoThis is definitely my bet on where things are going. And not just this particular example - i believe we will identify many recurring submodules and patterns in neural networks that can be extracted into conventional code, leaving a lightweight neural glue layer orchestrating them. This should be more efficient, faster to train, more interpretable, and more reliable, so better for users. But less mysterious, so worse for VCs.
- woeirua 3y agoThe big advantage here would be the ability to attribute entire blocks of text back to a specific source and cross domains just by building a database of embeddings. The downside is that these networks are probably not as creative as they're limited to only data that's available. It might work best to use something like this as an expert system for a GPT like agent to refer to when needed.
- MAXPOOL 3y agoWhat about LLM reasoning ability? Faith and Fate: Limits of Transformers on Compositionality https://arxiv.org/abs/2305.18654 https://arxiv.org/abs/2305.18654 Transformers solve compositional reasoning tasks by reducing multi-step compositional reasoning into linearized subgraph matching without problem-solving skills. They can solve problems when they have reasoning graphs in the memory.
- esjeon 3y agoLLMs do logic by mimicking logical structures on the text level (and that's why they often need be ordered to do step-by-step for correct answers), so this one may also have the same ability as long as memories are properly utilized.
- BSEdlMMldESB 3y agoI think this boils down to the capacity to match together parenthesis in a logical-syntax way however, the "parenthesis" can be any symbol. even grammatical clauses are one sort of "parenthesis" in the way I'm thinking about them
- abc_lisper 3y agoFunny, I write Clojure for my day job and fun, so I have tried to use ChatGPT to generate code. If anything, it sucks at paren matching. It reminded me of stable diffusion's "six finger problem".
- BSEdlMMldESB 3y agoas I said, it's not exactly "parenthesis" with the strictness that real programing needs. in fact, my whole idea has got me on a deep dive into the nature of the decimal point (up to which extent is the decimal point representation of numbers and instance of a "fixed point"? I don't know! I cannot understand a fix point just yet; and for me to say I get decimal notation actually means I understand something about p-adic representation; which I'm still working on figuring out) I thought these models got more 'logical' after training with computer code
- awestroke 3y agoFirst, hate the title Second, this approach seems equivalent to using larger tokens, which means the problems with using tokens instead of letters are just exacerbated
- collinc777 3y agoSlight tangent: I once worked with a programmer who, the vast majority of time, would only input text into a text editor via copy and paste. Think anti-vim. His fingers were locked on mouse and crtl+c/v. It was incredible to watch and his programming speed was very impressive.
- willsmith72 3y agoPlease tell me more. Where was he copying from? What about formatting and refactors? Was his quality as impressive as his speed?
- tylercrompton 3y agoStack Overflow, surely
- high_priest 3y agoOpenAI, surely.
- postalrat 3y agoStart with a file like: abcde...ABCD..1234.{}()*.... and go from there.
- ourmandave 3y agoVoice dictation, Shirley.
- esafak 3y agoDon't call me Shirley!
- jaredsohn 3y agoprogrammers he subcontracted to. Also explains why he was so fast
- collinc777 3y agoGenerally the repository he was working in, but really it was any application that he had open on his machine. He would remember where words, or portion of words that he needed were, go to them, and copy and paste what he needed. Just in case you're thinking this: He was not copying large portions of code from stack overflow or anything like that. He was line by line writing code, a few copy and pastes at a time. Often he times would copy and paste single characters to maintain his flow.
- js8 3y agoI remember that around 2004, before convnets became popular, there was a paper on image texture style transfer using approximate nearest neighbors based on some neighborhood of each point. This technique seems similar but for text.
- kastnerkyle 3y agoMaybe 'Image Quilting for Texture Synthesis and Transfer', Efros and Freeman [0]? There's some neural / patch blends from 2016 that I always thought were interesting (CNN-MRF) [1], and I think there's a renaissance in those approaches recently (combined with other generators / prompts etc.). You can also argue ViT is "patch based" in a major sense... I am still a big believer in patch + combinations + warping (non-parameteric synthesis) generally, some cool older work from Apple on that in speech land [2]. I go as far as arguing BPE / wordpiece / sentencepiece / tokenizers in general are key for modern approaches (as were word vocab selections in the earlier days of NMT), because they find 'good enough' patches (tokens) for a higher level model to stitch together while still having some creativity / generalization available... but we focus on the model details rather than the importance of the tokenizer (and tokenizer distribution) in publication many times. [0] http://people.eecs.berkeley.edu/~efros/research/quilting.html http://people.eecs.berkeley.edu/~efros/research/quilting.htm... [1] https://github.com/chuanli11/CNNMRF https://github.com/chuanli11/CNNMRF [2] https://machinelearning.apple.com/research/siri-voices https://machinelearning.apple.com/research/siri-voices
- thanatropism 3y ago> COG https://wiki.opencog.org/w/The_Open_Cognition_Project https://wiki.opencog.org/w/The_Open_Cognition_Project
- msoad 3y agoObvious immediate question is, is it as creative? There are a lot creativity left behind when you increase the token size (let's be real, it's just that). As an example creating a new word like "dickstracted"[1] would not ever happen in this model [1] https://www.urbandictionary.com/define.php?term=Dickstracted https://www.urbandictionary.com/define.php?term=Dickstracted
- 3cats-in-a-coat 3y agoWhy wouldn't it. It suggest it copies text spans, it doesn't say how big.
- soliton4 3y agothis made me think of a fun activity. ask chatgpt to come up with a new word and then google that word. sometimes the word exists in the context of a scify show or a plant. sometimes gpt just added a "se" or "us" to existing words. sometimes it changes a Z to a C but it never actually came up with a new word
- _ea1k 3y agoI asked it this: "Set your model temperature as high as possible an generate a completely new and random word" It acted acted like it understood and generated the word Blazivox. I don't see it on Google at least.
- jojobaskins 3y agoBlazeVox is a publishing company. I guess its still one character away but close enough that it could have just randomly swapped out the character.
- fsmv 3y agoIt cannot change the temperature intrinsically. Only OpenAI controls that in their API.
- vanjajaja1 3y agobut it does know the concept, so it can simulate it
- xg15 3y agoI think the "... is all you need" title here is particularly misleading as the paper does in fact use a BERT model for generating the vectors. So if the implication was that no language model was needed at all and you can just do nearest neighbour on string similarity and patch results together, that implication was clearly wrong. I think what the paper does show though is that there are methods that can make language models topic-specific without fine-tuning and that yield competitive results even with older models.
- moffkalast 3y agoNext thing you'll say the Beatles are being misleading with 'All you need is love' because people also need food and shelter.
- Zacharias030 3y agommd! <3
- xg15 3y agoEh, the "attention is all you need" paper was kinda arguing that. And this paper doesn't.
- usgroup 3y agoYeah I thought the same -- it struck me at first blush as if it was some kind of super simple architecture that didn't use transformers, and then in the diagram i saw they used BERT to produce the embeddings!
- amluto 3y agoThis is interesting coming on the heels of the gzip-based inference paper. gzip is based on LZ77, and the LZ family of compressors generate and store (and cleverly encode) instructions to copy blocks of text they have seen before to their output.
- xianshou 3y agoBehold, the true stochastic parrot.
- twic 3y agoAuto-dadaism: https://en.wikipedia.org/wiki/Cut-up_technique https://en.wikipedia.org/wiki/Cut-up_technique
- Animats 3y agoThis approach can probably handle most of the queries search engines and Siri-type chatbots handle. The big GPT-type engines can be reserved for the hard problems. Something along those lines is needed to keep the cost of search down. There's an estimate that using a large language model for search is 10x more expensive than existing search engines. Yet few queries really need that much heavy machinery.
- Der_Einzige 3y agoThis has deep connections with my attempt to implement an effective queryable word-level grammatically correct extractive text summarizer (AKA: The way most people actually summarize documents) - https://github.com/Hellisotherpeople/CX_DB8 https://github.com/Hellisotherpeople/CX_DB8 I will try to implement this with the necessary changes to actually make this work properly, where instead of generating a new answer, it simply highlights the most likely text spans.
- rapatel0 3y agoSurprised no one has mentioned the obvious issue: plagiarism (Not sure if the authors have indicated any method for attribution of the original data)
- enoch2090 3y ago[dead]
- sfmike 3y agoThought this was about how you just need good copywriting skills