10 ms·
The Secret Sauce behind 100K context window in LLMs: all tricks in one place
- version_five 3y agohttps://archive.md/bw2cN https://archive.md/bw2cN (Its a medium page that doesn't load for me)
- knodi123 3y agowhereas archive.md returns "ERR_SSL_VERSION_OR_CIPHER_MISMATCH"! Sometimes I wish there was a way to tell our browsers "I really don't care about SSL on this page, honestly, and I'm qualified to tell when it matters."
- version_five 3y agoHmmm.. hopefully between the two of them most can read it. The archive works for me.
- james-revisoai 3y agoAs far as I know, Firefox still allows this for any expired certificate which at least has correct domain details and authority (e.g. it once worked, which some dev should validate). SSL version or cipher mismatch can be from other causes. For example, the server might be responding with a html page that your browser is interpreting as https or vice versa, such as if the developers run http for local dev and https for prod and something gets confused.
- deathanatos 3y ago> SSL version or cipher mismatch can be from other causes. For example, the server might be responding with a html page that your browser is interpreting as https or vice versa, No, it's speaking TLS; it (the server) sends a TLS fatal alert & disconnects immediately after the ClientHello. It's odd, too; I asked nmap to show what ciphersuites the server offers, and it seems like what nmap was able to elicit indicates there is overlap between what's offered by the client and the server. So … IDK what is going on here. (It seems like the server isn't doing cipher suite negotiation correctly, AFAICT. The server-offered cipher suite set is a bit … unusual looking? E.g., no DHE, but ECDHE, but also non-DHE?)
- Izkata 3y ago> and it seems like what nmap was able to elicit indicates there is overlap between what's offered by the client and the server. On your client, maybe the person getting this is just out of date? (Or are you getting the same thing?)
- deathanatos 3y agoI'm getting the same error from Firefox, no cipher suite overlap. But I can see in Wireshark FF's ClientHello, and some of the cipher suites in that ClientHello seem to appear in the output that nmap says is the server's available cipher suites. So, I am perplexed. (And I'm on Arch; my FF can't be too far behind.)
- londons_explore 3y agoI wish the browser would just load the page without cookies whenever that happens. (ie. automatically switch to incognito mode for just that tab whenever security can't be guaranteed). Also, perhaps disable keyboard entry so you can't type a password in without acknowledging that you probably aren't visiting the site you think you are.
- atherton33 3y agoThere's probably heightened risk of having an unpatched vulnerability exploited if you keep processing the payload past the point where you suspect a bad actor is on the other end.
- sam_bristow 3y agoI believe you can type "thisisunsafe" on the SSL error page in Chrome to bypass any warnings.
- knodi123 3y agodoesn't work for me in chrome
- flakiness 3y agoThe primary source is the liked Twitter thread. I wonder how credible this source is. (I'm not familiar with the norm of ML community - They seem to be Twitter-heavy than other part of tech.)
- ShamelessC 3y agoCan you clarify what you’re referring to?
- Lerc 3y agoI only gave it a quick skim but it seems to match what I have learned so far, but I'm also learning from things that people said online so there remains the possibility of common misconceptions. The ALiBi stuff just makes sense to me. I don't understand why the Positional Sinusoidal Encoding was used initially. I assume there were good reasons for it but I haven't seen an explanation, (pointers to one appreciated).
- jimsimmons 3y agoDL practitioner for a decade here: The OP doesn’t explain anything. It just vaguely talks about a few things that might break when scaling context. But that means nothing. Take for example sinusoidal embeddings they talk about. Of course it breaks for large contexts but in no one uses it. GPT uses learned positional embeddings so the entire section is irrelevant. Copy this for pretty much everything else. Being an expert in a field has never been this exhausting
- it_citizen 3y agoTry virologist 3 years ago.
- oneseven 3y agoIt seems like learned positional encodings would still prevent you from doing fine tuning on a larger context size, though, so maybe using alibi is still relevant (although I have not read that paper).
- deleted 3y ago[deleted]
- asylteltine 3y ago[dead]
- treprinum 3y agoNot training full attention might score nicely in benchmarks but humans will instantly notice the whole spectrum is not represented. What you are proposing is basically get rid of infrequent combinations but those happen in the real world and will be missing from whatever your LLM will produce.
- cypress66 3y agoYour typical LLM benchmarks simply do not test or use large context sizes. We need benchmarks for tasks that requiere large context sizes (like recalling facts and understanding them in long stories). I'm sure OpenAI have internal benchmarks for these tasks.
- jumpCastle 3y agohttps://github.com/google-research/long-range-arena https://github.com/google-research/long-range-arena
- DavidSJ 3y agoThe attention is sparse in a single attention head in a single layer, but the model still has a way to route information where it needs it given multiple attention heads and multiple layers. Similarly, your computer is not physically connected to mine but I'm still able to read your comment.
- upthestake_s 3y ago[flagged]
- MintsJohn 3y agoWhat a weird take. These AIs pretty obviously so work. Sure some parts are surprising, more to me than to experts of course, yet why not take the less understood parts and try to understand them, experiment, see what works and why and how, see what doesn't, that's science, not "boy geniuses".
- upthestake_s 3y agoits only a weird take because the scientists who "do actually know how it works", are lopping this in as a software product. And the softwe=are engineers are still thinking they will be able to take credit for it. (wont be able to fix it) Applied math and software have nothing to do with each other. Any more than applied chemistry (reduced to math) written in a Visual Basic Excel macro is software. It aint computer science just because you typed it in.
- machdiamonds 3y agoI've been wondering about this, as simply extending the context window in a straightforward manner would lead to a significant increase in computational resources. I've had the opportunity to experiment with Anthropics' 100k model, and it's evident that they're employing some clever techniques to make it work, albeit with some imperfections. One interesting observation is that their prompt guide recommends placing instructions after the reference text when inputting lengthy text bodies. I noticed that the model often disregarded the instructions if placed beforehand. It's clear that the model doesn't allocate the same level of "attention" to all parts of the input across the entire context window. Moreover, the inability to cache transformers makes the use of large context windows quite costly, as all previous messages must be sent with each call. In this context, the RWKV-LM project on GitHub (https://github.com/BlinkDL/RWKV-LM https://github.com/BlinkDL/RWKV-LM) might offer a solution. They claim to achieve performance comparable to transformers using an RNN, which could potentially handle a 100-page document and cache it, thereby eliminating the need to process the entire document with each subsequent query. However, I suspect RWKV might fall short in handling complex tasks that require maintaining multiple variables in memory, such as mathematical computations, but it should suffice for many scenarios. On a related note, I believe Anthropics' Claude is somewhat underappreciated. In some instances, it outperforms GPT4, and I'd rank it somewhere between GPT4 and Bard overall.
- furyofantares 3y ago> One interesting observation is that their prompt guide recommends placing instructions after the reference text when inputting lengthy text bodies. I tend to do this with GPT-4 even on the context window in default ChatGPT (or more often I bookend it with instructions). I find it pays off at even 1000 tokens.
- littlestymaar 3y agoIsn't that weird? I mean weren't transformers/attention explicitly designed to avoid this problem faces by RNNs?
- 3y ago
- culopatin 3y agoTangent: where can a mortal go learn what this title means to the point where they can have something in their computer that allows them to change settings (and know which ones) and see what happens?
- cypress66 3y agoJust learn AI in general and the rest will be easy to process.
- taneq 3y ago“Learn to draw the rest of the owl first, then those first three ovals are super easy!”
- tekno45 3y agogood explanation on tokens and context https://www.youtube.com/watch?v=-4Oso9-9KTQ&pp=ygUJa3lsZSBoaWxs https://www.youtube.com/watch?v=-4Oso9-9KTQ&pp=ygUJa3lsZSBoa...
- quickthrower2 3y agoGood question. I have done the andrej karpathy course on youtube. It is not easy. And it is fast paced (about 12 or so hours that would be 50 if it were a university designed course plus same again for practice). Then even with all that debugging the model, and making architecture choices is a whole other thing he barely covers. Would love a good course on that. If you learn the nuts and bolts you can point to any part of the transformer model and describe what numbers and operations are happening in the forward and backward pass. That sort of is the reality. Reading other people’s fuzzy explanations is probably like understanding quantum mechanics by reading quanta (you get the layman example but don’t really understand… btw I know little about QM!) I needed to hop on other materials to exist. I van understand some of this but not all and running it and trying things out… probably not yet. Well never for 100k context size as I don’t want to sell my house to pay for compute :-).
- wokwokwok 3y agoThere are lots of places that explain LLMs. …but you will be disappointed if you expect/hope to be able to recreate or modify things yourself. You need a massive (multi TB) dataset of high quality data, and an array of 80GB graphics cards. The barrier to entry isn’t knowledge; it’s money. (So learning, yes. Try doing the fastai course. Change settings? No. Not unless you have a couple of million $$ in cloud credits to burn)
- mabbo 3y agoThe author mentions the n^2 nature of token size to memory and run time. Do we see any interesting work towards improving on that? Will we need a whole different paradigm to achieve that? Or is it simply the nature of the problem - we need to consider all pairs of tokens.
- gdiamos 3y agoOne thing that seems to be overlooked with very long prompts is that the compute still scales at best linearly with the input size. So a context size of 100k requires 100x more compute than a prompt size of 1k. For which applications is that worth it? Note you could reduce the cost to less than linear by using a retrieval method, but I don’t think that is what is being proposed.
- jimsimmons 3y agoNot sure why you’re downvoted
- jumpCastle 3y agoRead codebase and implement feature. Read paper and prove conjecture.
- safarimonkey 3y agoSome of the techniques improve over linear scaling of the baseline models. For example, from the article: > Conditional computation avoids applying all model parameters to all tokens from the input sequence. [CoLT5] applies heavy computations only to the most important tokens and processes the rest of the tokens with a lighter version of layers. It will speed up both training and inference. [CoLT5]: https://arxiv.org/abs/2303.094752 https://arxiv.org/abs/2303.094752
- gdiamos 3y agoThanks, that's interesting.
- jiggawatts 3y ago> For which applications is that worth it? Programming, primarily. Code takes many more tokens per kilobyte of text than written English. So even quite short blocks of code eat up a lot of tokens. The current AIs can do trivial, generic things using popular libraries. None can really help make changes in a large proprietary codebase where the prerequisite knowledge is the structure, design, and APIs of the private code. With 100K token windows, a model could be given entire database schemas, or reams of interface definitions, Rest API schemas, or whatever, and then make edits based on that context. It wouldn't even matter if it was slower than human, as long as it was cheaper. Look at it this way: An 8-GPU NVIDIA DGX server is what, $400K to purchase at retail pricing? That would be "good enough" to run really beefy LLMs. If you use that server for about 3 years, then even factoring in all ancillary costs, that's about $13/hour to run. Or about 30 cents per minute. So even if it takes some huge 100K token super-smart model a full minute to run through a prompt like "given the following reams of context, find the bugs in the given code below", then that's almost certainly worth it to most dev shops. Bugs can cost thousands of dollars to find and fix. Merely finding half of the bugs for mere cents per function could yield staggering savings.
- TechBro8615 3y agoNot that it matters, but it confused me: note that this blog is called "GoPenAI," and despite its domain having a one character difference from "openai," does not appear to be affiliated with OpenAI.
- two_in_one 3y agoDoesn't position become irrelevant after some distance in context window? I mean for long data table it's often doesn't matter how it's sorted. For meaningful text, text in between changes the meaning :). Transformers don't capture this. And just position may be too simplistic. RNNs (mentioned in other comments) with proper design can be better solution(?)
- HarHarVeryFunny 3y agoPositional encoding is relative, so it helps where word order is important at a short-range distance. This relative positioning will work around any absolute position in the context. The main way that long contexts are useful is via the key query "attention" mechanism, since the key may be matched at any position in the context. This allows something at the end of the context to refer to something at the beginning (or anyplace else).
- jiggawatts 3y agoCounter-intuitively, lossy compression can result in better quality than lossless compression! Sure, if you start off with a 4K raw video and compress it, by definition the quality gets worse. But if you compared 8K lossy that's the same size as 4K raw, then the 8K video would look better. That's because it allocates the bits more efficiently, putting them to work where it counts. It's a fairly safe bet that the same would apply to LLMs. If you start with a simple uncompressed LLMs that is 65B parameters and somehow compress or quantise it down to less than that, it will inevitably become a little dumber. But if you compared the raw LLM to one that utilised all of these tricks and was the same size, then the latter would be superior because it could use the available parameters more efficiently. If we can train and run GPT-3 cost effectively now with ~100B parameters, then it's a safe bet that we could train something as smart as GPT-4, with >200K window sizes, but as fast as GPT-3 for inference. (That's assuming all the recent quantization techniques are also applied.) I'm betting we'll have something like that generally available within two years. That'll be terrifying. An AI that can read and understand a book every few seconds...
- teaearlgraycold 3y agoWhat's interesting is we're currently training AI on how to use vector databases. Next generation LLMs trained on GitHub from the era of LangChain and FOSS vector DBs will be able to self-program their own long term memory recall. I don't think that just chunking and storing vectors for all of the text the LLM reads is the best approach, but it might be able to apply a strategy to each unique situation that is more optimal.
- rafterydj 3y agoThat's fascinating. As a novice to the problem, are there any resources you could link about this? I'm new to studying AI- but I've been prototyping connecting GPT to a nonrelational database to serve as a stand in for long term memory. My problem so far utilizing GPT3 has been difficulty getting it to use any consistent schema, as it will write to the database in a generated schema but try to recall in another. This is the first I've heard about using vector databases for the task.
- deleted 3y ago[deleted]
- weinzierl 3y agoAll the open an freely available models seem to have a short context window. Is there any one that I can play with, with a large one?
- btobolaski 3y agompt-7b and the falcon models have longer contexts available. They aren’t trained on longer contexts so, the results might not be very good.
- weinzierl 3y agoIsn't this true for any model. I always thought the context window exists on the training side and on the query side but not in the model itself and with query side bigger than training side not useful.
- aethelyon 3y agoThis article seems directionally correct, but poorly written
- neximo64 3y agoSparse attention is lossy, not sure i would count it as a good trick