25 ms·
100K Context Windows
- lee101 3y ago[dead]
- minimaxir 3y agoNo pricing, but given that OpenAI's GPT-4 doubles the cost-per-token if you go from 8k to a 32k context window, I suspect the pricing here will be 2-4x from the base Claude model which is 9k: https://cdn2.assets-servd.host/anthropic-website/production/images/apr-pricing-tokens_2023-05-10-213035_fhnc.pdf https://cdn2.assets-servd.host/anthropic-website/production/... Although with flash attention, who knows if marginal cost scales that consistently.
- adamkochanowicz 3y agohttps://cdn2.assets-servd.host/anthropic-website/production/images/apr-pricing-tokens.pdf https://cdn2.assets-servd.host/anthropic-website/production/...
- minimaxir 3y agoThose are the same SKUs I linked. The new model are a different model identified that's not listed in the pricing doc, although it sounds like the intent may be to replace the base from looking at the API docs: https://console.anthropic.com/docs/api/reference#-v1-complete https://console.anthropic.com/docs/api/reference#-v1-complet...
- f_devd 3y ago<4x would be quite optimistic, at ~11x the tokens the amount of compute/memory required would be n^10 (even with the lower starting point of flash attention) so unless they are already have excessive margins it wouldn't make much sense to go that low.
- sp332 3y agoI was assuming they used a different architecture to get the increase instead of just letting it eat hardware that way. Especially with the speed numbers in the post.
- l1n 3y agoPricing is the same as the base model.
- jimsimmons 3y agoConfirmation here: https://twitter.com/AnthropicAI/status/1656743460769259521?s=20 https://twitter.com/AnthropicAI/status/1656743460769259521?s...
- minimaxir 3y agoHuh. Well that changes things.
- rat9988 3y agoOnly for the duration of the beta
- jimsimmons 3y agoSource?
- felixgallo 3y agothe actual tweet you linked.
- jimsimmons 3y agoIt doesn’t say exclusively for the beta period
- scoopertrooper 3y agoWith an extremely literal reading you are correct, but there was clearly an implication.
- gigel82 3y agoNice, that's roughly a 250-page book based on average word counts.
- deleted 3y ago[deleted]
- meghan_rain 3y agoThe most interesting bit is that for the first time since the release of ChatGPT in December 2022, OpenAI does not have the lead on LLMs anymore. At least, for people who need large context windows, they would not be the first choice anymore.
- refulgentis 3y agoClaude’s very quietly better on everything but pricing, for a while, it just got buried because they announced on “AI Tuesday” (iirc gpt4 and Bing announcement day) The ChatGPT equivalent is 3x speed and was somewhere between ChatGPT and GPT4 on my TriviaQA benchmark replication I did Couple tweets with data and examples. Note they’re from 8 weeks ago, I know Claude got a version bump, GPT3.5/4 accessible via API seem the same. [1] brief and graphical summary of speed and TriviaQA https://twitter.com/jpohhhh/status/1638362982131351552?s=46&t=pxb6XO8RvMiZoZE1x8d58Q https://twitter.com/jpohhhh/status/1638362982131351552?s=46&... [2] ad hoc side by sides https://twitter.com/jpohhhh/status/1637316127314305024?s=46&t=pxb6XO8RvMiZoZE1x8d58Q https://twitter.com/jpohhhh/status/1637316127314305024?s=46&...
- com2kid 3y ago> I know Claude got a version bump, GPT3.5/4 accessible via API seem the same. GPT3.5 just got an update a few days ago that resulted in a pretty good improvement on its creativity. I saved some sample outputs from the previous March model, and for the same prompt the difference is quite dramatic. Prose is much less formulaic overall.
- refulgentis 3y agoThank you, every little comment I get from fellow boots on the ground is so valuable, lotta noise these days. Random Q, I don’t use the ChatGPT front end much past month or two, used it a week back and it seemed blazingly faster than my integration: do you have a sense of if it got faster too?
- com2kid 3y ago
- gdiamos 3y agoNice. Will we be able to get to 1M tokens?
- programmarchy 3y agoSeems like a good target. Even 100K seems too small. As a reference point, the Bible is ~750,000 words.
- smallerfish 3y ago"You are a hebrew god and below the dashes is The Word. Who will you smite today?"
- SeanAnderson 3y agobig if true? :) Exciting to see competition across LLMs for increasing context window size. I can't find updated pricing anywhere. Previous prices are here: https://cdn2.assets-servd.host/anthropic-website/production/images/apr-pricing-tokens.pdf https://cdn2.assets-servd.host/anthropic-website/production/... but don't seem to be embedded directly on the Anthropic website. I tried messing with the URL (apr -> may/jun) but 404'ed.
- kordlessagain 3y ago> Exciting to see competition across LLMs for increasing context window size. Maybe. I think the debate is going to continue about prompt optimization vs. context window size. A while ago, I had a rather interesting conversation with GPT-3.5 about forgetting things. Knowing what to forget, or delete from the prompt, may be just as important as what to put in it. Putting the kitchen sink into the prompt probably isn't going to help much, past a certain point and it may be putting certain things in there based on time and context is a better strategy.
- SeanAnderson 3y agoYeah, there's definitely diminishing returns. I just wanted to talk to ChatGPT about a game I'm developing. I have pages upon pages of product design notes and I'm not able to just copy/paste the whole thing in and start talking to it at 8k context length. There's not really duplicate information as far as I can tell since each section covers new topics. I'm sure there's a way to express the same ideas more succinctly, but I kind of want ChatGPT to do that for me rather than me figuring out how to do that just to interface the ideas into it.
- kordlessagain 3y agoHey! Try this: https://github.com/featurebasedb/DocGPT https://github.com/featurebasedb/DocGPT Holler if you want help with it. I have some more code I'm adding to it this week.
- skilled 3y agoMy wallet is hardly capable of handling 8k GPT-4.
- tikkun 3y agoThis is the first time I've felt like Anthropic may be a true competitor to OpenAI. I see 6 ways to improve foundation LLMs other than cost. If your product is best at one of the below, and has parity at the other 5 items, then customers will switch. I'm currently using GPT-4-8k. I regularly run into the context limit. If Claude-100K is close enough on "intelligence" then I will switch. Six Dimensions to Compare Foundation LLMs: 1. Smarter models 2. Larger context windows 3. More input and output modes 4. Lower time to first response token and to full response 5. Easier prompting 6. Integrations
- zomglings 3y agoAlso if you allow users to receive vector representations of context and provide such representations as side information when querying LLMs.
- danenania 3y agoOne question is how much other factors really matter compared to the raw "intelligence" of the model--how good its completions are. You're not going to care very much about context window, prompting, or integrations if the output isn't good. It would be sort of like a car that has the best steering and brakes on the market, but can't go above 5 mph.
- tikkun 3y agoStrongly agree. They are ordered by how much I think they generally will lead to users choosing one model over the other. Intelligence is the most important dimension by far, perhaps an order of magnitude or more above the second item on the list.
- danenania 3y agoOn that note, can anyone speak to how Anthropic (or other models) are doing on catching up to OpenAI for pure model intelligence/quality of completions? Are any others approaching GPT-4? I've only used GPT-based tools so I have no idea.
- justanotheratom 3y agoThis is nice, but it can get quite expensive. Let's say I have a book and I want to ask multiple questions about it. Every query will pay the price of the book's text. It would be awesome if I could "index" the book once, i.e. pay for the context once, and then ask multiple questions.
- wahnfrieden 3y agoNot sure about this one but you can usually ask multiple questions in one shot at least
- minimaxir 3y agoGeneration is more expensive than the prompt input (for Claude v1, generation is 3x the cost; for GPT-4 it's 2x the cost) It makes the economics slightly trickier.
- newhouseb 3y agoI wonder why this is? Naively there's no difference between the two from a transformer standpoint. Perhaps it's because under the hood there's additional safety analysis/candidate generate that is resource intensive?
- space_fountain 3y agoWell each additional token generated requires rerunning the model right to find the next likely token given the previous one
- newhouseb 3y agoNaively, yes, but you can cache the bulk of that "rerunning" [1]. That said the (non-flash) attention costs go up with the length of the sequence so perhaps this is just a simpler way to approximate these costs. [1] https://kipp.ly/blog/transformer-inference-arithmetic/ https://kipp.ly/blog/transformer-inference-arithmetic/
- helen___keller 3y agoThis seems like it could be a game changer. Modern LLM based applications face a balancing act of context limitations, which often results in some kind of mapreduce-type behavior when that context can’t fit the input If contexts keep growing, the landscape of LLM application engineering will as well
- whimsicalism 3y agoThe problem is there are no public benchmarks usually so it is hard to really compare on long context lengths to see if they are still performing equally intelligent of tasks.
- tempusalaria 3y agoWould be great to see some benchmarks on how loss changes across this very large context. It’s been technically possible to do 1mln+ token context for some time with performance deterioration so it would be interesting to see how this compares to those efforts
- ginger2016 3y agoHow do I sign-up? What is the cost?
- karmasimida 3y agoGoing to be absolutely expensive.
- whimsicalism 3y agoWe need public benchmarks. This is incredibly fast progress on large contexts and I would like to see if they are actually attending equally as well to all of the information or there is some sparse approximation leading to intelligence/reasoning degradation.
- monlockandkey 3y agohttps://lmsys.org/blog/2023-05-10-leaderboard/ https://lmsys.org/blog/2023-05-10-leaderboard/ https://chat.lmsys.org/?arena https://chat.lmsys.org/?arena Claude by Anthropic has more favourable responses then ChatGPT
- polishdude20 3y agoSo I tried this prompt in their chatbot arena multiple times. Each time getting the wrong answer: "Given that Beth is Sue's sister and Arnold is Sue's father and Beth Junior is Beth's Daughter and Jacob is Arnold's Great Grandfather, who is Jacob to Beth Junior?"
- Workaccount2 3y agoChatGPT3.5* It's still below GPT4, but it is closer to 4 than 3.5
- nynx 3y agoThere has got to be a number of fascinating tricks that they're using to support context lengths that long. Shame it's all closed-source.
- sweezyjeezy 3y agoCan LLMs take advantage of this bigger window to solve meaningful tasks though? I can't imagine in the training data, knowing what happened 100k tokens ago would be _that_ relevant to predicting the current token very often, so unless this is something that the model learns to leverage more implicitly, I'd be a bit pessimistic.
- ttul 3y agoYes. For instance, a large context window allows you to have a chat for months where the model can remember and make use of everything you’ve ever talked about. That enables creating a much more effective “assistant” that can remember key details months later that may be valuable. A second example is the analysis of long documents. Today, hacks like chunking and HyDE enable us to ask questions about a long document or a corpus of documents. But is far superior if the model can ingest the whole document and apply attention to everything, rather than just one chunk at a time. Chunking effectively means that the model is limited to drawing conclusions from one chunk at a time and cannot synthesize useful responses relating to the entire document.
- sweezyjeezy 3y agoI'm not questioning whether it would be useful, just whether it's actually something that token masking in training is going to work to make the model learn this.
- kaj_sotala 3y agoI would imagine that if you are training on the text of a novel, then anything that happened earlier in the text may be relevant for predicting the next events. Especially if it's something like a detective novel that has clues about the criminal's identity scattered across the story. Also if you are training on a database of code.
- sweezyjeezy 3y agoYeah but when you're training a neural net with backprop on a finite dataset, "this would help the model" ≠ "the model will learn this". This is 100% speculation, but my intuition is that it's not going to work very well unless it happens 'a lot' in the training data, or if they've curated the data specifically to try and make it learn long range signals.
- deleted 3y ago[deleted]
- ablyveiled 3y agoWhat's the catch? Using GPT-4 relative to its own marketing copy was a letdown.
- we_never_see_it 3y agoGoogle is really trying to catch up to OpenAI & MS. The truth is they have never been in the race to begin with. All they had and still have is PR stunts. Let's see if their copying of MS model will produce anything useful.
- andreyk 3y agoCurious why you think this? PaLM2 looks great, and Google has been productizing cutting edge AI pretty fast for years.
- sebzim4500 3y agoI guess PaLM2 is competitive with GPT-3.5 so for people not willing to pay it will be an attractive offering. I'm not sure that counts as 'great' though.
- rsstack 3y agoBased on what do you think it's comparable to GPT-3.5 and not to 4? Did we see a lot of public performance?
- sebzim4500 3y agoThey claim it is already being used in Bard, also if you read the paper it does much worse at the important benchmarks.
- MacsHeadroom 3y agoPaLM 2 can't even solve "Write three sentences ending in the word Apple." It's worse than GPT-3.5. Go see for yourself at bard.google.com, which is running on PaLM 2 everywhere but the EU as of yesterday.
- Garrrrrr 3y agoAh yes, the famous benchmark for all LLMs. I just tried your novel example with GPT-3.5 and it couldn't solve it either: > After lunch, I like to snack on a juicy and crisp apple to satisfy my sweet tooth. > In the fall, many families enjoy going to apple orchards to pick their own apples and make homemade apple pies. > The new MacBook Pro features a powerful M1 chip and a stunning Retina display, making it the perfect tool for creative professionals who work with Apple software.
- flerovium 3y agoIt means nothing as long as they don't actually let us test the API. Good luck waiting for it.
- nr2x 3y ago“POC or GTFO” as the security people say. :-)
- jackson1372 3y agoSee the pricing PDF[^1] and API docs[^2], but TL;DR: - Price per token doesn't change compared to regular models - Existing api users have access now by setting the `model` param to "claude-v1-100k" or "claude-instant-v1-100k" - New customers can join waitlist at anthropic.com/product [1]: https://cdn2.assets-servd.host/anthropic-website/production/images/model_pricing_may2023.pdf https://cdn2.assets-servd.host/anthropic-website/production/... [2]: https://console.anthropic.com/docs/api/reference#parameters https://console.anthropic.com/docs/api/reference#parameters
- 19h 3y agoWe have access and we’re already playing with the 100K model. It’s pretty insane. We‘re about to ditch all of our recursive summarization code. If you want me to test something for you, lmk and I’ll send it through the api.
- jcims 3y ago> We‘re about to ditch all of our recursive summarization code. I’m in an adjacent industry and this is what I’m looking forward to.
- ibitto 3y agoAnyone using Claude? How long did it take you to get access?
- harisec 3y agoClaude is available for free in the Poe app (poe.com). I think it's good and underappreciated.
- dkarras 3y agoyeah, been using it instead of ChatGPT and it performs better IMO. My conversational LLM of choice for sure.
- danysdragons 3y agoIt is good, but the free subscription to Poe only provides access to Claude Instant. It's impressively fast but not their smartest model (claude-v1.3).
- Mizza 3y agoI've got access, it's _blazing_ fast and seems very good. Solved some of my little puzzles that other models couldn't. I haven't tried ChatGPT-4 yet, but it's the best one that I have used.
- thewataccount 3y agoYou need to try GPT4 only because GPT3.5 really doesn't compare to it in a lot of ways.
- iEchoic 3y agoGPT-4 is a major leap ahead of everything else I've used (including GPT-3.5), so definitely worth trying for comparison.
- com2kid 3y agoI am curious how consistent Claude is at obeying detailed instructions. One issue ChatGPT 3.5 and 4 have, even with just a few hundred words of instructions, is it forgets instructions given to it earlier on.[1] This huge context window is awesome though, I'm trying to use LLMs to do small town social interaction simulations, with output in a structured format. Finding ways to compress existing state and pass it around, so the LLM knows the current state of what people in the town did for a given day is hard with a tiny token limit! [1] For my use cases, early instructions tend to be describing a DSL syntax for responses, if I add too much info after the instructions, the response syntax starts getting wonky!
- rescripting 3y agoA simple example I ran in to was I asked ChatGPT to generate me story in madlibs format for my 4 year old daughter. They're in the format "The young _____ went to the ______, ...", and she fills in the blanks with silly nouns/adjectives. As she kept asking for more, I prompted "great, do another one" and eventually my original instruction fell out of the context window. It continued to generate a children's story, but with no more blanks.
- com2kid 3y agoThis is actually a different issue, largely a UI one, although one I wish ChatGPT would fix it. There is no good way to tell it "this isn't a conversation, just repeat the answer to the initial prompt again". The solution is to just re-paste the initial prompt in each time, but still it isn't ideal. There isn't a good way to tell chatgpt "you can throw away all the context after the initial prompt and up until now". Of course the entire point of ChatGPT is that it maintains a conversation thread, so I get why they don't fix up this edge case. My problem is more of, I give ChatGPT some complicated instructions, and it'll start forgetting the early on instructions long before any token limit is reached. So for example, if early on I ask for certain tokens to be returned in parens, well my initial prompt is too long, it'll forget the parens thing and start returning tokens without the surrounding (), which then breaks my parser!
- 3y ago
- fire 3y agogod I'd love to work there
- Imnimo 3y ago>For example, we loaded the entire text of The Great Gatsby into Claude-Instant (72K tokens) and modified one line to say Mr. Carraway was “a software engineer that works on machine learning tooling at Anthropic.” When we asked the model to spot what was different, it responded with the correct answer in 22 seconds. This sort of needle-in-a-haystack retrieval is definitely impressive, and it makes a lot more sense to achieve this in-context rather than trying to use a vector database if you can afford it. I'm curious, though, whether there are diminishing returns in terms of how much analysis the model can do over those 100k tokens in a single forward pass. A human reading modified-Gatsby might eventually spot the altered line, but they'd also be able to answer questions about the overarching plot and themes of the novel, including ones that cannot be deduced from just a small number of salient snippets. I'd be curious to see whether huge-context models are also able to do this, or if they start to have trouble when the bottleneck becomes reasoning capacity rather than input length. I feel like it's hard to predict one way or the other without trying it, just because LLMs have already demonstrated a lot of surprising powers.
- tunesmith 3y agoI've been curious about this for a while, I have a hobby use-case of wanting to input in-progress novellas and then asking it questions about plot holes, open plot threads, and if new chapter "x" presents any serious plot contradiction problems. I haven't tried exploring that with a vectordb-embeddings approach yet.
- make3 3y agoThis is an exact example of something a vector dbs would be terrible at. Vector dbs work by fetching segments that are similar in topics to the question, so like "Where did <Character> go after <thing>" will retrieve segments with locations & the character & maybe talking about <thing> as a recent event. Your question has no similarity with the segments required in any way; & it's not the segments that are wrong it's the way they relate to the rest of the story
- toss1 3y ago
- sashank_1509 3y agoHow are LLM’s increasing their context size? I guess you just increase input size if it’s for the self supervised GPT3 style training but for RLHF? Are they creating datasets of books to input to the LLM and then making human labelers label the response? There might be a smart way that does not involve new datasets
- potatoman22 3y agoI don't think RLHF datasets need to take full advantage of the context window. There's also many ways to programatically generate NLP datasets.
- sp332 3y agoMosaic wrote about their new model here. https://www.mosaicml.com/blog/mpt-7b https://www.mosaicml.com/blog/mpt-7b It was trained on 65k inputs and has decent performance working with 80k+ tokens.
- jacooper 3y agoAnthropic is basically Google's openAI.
- deleted 3y ago[deleted]
- deleted 3y ago[deleted]
- mark_l_watson 3y agoWith quadratic time complexity for context size, that gets expensive.
- Workaccount2 3y agoIs there any path towards folding tokens into the actual model? That is, continual training rather than the current "training first then just tokens after"
- ilaksh 3y agoPaLM 2 on Vertex AI which Google just released yesterday has fine tuning the large models as a core part of their offering.
- RoddaWallPro 3y agoI requested & have been waiting for access to Claude for nearly 3 months now. Guess the waitlist must be really long...
- technics256 3y agoYou likely got rejected. Was the same for me and I reapplied with a good use case and was let in
- jazzyjackson 3y agoAPI access or just access to the chatbot? You can go through Poe.com
- melvinmelih 3y ago> You can drop multiple documents or even a book into the prompt and then ask Claude questions that require synthesis of knowledge across many parts of the text. This is cool but does it also work the other way around? Generate a book's worth of content based on a single prompt?
- Chabsff 3y agoKinda. But it`s going to be a lot like how data compression works. There will always be a somewhat fundamental limit to how much "creativity" you can get out of a small prompt generating large texts when using an isolated model.
- cubefox 3y agoThat's a good question. Can Claude write a coherent book?
- deleted 3y ago[deleted]
- deleted 3y ago[deleted]
- cheeselip420 3y agoMaybe this model can finish Winds of Winter and the rest of GoT for us...
- thepasswordis 3y agoThat's actually a really interesting use case!
- azakai 3y ago75,000 words is a drop in the bucket for A Song of Ice and Fire: https://blog.fostergrant.co.uk/2017/08/03/word-counts-popular-books-world/#:~:text=A%20Feast%20for%20Crows%20%E2%80%93%20300%2C000,(so%20far)%20%E2%80%93%201%2C770%2C000%20words https://blog.fostergrant.co.uk/2017/08/03/word-counts-popula...
- camel-cdr 3y agoMeanwhile web serial authors: [0] [1] [0] https://wanderinginn.neocities.org/statistics https://wanderinginn.neocities.org/statistics [1] https://www.reddit.com/r/Parahumans/comments/rz8ogt/wildbows_crazy_total_word_count_and_rate_of/ https://www.reddit.com/r/Parahumans/comments/rz8ogt/wildbows...
- akiselev 3y agoYou'd want to generate it in multiple steps to make it feasible to control the text generation anyway. First call generates the broad outline, several parallel calls flesh out character development and some other details so that they're consistent, then generate the story piece by piece by feeding in bits of the outline.
- nottorp 3y agoAnd then you end up with what the movie did which is not exactly a GRRM novel.
- pclmulqdq 3y agoThat may need a million tokens just for one book, though!
- m3kw9 3y agoIs this real input context or is it some vectordb in the background type trickery?
- HarHarVeryFunny 3y agoPretty sure it's not "real" (model) context width. Another wide context model is MosaicML's MPT-7B-StoryWriter-65k+ which they are describing as having a context width of 65k, but then give a bit more detail to say they are using ALiBi - a type of positional encoding that allows longer contexts at inference time than training (i.e beyond the real context width of the model). For these types of "extended context" models to actually reason over inputs longer than the native context width of the model, I assume that there is indeed some sort of vector DB trickery - maybe paging thru the input to generate vector DB content, then using some type of Retrieval Augmented Generation (RAG) to process that using the extended contexts ? Maybe someone from Anthropic or MosaicML could throw us a bone and give a bit more detail of how these are working ! https://www.mosaicml.com/blog/mpt-7b https://www.mosaicml.com/blog/mpt-7b https://arxiv.org/abs/2005.11401 https://arxiv.org/abs/2005.11401
- sanxiyn 3y agoThere is no trickery going on for MPT. The model is open source. MPT-7B-StoryWriter-65k+ was trained on books with 65k context length so there is nothing not "real" about it. The point of ALiBi is that you can reuse training you did with short context for long context to save compute, but of course you also just train with long context. Who knows about Anthropic.
- jxmorris12 3y agoappears to be real input context
- vrglvrglvrgl 3y ago[dead]
- dr_dshiv 3y agoI often prefer Claude over GPT4 (partially due to speed), but it degrades more quickly. Like I can get a better response early, but usually the quality drops faster. But, sometimes if it can really vibe with it, it gets better over time.
- timsuchanek 3y agoCurious what this will mean for the vector db vendors. Imagine finetuning would be quick and cheap. Could there be a world where vector dbs aren’t needed anymore?
- shri_krishna 3y ago100k context limit is still a limit (we have no idea how Anthropic is achieving this - if it is extension of the base model context limit itself or some vector db trickery in the backend or probably even RAG). Even in this example, though it could fit entire text of Great Gatsby it still is 1 book/text/document. Typical business use cases require searching through hundreds if not thousands of documents/books and finding similar vector embeddings through all of them and fetching top-K results (this is how Google search works when it has to scan through embeddings for billions of websites). These top-K results can be stuffed into the 100k context limit and produce an even more holistic picture rather than just stuff one book/pdf/file into the context. Depends on the requirements though. I don't see how it might affect vector db vendors who can process billions of vectors per query and provide top-K results. Also having a massive context length is not necessarily a good thing from perspective of cost. It also doesn't work great with a chatbot as you will have to feed the same 100k worth context back into the chatbot for every question which will turn out to be very expensive. At some point you will have to discard some parts of the context to be specific to the question being asked and that is where vector embeddings come into play. For one off research/Q&A 100k limit works great!
- gumballindie 3y agoI am noticing a different tone coming from Anthropic. Unlike openai they dont appear to be focused on fud and replacement. Gives the impression it’s run by adults instead of crypto bros turned ai experts. Curious how their models will work.
- lubesGordi 3y agoUm Ilya Sutskever isn't a crypto bro.
- gumballindie 3y agoNo but sam altman is. That company can go whistling.
- flangola7 3y ago>Unlike openai they dont appear to be focused on fud and replacement. I do not have a clue what you are talking about. What happened?
- alpark3 3y agoI use GPT-4 through the API, but I can't help but hate the token/characterization pricing of these LLM APIs we've seen so far. Because the entire context needs to be fed back into the model, as my conversation gets longer, it gets more expensive. Yeah, it's fractions of a cent and cheaper, but something about it is so psychologically taxing that I'd rather pay a flat sum/month and get unlimited access, even if it costs more considering my usage.
- WA 3y agoHave you tried to start a new chat after your first question, but refine your new prompt to include some infos you gathered from the first response? This way, you know exactly how many tokens you gonna send.
- throwaway012919 3y agoSounds expensive. I guess we know where the $580M 'investment' from SBF is going now.
- maxutility 3y agoI don’t see this in the article. Has Anthropic explained the mechanism by which they were able to cost-effectively expand the context window, and whether there was additional training or a design decision (e.g. alternative positional embedding approach) that helped the model optimize for a larger window?
- swyx 3y agoa lot of people are speculating online (https://twitter.com/search?q=anthropicai%20alibi&src=typed_query&f=top https://twitter.com/search?q=anthropicai%20alibi&src=typed_q...) but i'm guessing it's ALiBi, which was also used by MPT-7B to get up to 85k long context
- sanxiyn 3y agoNo. As far as I know, they haven't said anything about this. Neither did OpenAI about gpt-4-32k. MosaicML did say something about MPT-7B-StoryWriter-65k+: https://www.mosaicml.com/blog/mpt-7b https://www.mosaicml.com/blog/mpt-7b. They are using ALiBi (Attention with Linear Biases): https://arxiv.org/abs/2108.12409 https://arxiv.org/abs/2108.12409. I think OpenAI and Anthropic are using ALiBi or their own proprietary advances. Both seem possible.
- maxutility 3y agoInteresting. Does the decision to use ALiBi have to be done before the model weights are first trained, or is there a way that these models could have incorporated ALiBi instead or in addition to an alternate positional encoding method to ALiBi after they were first trained?
- sanxiyn 3y agoThe decision needs to be made before starting training. Maybe there is a clever way to add it after the fact in the style of LoRA? First, that would be a different method in its own right (just as LoRA is), second, I can't see how to do so easily. But then I just thought about it for a minute.
- 3y ago
- ilaksh 3y agoDid anyone else get on the waitlist, get in, and now their console link doesn't work? I remember deciding the code generation wasn't good enough to bother. Not sure if I actually ever activated it but I guess not. Now I tried to request access again on their form and it just redirected. Can't even tell if that worked. Does anyone know if this can program as well as GPT-4? Because if so then the larger context window is a big improvement.
- swiftcoder 3y ago> When we asked the model to spot what was different, it responded with the correct answer in 22 seconds. Now we've gone from using ML to implement slow, unreliable databases, to using ML to implement slow, unreliable string comparison, I guess
- darig 3y ago[dead]
- terabytest 3y agoHow does Claude stack up to GPT-4?
- worik 3y agoTheir sign up form does not let me sign in for early access. A bit disappointing
- qwertox 3y agoThe day a quantum computer is able to host a huge LLM, things will get really interesting for humanity. I say this, because I'm not sure how all of this is really going to scale on GPUs. It feels like LLM's are just as magical as quantum computing.
- pk-protect-ai 3y agoOk. It has spatial comprehension of some level. Unlike GPT-4 it lacks proper time comprehension because it is bad at calculus. Unlike GPT-4 it can't properly solve traveling salesman problem.
- PeterisP 3y agoAll I see in the link is empty PR claims - is there any information about how they're doing that? There are all kinds of known techniques that "expand" context window without really doing so, with different tradeoffs, and unless they provide actual information, any claims should be taken with a pile of salt, we shouldn't just assume that they actually have "true" 100k context windows.
- seydor 3y agoso i m going to just paste a few physics book and ask it "make fusion" What is the approach to increase the sequence length here?
- deleted 3y ago[deleted]
- hn_throawlles 3y agothis is an advertisement, wouldn't it be nice to know this in advance?
- flyingcircus3 3y agoIn advance of what? The first three words on this article's page, beyond the top banner, are "product", "announcements", and "Introducing". What other category of article could you more reasonably expect to encounter after reading those three words?
- hn_throawlles 3y agoin advance of clicking
- epicprogrammer 3y agoThis is a fascinating study on the impact of context windows on language models. It's interesting to see how smaller context windows can lead to more efficient and accurate language models, even when dealing with complex natural language tasks like question answering. I think this research could have important implications for a wide range of applications, from chatbots and virtual assistants to machine translation and text summarization. I'm looking forward to seeing how these findings are further developed and applied in real-world scenarios.
- Anil1331 3y agoThis is incredible
- maskedpirate 3y agoIs there any paper or architecture about this model that explains how they achieved it? is it unlimiformer [https://arxiv.org/abs/2305.01625?utm_source=tldrai https://arxiv.org/abs/2305.01625?utm_source=tldrai] or something different?
- dsrtslnd23 3y agoAnyone with API access tried this for coding already?
- coldblues 3y agoIt's very unfortunate that all of these AI models are so impressive, yet they're all heavily filtered and bogged down by these massive AI corporations. All the filtering that they do heavily impacts the performance of the language models. A 100K context would also be incredible for roleplay but infeasible because of the heavy filtering.
- 19h 3y agoClaude may reject answers, but other than OpenAI‘s GPT you can put words into the mouth of the assistant and essentially bypass safety checks. In fact, Anthropic explicitly discusses putting words into the assistants‘ mouth to be able to shape it’s responses and make it better align with the desired output.
- coldblues 3y agoEventually you will get your account banned, not to mention that the filtering that they do decreases the quality of the results you will get compared to an uncensored model, even if you can "jailbreak" it.
- 19h 3y agoIf we were to get banned for this, we'd have been banned long ago. We literally process "questionable" content "as a service" and this use case was explicitly approved. (We do heuristics and ML-assisted background checks on unstructured OSINT data.)
- goertzen 3y agoDoes context window size matter beyond a certain point?
- dshah 3y agoThis is a really useful thread with posts from people actually using LLMs. I wonder if Claude-100k could be used to ingest this entire thread and then answer questions based on it, or summarize or identify the pros/cons of certain aspects of Claude, large context windows, vector embeddings, etc.
- havercosine 3y agoHas anyone actually tried to talk with Anthropic team for obtaining commercial usage permissions? I'm slightly vary of anything that says "talk to our sales team and explore partnerships" to be a time waste. But happy to change my mind if anyone has gone through the pain and found it worth.
- dataangel 3y agoThe "Request Access" button currently does nothing