8 ms·

Access is invite only for the API, and rate limited for paid GPT+. > gpt-4 has a context length of 8,192 tokens. We are also providing limited access to our 32

Access is invite only for the API, and rate limited for paid GPT+. > gpt-4 has a context length of 8,192 tokens. We are also providing limited access to our 32,768–context (about 50 pages of text) version, gpt-4-32k, which will also be updated automatically over time (current version gpt-4-32k-0314, also supported until June 14). Pricing is $0.06 per 1K prompt tokens and $0.12 per 1k completion tokens. The context length should be a huge help for many uses.

chis 4y ago
I'm really curious to see if expanding the context length this much will allow GPT to do typical software development tasks on a big codebase. If it can take in a github issue and produce decent code solving a complex issue across many files... will certainly be an interesting time.
- layer8 4y ago
  My guess is that anything requiring nontrivial business/technical domain knowledge will be fairly safe. Also anything with a visual (or auditory) correlate, like UI work.
  - oezi 4y ago
    Why would you think this? As long as the technical domain knowledge is at least partially published, I don't see them stopping becoming better. UI stuff just has an input problem. But it is not that hard to think that ChatGPT could place widgets once it can consume images and has a way to move a mouse.
    - layer8 4y ago
      > As long as the technical domain knowledge is at least partially published Most internal technical and business domain logic of companies isn’t published, though. Every time I asked ChatGPT about topics I had actually worked on over the past decade or two, or that I’m currently working on, it basically drew a blank, because it’s just not the category of topics that are discussed in detail (if at all) on the internet. At best it produced some vague generalisms. > once it can consume images and has a way to move a mouse. That’s quite far from ChatGPTs current capabilities, which is strongly tied to processing a linear sequence of tokens. We will certainly improve in that direction as we start combining it with image-processing AIs, but that will take a while.
      - cma 4y ago
        I wonder if there will be a race to buy defunct companies for access to their now valuable junky tech-debt ridden hairball code, so they can train on it and benchmark on fixing bugs and stuff. With full source control history they could also find bug resolution diffs.
        layer8 4y ago
        That source code isn’t worth much without the underlying domain knowledge, large parts of which only exist in the employees’ heads, more often than not. Maybe if the code is really, really well documented. ;) Companies could in principle train an in-house AI with their corporate knowledge, and will likely be tempted to do so in the future. But that also creates a big risk, because whoever manages to get their hand on a copy of that model (a single file) will instantly have unrestrained access to that valuable knowledge. It will be interesting to see what mechanisms are found to mitigate that risk.
        in3d 4y ago
        The weights file could be encrypted and require a password before becoming usable.
      - dwaltrip 4y ago
        Check out the announcement. GPT-4 accepts mixed-mode inputs of text and images. Mouse cursor instructions aren’t a massive leap from the current capabilities, given the rate of progress and recent developments around LLM tool use and the like.
      - dw_arthur 4y ago
        I think what you say goes for most jobs. Why would GPT know much detail about being a machinist or luthier? Eventually job and role specific information will be fed into these models. I imagine corporations will have GPTs training on all internal communications, technical documentation, and code bases. Theoretically, this should result in a big increase in productivity.
    - barking_biscuit 4y ago
      >UI stuff just has an input problem. But it is not that hard to think that ChatGPT could place widgets once it can consume images and has a way to move a mouse. I remember one of the OpenAI guys on Lex Fridman podcast talking about how one of the early things they tried and failed at was training a model that could use websites, and he alluded to maybe giving it another go once the tech had matured a bit. I think with GPT-4 being multi-modal, it's potentially a very close to being able to do this with the right architecture wrapped around it. I can imaging an agent using LangChain and feed it a series of screenshots and maybe it feeds you back a series of co-ordinates for where the mouse should go and what action to take (i.e. click). Alternatively, updating the model itself to be able to produce those outputs directly somehow. Either way, I think that's going to happen.
  - dirheist 4y ago
    Yeah, the example given in the OpenAI GPT4 twitter video is someone asking it to write a python script to analyze their monthly finances and it simply just importing dataframes, importing "finances.csv", running a columnar sum for all finances and then displaying the sum and the dataframe. I'm sure it's capable of some deeper software development but it almost always makes radical assumptions and is rarely ever self sufficient (you don't need to look it over and don't need to change the architecture of the code it produced).
- graypegg 4y ago
  How would you have it suggest solutions for multiple files? Has anyone gotten GPT-X to output a valid git patch or something?
  - fabiospampinato 4y ago
    You just kind of concatenate the entire codebase into one file, tell the model to do something and output the modified codebase into another file, diff the two and produce a patch automatically.
    - graypegg 4y ago
      That codebase=>token stream=>codebase step feels like it could be lossy depending on how you encode things like file paths when concatenating everything, would be interesting to see in practice though!
    - dom96 4y ago
      or you might even be able to feed it individual files with their filenames, then ask it what modifications it would make as a diff for each of the files
  - alexwebb2 4y ago
    Yep, I know that’s been possible since at least GPT-3 davinci
    - JyB 4y ago
      How would you go about it?
  - gremlinsinc 4y ago
    I think there's ways but you might have to use pinecone db or something like lang chain to essentially give it a long term memory... or another option is having one instance or chat order code page and one that basically just has an API index and knows which chat has the related things.
- 2OEH8eoCRo0 4y ago
  I'd love to get to a point where I can go: Add a cast button to this open source android video app. I see some FOSS-boosting silver linings in all of this.
- barking_biscuit 4y ago
  >If it can take in a github issue and produce decent code solving a complex issue across many files... will certainly be an interesting time. Oh snap. I didn't even think about that! That gives me a fun idea! I've got a repo that I built and setup CI/CD and setup renovate to automatically upgrade dependencies and merge them when all the tests pass, but of course sometimes there are breaking changes. I don't actively work on this thing and hence it's just got issues sitting there when upgrades fail. It's the perfect testing ground to see if I can leverage it to submit PRs to perform the fixes required for the upgrade to succeed! That'll be hectic if it works.
- amelius 4y ago
  It can't even do simple sysadmin tasks like fixing a broken installation, or fixing simple configure/make/make install issues.
sebastianconcpt 4y ago
And how it works? you can build a context and then ask something in a prompt using it?
- teruakohatu 4y ago
  Context is how many tokens it can be fed to produce an output. So now you can feed it up to 32k words (tokens).
  - O__________O 4y ago
    Token is 0.75 words on average per OpenAI; 32k in tokens would be roughly 24k in words. https://help.openai.com/en/articles/4936856-what-are-tokens-and-how-to-count-them https://help.openai.com/en/articles/4936856-what-are-tokens-...
minimaxir 4y ago
$0.12 per 1k completion tokens is high enough that it makes it prohibitively expensive to use the 32k context model. Especially in a chatbot use case with cumulative prompting, which is the best use case for such a large context vs. the default cheaper 8k window. In contrast, GPT-3.5 text-davinci-003 was $0.02/1k tokens, and let's not get into the ChatGPT API.
- weird-eye-issue 4y ago
  I disagree that out of all possible use cases for a large context model that a chatbot is really the "best use case".
- deleted 4y ago
  [deleted]
- ml_basics 4y ago
  > Especially in a chatbot use case with cumulative prompting, which is the best use case for such a large context vs. the default cheaper 8k window. Depends on what is up with the images and how they translate into tokens. I really have no idea, but could be that 32k tokens (lots of text) translates to only a few images for few-shot prompting. The paper seems not to mention image tokenization, but I guess it should be possible to infer something about token rate when actually using the API and looking at how one is charged.
  - minimaxir 4y ago
    Currently, CLIP's largest size is at patch-14 for 336x336 images, which translates to 577 ViT tokens [(336/14)^2+1]. It might end up being token-efficient depending on how it's implemented. (the paper doesn't elaborate)
- sebzim4500 4y ago
  I would imagine most usecases for the 32k model have much longer prompts than completions, so the $0.06 per prompt token will be the real problem. I can't think of a usecase yet, but that might be because I haven't got a sense of how smart it is.
James_Henry 4y ago
Also note that image input isn't available to the public yet. >Image inputs are still a research preview and not publicly available.
anileated 4y ago
Will any of the profits be shared with original authors whose work powers the model?
- James_Henry 4y ago
  Which authors? Machine Learning research authors?
  - anileated 4y ago
    I mean the authors who wrote the works which are resold by Microsoft for its own profit without any opt-in or even opt-out, much less compensation.
- cma 4y ago
  It doesn't seem to be answered in the article, but if it was and you read it should you have to pay them a fee for the knowledge if it was published openly on the net?
  - anileated 4y ago
    You are confusing two distinct cases. In the first case, you found/bought a book and read it. No one can or should make you pay for it, unless you stole the book. In the second case, you found/bought a book then reprinted it infinitely and sold it for profit, ethically you should pay the author and legally you should be in violation of the law. Even if you made a machine that ingests and recombines books automatically, and you keep that machine locked up and charge people for its use, it is the same scenario: the machine would be absolutely useless without the original books, those books cost people effort and money to produce, yet you pay those people nothing while the machine is basically an infinite money maker for you. I hope the analogy makes sense.
- sebzim4500 4y ago
  No. Now that you have read my answer, you owe me $0.01 because your brain might use this information in the future.
doomleika 4y ago
Poe.com(by Quora) have gpt4 now, give it a shot
- daydream 4y ago
  Seems limited to only one prompt per day. I just tried it.
  - acallaha 4y ago
    You can pay $17/month (Apple subscription only for now), for ~unlimited queries to GPT-4 (and Claude+)
    - daydream 4y ago
      I missed that, thanks
nealabq 4y ago
> Image inputs are still a research preview and not publicly available. Will input-images also be tokenized? Multi-modal input is an area of research, but an image could be converted into a text description (?) before being inserted into the input stream.
- teruakohatu 4y ago
  My understanding is thta the image embedding is included, rather than converting to text.
  - 2sk21 4y ago
    My understanding is that image embeddings are a rather abstract representation of the image. What about if the image itself contains text, such as street signs etc?
fzliu 4y ago
One way to get around context length is to perform embedding and retrieval of your entire corpus. Langchain (https://langchain.readthedocs.io/en/latest/ https://langchain.readthedocs.io/en/latest/) and Milvus (https://milvus.io https://milvus.io) is one of the stacks you can use.
- deleted 4y ago
  [deleted]
- ComplexSystems 4y ago
  Can you elaborate on how this works?
  - teaearlgraycold 4y ago
    You run the corpus through the model piecemeal, recording the model's interpretation for each chunk as a vector of floating point numbers. Then when performing a completions request you first query the vectors and include the closest matches as context.
soheil 4y ago
I still doesn't understand how can content length not be limited if you have a conversation composed of several messages each with length nearing the limit of what is allowed. Does it not have to in some way incorporate all the input albeit in one input or multiple inputs?

chis 4y ago

I'm really curious to see if expanding the context length this much will allow GPT to do typical software development tasks on a big codebase. If it can take in a github issue and produce decent code solving a complex issue across many files... will certainly be an interesting time.

layer8 4y ago
My guess is that anything requiring nontrivial business/technical domain knowledge will be fairly safe. Also anything with a visual (or auditory) correlate, like UI work.
- oezi 4y ago
  Why would you think this? As long as the technical domain knowledge is at least partially published, I don't see them stopping becoming better. UI stuff just has an input problem. But it is not that hard to think that ChatGPT could place widgets once it can consume images and has a way to move a mouse.
  - layer8 4y ago
    > As long as the technical domain knowledge is at least partially published Most internal technical and business domain logic of companies isn’t published, though. Every time I asked ChatGPT about topics I had actually worked on over the past decade or two, or that I’m currently working on, it basically drew a blank, because it’s just not the category of topics that are discussed in detail (if at all) on the internet. At best it produced some vague generalisms. > once it can consume images and has a way to move a mouse. That’s quite far from ChatGPTs current capabilities, which is strongly tied to processing a linear sequence of tokens. We will certainly improve in that direction as we start combining it with image-processing AIs, but that will take a while.
    - cma 4y ago
      I wonder if there will be a race to buy defunct companies for access to their now valuable junky tech-debt ridden hairball code, so they can train on it and benchmark on fixing bugs and stuff. With full source control history they could also find bug resolution diffs.
      - layer8 4y ago
        That source code isn’t worth much without the underlying domain knowledge, large parts of which only exist in the employees’ heads, more often than not. Maybe if the code is really, really well documented. ;) Companies could in principle train an in-house AI with their corporate knowledge, and will likely be tempted to do so in the future. But that also creates a big risk, because whoever manages to get their hand on a copy of that model (a single file) will instantly have unrestrained access to that valuable knowledge. It will be interesting to see what mechanisms are found to mitigate that risk.
        in3d 4y ago
        The weights file could be encrypted and require a password before becoming usable.
    - dwaltrip 4y ago
      Check out the announcement. GPT-4 accepts mixed-mode inputs of text and images. Mouse cursor instructions aren’t a massive leap from the current capabilities, given the rate of progress and recent developments around LLM tool use and the like.
    - dw_arthur 4y ago
      I think what you say goes for most jobs. Why would GPT know much detail about being a machinist or luthier? Eventually job and role specific information will be fed into these models. I imagine corporations will have GPTs training on all internal communications, technical documentation, and code bases. Theoretically, this should result in a big increase in productivity.
  - barking_biscuit 4y ago
    >UI stuff just has an input problem. But it is not that hard to think that ChatGPT could place widgets once it can consume images and has a way to move a mouse. I remember one of the OpenAI guys on Lex Fridman podcast talking about how one of the early things they tried and failed at was training a model that could use websites, and he alluded to maybe giving it another go once the tech had matured a bit. I think with GPT-4 being multi-modal, it's potentially a very close to being able to do this with the right architecture wrapped around it. I can imaging an agent using LangChain and feed it a series of screenshots and maybe it feeds you back a series of co-ordinates for where the mouse should go and what action to take (i.e. click). Alternatively, updating the model itself to be able to produce those outputs directly somehow. Either way, I think that's going to happen.
- dirheist 4y ago
  Yeah, the example given in the OpenAI GPT4 twitter video is someone asking it to write a python script to analyze their monthly finances and it simply just importing dataframes, importing "finances.csv", running a columnar sum for all finances and then displaying the sum and the dataframe. I'm sure it's capable of some deeper software development but it almost always makes radical assumptions and is rarely ever self sufficient (you don't need to look it over and don't need to change the architecture of the code it produced).
graypegg 4y ago
How would you have it suggest solutions for multiple files? Has anyone gotten GPT-X to output a valid git patch or something?
- fabiospampinato 4y ago
  You just kind of concatenate the entire codebase into one file, tell the model to do something and output the modified codebase into another file, diff the two and produce a patch automatically.
  - graypegg 4y ago
    That codebase=>token stream=>codebase step feels like it could be lossy depending on how you encode things like file paths when concatenating everything, would be interesting to see in practice though!
  - dom96 4y ago
    or you might even be able to feed it individual files with their filenames, then ask it what modifications it would make as a diff for each of the files
- alexwebb2 4y ago
  Yep, I know that’s been possible since at least GPT-3 davinci
  - JyB 4y ago
    How would you go about it?
- gremlinsinc 4y ago
  I think there's ways but you might have to use pinecone db or something like lang chain to essentially give it a long term memory... or another option is having one instance or chat order code page and one that basically just has an API index and knows which chat has the related things.
2OEH8eoCRo0 4y ago
I'd love to get to a point where I can go: Add a cast button to this open source android video app. I see some FOSS-boosting silver linings in all of this.
barking_biscuit 4y ago
>If it can take in a github issue and produce decent code solving a complex issue across many files... will certainly be an interesting time. Oh snap. I didn't even think about that! That gives me a fun idea! I've got a repo that I built and setup CI/CD and setup renovate to automatically upgrade dependencies and merge them when all the tests pass, but of course sometimes there are breaking changes. I don't actively work on this thing and hence it's just got issues sitting there when upgrades fail. It's the perfect testing ground to see if I can leverage it to submit PRs to perform the fixes required for the upgrade to succeed! That'll be hectic if it works.
amelius 4y ago
It can't even do simple sysadmin tasks like fixing a broken installation, or fixing simple configure/make/make install issues.

sebastianconcpt 4y ago

And how it works? you can build a context and then ask something in a prompt using it?

teruakohatu 4y ago
Context is how many tokens it can be fed to produce an output. So now you can feed it up to 32k words (tokens).
- O__________O 4y ago
  Token is 0.75 words on average per OpenAI; 32k in tokens would be roughly 24k in words. https://help.openai.com/en/articles/4936856-what-are-tokens-and-how-to-count-them https://help.openai.com/en/articles/4936856-what-are-tokens-...

minimaxir 4y ago

$0.12 per 1k completion tokens is high enough that it makes it prohibitively expensive to use the 32k context model. Especially in a chatbot use case with cumulative prompting, which is the best use case for such a large context vs. the default cheaper 8k window. In contrast, GPT-3.5 text-davinci-003 was $0.02/1k tokens, and let's not get into the ChatGPT API.

weird-eye-issue 4y ago
I disagree that out of all possible use cases for a large context model that a chatbot is really the "best use case".
deleted 4y ago
[deleted]
ml_basics 4y ago
> Especially in a chatbot use case with cumulative prompting, which is the best use case for such a large context vs. the default cheaper 8k window. Depends on what is up with the images and how they translate into tokens. I really have no idea, but could be that 32k tokens (lots of text) translates to only a few images for few-shot prompting. The paper seems not to mention image tokenization, but I guess it should be possible to infer something about token rate when actually using the API and looking at how one is charged.
- minimaxir 4y ago
  Currently, CLIP's largest size is at patch-14 for 336x336 images, which translates to 577 ViT tokens [(336/14)^2+1]. It might end up being token-efficient depending on how it's implemented. (the paper doesn't elaborate)
sebzim4500 4y ago
I would imagine most usecases for the 32k model have much longer prompts than completions, so the $0.06 per prompt token will be the real problem. I can't think of a usecase yet, but that might be because I haven't got a sense of how smart it is.

James_Henry 4y ago

Also note that image input isn't available to the public yet. >Image inputs are still a research preview and not publicly available.

anileated 4y ago

Will any of the profits be shared with original authors whose work powers the model?

James_Henry 4y ago
Which authors? Machine Learning research authors?
- anileated 4y ago
  I mean the authors who wrote the works which are resold by Microsoft for its own profit without any opt-in or even opt-out, much less compensation.
cma 4y ago
It doesn't seem to be answered in the article, but if it was and you read it should you have to pay them a fee for the knowledge if it was published openly on the net?
- anileated 4y ago
  You are confusing two distinct cases. In the first case, you found/bought a book and read it. No one can or should make you pay for it, unless you stole the book. In the second case, you found/bought a book then reprinted it infinitely and sold it for profit, ethically you should pay the author and legally you should be in violation of the law. Even if you made a machine that ingests and recombines books automatically, and you keep that machine locked up and charge people for its use, it is the same scenario: the machine would be absolutely useless without the original books, those books cost people effort and money to produce, yet you pay those people nothing while the machine is basically an infinite money maker for you. I hope the analogy makes sense.
sebzim4500 4y ago
No. Now that you have read my answer, you owe me $0.01 because your brain might use this information in the future.

doomleika 4y ago

Poe.com(by Quora) have gpt4 now, give it a shot

daydream 4y ago
Seems limited to only one prompt per day. I just tried it.
- acallaha 4y ago
  You can pay $17/month (Apple subscription only for now), for ~unlimited queries to GPT-4 (and Claude+)
  - daydream 4y ago
    I missed that, thanks

nealabq 4y ago

> Image inputs are still a research preview and not publicly available. Will input-images also be tokenized? Multi-modal input is an area of research, but an image could be converted into a text description (?) before being inserted into the input stream.

teruakohatu 4y ago
My understanding is thta the image embedding is included, rather than converting to text.
- 2sk21 4y ago
  My understanding is that image embeddings are a rather abstract representation of the image. What about if the image itself contains text, such as street signs etc?

fzliu 4y ago

One way to get around context length is to perform embedding and retrieval of your entire corpus. Langchain (https://langchain.readthedocs.io/en/latest/ https://langchain.readthedocs.io/en/latest/) and Milvus (https://milvus.io https://milvus.io) is one of the stacks you can use.

deleted 4y ago
[deleted]
ComplexSystems 4y ago
Can you elaborate on how this works?
- teaearlgraycold 4y ago
  You run the corpus through the model piecemeal, recording the model's interpretation for each chunk as a vector of floating point numbers. Then when performing a completions request you first query the vectors and include the closest matches as context.

soheil 4y ago

I still doesn't understand how can content length not be limited if you have a conversation composed of several messages each with length nearing the limit of what is allowed. Does it not have to in some way incorporate all the input albeit in one input or multiple inputs?