3 ms·

Google's docs on the APIs are up: https://cloud.google.com/vertex-ai/docs/generative-ai/learn/overview https://cloud.google.com/vertex-ai/docs/generative-ai/lea

Google's docs on the APIs are up: https://cloud.google.com/vertex-ai/docs/generative-ai/learn/overview https://cloud.google.com/vertex-ai/docs/generative-ai/learn/... The pricing is also now listed but free during the trial period, although it's annoyingly priced by character: https://cloud.google.com/vertex-ai/pricing#generative_ai_models https://cloud.google.com/vertex-ai/pricing#generative_ai_mod... Assuming ChatGPT's tokens are the equivalent of 4 characters on average (a fair assumption), the pricing of PaLM's chat and embedding APIs are the same cost as OpenAI's equivalents.

ntonozzi 4y ago
Why would that be annoying? It’s much easier to understand, predict and truncate appropriately than having to explain all of these different tokenization schemes to devs.
- rcoveson 4y ago
  Yeah, everybody agrees on what a character is, right? It's just {an ASCII byte|a UTF8 code unit|a UTF16 code unit|a Unicode code point|a Unicode grapheme}.
  - ntonozzi 4y ago
    I’m not saying it’s easy but it’s much better than tokens IMO. I think bytes would be understandable too.
    - criddell 4y ago
      Bytes are understandable but make no sense from a business point of view. If you submit the same simple query with UTF-8 and UTF-32, the latter will cost 4x as much.
      - xyzzyz 4y ago
        No API accepts input in UTF-32. Nobody uses this on the internet.
  - geysersam 4y ago
    At least there are standards for characters. Nothing like that for tokens.
  - sheepscreek 4y ago
    And we think tokens solve that problem? Spoiler alert: they don’t https://www.reddit.com/r/OpenAI/comments/124v2oi/hindi_8_times_more_expensive_than_english_the/ https://www.reddit.com/r/OpenAI/comments/124v2oi/hindi_8_tim...
    - est31 4y ago
      They don't but Google could have been more precise with which of the definitions listed by GP they mean by "character".
jxy 4y ago
There is a limit of maxOutputTokens, 1024! Is this the true capability of PaLM 2? However I couldn't find anything about the context length of their model anywhere. And the API didn't tell me how long the prompt could be.
- sanxiyn 4y ago
  No. Autoregressive models don't have model specific limit to output tokens, it's just when to stop looping.
  - jxy 4y ago
    I guess I only know transformers and how BERT or GPT works, as there would be a limit in the context length. With GPT, you can certainly generate infinite amount of tokens but the previous tokens outside of the maximum context length would be outside of the context window. LLaMa has 2k, GPT-4 has 32k. Are you saying I can give unlimited tokens to PaLM and generate unlimited amount of tokens? So PaLM doesn't have a context limit?
    - sanxiyn 4y ago
      No, I am not saying that. Since PaLM 2 is a transformer model (they didn't disclose almost anything about the model architecture, but they did disclose that), it has a context length limit. What I am saying is that you can't infer that limit from the limit of maxOutputTokens parameter in the API.
danpalmer 4y ago
Per token might be 4 characters on average, but that can vary wildly. Pricing per character is easier to understand and means more flexibility to change tokenisation without affecting pricing. So far OpenAI has charged very different prices per model, but I expect we’ll see more granular changes in the future that might not change pricing… except for changing the tokenisation.
fomine3 4y ago
It's happy pricing as a Japanese user, maybe also for Korean and Chinese.

ntonozzi 4y ago

Why would that be annoying? It’s much easier to understand, predict and truncate appropriately than having to explain all of these different tokenization schemes to devs.

rcoveson 4y ago
Yeah, everybody agrees on what a character is, right? It's just {an ASCII byte|a UTF8 code unit|a UTF16 code unit|a Unicode code point|a Unicode grapheme}.
- ntonozzi 4y ago
  I’m not saying it’s easy but it’s much better than tokens IMO. I think bytes would be understandable too.
  - criddell 4y ago
    Bytes are understandable but make no sense from a business point of view. If you submit the same simple query with UTF-8 and UTF-32, the latter will cost 4x as much.
    - xyzzyz 4y ago
      No API accepts input in UTF-32. Nobody uses this on the internet.
- geysersam 4y ago
  At least there are standards for characters. Nothing like that for tokens.
- sheepscreek 4y ago
  And we think tokens solve that problem? Spoiler alert: they don’t https://www.reddit.com/r/OpenAI/comments/124v2oi/hindi_8_times_more_expensive_than_english_the/ https://www.reddit.com/r/OpenAI/comments/124v2oi/hindi_8_tim...
  - est31 4y ago
    They don't but Google could have been more precise with which of the definitions listed by GP they mean by "character".

jxy 4y ago

There is a limit of maxOutputTokens, 1024! Is this the true capability of PaLM 2? However I couldn't find anything about the context length of their model anywhere. And the API didn't tell me how long the prompt could be.

sanxiyn 4y ago
No. Autoregressive models don't have model specific limit to output tokens, it's just when to stop looping.
- jxy 4y ago
  I guess I only know transformers and how BERT or GPT works, as there would be a limit in the context length. With GPT, you can certainly generate infinite amount of tokens but the previous tokens outside of the maximum context length would be outside of the context window. LLaMa has 2k, GPT-4 has 32k. Are you saying I can give unlimited tokens to PaLM and generate unlimited amount of tokens? So PaLM doesn't have a context limit?
  - sanxiyn 4y ago
    No, I am not saying that. Since PaLM 2 is a transformer model (they didn't disclose almost anything about the model architecture, but they did disclose that), it has a context length limit. What I am saying is that you can't infer that limit from the limit of maxOutputTokens parameter in the API.

danpalmer 4y ago

Per token might be 4 characters on average, but that can vary wildly. Pricing per character is easier to understand and means more flexibility to change tokenisation without affecting pricing. So far OpenAI has charged very different prices per model, but I expect we’ll see more granular changes in the future that might not change pricing… except for changing the tokenisation.

fomine3 4y ago

It's happy pricing as a Japanese user, maybe also for Korean and Chinese.