4 ms·

Gemini models on Vertex AI can be called via a preview OpenAI-compatible endpoint [1], but shoving it into existing tooling where you don't have programmatic co

Gemini models on Vertex AI can be called via a preview OpenAI-compatible endpoint [1], but shoving it into existing tooling where you don't have programmatic control over the API key and is long lived is non-trivial because GCP uses short lived access tokens (and long-lived ones are not great security-wise). Billing for the Gemini models (on Vertex AI, the Generative Language AI variant still charges by tokens) I would argue is simpler than every other provider, simply because you're charged by characters/image/video-second/audio-second and don't need to run a tokenizer (if it's even available cough Claude 3 and Gemini) and having to figure out what the chat template is to calculate the token cost per message [2] or figure out how to calculate tokens for an image [3] to get cost estimates before actually submitting the request and getting usage info back. [1]: https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/call-gemini-using-openai-library https://cloud.google.com/vertex-ai/generative-ai/docs/multim... [2]: https://platform.openai.com/docs/guides/text-generation/managing-tokens https://platform.openai.com/docs/guides/text-generation/mana... [3]: https://platform.openai.com/docs/guides/vision/calculating-costs https://platform.openai.com/docs/guides/vision/calculating-c...

luke-stanley 2y ago
Good to know about this API preview. Hopefully the billing problem and UI maze of Vertex AI can be sorted too?
- Flumio 2y ago
  Google does plenty of ux studies on gcp. I took part in at least 3 of them. I'm also not sure if I understand your problem with pricing? Depending on what you do with it, it's not just an LLM. It actually started before llms. Pricing for image classification and other features are completely different products like an LLM.
  - luke-stanley 2y ago
    They should do a whole lot more then! Ideally they'd have effective impact. It's a busy mess on GCP. If they wanted to compete well, they should do much better with UX design, especially for onboarding. Compare how easy setting up a Mistral account is with GCP to do some generative LLM in a Python script. GCP is a maze. Did you make an account to reply to this? I'm curious what you do with GCP? Are you a heavy user?
    - Flumio 2y ago
      I create new accounts because I use hn too much. I use gcp professional every day and always found it quite intuitive. Did plenty of image classification with vertex ai too
      - luke-stanley 2y ago
        Why would you make new accounts because you use HN too much? Doesn't make sense to me. Anyhow if you use GCP every day, you're going to have learned it's weird clunky behaviour. GCP's main problem is that they've steadily become a sprawling mess of complexity, which is in big contrast to quite a few LLM specific cloud services that are happy to take peoples money without extra complexity?
        Flumio 2y ago
        Not being logged in feels like a bigger hurdle to comment and check if someone responded to it. It's a shitty solution to a stupid problem ;) But I did mention that vertex AI is more than just hosting llms though

luke-stanley 2y ago

Good to know about this API preview. Hopefully the billing problem and UI maze of Vertex AI can be sorted too?

Flumio 2y ago
Google does plenty of ux studies on gcp. I took part in at least 3 of them. I'm also not sure if I understand your problem with pricing? Depending on what you do with it, it's not just an LLM. It actually started before llms. Pricing for image classification and other features are completely different products like an LLM.
- luke-stanley 2y ago
  They should do a whole lot more then! Ideally they'd have effective impact. It's a busy mess on GCP. If they wanted to compete well, they should do much better with UX design, especially for onboarding. Compare how easy setting up a Mistral account is with GCP to do some generative LLM in a Python script. GCP is a maze. Did you make an account to reply to this? I'm curious what you do with GCP? Are you a heavy user?
  - Flumio 2y ago
    I create new accounts because I use hn too much. I use gcp professional every day and always found it quite intuitive. Did plenty of image classification with vertex ai too
    - luke-stanley 2y ago
      Why would you make new accounts because you use HN too much? Doesn't make sense to me. Anyhow if you use GCP every day, you're going to have learned it's weird clunky behaviour. GCP's main problem is that they've steadily become a sprawling mess of complexity, which is in big contrast to quite a few LLM specific cloud services that are happy to take peoples money without extra complexity?
      - Flumio 2y ago
        Not being logged in feels like a bigger hurdle to comment and check if someone responded to it. It's a shitty solution to a stupid problem ;) But I did mention that vertex AI is more than just hosting llms though