4 ms·
OpenAI compatible API is missing important parameters, for example I don't think there is a way to disable flash 2 thinking with it. Vertex AI is for grpc, ser
by mgraczyk 1y ago
OpenAI compatible API is missing important parameters, for example I don't think there is a way to disable flash 2 thinking with it.
Vertex AI is for grpc, service auth, and region control (amongst other things). Ensuring data remains in a specific region, allowing you to auth with the instance service account, and slightly better latency and ttft
- minimaxir 1y agoFrom the linked docs: > If you want to disable thinking, you can set the reasoning effort to "none". For other APIs, you can set the thinking tokens to 0 and that also works.
- mgraczyk 1y agoWow thanks I did not know
- logankilpatrick 1y agoWe added it to the docs. The downside of the OAI compat endpoint is we have to design the API twice, once for our API, then once through the OAI compat layer which makes it slower sometimes to have certain features, especially if we diverge at all.
- mgraczyk 1y agoThanks, yes makes sense. BTW, I have noticed that when tested outside GCP, the OpenAI compat endpoint has significantly lower latency for most requests (vs using the genai library). VertexAI is better than both. Any idea why or if that will change?
- simonw 1y agoI find Google's service auth SO hard to figure out. I've been meaning to solve deploying to Cloud Run via service with for several years now but it just doesn't fit in my brain well enough for me to make the switch.
- mgraczyk 1y agoIf you're on cloud run it should just work automatically. For deploying, on GitHub I just use a special service account for CI/CD and put the json payload in an environment secret like an API key. The only extra thing is that you need to copy it to the filesystem for some things to work, usually a file named google_application_credentials.json If you use cloud build you shouldn't need to do anything
- candiddevmike 1y agoYou should consider setting up Workload Identity Federation and authentication to Google Cloud using your GitHub runner OIDC token. Google Cloud will "trust" the token and allow you to impersonate service accounts. No static keys!
- mgraczyk 1y agoDoes not work for many Google services, including firebase
- progbits 1y agoYes it does. We deploy firebase and bunch of other GCP things from github actions and there are zero API keys or JSON credentials anywhere. Everything is service accounts and workload identity federation, with restrictions such as only letting main branch in specific repo to use it (so no problem with unreviewed PRs getting production access). Edit: if you have a specific error or issue where this doesn't work for you, and can share the code, I can have a look.
- mgraczyk 1y agoNo thank you, there is zero benefit to migrating and no risk in using credentials the way I do. How do you sign a firebase custom auth token with workload identity federation? How about a pre signed storage URL? Off the top of my head I think those were two things that don't work
- omneity 1y agoJSONSchema support on Google's OpenAI-compatible API is very lackluster and limiting. My biggest gripe really.
- shresbm123 1y agoyeah we are looking into it
- omneity 1y agoThank you! Adding support for `additionalProperties`[0] (and perhaps `patternProperties` too) would be particularly great! Happy to provide test cases as well if helpful. 0: https://datatracker.ietf.org/doc/html/draft-fge-json-schema-validation-00#page-13 https://datatracker.ietf.org/doc/html/draft-fge-json-schema-...
- Aeolun 1y agoWhen I used the openai compatible stuff my API’s just didn’t work at all. I switched back to direct HTTP calls, which seems to be the only thing that works…
- chrisheecho 1y agoWe built the OpenAI Compatible API (https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/call-vertex-using-openai-library https://cloud.google.com/vertex-ai/generative-ai/docs/multim...) layer to help customers that are already using OAI library to test out Gemini easily with basic inference but not as a replacement library for the genai sdk (https://github.com/googleapis/python-genai https://github.com/googleapis/python-genai). We recommend using th genai SDK for working with Gemini.
- mike_hearn 1y agoSo, to be clear, Google only supports Python as a language for accessing your models? Nothing else?
- chrisheecho 1y agoWe have Python/Go in GA. Java/JS is in preview (not ready for production) and will be GA soon!
- troupo 1y agoWhat about providing an actual API people can call without needing to rely on Google SDKs?
- logankilpatrick 1y agoyou can do so with the AI SDK from Vercel, open router, etc or just sending raw http requests
- franze 1y agoyeah, 2 days to get Google OAuth flow integrated into an background app/script, 1 day coding for the actual app ...
- arccy 1y agoshould have used ai to write the integrations...
- franze 1y agothats with AI as there are so many variations out there the AI gets majorly confused, as a matter of fact, the google oauth part is the one thing that gemini 2.5 pro cant code should be its own benchmark
- enneff 1y agoMaybe you should just read the docs and use the examples there. I have used all kinds of GCP services for many years and auth is not remotely complicated imo.
- jpc0 1y agoIs this vertexAI related or in general, I find googles oauth flow to be extremely well documented and easy to setup…
- jacob019 1y agoI got claude to write me an auth layer using only python http.client and cryptography. One shot no problem, now I can get a token from the service key any time, just have to track expiration. Annoying that they don't follow industry standard though.
- shresbm123 1y agoWe support reasoning_effort = none. That will let you disable flash 2 thinking. We will document it better.
- logankilpatrick 1y agoThis is documented for AI Studio here: https://ai.google.dev/gemini-api/docs/openai#thinking https://ai.google.dev/gemini-api/docs/openai#thinking