6 ms·
I still don't really understand what Vertex AI is. If you can ignore Vertex most of the complaints here are solved - the non-Vertex APIs have easy to use API k
by simonw 2y ago
I still don't really understand what Vertex AI is.
If you can ignore Vertex most of the complaints here are solved - the non-Vertex APIs have easy to use API keys, a great debugging tool (https://aistudio.google.com https://aistudio.google.com), a well documented HTTP API and good client libraries too.
I actually use their HTTP API directly (with the ijson streaming JSON parser for Python) and the code is reasonably straight-forward: https://github.com/simonw/llm-gemini/blob/61a97766ff0873936ad80b30e83d12439736d126/llm_gemini.py#L384 https://github.com/simonw/llm-gemini/blob/61a97766ff0873936a...
You have to be very careful when searching (using Google, haha) that you don't accidentally end up in the Vertext documentation though.
Worth noting that Gemini does now have an OpenAI-compatible API endpoint which makes it very easy to switch apps that use an OpenAI client library over to backing against Gemini instead: https://ai.google.dev/gemini-api/docs/openai https://ai.google.dev/gemini-api/docs/openai
Anthropic have the same feature now as well: https://docs.anthropic.com/en/api/openai-sdk https://docs.anthropic.com/en/api/openai-sdk
- mgraczyk 2y agoOpenAI compatible API is missing important parameters, for example I don't think there is a way to disable flash 2 thinking with it. Vertex AI is for grpc, service auth, and region control (amongst other things). Ensuring data remains in a specific region, allowing you to auth with the instance service account, and slightly better latency and ttft
- minimaxir 2y agoFrom the linked docs: > If you want to disable thinking, you can set the reasoning effort to "none". For other APIs, you can set the thinking tokens to 0 and that also works.
- mgraczyk 2y agoWow thanks I did not know
- logankilpatrick 2y agoWe added it to the docs. The downside of the OAI compat endpoint is we have to design the API twice, once for our API, then once through the OAI compat layer which makes it slower sometimes to have certain features, especially if we diverge at all.
- mgraczyk 2y agoThanks, yes makes sense. BTW, I have noticed that when tested outside GCP, the OpenAI compat endpoint has significantly lower latency for most requests (vs using the genai library). VertexAI is better than both. Any idea why or if that will change?
- simonw 2y agoI find Google's service auth SO hard to figure out. I've been meaning to solve deploying to Cloud Run via service with for several years now but it just doesn't fit in my brain well enough for me to make the switch.
- mgraczyk 2y agoIf you're on cloud run it should just work automatically. For deploying, on GitHub I just use a special service account for CI/CD and put the json payload in an environment secret like an API key. The only extra thing is that you need to copy it to the filesystem for some things to work, usually a file named google_application_credentials.json If you use cloud build you shouldn't need to do anything
- candiddevmike 2y agoYou should consider setting up Workload Identity Federation and authentication to Google Cloud using your GitHub runner OIDC token. Google Cloud will "trust" the token and allow you to impersonate service accounts. No static keys!
- mgraczyk 2y agoDoes not work for many Google services, including firebase
- progbits 2y agoYes it does. We deploy firebase and bunch of other GCP things from github actions and there are zero API keys or JSON credentials anywhere. Everything is service accounts and workload identity federation, with restrictions such as only letting main branch in specific repo to use it (so no problem with unreviewed PRs getting production access). Edit: if you have a specific error or issue where this doesn't work for you, and can share the code, I can have a look.
- 2y ago
- omneity 2y agoJSONSchema support on Google's OpenAI-compatible API is very lackluster and limiting. My biggest gripe really.
- shresbm123 2y agoyeah we are looking into it
- omneity 2y agoThank you! Adding support for `additionalProperties`[0] (and perhaps `patternProperties` too) would be particularly great! Happy to provide test cases as well if helpful. 0: https://datatracker.ietf.org/doc/html/draft-fge-json-schema-validation-00#page-13 https://datatracker.ietf.org/doc/html/draft-fge-json-schema-...
- Aeolun 2y agoWhen I used the openai compatible stuff my API’s just didn’t work at all. I switched back to direct HTTP calls, which seems to be the only thing that works…
- chrisheecho 2y agoWe built the OpenAI Compatible API (https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/call-vertex-using-openai-library https://cloud.google.com/vertex-ai/generative-ai/docs/multim...) layer to help customers that are already using OAI library to test out Gemini easily with basic inference but not as a replacement library for the genai sdk (https://github.com/googleapis/python-genai https://github.com/googleapis/python-genai). We recommend using th genai SDK for working with Gemini.
- mike_hearn 2y agoSo, to be clear, Google only supports Python as a language for accessing your models? Nothing else?
- chrisheecho 2y agoWe have Python/Go in GA. Java/JS is in preview (not ready for production) and will be GA soon!
- troupo 2y agoWhat about providing an actual API people can call without needing to rely on Google SDKs?
- logankilpatrick 2y agoyou can do so with the AI SDK from Vercel, open router, etc or just sending raw http requests
- franze 2y agoyeah, 2 days to get Google OAuth flow integrated into an background app/script, 1 day coding for the actual app ...
- arccy 2y agoshould have used ai to write the integrations...
- franze 2y agothats with AI as there are so many variations out there the AI gets majorly confused, as a matter of fact, the google oauth part is the one thing that gemini 2.5 pro cant code should be its own benchmark
- enneff 2y agoMaybe you should just read the docs and use the examples there. I have used all kinds of GCP services for many years and auth is not remotely complicated imo.
- jpc0 2y agoIs this vertexAI related or in general, I find googles oauth flow to be extremely well documented and easy to setup…
- jacob019 2y agoI got claude to write me an auth layer using only python http.client and cryptography. One shot no problem, now I can get a token from the service key any time, just have to track expiration. Annoying that they don't follow industry standard though.
- shresbm123 2y agoWe support reasoning_effort = none. That will let you disable flash 2 thinking. We will document it better.
- logankilpatrick 2y agoThis is documented for AI Studio here: https://ai.google.dev/gemini-api/docs/openai#thinking https://ai.google.dev/gemini-api/docs/openai#thinking
- minimaxir 2y agoVertex AI is essentially a rebranding of their more enterprise platform on GCP, nothing explicitly "new."
- unknown_user_84 2y agoIndeed. Though the billing dashboard feels like an over engineered April fool's joke compared to Anthropic or OpenAI. And it takes too long to update with usage. I understand they tacked it into GCP, but if they're making those devs work 60 hours a week can we get a nicer, and real time, dashboard out of it at least?
- coredog64 2y agoWait until you see how to check Bedrock usage in AWS. (While you can certainly try to use CloudWatch, it’s not exact. Your other options are “Wait for the bill” or log all Bedrock invocations to CloudWatch/S3 and aggregate there)
- logankilpatrick 2y agowe will have a dashboard in AI Studio very soon! Then will work to drive down delay.
- laborcontract 2y agoGoogle Cloud Console's billing console for Vertex is so poor. I'm trying to figure out how much i spent on which models and I still cannot for the life of me figure it out. I'm assuming the only way to do it is to use the gemini billing assistant chatbot, but that requires me to turn on another api permission. I still don't understand the distinction between Gemini and Vertex AI apis. It's like Logan K heard the criticisms about the API and helped push to split Gemini from the broader Google API ecosystem but it's only created more confusion, for me at least.
- tyre 2y agoGemini’s is no better. Their data can be up to 24h stale and you can’t set hard caps on API keys. The best you can do is email notification billing alerts, which they acknowledge can be hours late.
- chrisheecho 2y agoI couldn’t have said it better. My billing friends are working to address some of these concerns along with the Vertex team. We are planning to address this issue. Please stay tuned, we will come back to this thread to announce when we can In fact, if you can DM me (@chrischo_pm on X) with, I would love to learn more if you are interested.
- jeswin 2y agoCan you allow prepaid credits as well please?
- byefruit 2y ago100% this. We actually use OpenRouter (and pay their surcharge) with Gemini 2.5 Pro just because we can actually control spend via spent limit on keys (A++ feature) and prepaid credit.
- chrisheecho 2y agoone step ahead of you ;)
- tzury 2y agoVertex by example: creds = service_account.Credentials.from_service_account_file( SA_FILE, scopes=[ "https://www.googleapis.com/auth/cloud-platform", "https://www.googleapis.com/auth/generative-language", ] ) google.genai.Client( vertexai=True, project=PROJECT_ID, location=LOCATION, http_options={"api_version": "v1beta1"}, credentials=sa_creds, ) That `vertexai=True` does the trick - you can use same code without this option, and you will not be using "Vertex". Also, note, with Vertex, I am providing service account rather than API key, which should improve security and performance. For me, the main aspect of "using Vertex", as in this example is the fact Start AI Cloud Credit ($350K) are only useable under Vertex. That is, one must use this platform to benefit from this generous credit. Feels like the "Anthos" days for me, when Google now pushing their Enterprise Grade ML Ops platform, but all in all I am grateful for their generosity and the great Gemini model.
- sitefail1 2y agoI don't think a service account vs an API key would improve performance in any meaningful way. I doubt the AI endpoint is authenticating the API key against a central database every request, it will most certainly be cached against a service key in the same AZ or whatever GCP call it.
- ivanvanderbyl 2y agoService account file vs API Key have similar security risks if provided the way you are using them. Google recommends using ADC and it’s actually an org policy recommendation to disable SA files.
- wanderer2323 2y agoADC (Application Default Credentials) is a specification for finding credentials (1. look here 2. look there etc.) not an alternative for credentials. Using ADC one can e.g. find an SA file. As a replacement for SA files one can have e.g. user accounts using SA impersonation, external identity providers, or run on GCP VM or GKE and use built-in identities. (ref: https://cloud.google.com/iam/docs/migrate-from-service-account-keys https://cloud.google.com/iam/docs/migrate-from-service-accou...)
- KTibow 2y agoVertex is the enterprise platform. It also happens to have much higher rate limits, even for free models.
- chrisheecho 2y agosimonw, good points. The Vertex vs. non-Vertex Gemini API (via AI Studio at aistudio.google.com) could use more clarity. For folks just wanting to get started quickly with Gemini models without the broader platform capabilities of Google Cloud, AI Studio and its associated APIs are recommended as you noted. However, if you anticipate your use case to grow and scale 10-1000x in production, Vertex would be a worthwhile investment.
- troupo 2y agoWhy create two different APIs that are the same, but only subtly different, and have several different SDKs?
- chrisheecho 2y agoI think you are talking about generativeai vs. vertexai vs. genai sdk. And you are watching us evolve overtime to do better. Couple clarifications 1. Going forward we only recommend using genai SDK 2. Subtle API differences - this is a bit harder to articulate but we are working to improve this. Please dm at @chrischo_pm if you would like to discuss further :)
- troupo 2y agoSo. Three different SDKs. No idea what any of those SDK names mean. But sure enoough searching will bring up all three of them for different combination of search terms, and none of them will point to the "recommend only using <a random name that is indistinguishable form other names>" Oh, And some of these SDKs (and docs) do have a way to use this functionality without the SDKs, but not others. Because there are only 4 languages in the world, and everyone should be happy using them.
- mark_l_watson 2y agoI think you can strongly influence which SDK your customers use by keeping the Python, Typescript, and Curl examples in the documentation up to date and uniformly use what you consider the ‘best’ SDK in the examples. Overall, I think that Google has done a great job recently in productizing access to your models. For a few years I wrote my own utilities to get stuff done, now I do much less coding using Gemini (and less often ChatGPT) because the product offerings do mostly what I want. One thing I would like to see Google offer is easier integrated search with LLM generation. The ‘grounding’ examples are OK, but for use in Python I buy a few Perplexity API credits and use that for now. That is the single thing I would most like to see you roll out. EDIT: just looked at your latest doc pages, I like the express mode setup with a unified access to regular APIs vs. Vertex.
- egamirorrim 2y agoI use Vertex because that's the one that makes enterprise security people happy about how our datas handled. Do Google use all the AI studio traffic to train etc?
- sunaookami 2y agoNot if you have billing enabled: https://ai.google.dev/gemini-api/docs/pricing https://ai.google.dev/gemini-api/docs/pricing
- kmod 2y agoThere are a few conditions that take precedence over having-billing-enabled and will cause AI Studio to train on your data. This is why I personally use Vertex
- logankilpatrick 2y agoThis is correct, "When you activate a Cloud Billing account, all use of Gemini API and Google AI Studio is a "Paid Service" with respect to how Google Uses Your Data, even when using Services that are offered free of charge, such as Google AI Studio and unpaid quota of Gemini API."
- anaisbetts 2y agoIt's a way for you to have your AI billing under the same invoice as all of your other cloud purchases. If you're a startup this is a dumb feature, if you work at a $ENTERPRISE_BIGCO, it just saved you 6mo+ of fighting with IT / Legal / various annoying middle managers
- progbits 2y agoIt's also useful in a startup, I just start using it with zero effort. For external service I have to get a unique card for billing and then upload monthly receipts, or ask our ops to get it setup and then wait for weeks as the sales/legal/compliance teams on each side talk to each other.
- arresin 2y agoah! thank you. I was also struggling with where vertex fitted.
- NoahZuniga 2y agoThis is not true??? The AI studio surface is also billed on a per project basis?
- blitzar 2y ago> $ENTERPRISE_BIGCO, it just saved you 6mo+ of fighting with IT / Legal / various annoying middle managers What's the point of working at $ENTERPRISE_BIGCO if you don't fight with IT & Legal & various annoying middle managers. Anyway let's table this for now and circle back later after we take care of some of the low hanging fruit. Keep me in the loop and I will do a deep dive into how we can think outside the box and turn this into a win-win. I will touch base with you when I have all my ducks in a row and we can hop on a call.
- kridsdale1 2y agoI work AT Google and 99% of my conversations must have been the training set for your paragraph.
- 2y ago
- jacob019 2y agoExcept that the OpenAI compatible endpoint isn't actually compatible. Doesn't support string enum values for function calls and throws a confusing error. Vertex at least has better error messages. My solution, just use text completions and emulate the tool call support client side, validate the responses against the schema, and retry on failure. It rarely has to retry and always works the 2nd time even without feedback.
- ashu1461 2y agoThere is also no way to over-write content moderation settings, and half of the responses you generate via open ai endpoint end up being moderated.
- __jl__ 2y agoOnly problem is that the genai API at https://ai.google.dev https://ai.google.dev is far less reliable and can be problematic for production use cases. Right around the time Gemini 2.0 launched, it was done for days on end without any communication. They are putting a lot of effort into improving it but it's much less reliable than openai, which matters for production. They can also reject your request based on overall system load (not your individual limits), which is very unpredictable. They advertise 2000 requests per minute. When I tried several weeks ago, I couldn't even get 500 per minute.
- logankilpatrick 2y agoPls ping me if you run into any production issues, will raise right away to the team. We have massive at scale products operating on AI Studio, so we are set up to ensure stability.
- fzysingularity 2y agoVertex AI is essentially equivalent to Azure OpenAI - enterprise-ready, with HIPAA/SOC2 compliance and data-privacy guarantees. FWIW OpenAI compatibility only gets you so far with Gemini. Gemini’s video/audio capabilities and context caching are unparalleled and you’ll likely need to use their SDKs instead to fully take advantage of them.
- ashu1461 2y agoHave to work hard to figure out the difference between - Vertex AI - AI Studio - Gemini - Firebase Gen AI
- hustwindmaple1 2y agoIf you are not a paying GCP user, there is really no point to even look at Vertex AI. Just stick with AI Studio and the free developer AI along with it; you will be much much happier.