3 ms·

Site seems to be down - I can’t get the article to load - but by far the most maddening part of Vertex AI is the way it deals with multimodal inputs. You can’t

by rafram 1y ago

Site seems to be down - I can’t get the article to load - but by far the most maddening part of Vertex AI is the way it deals with multimodal inputs. You can’t just attach an image to your request. You have to use their file manager to upload the file, then make sure it gets deleted once you’re done. That would all still be OK-ish except that their JS library only accepts a local path, which it then attempts to read using the Node `fs` API. Serverless? Better figure out how to shim `fs`! It would be trivial to accept standard JS buffers. But it’s not clear that anyone at Google cares enough about this crappy API to fix it.

Deathmax 1y ago
> You can’t just attach an image to your request. You can? Google limits HTTP requests to 20MB, but both the Gemini API and Vertex AI API support embedded base64-encoded files and public URLs. The Gemini API supports attaching files that are uploaded to their Files API, and the Vertex AI API supports files uploaded to Google Cloud Storage.
- rafram 1y ago
  Their JavaScript library didn’t support that as of whenever I tried.
  - simonw 1y ago
    I got their most recent JavaScript API library to work for images here: https://tools.simonwillison.net/gemini-mask https://tools.simonwillison.net/gemini-mask Here's the code: https://github.com/simonw/tools/blob/main/gemini-mask.html https://github.com/simonw/tools/blob/main/gemini-mask.html
mofunnyman 1y ago
Semi hugged.
chrisheecho 1y ago
That’s correct! You can send images through uploading either the Files API from Gemini API or Google Cloud Storage (GCS) bucket reference. What we DON’T have a sample on is sending images through bytes. Here is a screenshot of the code sample from the “Get Code” function in the Vertex AI studio. https://drive.google.com/file/d/1rQRyS4ztJmVgL2ZW35NXY0TW-S0E-p3x/view https://drive.google.com/file/d/1rQRyS4ztJmVgL2ZW35NXY0TW-S0... Let me create a feature request to get these samples in our docs because I could not find a sample too. Fixing it

by rafram 1y ago

Deathmax 1y ago

> You can’t just attach an image to your request. You can? Google limits HTTP requests to 20MB, but both the Gemini API and Vertex AI API support embedded base64-encoded files and public URLs. The Gemini API supports attaching files that are uploaded to their Files API, and the Vertex AI API supports files uploaded to Google Cloud Storage.

rafram 1y ago
Their JavaScript library didn’t support that as of whenever I tried.
- simonw 1y ago
  I got their most recent JavaScript API library to work for images here: https://tools.simonwillison.net/gemini-mask https://tools.simonwillison.net/gemini-mask Here's the code: https://github.com/simonw/tools/blob/main/gemini-mask.html https://github.com/simonw/tools/blob/main/gemini-mask.html

mofunnyman 1y ago

Semi hugged.

chrisheecho 1y ago

That’s correct! You can send images through uploading either the Files API from Gemini API or Google Cloud Storage (GCS) bucket reference. What we DON’T have a sample on is sending images through bytes. Here is a screenshot of the code sample from the “Get Code” function in the Vertex AI studio. https://drive.google.com/file/d/1rQRyS4ztJmVgL2ZW35NXY0TW-S0E-p3x/view https://drive.google.com/file/d/1rQRyS4ztJmVgL2ZW35NXY0TW-S0... Let me create a feature request to get these samples in our docs because I could not find a sample too. Fixing it