4 ms·

I could have used this just yesterday! I've been evaluating Gemini Embedding 2 using Hacker News comments and I wasted half a day making a wrapper for the HN A

I could have used this just yesterday! I've been evaluating Gemini Embedding 2 using Hacker News comments and I wasted half a day making a wrapper for the HN API to collect some sample data to play with. In case anyone is curious: - The ability to simply truncate the provided embedding to a prefix (and then renormalize) is useful because it lets users re-use the same (paid!) embedding API response for multiple indexes at different qualities. - Traditional enterprise software vendors are struggling to keep up with the pace of AI development. Microsoft SQL Server for example can't store a 3072 element vector with 32-bit floats (because that would be 12 KB and the page size is only 8 KB). It supports bfloat16 but... the SQL client doesn't! Or Entity Framework. Or anything else. - Holy cow everything is so slow compared to full text search! The model is deployed in only one US region, so from Australia the turnaround time is something like 900 milliseconds. Then the vector search over just a few thousand entries with DiskANN is another 600-800 ms! I guess search-as-you-type is out of the question for... a while. - Speaking of slow, the first thing I had to do was write an asynchronous parallel bounded queue data processor utility class in C# that supports chunking of the input and rate limit retries. This feels like it ought to be baked into the standard library or at least the AI SDKs because it's pretty much mandatory if working with anything other than "hello world" scenarios. - Gemini Embedding 2 has the headline feature of multi-modal input, but they forgot to implement anything other than "string" for their IEmbeddingGenerator abstraction when used with Microsoft libraries. I guess the next "Preview v0.0.3-alpha" version or whatever will include it.

pjot 7mo ago
I did this but used duckdb as the vector store. Works really well, quite fast too. https://github.com/patricktrainer/duckdb-embedding-search https://github.com/patricktrainer/duckdb-embedding-search
- jiggawatts 7mo ago
  Unless I'm missing something, this uses a simple synchronous for loop: for text in texts: key = (text, model) if key not in pickle_cache: pickle_cache[key] = openai_client.create_embedding(text, model=model) embeddings.append(pickle_cache[key]) operations.save_pickle_cache(pickle_cache, pickle_path) return embeddings At the throughput rates I was seeing of one embedding per second, a million comments would take over a week to process! I had to call the Gemini model with ten comments at a time from eight threads to reach even the paltry 3K rpm rate limit they offer to "Tier 1" customers. Based on this experience, for real "enterprise" customers I might implement a generic wrapper for Google's Batch API that could handle continuous streaming from a database, chunking it, uploading, and then in parallel checking the status of the pending jobs and streaming the results back into a database.
  - pjot 7mo ago
    Re-reading your comment :) Yes, my demo has just a simple loop when loading the embeddings. I was replying more towards the latency you mentioned. Because duckdb runs on device, you save yourself the additional round trip network time when comparing similarities.
    - jiggawatts 7mo ago
      I was running SQL Server 2025 on my laptop. The source of latency is calling the Google Gemini API to compute the embedding of the query text. I was hoping to make a demo that searches as you type, but the two second delay makes it more annoying than useful. Looking at your sample you may be only grouping or categorising based on similarity between comments. I was experimenting with a question -> answer tool for RAG applications.
  - vienneraphael 7mo ago
    Hey, idk if that helps but I developed something similar to the wrapper you're mentioning as an open-source python library. Just plug any async function into the provided async context manager and you get Batch APIs in two lines of code with any existing framework you currently have: https://github.com/vienneraphael/batchling https://github.com/vienneraphael/batchling Let me know if you have any questions, looking forward to having your feedback!
    - jiggawatts 7mo ago
      Looks very nice! This is exactly what I was thinking of doing, except that I work mostly with C# in enterprise settings. Looking at your approach, the equivalent in .NET land would be if the Microsoft.AI.Extensions package added some sort of batch abstraction side-by-side (or on top of) their existing IChatClient or IEmbeddingGenerator interfaces.

pjot 7mo ago

I did this but used duckdb as the vector store. Works really well, quite fast too. https://github.com/patricktrainer/duckdb-embedding-search https://github.com/patricktrainer/duckdb-embedding-search

jiggawatts 7mo ago
Unless I'm missing something, this uses a simple synchronous for loop: for text in texts: key = (text, model) if key not in pickle_cache: pickle_cache[key] = openai_client.create_embedding(text, model=model) embeddings.append(pickle_cache[key]) operations.save_pickle_cache(pickle_cache, pickle_path) return embeddings At the throughput rates I was seeing of one embedding per second, a million comments would take over a week to process! I had to call the Gemini model with ten comments at a time from eight threads to reach even the paltry 3K rpm rate limit they offer to "Tier 1" customers. Based on this experience, for real "enterprise" customers I might implement a generic wrapper for Google's Batch API that could handle continuous streaming from a database, chunking it, uploading, and then in parallel checking the status of the pending jobs and streaming the results back into a database.
- pjot 7mo ago
  Re-reading your comment :) Yes, my demo has just a simple loop when loading the embeddings. I was replying more towards the latency you mentioned. Because duckdb runs on device, you save yourself the additional round trip network time when comparing similarities.
  - jiggawatts 7mo ago
    I was running SQL Server 2025 on my laptop. The source of latency is calling the Google Gemini API to compute the embedding of the query text. I was hoping to make a demo that searches as you type, but the two second delay makes it more annoying than useful. Looking at your sample you may be only grouping or categorising based on similarity between comments. I was experimenting with a question -> answer tool for RAG applications.
- vienneraphael 7mo ago
  Hey, idk if that helps but I developed something similar to the wrapper you're mentioning as an open-source python library. Just plug any async function into the provided async context manager and you get Batch APIs in two lines of code with any existing framework you currently have: https://github.com/vienneraphael/batchling https://github.com/vienneraphael/batchling Let me know if you have any questions, looking forward to having your feedback!
  - jiggawatts 7mo ago
    Looks very nice! This is exactly what I was thinking of doing, except that I work mostly with C# in enterprise settings. Looking at your approach, the equivalent in .NET land would be if the Microsoft.AI.Extensions package added some sort of batch abstraction side-by-side (or on top of) their existing IChatClient or IEmbeddingGenerator interfaces.