4 ms·

Voyage-code-3

111 points by fzliu 2y ago

potatoman22 2y ago
What's their ranking on SWEBench?
- swyx 2y ago
  for an embedding model, this is a very very bad question, to the point where i am wondering if you are trolling and i missed the joke i can be constructive but i dont have space in the margins of this paper
  - moojacob 2y ago
    Pardon my ignorance, but are you saying that embedding model leaderboards aren't useful? When I look at https://huggingface.co/spaces/mteb/leaderboard https://huggingface.co/spaces/mteb/leaderboard I see voyage's embedding model is at the top. Does that mean they have the best model?
    - fzliu 2y ago
      Although voyage-3-m-exp is at the top of the leaderboard, I would not use it for production use cases unless your data is extremely similar to "in-domain" MTEB data. Check out this post from Nils (one of the authors of the original MTEB paper) for more info: https://x.com/Nils_Reimers/status/1870812625505849849 https://x.com/Nils_Reimers/status/1870812625505849849 voyage-3-large will work better for almost all real-world production use cases.
      - moojacob 2y ago
        Which model would you choose?
    - evilduck 2y ago
      They did not claim that leaderboards were useless. There’s a big difference between an embedding model leaderboard and the SWEbench leaderboard though, given that they measure completely different categories of models.
  - potatoman22 2y ago
    Woops. I should have done more reading before posting.
- fzliu 2y ago
  We have a repurposed version of SWE-bench Lite for retrieval; you can see the numbers in the last column of this spreadsheet: https://docs.google.com/spreadsheets/d/1Q5GDXOXueHuBT9demPrL9bz3_LMgajcZs_-GPeawrYk https://docs.google.com/spreadsheets/d/1Q5GDXOXueHuBT9demPrL...
qeternity 2y ago
Their 1024 dimension outperforms their 2048 dimension? What am I missing here.
- fzliu 2y ago
  For float and int8, 1024 does indeed outperform 2048. However, for binary, 2048 outperforms 1024. For our general-purpose embedding model (voyage-3-large), embedding vectors with 2048 dimensions outperform 1024 across the board: https://blog.voyageai.com/2025/01/07/voyage-3-large/ https://blog.voyageai.com/2025/01/07/voyage-3-large/
  - qeternity 2y ago
    Do you have any insights into this? Perhaps scaling degradation with Matryoshka learning?
doctorpangloss 2y ago
OpenAI is a big famous company. Why should I trust you guys with code, when I’m too lazy to clean out sensitive stuff?
- benreesman 2y ago
  It’s Fei-Fei and Christopher Ré. These are serious people.
  - swyx 2y ago
    highly doubt either are deeply involved in the day to day lol
    - benreesman 2y ago
      Have some respect.
serjester 2y ago
I don’t understand why we have so many companies working on vector databases when the real value is in fine tuned embeddings. Having recently evaluated vector DB’s to handle 100M+ nodes, everything will run you $1 - 8k per month. Yet using embeddings fine tuned on a specific use case, will lower that 10X since you can get away with way coarser vectors. This makes so much intuitive sense - voyage please release an insurance focused model.
- digdugdirk 2y ago
  Can you explain more about this, or point someone in the right direction with a few resources for reading more about it?
- apricot0 2y ago
  Can you recommend any resources/tutorials about fine-tuning embedding models? Which base model did you fine-tune?
- ghxst 2y ago
  What's an insurance focused model in this context? Sorry if it's a stupid question.
- parentheses 2y ago
  The answer is obvious. Vector DBs have clear line of sight. Experimental embeddings do not.
uncomplexity_ 2y ago
Currently using voyage for my project for the following reasons: 1. They are recommended by Anthropic on their docs 2. They're focused on embeddings as a service, I somehow prefer this other than spread-thin large orgs like OpenAI, GoogleAI, etc. 3. They got good standing on the huggingface mteb leaderboard: https://huggingface.co/spaces/mteb/leaderboard https://huggingface.co/spaces/mteb/leaderboard
zelcon 2y ago
Release the weights or buy an ad. This doesn’t deserve front page.
underlines 2y ago
For 2 years I built RAG apps for clients. They are not Code-Assistants but everytime I see code assistants solely relying on embeddings to find the right pieces as context, it feels wrong to me: Code is very well structured. Based on a starting point (current cursor, current file, or results of an embedding search result) you would probably fair better to traverse the code tree up and down building a tree or using Abstract Syntax Trees (ASTs) as described in this blog post [4]. It's like a tree search in order to get relevant code pieces for a certain task, and it imitates what human coders do. It would integrate well into an agent loop to search relevant code. Aren't there any open source code assistants and plugins that do this? All I see are embedding searches for the big projects such as cursor, cline or continue. All I ever found were a few research efforts such as RepoGraph [1], CodeGraph [2] and one one codebase open sourced by Deutsche Telekom called advanced-coding-assistant [3] 1 https://github.com/ozyyshr/RepoGraph https://github.com/ozyyshr/RepoGraph 2 https://arxiv.org/abs/2408.13863 https://arxiv.org/abs/2408.13863 3 https://github.com/telekom/advanced-coding-assistant-backend https://github.com/telekom/advanced-coding-assistant-backend 4 https://cyrilsadovsky.substack.com/p/advanced-coding-chatbot-knowledge-graphs-and-asts-0c18c90373be https://cyrilsadovsky.substack.com/p/advanced-coding-chatbot...
- littlestymaar 2y ago
  Definitely, why isn't there a code assistant that just uses the LSP to query the info you need straight from your language-specific tooling.
  - lexoj 2y ago
    Similarly, Aider sends a repo map (ranked) https://aider.chat/docs/repomap.html https://aider.chat/docs/repomap.html
  - popinman322 2y ago
    The LSP is limited in scope and doesn't provide access to things like the AST (which can vary by language). If you want to navigate by symbols, that can be done. If you want to know whether a given import is valid, to verify LLM output, that's not possible. Similarly, you can't use the LSP to determine all valid in-scope objects for an assignment. You can get a hierarchy of symbol information from some servers, allowing selection of particular lexical scopes within the file, but you'll need to perform type analysis yourself to determine which of the available variables could make for a reasonable completion. That type analysis is also a bit tricky because you'll likely need a lot of information about the type hierarchy at that lexical scope-- something you can't get from the LSP. It might be feasible to edit an open source LSP implementation for your target language to expose the extra information you'd want, but they're relatively heavy pieces of software and, of course, they don't exist for all languages. Compared to the development cost of "just" using embeddings-- it's pretty clear why teams choose embeddings. Also, if you assume that the performance improvements we've seen in embeddings for retrieval will continue, it makes less sense to invest weeks of time on something that would otherwise improve passively with time.
maCDzP 2y ago
I used the embeddings from Voyage for a project in Swedish this summer. The neat thing about Voyage was besides the speed of the service. I think I had 250 million tokens and Voyage was the fastest. It took a couple of days on and off. I believe my napkin calculation showed that OpenAI would have taken months.

Voyage-code-3

111 points by fzliu 2y ago

potatoman22 2y ago

What's their ranking on SWEBench?

swyx 2y ago
for an embedding model, this is a very very bad question, to the point where i am wondering if you are trolling and i missed the joke i can be constructive but i dont have space in the margins of this paper
- moojacob 2y ago
  Pardon my ignorance, but are you saying that embedding model leaderboards aren't useful? When I look at https://huggingface.co/spaces/mteb/leaderboard https://huggingface.co/spaces/mteb/leaderboard I see voyage's embedding model is at the top. Does that mean they have the best model?
  - fzliu 2y ago
    Although voyage-3-m-exp is at the top of the leaderboard, I would not use it for production use cases unless your data is extremely similar to "in-domain" MTEB data. Check out this post from Nils (one of the authors of the original MTEB paper) for more info: https://x.com/Nils_Reimers/status/1870812625505849849 https://x.com/Nils_Reimers/status/1870812625505849849 voyage-3-large will work better for almost all real-world production use cases.
    - moojacob 2y ago
      Which model would you choose?
  - evilduck 2y ago
    They did not claim that leaderboards were useless. There’s a big difference between an embedding model leaderboard and the SWEbench leaderboard though, given that they measure completely different categories of models.
- potatoman22 2y ago
  Woops. I should have done more reading before posting.
fzliu 2y ago
We have a repurposed version of SWE-bench Lite for retrieval; you can see the numbers in the last column of this spreadsheet: https://docs.google.com/spreadsheets/d/1Q5GDXOXueHuBT9demPrL9bz3_LMgajcZs_-GPeawrYk https://docs.google.com/spreadsheets/d/1Q5GDXOXueHuBT9demPrL...

qeternity 2y ago

Their 1024 dimension outperforms their 2048 dimension? What am I missing here.

fzliu 2y ago
For float and int8, 1024 does indeed outperform 2048. However, for binary, 2048 outperforms 1024. For our general-purpose embedding model (voyage-3-large), embedding vectors with 2048 dimensions outperform 1024 across the board: https://blog.voyageai.com/2025/01/07/voyage-3-large/ https://blog.voyageai.com/2025/01/07/voyage-3-large/
- qeternity 2y ago
  Do you have any insights into this? Perhaps scaling degradation with Matryoshka learning?

doctorpangloss 2y ago

OpenAI is a big famous company. Why should I trust you guys with code, when I’m too lazy to clean out sensitive stuff?

benreesman 2y ago
It’s Fei-Fei and Christopher Ré. These are serious people.
- swyx 2y ago
  highly doubt either are deeply involved in the day to day lol
  - benreesman 2y ago
    Have some respect.

serjester 2y ago

I don’t understand why we have so many companies working on vector databases when the real value is in fine tuned embeddings. Having recently evaluated vector DB’s to handle 100M+ nodes, everything will run you $1 - 8k per month. Yet using embeddings fine tuned on a specific use case, will lower that 10X since you can get away with way coarser vectors. This makes so much intuitive sense - voyage please release an insurance focused model.

digdugdirk 2y ago
Can you explain more about this, or point someone in the right direction with a few resources for reading more about it?
apricot0 2y ago
Can you recommend any resources/tutorials about fine-tuning embedding models? Which base model did you fine-tune?
ghxst 2y ago
What's an insurance focused model in this context? Sorry if it's a stupid question.
parentheses 2y ago
The answer is obvious. Vector DBs have clear line of sight. Experimental embeddings do not.

uncomplexity_ 2y ago

Currently using voyage for my project for the following reasons: 1. They are recommended by Anthropic on their docs 2. They're focused on embeddings as a service, I somehow prefer this other than spread-thin large orgs like OpenAI, GoogleAI, etc. 3. They got good standing on the huggingface mteb leaderboard: https://huggingface.co/spaces/mteb/leaderboard https://huggingface.co/spaces/mteb/leaderboard

zelcon 2y ago

Release the weights or buy an ad. This doesn’t deserve front page.

underlines 2y ago

For 2 years I built RAG apps for clients. They are not Code-Assistants but everytime I see code assistants solely relying on embeddings to find the right pieces as context, it feels wrong to me: Code is very well structured. Based on a starting point (current cursor, current file, or results of an embedding search result) you would probably fair better to traverse the code tree up and down building a tree or using Abstract Syntax Trees (ASTs) as described in this blog post [4]. It's like a tree search in order to get relevant code pieces for a certain task, and it imitates what human coders do. It would integrate well into an agent loop to search relevant code. Aren't there any open source code assistants and plugins that do this? All I see are embedding searches for the big projects such as cursor, cline or continue. All I ever found were a few research efforts such as RepoGraph [1], CodeGraph [2] and one one codebase open sourced by Deutsche Telekom called advanced-coding-assistant [3] 1 https://github.com/ozyyshr/RepoGraph https://github.com/ozyyshr/RepoGraph 2 https://arxiv.org/abs/2408.13863 https://arxiv.org/abs/2408.13863 3 https://github.com/telekom/advanced-coding-assistant-backend https://github.com/telekom/advanced-coding-assistant-backend 4 https://cyrilsadovsky.substack.com/p/advanced-coding-chatbot-knowledge-graphs-and-asts-0c18c90373be https://cyrilsadovsky.substack.com/p/advanced-coding-chatbot...

littlestymaar 2y ago
Definitely, why isn't there a code assistant that just uses the LSP to query the info you need straight from your language-specific tooling.
- lexoj 2y ago
  Similarly, Aider sends a repo map (ranked) https://aider.chat/docs/repomap.html https://aider.chat/docs/repomap.html
- popinman322 2y ago
  The LSP is limited in scope and doesn't provide access to things like the AST (which can vary by language). If you want to navigate by symbols, that can be done. If you want to know whether a given import is valid, to verify LLM output, that's not possible. Similarly, you can't use the LSP to determine all valid in-scope objects for an assignment. You can get a hierarchy of symbol information from some servers, allowing selection of particular lexical scopes within the file, but you'll need to perform type analysis yourself to determine which of the available variables could make for a reasonable completion. That type analysis is also a bit tricky because you'll likely need a lot of information about the type hierarchy at that lexical scope-- something you can't get from the LSP. It might be feasible to edit an open source LSP implementation for your target language to expose the extra information you'd want, but they're relatively heavy pieces of software and, of course, they don't exist for all languages. Compared to the development cost of "just" using embeddings-- it's pretty clear why teams choose embeddings. Also, if you assume that the performance improvements we've seen in embeddings for retrieval will continue, it makes less sense to invest weeks of time on something that would otherwise improve passively with time.

maCDzP 2y ago

I used the embeddings from Voyage for a project in Swedish this summer. The neat thing about Voyage was besides the speed of the service. I think I had 250 million tokens and Voyage was the fastest. It took a couple of days on and off. I believe my napkin calculation showed that OpenAI would have taken months.