4 ms·

I'm not sure why the voyage-3 models aren't on the MTEB leaderboard. The code for the leaderboard suggests they should be there: https://huggingface.co/spaces/m

I'm not sure why the voyage-3 models aren't on the MTEB leaderboard. The code for the leaderboard suggests they should be there: https://huggingface.co/spaces/mteb/leaderboard/commit/b7faae9e2db6d721cacc15cb923f29b8bb9115a4 https://huggingface.co/spaces/mteb/leaderboard/commit/b7faae... But I don't see them when I filter the list for 'voyage'.

newrotik 2y ago
It is unclear this model should be on that leaderboard because we don't know whether it has been trained on mteb test data. It is worth noting that their own published material [0] does not entail any score from any dataset from the mteb benchmark. This may sound nit picky, but considering transformers' parroting capabilities, having seen test data during training should be expected to completely invalidate those scores. [0] see excel spreadsheet linked here https://blog.voyageai.com/2024/09/18/voyage-3/ https://blog.voyageai.com/2024/09/18/voyage-3/
- jdthedisciple 2y ago
  I'm critical of the low number of embedding dims. Could hurt performance in niche applications, in my estimation. Looking forward to try the announced large models though.
fzliu 2y ago
(I work at Voyage) Many of the top-performing models that you see on the MTEB retrieval for English and Chinese tend to overfit to the benchmark nowadays. voyage-3 and voyage-3-lite are also pretty small in size compared to a lot of the 7B models that take the top spots, and we don't want to hurt performance on other real-world tasks just to do well on MTEB.
- jdthedisciple 2y ago
  It would still be great to know how it compares? Why should I pick voyage-3 if for all I know it sucks when it comes to retrieval accuracy (my personally most important metric)?
  - fzliu 2y ago
    We provide retrieval metrics for a variety of datasets and languages: https://blog.voyageai.com/2024/09/18/voyage-3/ https://blog.voyageai.com/2024/09/18/voyage-3/. I also personally encourage folks to either test on their own data or to find an open source dataset that closely resembles the documents they are trying to search (we provide a ton of free tokens for the evaluating our models).
- kkielhofner 2y ago
  > we don't want to hurt performance on other real-world tasks just to do well on MTEB Nice! Fortunately MTEB lets you sort by model parameter size because using 7B parameter LLMs for embeddings is just... Yuck.

newrotik 2y ago

It is unclear this model should be on that leaderboard because we don't know whether it has been trained on mteb test data. It is worth noting that their own published material [0] does not entail any score from any dataset from the mteb benchmark. This may sound nit picky, but considering transformers' parroting capabilities, having seen test data during training should be expected to completely invalidate those scores. [0] see excel spreadsheet linked here https://blog.voyageai.com/2024/09/18/voyage-3/ https://blog.voyageai.com/2024/09/18/voyage-3/

jdthedisciple 2y ago
I'm critical of the low number of embedding dims. Could hurt performance in niche applications, in my estimation. Looking forward to try the announced large models though.

fzliu 2y ago

(I work at Voyage) Many of the top-performing models that you see on the MTEB retrieval for English and Chinese tend to overfit to the benchmark nowadays. voyage-3 and voyage-3-lite are also pretty small in size compared to a lot of the 7B models that take the top spots, and we don't want to hurt performance on other real-world tasks just to do well on MTEB.

jdthedisciple 2y ago
It would still be great to know how it compares? Why should I pick voyage-3 if for all I know it sucks when it comes to retrieval accuracy (my personally most important metric)?
- fzliu 2y ago
  We provide retrieval metrics for a variety of datasets and languages: https://blog.voyageai.com/2024/09/18/voyage-3/ https://blog.voyageai.com/2024/09/18/voyage-3/. I also personally encourage folks to either test on their own data or to find an open source dataset that closely resembles the documents they are trying to search (we provide a ton of free tokens for the evaluating our models).
kkielhofner 2y ago
> we don't want to hurt performance on other real-world tasks just to do well on MTEB Nice! Fortunately MTEB lets you sort by model parameter size because using 7B parameter LLMs for embeddings is just... Yuck.