3 ms·

10x exit in a couple years, quite nice on the VC side! On the tech side ... no idea what Mongo's plan is ... their embedding model is not SOTA, does not even o

10x exit in a couple years, quite nice on the VC side! On the tech side ... no idea what Mongo's plan is ... their embedding model is not SOTA, does not even outperform the open ones out there, and reranking is a dead end in 2025. I think the value is on Voyage's team, their user base and having a vision that aligned with Mongo's. Congrats!

serjester 2y ago
We've benchmarked a ton of the open models and voyage dramatically outperforms them. I think MTEB is a bad benchmark.
touche_bag 2y ago
Interesting take. Have you benchmarked models on your own data? Cause at this point everything is contaminated so I find it impossible to tell what proper sota is. Also - most folks still just use openai. Last time I checked, reranking always performs better than pure vector search. And to my knowledge it's still the superior fusion method for keyword and vector results.
- moralestapia 2y ago
  In my experience, storing RAG chunks with a little bit of context helps a lot when doing the retrieval, then you can skip the whole "rerank" bit and halve your cost and latency. With embedding/generative models becoming better with time, the need for a rerank step will be optimized away.
  - touche_bag 2y ago
    Huh? Rerank is always a boost on top of retrieval. So regardless of the chunking method or model you use, reranking with a good model will always result in higher MRR. And improvements in embedding models also will never solve the problem of merging lexical and vector search results. Rank/score fusion are flawed since both are hardly comparable and boosting only works sometimes. Whereas rerankers generally do a pretty good job at this. Performance is indeed the biggest issue here. Rerankers are slow as hell and simply not feasible for some use cases.
hweller 2y ago
>their embedding model is not SOTA, does not even outperform the open ones out there, and reranking is a dead end in 2025. Are you referring to the MTEB leaderboard? It's widely believed many of those test datasets are considered during the training of most open-source text embedding models, hence why you see novel + private benchmarks discussed in many launch blogs that don't exclusively refer to MTEB. There are problems there, and it would be great to see more folks in the search benchmark dataset production space like what Marqo AI has done in recent months. Also what makes you say reranking is dead? Mongo doesn't provide it out of the box but many other search providers like ES, Pinecone, Opensearch do so it must provide some value to their customers? Maybe you're saying it's overrated in terms of how many apps actually need it? disclosure: I work on vector search at Mongo
- moralestapia 2y ago
  >Maybe you're saying it's overrated in terms of how many apps actually need it? Yes, my comment leans more towards that, rather than suggesting is useless.
  - redwood 2y ago
    Taking a step back, accuracy/quality of retrieval is critical as input to anything generated b/c your generated output is only as good as your input. And right now folks are struggling to adopt generative use cases due to risk and fear of how to control outputs. Therefore I think this could be bigger than you think.

serjester 2y ago

We've benchmarked a ton of the open models and voyage dramatically outperforms them. I think MTEB is a bad benchmark.

touche_bag 2y ago

Interesting take. Have you benchmarked models on your own data? Cause at this point everything is contaminated so I find it impossible to tell what proper sota is. Also - most folks still just use openai. Last time I checked, reranking always performs better than pure vector search. And to my knowledge it's still the superior fusion method for keyword and vector results.

moralestapia 2y ago
In my experience, storing RAG chunks with a little bit of context helps a lot when doing the retrieval, then you can skip the whole "rerank" bit and halve your cost and latency. With embedding/generative models becoming better with time, the need for a rerank step will be optimized away.
- touche_bag 2y ago
  Huh? Rerank is always a boost on top of retrieval. So regardless of the chunking method or model you use, reranking with a good model will always result in higher MRR. And improvements in embedding models also will never solve the problem of merging lexical and vector search results. Rank/score fusion are flawed since both are hardly comparable and boosting only works sometimes. Whereas rerankers generally do a pretty good job at this. Performance is indeed the biggest issue here. Rerankers are slow as hell and simply not feasible for some use cases.

hweller 2y ago

>their embedding model is not SOTA, does not even outperform the open ones out there, and reranking is a dead end in 2025. Are you referring to the MTEB leaderboard? It's widely believed many of those test datasets are considered during the training of most open-source text embedding models, hence why you see novel + private benchmarks discussed in many launch blogs that don't exclusively refer to MTEB. There are problems there, and it would be great to see more folks in the search benchmark dataset production space like what Marqo AI has done in recent months. Also what makes you say reranking is dead? Mongo doesn't provide it out of the box but many other search providers like ES, Pinecone, Opensearch do so it must provide some value to their customers? Maybe you're saying it's overrated in terms of how many apps actually need it? disclosure: I work on vector search at Mongo

moralestapia 2y ago
>Maybe you're saying it's overrated in terms of how many apps actually need it? Yes, my comment leans more towards that, rather than suggesting is useless.
- redwood 2y ago
  Taking a step back, accuracy/quality of retrieval is critical as input to anything generated b/c your generated output is only as good as your input. And right now folks are struggling to adopt generative use cases due to risk and fear of how to control outputs. Therefore I think this could be bigger than you think.