Yes that's true, but performance vs run time isn't an easy win as compared to say using bicoders with pre-embedded retrieval embeddings and just embedding the q

Yes that's true, but performance vs run time isn't an easy win as compared to say using bicoders with pre-embedded retrieval embeddings and just embedding the query (which is almost always a < 0.1s operation even on a low-RAM EC2 instnace). It takes > 1 second(perhaps 1.5-3s if you embed paragraphs not sentences) for re-ranking k >= 8 on a decent say 32GB CPU instance..(requires encoding each document passage with the search query top k times)... if you have GPUs available for your search inference sure it's just a fraction of a second, but right now I would say it's significantly more onerous than bicoders that get you the first jump in semantic search performance (10-40x slower in practise on CPUs) so the use case differs.