This is beautiful, but is there a decent way to plow through say, 20TB of text and put that into a vector database (encoder only)? It would be quite a great add

This is beautiful, but is there a decent way to plow through say, 20TB of text and put that into a vector database (encoder only)? It would be quite a great addition, especially if the vectors could then be translated into other forms (different language, json representation, pull out names/NER, etc) by just applying a decoder to the database.