5 ms·
Once the context windows for these models grows by one or two orders of magnitude, stuff is going to be very interesting.
by Maxion 3y ago
Once the context windows for these models grows by one or two orders of magnitude, stuff is going to be very interesting.
- alexpetralia 3y agoThe new register size!
- jesusofnazarath 3y ago[dead]
- nighmi 3y agoWe had seen interesting developpments around vector databases, but then people stopped hyping them as you could just save them in normal databases without real differences. I wonder what will happen when the models can freely access them though.
- ENGNR 3y agoCurious, do it mean saving into Postgres with pgvector, or some other technique?
- EGreg 3y agoI really don't understand how people figure out the vectors to actually store in the databases, regardless of the underlying storage model. Isn't that itself the province of an LLM? Say I have a bunch of text. How do I save the text search "by similarity"? Sphinx and semantic search was hard, I remember. Facebook had Faiss. And here we are supposed to just save vectors on commodity hardware BEFORE using an LLM?
- martythemaniak 3y ago> Isn't that itself the province of an LLM? It is! The steps are: 1. Take a bunch of text, run it through an LLM in embedding mode. The LLM turns the text into a vector. If the text is longer than the LLM context window, chunk it. 2. Store the vector in a vector DB. 3. Use the LLM to generate a vector of your question. 4. Query the vectordb for all similar vectors (that fit in the context window) 5. Get the text from all those vectors. Concatenate the text with the question from step 3. 6. Step 5 is your prompt. The LLM can now answer your question with a collection of similar/relevant text already provided to the LLM in the context window along with your question.
- sterlind 3y agoYou don't even need an LLM. You can use Word2Vec, or even yank the embeddings matrix from the bottom layer of an LLM. And you can use CLIP and BLIP for images and audio, respectively.
- mewpmewp2 3y agoI had scaling issues with pgvector. So would also like to know what regular db can scale?
- RivieraKid 3y ago> stuff is going to be very interesting So no change from today?
- morkalork 3y agoSAP and IBM will be able to boondoggle government projects at scales never seen before!
- l5870uoo9y 3y agoOn a side note, the largest context for GPT4 is 32K. That will cost around $3 (rough estimate). If GPT5 is 20 times the cost of its predecessor as well, then that will cost around $60 per generation. GPT6 will then be $1200.
- kridsdale3 3y agoBut Nvidia making future GPUs on ever smaller TSMC nodes will be a counter-force, converging those future model-use costs to similar zones to today's. Ultimately the market will bear what it will, and if the costs are outside that zone, those products won't launch.