3 ms·
Counter-intuitively, lossy compression can result in better quality than lossless compression! Sure, if you start off with a 4K raw video and compress it, by de
by jiggawatts 3y ago
Counter-intuitively, lossy compression can result in better quality than lossless compression! Sure, if you start off with a 4K raw video and compress it, by definition the quality gets worse. But if you compared 8K lossy that's the same size as 4K raw, then the 8K video would look better. That's because it allocates the bits more efficiently, putting them to work where it counts.
It's a fairly safe bet that the same would apply to LLMs. If you start with a simple uncompressed LLMs that is 65B parameters and somehow compress or quantise it down to less than that, it will inevitably become a little dumber.
But if you compared the raw LLM to one that utilised all of these tricks and was the same size, then the latter would be superior because it could use the available parameters more efficiently.
If we can train and run GPT-3 cost effectively now with ~100B parameters, then it's a safe bet that we could train something as smart as GPT-4, with >200K window sizes, but as fast as GPT-3 for inference. (That's assuming all the recent quantization techniques are also applied.)
I'm betting we'll have something like that generally available within two years.
That'll be terrifying. An AI that can read and understand a book every few seconds...
- teaearlgraycold 3y agoWhat's interesting is we're currently training AI on how to use vector databases. Next generation LLMs trained on GitHub from the era of LangChain and FOSS vector DBs will be able to self-program their own long term memory recall. I don't think that just chunking and storing vectors for all of the text the LLM reads is the best approach, but it might be able to apply a strategy to each unique situation that is more optimal.
- rafterydj 3y agoThat's fascinating. As a novice to the problem, are there any resources you could link about this? I'm new to studying AI- but I've been prototyping connecting GPT to a nonrelational database to serve as a stand in for long term memory. My problem so far utilizing GPT3 has been difficulty getting it to use any consistent schema, as it will write to the database in a generated schema but try to recall in another. This is the first I've heard about using vector databases for the task.
- deleted 3y ago[deleted]
- fzliu 3y agoI don't think chunking is the most optimal approach either. I can envision a future where embeddings have variable length and comparing two variable-length embeddings would require a more complex similarity metric. Vector databases will need to adjust to this reality.
- gdiamos 3y agoThe future is exciting and terrifying. Scaling laws have resulted in GPT-4 and they keep going. The design space for LLMs is bigger than just transformer architectures. It includes long prompts, retrieval, chains, data quality, compression, etc, which potentially stack on top of HW acceleration. These are the worst models that we will see over the next decade. It’s going to be wild.