3 ms·

Limitations of single point in vector space of what dimension? I’m not sure it’s public knowledge, but it’s an architecture choice. They choose how big to make

Limitations of single point in vector space of what dimension? I’m not sure it’s public knowledge, but it’s an architecture choice. They choose how big to make the embedding dimension. My point is just that there’s no limitation in principle, it’s just a matter of how they design it and resource constraints.

causal 2y ago
Thanks for responding to that point - it's the one most on my mind. So OpenAI's large embedding model has 3072 dimensions, though in practice far fewer are probably used. Clearly you can't compress 1M tokens down to 3072. Yet those 3072 numbers are all you've got for capturing the full meaning of the previous token when predicting the next one; including all 1M tokens of modifying context. So perhaps human language is simply never complex enough to need more than 3072 numbers to represent a given train of thought, but that doesn't seem clear to me. Edit: Since Gemini is relevant here, it looks like their text embedding model is 768 dimensions.
- neverokay 2y ago
  So perhaps human language is simply never complex enough to need more than 3072 numbers to represent a given train of thought, but that doesn't seem clear to me. Will compute allow that number to go up? Or is that an optimal number?
  - WhitneyLand 2y ago
    Definitely has trended upward, there’s no special number. It’s just a matter of how much compute, storage, time to allocate to that part of the architecture.
    - causal 2y ago
      Has it? It's my understanding that GPT-3 was 12,288 and GPT-4 went down to 3072.
      - WhitneyLand 2y ago
        I’m not aware that internal embedding dimension size has been made public for Gpt4 et al. In general for models we know its trended upward, but for sure it’d be interesting to know what they’re using now.
- WhitneyLand 2y ago
  Yes but we can distinguish between embedding provided to customers, and internal embeddings. One is optimized for usage in certain types of applications, but the internal embeddings need to be optimized to support long contexts and are not constrained by the customer facing embeddings. For example, with Open AI I believe it’s known that the internal dimension for Gpt3 was 12,288.
  - causal 2y ago
    Are the same embeddings not used internally? I thought they were. Maybe I'm wrong about that. Mistral uses a 1024 dimension embedding for 8K context. I think the point about trying to capture that rich of a context into a smaller number of dimensions still stands?
    - WhitneyLand 2y ago
      Yes definitely, you have a valid concern. For long contexts this is a key consideration along with what self attention optimizations the model chooses to implement. They don’t make this public, but we can infer they can’t be using full self attention pairs at 1,000,000 tokens because it scales quadratically and would take Terabytes of RAM. There are different approaches like sparse attention, and the only way to really know how well their choices work is to test it.

causal 2y ago

Thanks for responding to that point - it's the one most on my mind. So OpenAI's large embedding model has 3072 dimensions, though in practice far fewer are probably used. Clearly you can't compress 1M tokens down to 3072. Yet those 3072 numbers are all you've got for capturing the full meaning of the previous token when predicting the next one; including all 1M tokens of modifying context. So perhaps human language is simply never complex enough to need more than 3072 numbers to represent a given train of thought, but that doesn't seem clear to me. Edit: Since Gemini is relevant here, it looks like their text embedding model is 768 dimensions.

neverokay 2y ago
So perhaps human language is simply never complex enough to need more than 3072 numbers to represent a given train of thought, but that doesn't seem clear to me. Will compute allow that number to go up? Or is that an optimal number?
- WhitneyLand 2y ago
  Definitely has trended upward, there’s no special number. It’s just a matter of how much compute, storage, time to allocate to that part of the architecture.
  - causal 2y ago
    Has it? It's my understanding that GPT-3 was 12,288 and GPT-4 went down to 3072.
    - WhitneyLand 2y ago
      I’m not aware that internal embedding dimension size has been made public for Gpt4 et al. In general for models we know its trended upward, but for sure it’d be interesting to know what they’re using now.
WhitneyLand 2y ago
Yes but we can distinguish between embedding provided to customers, and internal embeddings. One is optimized for usage in certain types of applications, but the internal embeddings need to be optimized to support long contexts and are not constrained by the customer facing embeddings. For example, with Open AI I believe it’s known that the internal dimension for Gpt3 was 12,288.
- causal 2y ago
  Are the same embeddings not used internally? I thought they were. Maybe I'm wrong about that. Mistral uses a 1024 dimension embedding for 8K context. I think the point about trying to capture that rich of a context into a smaller number of dimensions still stands?
  - WhitneyLand 2y ago
    Yes definitely, you have a valid concern. For long contexts this is a key consideration along with what self attention optimizations the model chooses to implement. They don’t make this public, but we can infer they can’t be using full self attention pairs at 1,000,000 tokens because it scales quadratically and would take Terabytes of RAM. There are different approaches like sparse attention, and the only way to really know how well their choices work is to test it.