5 ms·

1M token context by default is the big feature here IMO, but we need better benchmarks to measure what that really means. My intuition is that as contexts get

by causal 2y ago

1M token context by default is the big feature here IMO, but we need better benchmarks to measure what that really means. My intuition is that as contexts get longer we start hitting the limits of how much comprehension can be embedded in a single point of vector space, and will need better architectures for selecting the relevant portions of the context.

dragonwriter 2y ago
> 1M token context by default is the big feature here IMO, but we need better benchmarks to measure what that really means. Multimodality in a model That's between 4-7% the cost per token of OpenAI’s cheapest multimodal model is an important feature when you are talking about production use and not just economically unsustainable demos.
- refulgentis 2y ago
  In preview, can't be used in production, they already rug-pulled people building on Gemini w/r/t cost and RPM, and they're pointedly not putting any RPM or cost on the page. (seriously, try finding info on cost, RPM, or release right now, you're linked in circles.) Agree on OpenAI multimodal but it's sort of a stilted example itself, it's because OpenAI has a hole in its lineup - ex. Claude Haiku is multimodal, faster, and significantly cheaper than GPT 3.5.
  - dragonwriter 2y ago
    > they're pointedly not putting any RPM or cost on the page 360 RPM base limit, pricing is posted. > seriously, try finding info on cost, RPM, or release right now, I wasn't making up numbers, its on their Gemini API pricing page: https://ai.google.dev/pricing https://ai.google.dev/pricing
    - refulgentis 2y ago
      Nice, thanks (btw, I didn't think you were making it up, it was in the keynote!)
  - causal 2y ago
    +1 on Haiku being oft overlooked.
    - verdverm 2y ago
      Shows the power of the brand and the limit of names consumers will recall long term "Who are the biggest soda or potato chip makers?"
      - refulgentis 2y ago
        [flagged]
        verdverm 2y ago
        1. Not an AI specific comment series, it applies generally 2. Comments like yours are against the guidelines of HN 3. You don't know what I do and are incorrect in your assessment of my work and experience (another common HN mishap) --- (re: #2/3) Please make your substantive points without crossing into personal attack. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
        anoncareer0212 2y ago
        What do you mean? A) This fills a gaping hole for cheap multimodal models, OpenAI doesn't have one B) Anthropic's Haiku is a good choice. You) wow A didn't know Anthropic. Goes to show power of brands, much like snack foods B) Eh I wouldn't conclude anything from A. Its one comment. some people don't know what an Anthropic is because there's high interest in AI relative to interest in AI APIs. you can expect a low SNR ratio, even on HN You) Stop personally attacking me! It's against the rules!!
        verdverm 2y ago
        > You) Stop personally attacking me! It's against the rules!! Your comment history certainly shows a disregard for the HN guidelines https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
- leetharris 2y ago
  The problem is that even 1.5 Pro seems completely useless for long context multimodal stuff. I have tried it for so many use cases in video / audio and it hallucinates an unbelievable amount. More than any other model I've ever used. So if 1.5 Pro can't even handle simple tasks without hallucination, I imagine this tiny model is even more useless.
refulgentis 2y ago
Yeah it's not very good in practice, you can get a halfway decent demo out of it ("look I gave it 6.5 harry potters and it made an SVG map connecting characters with annotations!!"...some of the characters...spare annotations...cost $20). Just good enough to fool you a couple times when you try to make it work 10 times.
WhitneyLand 2y ago
Limitations of single point in vector space of what dimension? I’m not sure it’s public knowledge, but it’s an architecture choice. They choose how big to make the embedding dimension. My point is just that there’s no limitation in principle, it’s just a matter of how they design it and resource constraints.
- causal 2y ago
  Thanks for responding to that point - it's the one most on my mind. So OpenAI's large embedding model has 3072 dimensions, though in practice far fewer are probably used. Clearly you can't compress 1M tokens down to 3072. Yet those 3072 numbers are all you've got for capturing the full meaning of the previous token when predicting the next one; including all 1M tokens of modifying context. So perhaps human language is simply never complex enough to need more than 3072 numbers to represent a given train of thought, but that doesn't seem clear to me. Edit: Since Gemini is relevant here, it looks like their text embedding model is 768 dimensions.
  - neverokay 2y ago
    So perhaps human language is simply never complex enough to need more than 3072 numbers to represent a given train of thought, but that doesn't seem clear to me. Will compute allow that number to go up? Or is that an optimal number?
    - WhitneyLand 2y ago
      Definitely has trended upward, there’s no special number. It’s just a matter of how much compute, storage, time to allocate to that part of the architecture.
      - causal 2y ago
        Has it? It's my understanding that GPT-3 was 12,288 and GPT-4 went down to 3072.
shoelessone 2y ago
> My intuition is that as contexts get longer we start hitting the limits of how much comprehension can be embedded in a single point of vector space, and will need better architectures for selecting the relevant portions of the context. Is it possible to explain what this means in a way that somebody only roughly familiar with vectors and vector databases? Or recommend an article or further reading on the topic?
- causal 2y ago
  So most of my understanding comes from this series, particularly the last two videos: https://www.3blue1brown.com/topics/neural-networks https://www.3blue1brown.com/topics/neural-networks Essentially each token of a text occupies a point in a many-dimensional model that represents meaning, and LLMs predict the next token by modifying the last token with the context of all the tokens before it. Attention heads are basically a way of choosing which prior tokens are most relevant and adjusting the last token's point in vector-space accordingly.
sojuz151 2y ago
>My intuition is that as contexts get longer we start hitting the limits of how much comprehension can be embedded in a single point of vector space, and will need better architectures for selecting the relevant portions of the context. We are dealing with multi-headed attention, therefore we have multiple points per token. You can always increase the number of heads or the size of the key vector.
- causal 2y ago
  The token embedding is what ultimately gets nudged around by the heads though, right? The key vector just relates to the context size, not the token embedding size, afaik.

by causal 2y ago

dragonwriter 2y ago

> 1M token context by default is the big feature here IMO, but we need better benchmarks to measure what that really means. Multimodality in a model That's between 4-7% the cost per token of OpenAI’s cheapest multimodal model is an important feature when you are talking about production use and not just economically unsustainable demos.

refulgentis 2y ago
In preview, can't be used in production, they already rug-pulled people building on Gemini w/r/t cost and RPM, and they're pointedly not putting any RPM or cost on the page. (seriously, try finding info on cost, RPM, or release right now, you're linked in circles.) Agree on OpenAI multimodal but it's sort of a stilted example itself, it's because OpenAI has a hole in its lineup - ex. Claude Haiku is multimodal, faster, and significantly cheaper than GPT 3.5.
- dragonwriter 2y ago
  > they're pointedly not putting any RPM or cost on the page 360 RPM base limit, pricing is posted. > seriously, try finding info on cost, RPM, or release right now, I wasn't making up numbers, its on their Gemini API pricing page: https://ai.google.dev/pricing https://ai.google.dev/pricing
  - refulgentis 2y ago
    Nice, thanks (btw, I didn't think you were making it up, it was in the keynote!)
- causal 2y ago
  +1 on Haiku being oft overlooked.
  - verdverm 2y ago
    Shows the power of the brand and the limit of names consumers will recall long term "Who are the biggest soda or potato chip makers?"
    - refulgentis 2y ago
      [flagged]
      - verdverm 2y ago
        1. Not an AI specific comment series, it applies generally 2. Comments like yours are against the guidelines of HN 3. You don't know what I do and are incorrect in your assessment of my work and experience (another common HN mishap) --- (re: #2/3) Please make your substantive points without crossing into personal attack. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
        anoncareer0212 2y ago
        What do you mean? A) This fills a gaping hole for cheap multimodal models, OpenAI doesn't have one B) Anthropic's Haiku is a good choice. You) wow A didn't know Anthropic. Goes to show power of brands, much like snack foods B) Eh I wouldn't conclude anything from A. Its one comment. some people don't know what an Anthropic is because there's high interest in AI relative to interest in AI APIs. you can expect a low SNR ratio, even on HN You) Stop personally attacking me! It's against the rules!!
        verdverm 2y ago
        > You) Stop personally attacking me! It's against the rules!! Your comment history certainly shows a disregard for the HN guidelines https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
leetharris 2y ago
The problem is that even 1.5 Pro seems completely useless for long context multimodal stuff. I have tried it for so many use cases in video / audio and it hallucinates an unbelievable amount. More than any other model I've ever used. So if 1.5 Pro can't even handle simple tasks without hallucination, I imagine this tiny model is even more useless.

refulgentis 2y ago

Yeah it's not very good in practice, you can get a halfway decent demo out of it ("look I gave it 6.5 harry potters and it made an SVG map connecting characters with annotations!!"...some of the characters...spare annotations...cost $20). Just good enough to fool you a couple times when you try to make it work 10 times.

WhitneyLand 2y ago

Limitations of single point in vector space of what dimension? I’m not sure it’s public knowledge, but it’s an architecture choice. They choose how big to make the embedding dimension. My point is just that there’s no limitation in principle, it’s just a matter of how they design it and resource constraints.

causal 2y ago
Thanks for responding to that point - it's the one most on my mind. So OpenAI's large embedding model has 3072 dimensions, though in practice far fewer are probably used. Clearly you can't compress 1M tokens down to 3072. Yet those 3072 numbers are all you've got for capturing the full meaning of the previous token when predicting the next one; including all 1M tokens of modifying context. So perhaps human language is simply never complex enough to need more than 3072 numbers to represent a given train of thought, but that doesn't seem clear to me. Edit: Since Gemini is relevant here, it looks like their text embedding model is 768 dimensions.
- neverokay 2y ago
  So perhaps human language is simply never complex enough to need more than 3072 numbers to represent a given train of thought, but that doesn't seem clear to me. Will compute allow that number to go up? Or is that an optimal number?
  - WhitneyLand 2y ago
    Definitely has trended upward, there’s no special number. It’s just a matter of how much compute, storage, time to allocate to that part of the architecture.
    - causal 2y ago
      Has it? It's my understanding that GPT-3 was 12,288 and GPT-4 went down to 3072.

shoelessone 2y ago

> My intuition is that as contexts get longer we start hitting the limits of how much comprehension can be embedded in a single point of vector space, and will need better architectures for selecting the relevant portions of the context. Is it possible to explain what this means in a way that somebody only roughly familiar with vectors and vector databases? Or recommend an article or further reading on the topic?

causal 2y ago
So most of my understanding comes from this series, particularly the last two videos: https://www.3blue1brown.com/topics/neural-networks https://www.3blue1brown.com/topics/neural-networks Essentially each token of a text occupies a point in a many-dimensional model that represents meaning, and LLMs predict the next token by modifying the last token with the context of all the tokens before it. Attention heads are basically a way of choosing which prior tokens are most relevant and adjusting the last token's point in vector-space accordingly.

sojuz151 2y ago

>My intuition is that as contexts get longer we start hitting the limits of how much comprehension can be embedded in a single point of vector space, and will need better architectures for selecting the relevant portions of the context. We are dealing with multi-headed attention, therefore we have multiple points per token. You can always increase the number of heads or the size of the key vector.

causal 2y ago
The token embedding is what ultimately gets nudged around by the heads though, right? The key vector just relates to the context size, not the token embedding size, afaik.