2 ms·

So a new set of small models that are competitive with and beat phi-3.5 on some benchmarks is extremely impressive. Context length is the next frontier for mod

So a new set of small models that are competitive with and beat phi-3.5 on some benchmarks is extremely impressive. Context length is the next frontier for models in this range - getting to a true 128-200k token length in a small model would be very very useful. Hallucinations are less of an issue because you can just dump all the source data in, whole codebases can go in for stuff ranging from a refactor to ‘write documentation of the API’. Claude and gpto-preview are the only games in town for these long context tasks right now, and they are slow. Some of the tasks need the extra intelligence, but a lot don’t. In those cases a lightweight or local model will be greatly appreciated. Not to mention context length that long will more easily enable multimodal parsing.

luckydata 2y ago
Gemini has the largest context of any model out there FYI.
- deleted 2y ago
  [deleted]
- machiaweliczny 2y ago
  From my experience these large context are just techinical feasability but there needs to be a lot better internal benchamarks to claim it really works. I've tested on real world task and it all fails so far.
startupsfail 2y ago
It seems like this is a new architecture, alongside Mamba, Hyena, RWKV and Hybrid Transformer.
- zwaps 2y ago
  The site insinuates that it is a set of flexible and probably parametric mix between full attention and linear conplexity models
jadbox 2y ago
Looking at 3B, the performance is roughly on par with phi 3.5.. not sure where how they gauge on their graph that it is better. Agreed though, I don't think I would switch my 3B model from phi unless Liquid was more transparent in data and research.
simonw 2y ago
Llama 3.2 1B and 3B both claim a 128,000 token length.
- GaggiX 2y ago
  That is indeed their claim, but if you read the linked page you will see that it is not their effective token window.
- dragonwriter 2y ago
  The issue isn't the nominal size of the context window which is easy to objectively measure, but effective use of the context window, which is harder to measure precisely, but a big issue: it doesn't matter how much stuff you can throw at the model if it effectively ignores most of it past a certain point.
- vessenes 2y ago
  They point out in the paper stats around where effectiveness drops off hard. It's at 32k at most everywhere now. Some models still at 4 to 8k. Getting to 200k in a smaller model is an open research area, as far as I'm aware. Ideas so far are pretty broad ranging, including using fourier transforms to try and capture cyclicality in inputs (mamba et al), RWKV (which I do not fully understand, but claims very long input context support), larger range of possible tokenizations shortening input length (llama 3), ..
simion314 2y ago
>Hallucinations are less of an issue because you can just dump all the source data in, whole codebases can go in for stuff ranging from a refactor to ‘write documentation of the API’. Is there no risk ? I mean say for testing purposes we give the AI a giant CSV file and ask it to make it a json is the chance for error 0% ? Because today we need to double check when we ask AI to transform some data or transform some code, there is the risk of it messing something up but if it is not something that would crash immediately you risk introducing a ton of new bugs by asking an AI to refactor instead of using some good tools.
- vessenes 2y ago
  There's always risk! But when you ask a model to rely on just the input data, you are (mostly) trying to tap into its reasoning, not knowledge side. Obviously what's kind of magical is that some knowledge will be needed for reasoning, and you have it. But LLMs small and large are pretty good at doing the in-context stuff. It is precisely what they're trained on, and in fact it was kind of a surprise how well they seemed to generalize outside of this task in the first place.

luckydata 2y ago

Gemini has the largest context of any model out there FYI.

deleted 2y ago
[deleted]
machiaweliczny 2y ago
From my experience these large context are just techinical feasability but there needs to be a lot better internal benchamarks to claim it really works. I've tested on real world task and it all fails so far.

startupsfail 2y ago

It seems like this is a new architecture, alongside Mamba, Hyena, RWKV and Hybrid Transformer.

zwaps 2y ago
The site insinuates that it is a set of flexible and probably parametric mix between full attention and linear conplexity models

jadbox 2y ago

Looking at 3B, the performance is roughly on par with phi 3.5.. not sure where how they gauge on their graph that it is better. Agreed though, I don't think I would switch my 3B model from phi unless Liquid was more transparent in data and research.

simonw 2y ago

Llama 3.2 1B and 3B both claim a 128,000 token length.

GaggiX 2y ago
That is indeed their claim, but if you read the linked page you will see that it is not their effective token window.
dragonwriter 2y ago
The issue isn't the nominal size of the context window which is easy to objectively measure, but effective use of the context window, which is harder to measure precisely, but a big issue: it doesn't matter how much stuff you can throw at the model if it effectively ignores most of it past a certain point.
vessenes 2y ago
They point out in the paper stats around where effectiveness drops off hard. It's at 32k at most everywhere now. Some models still at 4 to 8k. Getting to 200k in a smaller model is an open research area, as far as I'm aware. Ideas so far are pretty broad ranging, including using fourier transforms to try and capture cyclicality in inputs (mamba et al), RWKV (which I do not fully understand, but claims very long input context support), larger range of possible tokenizations shortening input length (llama 3), ..

simion314 2y ago

>Hallucinations are less of an issue because you can just dump all the source data in, whole codebases can go in for stuff ranging from a refactor to ‘write documentation of the API’. Is there no risk ? I mean say for testing purposes we give the AI a giant CSV file and ask it to make it a json is the chance for error 0% ? Because today we need to double check when we ask AI to transform some data or transform some code, there is the risk of it messing something up but if it is not something that would crash immediately you risk introducing a ton of new bugs by asking an AI to refactor instead of using some good tools.

vessenes 2y ago
There's always risk! But when you ask a model to rely on just the input data, you are (mostly) trying to tap into its reasoning, not knowledge side. Obviously what's kind of magical is that some knowledge will be needed for reasoning, and you have it. But LLMs small and large are pretty good at doing the in-context stuff. It is precisely what they're trained on, and in fact it was kind of a surprise how well they seemed to generalize outside of this task in the first place.