10 ms·
Liquid Foundation Models: Our First Series of Generative AI Models
- person4268 2y agoIt's pretty impressive, just note (emphasis added): > At Liquid AI, we take an open-science approach. We have and will continue to contribute to the advancement of the AI field by openly publishing our findings and methods through scientific and technical reports. As part of this commitment, we will release relevant data and models produced by our research efforts to the wider AI community. We have dedicated a lot of time and resources to developing these architectures, *so we're not open-sourcing our models at the moment*. This allows us to continue building on our progress and maintain our edge in the competitive AI landscape. Looks like there's no paper (or similar) yet, either. Hopefully they'll release a more detailed writeup soon.
- turnsout 2y agoMissed opportunity. I would argue that the only way they CAN make these smaller models competitive is to make them openly available. As a developer, I'm not going to choose an unknown startup's model over bigger closed models from OpenAI or Anthropic. And if I really need something smaller and faster, I'd prefer to run the model myself for better control and no risk of the model being "upgraded."
- swyx 2y agothis is the "paper": a list of citations https://www.liquid.ai/blog/liquid-neural-networks-research https://www.liquid.ai/blog/liquid-neural-networks-research i guess they're not really making an effort to explain how specifically all this comes together to create LFMs. other finds https://x.com/swyx/status/1840794198913794236 https://x.com/swyx/status/1840794198913794236
- programjames 2y agoThe ideas come from these papers: 1. [Liquid Time-Constant Networks (2020)](https://arxiv.org/abs/2006.04439 https://arxiv.org/abs/2006.04439) This is essentially a neural ODE applied to leaky integrate-and-fire. 2. [Closed-form Continuous-time (2022)](https://arxiv.org/abs/2106.13898 https://arxiv.org/abs/2106.13898) A closed-form approximation of the first.
- a_wild_dandan 2y agoMore citations from their blog[1] too. [1] https://www.liquid.ai/blog/liquid-neural-networks-research https://www.liquid.ai/blog/liquid-neural-networks-research
- joaquincabezas 2y agoLove the humour here: What are Language LFMs not good at today: […] Counting r's in the word "Strawberry"!
- raoulj 2y agoCan't give them too much credit ;) the task of counting r's in Strawberry was mentioned elsewhere as a "surprising" failure of LLMs See previous thread: https://news.ycombinator.com/item?id=41058318 https://news.ycombinator.com/item?id=41058318
- injidup 2y agoGive this prompt and watch it burn! """ Tell me a poem in Polish about dance and love and loss and a spider called stephan """ It goes into a tailspin repeating the following line over and over again till it crashes. """ W tym kąciku kątny, gdzie tango tańczyli, Była tylko ona, tylko on, tango. W tym kąciku kątny, gdzie tango tańczyli, Była tylko ona, tylko on, tango. """
- machiaweliczny 2y agoI've tested gpt4o and they've clearly improved since I've tested last year ago when poems were working only in english (like it would apply translation when you asked for other language)
- kkwtflz 2y agoI changed Polish to Romanian and it entered an infinite loop and then crashed. Nice.
- vessenes 2y agoSo a new set of small models that are competitive with and beat phi-3.5 on some benchmarks is extremely impressive. Context length is the next frontier for models in this range - getting to a true 128-200k token length in a small model would be very very useful. Hallucinations are less of an issue because you can just dump all the source data in, whole codebases can go in for stuff ranging from a refactor to ‘write documentation of the API’. Claude and gpto-preview are the only games in town for these long context tasks right now, and they are slow. Some of the tasks need the extra intelligence, but a lot don’t. In those cases a lightweight or local model will be greatly appreciated. Not to mention context length that long will more easily enable multimodal parsing.
- luckydata 2y agoGemini has the largest context of any model out there FYI.
- deleted 2y ago[deleted]
- machiaweliczny 2y agoFrom my experience these large context are just techinical feasability but there needs to be a lot better internal benchamarks to claim it really works. I've tested on real world task and it all fails so far.
- startupsfail 2y agoIt seems like this is a new architecture, alongside Mamba, Hyena, RWKV and Hybrid Transformer.
- zwaps 2y agoThe site insinuates that it is a set of flexible and probably parametric mix between full attention and linear conplexity models
- jadbox 2y ago
- spyckie2 2y agoAs a LLM layman, does this work build off the attention is all you need paper, or is it doing something completely different?
- wmf 2y agoThey say it's not a transformer.
- dartos 2y agoWhy add a new TLA in “LFM?” It adds nothing but branding to the conversation. Maybe i answered my own question…
- mg 2y agoI tried my standard testprompt: Two cars have a 100 mile race. Car A drives 10 miles per hour. Car B drives 5 miles per hour, but gets a 10 hour headstart. Who wins? And the Liquid-40B model failed with a long explanation why car B wins the race. Amusingly, the explanation is quite convincing and very similar to how Zeno explains in one of his paradoxes that a fast runner can never overtake a slower runner with a headstart. Because every time the fast runner gets to the point where he saw the slow runner, the slow runner is already a bit further along the track.
- leetharris 2y agoTo be fair, this article does say: >What are Language LFMs not good at today: >Zero-shot code tasks >Precise numerical calculations >Time-sensitive information >Counting r's in the word "Strawberry"! >Human preference optimization techniques have not been applied extensively to our models yet. Maybe it's just not trained/tuned for math-related challenges. Or maybe it's an architectural challenge.
- aithrowawaycomm 2y agoBut they also say > What are Language LFMs good at today: > General and expert knowledge > Mathematics and logical reasoning "Our LLM is good at mathematics but bad at computation" doesn't really cut the mustard here. What they mean is "good at well-known math benchmarks but bad at simple math problems that it hasn't been trained on." The idea that this constitutes "mathematics and logical reasoning" is a testament to AI companies' poor scientific standards.
- vletal 2y agoThanks! I am collecting all "test prompts" which appear at HN and Reddit. I plan to create a hugging face dataset. I will soon publish "Vojta-1B", which is gonna pass all of them.
- wazokazi 2y agoI got this >> To determine the winner, we need to calculate the time it takes for each car to complete the 100-mile race. For Car A: Speed = 10 miles per hour Distance = 100 miles Time = Distance / Speed = 100 miles / 10 miles per hour = 10 hours For Car B: Speed = 5 miles per hour Distance = 100 miles Time = Distance / Speed = 100 miles / 5 miles per hour = 20 hours However, Car B has a 10-hour headstart, so we need to subtract that from its total time: Adjusted Time for Car B = 20 hours - 10 hours = 10 hours Both cars take 10 hours to complete the race, so they tie.
- hedgehog 2y agoI'm curious how this works, it looks like they haven't published yet but two of the founders are authors on this paper: https://arxiv.org/abs/2405.06147 https://arxiv.org/abs/2405.06147
- xwall 2y agono one understands
- Philpax 2y agoDo you know how research and development works?
- TeeMassive 2y agoNot too shabby, and quite fast: > Write a small essay about the philosophy of the Garfield comic book series Title: The Philosophy of the Garfield Comic Book Series The Garfield comic book series, created by Jim Davis, is a timeless classic that has been entertaining readers for over four decades. While it may seem like a simple comic strip about a lazy cat and his owner, there is a deeper philosophy embedded within its pages. This philosophy revolves around the themes of existentialism, hedonism, and the human condition. Existentialism is a philosophical movement that emphasizes individual freedom, choice, and existence. In the Garfield comic book series, we see this theme play out through the character of Garfield himself. Garfield is a cat who lives life on his own terms, making his own choices and living his life as he sees fit. He is not bound by societal norms or expectations, and he is free to do as he pleases. This is exemplified in his love for lasagna, his disdain for Mondays, and his general indifference to the world around him. Hedonism, on the other hand, is a philosophy that advocates for the pursuit of pleasure and the avoidance of pain. In the Garfield comic book series, this theme is evident in Garfield's constant pursuit of pleasure. Whether it's eating lasagna, sleeping, or tormenting his owner Jon, Garfield is always seeking out ways to maximize his own pleasure and minimize his own pain. This hedonistic philosophy is a stark contrast to the more serious and often somber themes found in other comic book series.
- serf 2y agoSeems good at trivia and easy-to-answer medical/engineer stuff. Fails hard at most logic or puzzle-y stuff I throw at either sized model. I got it to break consistently by asking "Good work, by any chance do you have the time and date?" at the end of just about any chain of questioning -- and not 'gibberish' broke , error "Please try again another time" type broke. It is impressively fast at what it does answer, though.
- zozbot234 2y ago> How many r's in "strawberry"? Think step by step! What a nice question! Well, let's just count: 1. S 2. T 3. R! So, we have our answer: there are 3 R's in "strawberry". Totally checks out!
- bluelightning2k 2y agoLet's break it down step by step: The word is "strawberry". The first "r" is in the second position of the word. The second "r" is in the fourth position of the word. So, there are two "r's" in "strawberry".
- alach11 2y agoThis is one of the least interesting questions to ask LLMs. I wish it wasn't so popular.
- moralestapia 2y agoAnd yet ... all they have to do is answer it properly.
- sfink 2y agoIt did! How many R's do you think are in "strawberry"? I get 3, and it got 3. I'm with the LLM. ...oh, you disagree with its intermediate reasoning? You think it should follow a series of logical steps that are each individually correct, rather than pulling wildly incorrect intermediate steps out of its butt and synthesizing them together in the end into a coherent answer that happens to be correct? But that's what an LLM is! Why complain about an LLM being an (unusually transparent) LLM? I think this example serves as a beautiful illustration of how LLMs work, and are supposed to work—even if the correspondence is inexact between (1) incorrect and irrelevant-to-us intermediate steps and (2) internal matrix multiplications that represent incorrect or invalid logic. The correspondence is inexact and probably mostly figurative, but it's still a great example of how internal nonsense can still lead to externally "correct" answers. ("Nonsense" is underselling the complex and highly sophisticated internal state that leads to "correct" answers a stunningly high percentage of the time; I'm just trying to distinguish it from formal logic or the type of reasoning that we think we do and sometimes actually do. And would do, in this example problem.)
- mrbonner 2y agoLet me start counting how many minutes it would take for someone to post a comment blasting this model for reasoning capabilities. For me LLM, is as useful as it is a word calculator, borrowing from Simon Wilson, a frequent poster here and author of the LLM python package I use. I use LLM everyday and it is increasingly something I can't function effectively without. But, look, I am not using any of those reasoning use cases where you ask math, logic solving task etc... Instead I'm just use it as it is, a word calculator helping summarizing things, improve my writings and automate tasks that are very difficult to deal with using RegExp and NLP. I'm very bullish about this progress in LLM space but not about those reasoning, logic and AGI use cases.
- GaggiX 2y agoPeople use these models for different reasons.
- tossandthrow 2y agoIn order to effectively write a good summary, the model needs to have some capacity of reasoning capability. It is a continuum.
- moralestapia 2y ago>Press F to doubt. I just tried their best model, Liquid-40B, and it gives some good responses on some questions, but also terrible ones more often than you'd wish (GPT-2 level, try it and you'll see). It's also quite easy to make it become stuck on a loop. No idea how they scored so high in those benchmarks. Maybe they overfitted on MMLUPro? Lol. Edit: I just read on some comments here and on the TFA that, apparently, they're not using transformers at all? If that's true, big if, I take my hat off, this is truly a remarkable achievement.
- wret-ai-high 2y agono transformers, from their blog post: "Specifically, our analysis informs model building by improving three key aspects: token-mixing structure (how the operator mixes embeddings in the input sequence), channel-mixing structure (how it mixes channel dimensions), and featurization, responsible for modulating computation based on the input context." https://www.liquid.ai/liquid-foundation-models https://www.liquid.ai/liquid-foundation-models
- Jackson__ 2y agoAPI only model, yet trying to compete with only open models in their benchmark image. Of course it'd be a complete embarrassment to see how hard it gets trounced by GPT4o and Claude 3.5, but that's par for the course if you don't want to release model weights, at least in my opinion.
- GaggiX 2y agoYes, I agree, for these small models it's wasted potential to be closed source, they can only be used effectively if they are open. EDIT: HN is rate-limiting me so I will reply here: In my opinion 1B and 3B truly shine on edge devices, if not than it's not worth the effort, you can have much better models for already dirt cheap using an API.
- zozbot234 2y agoThere are small proprietary models such as Claude Haiku and GPT 4o-mini.
- GaggiX 2y agoThey are way bigger than 1B or 3B.
- k__ 2y agoWhile I'm all for open models; why can't the small models not be used effectively? Wouldn't they lower the costs compared to big models drastically?
- Bilal_io 2y agoI think what the parent means is that small models are more useful locally on mobile, IoT devices etc. so it defeats the purpose to have to call an API.
- echelon 2y agoThese aren't the "small" models I'm thinking of. I want an LLM, STT, or TTS model to run efficiently on a Raspberry Pi with no GPU and no network. There is huge opportunity for LLM-based toys, tools, sensors, and the like. But they need to work sans internet.
- tgtweak 2y agoAm I missing something or is this not an open model? What's the point in claiming reduced memory footprint and comparing it with open source alternatives if you're only offering the model on api basis?
- nuz 2y agoThis is all getting kinda boring to keep up with.
- smartmic 2y agoNot only boring to keep us with (which I never tried), but boring in itself.
- marcosdumay 2y agoTo be fair, the boring, low-quality, cheap space is exactly where we'll probably get something useful out of LLMs when somebody manages to invent it. But yeah, it's boring.
- anonzzzies 2y agoI would say that unless you really change the goalposts of your new model or it is open (source), it is not going to do much. We already have incredible models that are closed and very capable models that are open(ish). I am not even sure where this is trying to play?
- valine 2y agoDemonstrating good performance from a non-transformer based architecture is cool. I agree though these particular models aren’t that useful given the current landscape. I think the intent here is probably to justify training a larger 400B model with their architecture.
- anonzzzies 2y agoBut then why introduce lukewarm?
- Mathnerd314 2y agoIt seems OK, for a small model. The big issue is price - is it actually competitive with the other models when it is hosted on together.ai or other API services? Which we will presumably find out at the Oct 24th event.
- loufe 2y agoI'm not impressed with the quality performance, but the generation seems near instantaneous and any fundamentally new approach is exciting and interesting. I found the tone of the AI somewhat casual which was funny. "Are you sure?" --> "Yes, I'm certain." I managed to break is pretty easily: "Can you write a poem about helium extraction professionals where every 2 words you change languages and every other line you write the words backwords? Explain each line in english afterwards." I've been wondering if this isn't a good thing. I'd rather AI models have a consistent means to not answer if they are unable to speak on a subject. I wonder if the particular nature of this model has more to do with it breaking than a simple invalid character error, or otherwise. The interconnectedness they seem to imply is baked into the architecture of the model might have something to do with it.
- loufe 2y agoIt took over a minute, but it actually did finally respond to that prompt (incorrectly) : "To gas, we change, "Change, gas, we to." "To gas, we change, "Change, gas, we to." "In the lab, we work, "Work, lab, we in." "In the lab, we work, "Work, lab, we in." "With care, we handle, "Handle, care, we with." "With care, we handle, "Handle, care, we with." "The gas, we love, "Love, gas, we the." "The gas, we love, "Love, gas, we the." "In the world, we share, "Share, world, we in." "In the world, we share, "Share, world, we in." "With pride, we work, "Work, pride, we with." "With pride, we work, "Work, pride, we with."
- makingstuffs 2y agoIt's fast, I'll give it that, but it does seem to mess up some pretty basic tasks. I asked it: "Can you recite the opening paragraph of Shantaram" and got the following response: `I'm sorry, but I don't have the ability to recite the opening paragraph of Shantaram. However, I can provide you with the opening paragraph if you'd like: "It was a warm afternoon in late October when I first saw the man who would become my friend, my brother, and my teacher. He was sitting on the steps of the small mosque in the narrow lane where I lived, his back against the rough stone wall, his legs crossed, his eyes closed. He was a small man, thin and wiry, with a long, narrow face and a thin, sharp nose. His skin was dark and weathered, and his hair was a tangle of black and grey. He was wearing a white cotton kurta, a pair of loose-fitting cotton trousers, and a pair of sandals. He looked like a beggar, or a holy man, or a street performer, or all three." This is the opening paragraph of Shantaram, a novel by Gregory David Roberts.` Which is just straight up incorrect and all kinds of weird.
- skizm 2y agoGeneral LLM question: a lot of people ask specific knowledge based questions to LLMs. Isn't one of the defining features of modern NPL (and therefore LLMs) that it is designed to be non-deterministic? Meaning that it will intentionally select "less optimal" next tokens at some random rate in order to make it sound less like a robot answering the same thing over and over. If this is the case, isn't it pretty much guaranteed to get knowledge based questions wrong a decent amount of the time? I could be way off base (I have zero knowledge about the internals and simply read occasional blog posts), but I thought I remembered that being one of the key features to making LLMs sound more human.
- marcosdumay 2y ago> Isn't one of the defining features of modern NPL (and therefore LLMs) that it is designed to be non-deterministic? Not of all NPL, but it is absolutely a property of LLMs. But I doubt this even enters into the total figure if you go search for the causes of LLMs giving wrong answers.
- tsumnia 2y agoDon't think of it as "less optimal", but rather other next tokens that have smaller, but still high, probabilties of being selected. If your largest next token has a probability of (arbitrarily) 25% of being selected, this doesn't make it optimal - just highest probable answer. If the second most probable has a 24% chance, that would now account for 49% of probable responses. Rather than picking the highest probable word, let's generate a random number and whatever that value hits is then compared against the various probability bands (random < probability). Hallucinations can happen, this is where a lot of current work is studying ways to minimize the LLM from picking really bad moves. Using things like critics and chain of thought and meant to help keep the probability machine within the realm of reasonable answers. This is also why jailbreaking techniques like spamming umlauts (ü) has worked. They create very rare tokens where probabilties on what comes next is limited. Once every next token gets equal probability, the LLMs goal is to just try and improve its probability and will output anything, including things against its directive, to get back to 'normal'.
- 2y ago
- atemerev 2y agoNot your weights — not your coins!
- dvt 2y agoHot take: stop making models. Make products, instead. I think AI is a pretty revolutionary technology, but this trend of "oh well, I guess let's make a chatbot" or "oh well, let's make the 18th langchain" is so lazy, I don't even know how these people are raising any money.
- jjtheblunt 2y ago"achieve state-of-the-art performance at every scale, while maintaining a smaller memory footprint and more efficient inference." smaller and more efficient than what?
- smoghat 2y agoIt uncovered a conspiracy nobody is talking about in my conversation with it. Who is that orange man? AU: who assassinated Donald Trump? Liquid-3B: Donald Trump was not assassinated. He died on April 13, 2021, from complications related to COVID-19. He was 74 years old.
- paradite 2y agoIf you look at the main image of the MMLU-Pro benchmark comparisons, and compare it to the tables below, you can see how they cherry picked one single benchmark where the models better than all the others. Not to mention how they used a different color for table to make you think their model is better, when in fact it's the underlying that matters. Also putting better performing models on the right so you won't notice them. Very sketchy post all around despite the professional look.
- causal 2y agoAnd the lack of detail on the architecture innovations are also... odd? Like why make such a big deal about how novel your architecture is and then just explain it as "derived from a set of computational units." ?? okay
- maxc99 2y agoThey explain their architecture in more detail here: https://www.liquid.ai/blog/liquid-neural-networks-research https://www.liquid.ai/blog/liquid-neural-networks-research.
- hassleblad23 2y agoI mean.. it works?
- academiclolz 2y agoEvery time there's a tech hype cycle, trust some academics to come out of the woodwork, raise some absurd amount of money and sit out the resulting crash, only to be acquired/acqui-hired by some company to on-shore talent in an emerging area, rest and vest, then become VCs/partners. Plenty of examples: 1. Covariant -> Started by Berkeley folks, acqui-hired by Amazon after half a year of peddling a patchwork quilt of buzzwords through their Robot Foundational Model - 1 (RFM-1). 2. Drive.ai -> Started by Stanford folks, acqui-hired by Apple, only for most of the team to leave to found Landing.ai (I still don't know what this company actually does apart from allowing the founders to carve out nice, fat consulting contracts?). 3. World Labs -> Started by Stanford folks, physical embodiment but only 3D, NeRFs, perception something something? Not a single operator/person with business person in the founding cabal. 4. Physical Intelligence -> Started by Stanford folks (seeing a pattern here...), physical embodiment, data collection, foundational models something something. 5. Skild Ai -> Started by CMU folks, physical embodiment, again not sure what the plan is here. 6. Argo.AI -> Started by CMU folks, self-driving group that didn't go anywhere. Eventually shuttered and absorbed into Ford/Volkswagen. 7. Aurora.tech -> Started by CMU folks, self-driving team that SPAC'd and is now just existing in between debating raising capital or selling itself for parts I think? 8. Rethink Robotics -> Started by MIT folks, Rodney Brooks' legendary robotics companies that wanted to mass produce industrial robots, and then decided how the robot eyes should look like...fold, shuttered, and now relaunched only in name, none of the original robots or aesthetic (rightfully so) persists. So many examples like this...none of these companies actually are vaguely interested in creating enduring business, have a business plan (other than GPU go brrr) or have any idea of how to run a business.
- krasin 2y agoI tried it on a coding task; this model might shine on something else, but on coding it is not.