6 ms·
Laguna XS.2 and M.1
- rohitpaulk 6mo agoBeen testing these via their "pool" agent. It's fast, and the agent adheres to the ACP spec pretty well (better than codex, opencode etc.) so it's a good experience in Zed.
- nateb2022 6mo agoI've tried their "shimmer" site https://shimmer.poolside.ai https://shimmer.poolside.ai (seems in the same vein of products as AI Studio/ Repl.it) Either the harness or the models are very bad in it, I'd say they feels less capable than Gemma4-E2B in virtually any harness. The larger model would plan out some steps and never actually perform them even when prompted to several times. The smaller model actually got more done. My guess is it's the harness, since you've had a good experience. Haven't tried the pool cli yet.
- throwaw12 6mo agoHas anyone tried these models? I like their honesty in benchmarks, looks like Qwen3.6 35B is outperforming their Laguna M.1 225B model
- kingjimmy 6mo agothe color-codes make those benchmarks charts impossible to understand. very pretty though.
- data-ottawa 6mo agoFor what it's worth, the bars correspond in order with the legend. Plus there’s hover text.
- franksiem 6mo agoFelt like they would never come out of stealth mode but very nice to see it materialized into something competitive.
- refulgentis 6mo agoWhat makes them distinctive?
- refulgentis 6mo agoanswering my own question: tl;dr they speak enterprise, we'll train using YOUR code and run on YOUR stuff and we have a Model Factory.
- throwaw12 6mo agoNot sure if this is competitive, look at the numbers for Qwen3.6
- jaen 6mo agoFor similarly sized models, not looking very good on the slightly-less-benchmaxxed Terminal-Bench 2.0: Laguna XS.2 33B-A3B params: 30.6 Qwen 3.6 35B-A3B : 51.5 Devstral 2 123B : 31.2 Quite a huge lead for Qwen... well, at least it's catching up to other smaller Western labs.
- megavon 6mo agoNeed to look at SWEBench-Pro, it's super competitive. Suspect they'll catch up given the longer-tail on TB scores.
- jaen 6mo agoJust by the (lack of) inter-model variance, I don't think SWEBench-Pro does a very good job of representing model capability. Terminal-Bench seems more challenging and separates the wheat from the chaff. Also, *ops work, which in my experience can actually be more complicated than SWE is underrepresented there obviously.
- speedgoose 6mo agoPlease update the charts. Consider using textures or filling patterns. I usually score pretty well in colour perception tests but distinguishing between those two purples made me doubt myself.
- matthewfcarlson 6mo agoMy phone is in grayscale to make it less interesting (I still watch way too many videos in grayscale but it helps) so I’m right with you
- sgottit 6mo agoWe got a lot of feedback about this. We've just pushed new chart components live across the site – thanks. https://poolside.ai/blog/laguna-a-deeper-dive https://poolside.ai/blog/laguna-a-deeper-dive
- esafak 6mo agoThey're not winning any popular benchmark. Is there some niche where it excels?
- vmarkovtsev2 6mo agoWell there are benchmarks, and there is real experience, right? They are not the same.
- deleted 6mo ago[deleted]
- gslepak 6mo agoVery cool to see more small open models being worked on! One nit: I've seen on this homepage, and many others, this notion that the people behind the models are "working towards AGI". I get that this is marketing speak, but transformers are not AGI, and they will never be AGI, so it'd be great if people stopped saying that as it sort of wears out the meaning of "working towards AGI".
- altruios 6mo agoWhat does AGI mean to you? Transformers have approximate knowledge of many things. Is this not 'general'? Where is the goalpost here?
- gslepak 6mo ago> Transformers have approximate knowledge of many things. Is this not 'general'? Of course not. That's like saying the Encyclopedia Britannica is AGI. > What does AGI mean to you? I would define AGI as human-like machine intelligence (or superior). This is difficult for some people to understand because they don't understand what "human-like" means in the first place. Neuroscientists would be able to set some of these wayward computer scientists straight on this question.
- altruios 6mo ago> human-like But is that a hard requirement? Can a machine have Rat-like intelligence? Is all intelligence human-like (human-centric-mind-blindness-much?)? > Of course not. That's like saying the Encyclopedia Britannica is AGI. Well, I'd classify that as GK, general knowledge. Not artificial or intelligent. Let's consider a definition of intelligence as the act of 'manipulating data', have you a better general definition of intelligence?
- gslepak 6mo ago> But is that a hard requirement? Yes. > Can a machine have Rat-like intelligence? Yes, and that would be closer to AGI than today's LLMs, because the fundamental principles and architecture is there.
- simjnd 6mo agoProbably a testament to how good Qwen3.6 is considering Qwen3.6-35B-A3B is not only ahead of their similar weight class XS.2 but also their M.1 (close to 10x bigger at 225B-A23B). Interestingly, Gemma 4 26B-A4B and Qwen3.6 27B (dense) have been left out of the comparison. The smaller models are becoming very good and quantization techniques like importance weighting and TurboQuant on model weights let you run aggressively quantized version (IQ2, TQ3_4S) on consumer hardware with extremely acceptable perplexity and quality loss. Very exciting times for local LLMs.
- deleted 6mo ago[deleted]
- orliesaurus 6mo agoThe colors used in the charts are borderline criminal
- loufe 6mo agoThe order of the bars does not even follow the order in the legend unless I'm mistaken, that's insane.
- vijgaurav 6mo agoThe fact theyre shipping the actual agent harness alongside the weights is the part that matters. Most labs dump the model and make you figure out the agent layer yourself. If its the same runtime they use for RL training, its actually been exercised in production rather than being some demo wrapper.
- simonw 6mo agoPelicans via OpenRouter - the M.1 one is better, neither are particularly great though: https://gist.github.com/simonw/382464026d2e3535986e06437fb6d4e3?permalink_comment_id=6123893#gistcomment-6123893 https://gist.github.com/simonw/382464026d2e3535986e06437fb6d...
- Bolwin 6mo agoI actually like the XS one. It's broken, but it's very aesthetic
- sudb 6mo agoI'm not sure I understand why Poolside are training their own models at all - what's the perceived or real advantage of splitting up model training efforts into smaller companies and dividing up resources like this? Is it just to have a US-domiciled LLM lab?
- lostmsu 6mo agoLimiting resources steers research towards different routes.
- sgottit 6mo agoDesigner @ Poolside here. Thank you for all the feedback – specifically we rebuilt all the eval charts to be clearer. https://poolside.ai/blog/laguna-a-deeper-dive https://poolside.ai/blog/laguna-a-deeper-dive