10 ms·
Small Models Have Arrived
- oybng 2mo agoAn absolutely nothing post at #2 on the frontpage
- glimshe 2mo ago> There's obviously a lot we can optimize here, but if you're charging what the WSJ or The Economist charges, you'd better be delivering similar value. Gosh, watching paint dry has been a better value than reading The Economist in the last 5 years or so. That aside, I had good results with Luna. I'd be interested in hearing about a comparison that takes into consideration response time (not TPS), cost and performance of the popular models at different settings. That chart has some of that. For instance, is Luna Max a better value than Terra Medium?
- yousif_123123 2mo ago[flagged]
- deleted 2mo ago[deleted]
- my-next-account 2mo agoAre ya kidding me
- deleted 2mo ago[deleted]
- swiftcoder 2mo agoI find it quite funny all these folks who are addicted to chasing frontier models, only just noticing that small models became "good enough" for most tasks. Those of us without fable-sized expense accounts noticed this quite a while back
- SomeonesAccount 2mo agoExactly! Composer 2/2.5 were amazing, cheap, and fast. Everyone else was Gaga about GPT 5.5 and such, while we were over here doing the work with less cost and more speed
- sickcodebruh 2mo agoComposer 2.5 is phenomenal for so many tasks!
- jlkuester7 2mo agoExactly. Even 32b parameter models you can run locally on consumer hardware are "good enough" at this point for some workflows!
- jbjbjbjb 2mo agoI’ve been playing around with Luna, Terra and Sol and for the type of work I’ve been doing lately I actually think Sol is just a likely to trip up as Luna. Examples were Sol over assuming, persisting in the wrong direction, over engineering a little script to do some exploration of api. They can all be fixed but it’s a waste of tokens, I rather have Luna do it because course correction on small pieces of work is cheaper.
- scoring1774 2mo agoI've found the distinction to be in how much I care about how the final product looks. If I want high-quality code I typically find a smaller model with a well-designed spec to do better, if I want it to just run and produce something close to my vague description typically Sol does better. For most actual business use-cases I think the first is likely better but the experimentation speed up with the frontier is very nice.
- ZeWaka 2mo agoI've found Luna to be quite capable.
- mchusma 2mo ago
- tosh 2mo agoI think we'll see more of this soon replit is already leading the way with free luna usage
- michael0church 2mo ago[dead]
- LPisGood 2mo agoSmall amounts of world knowledge seems like it would inherently be tied to more hallucinations.
- TJTorola 2mo agoPerhaps we'll get to a point where believing any un-sourced information from an LLM will feel crazy. I don't want my model to know more than it needs to perform logic and use tools. Once it is capable of using tools I would much rather it looked up information or sourced it from existing context rather than just divine it from it's weights.
- pinkmuffinere 2mo agoI empathize, and I have the same preference, but I wonder how this interacts with other people (many of them being our coworkers) using LLMs. There is no authoritative source for the models to pull info from, so either people will have to exercise good judgement and double check important claims, or they will trust too blindly and fall close to the level of whatever LLM they use. In that case, I prefer my coworkers to use an LLM that does have world-knowledge -- I will still hear them spout ridiculous claims, but at least it should be less frequent. It strikes me there's a sort of prisoners dilemma here, where if nobody trusts others to critically evaluate info, it's in our interest to make the tooling do it instead, to whatever degree that is possible. Maybe I'm too cynical about working with others though.
- ImaCake 2mo agoThe problem is it needs world knowledge to know what to lookup. This puts a floor on how little it can know while being able to look up what it doesn't know. Maybe its better if it knows a lot but has a good instinct for verifying that.
- caust1c 2mo agoIMO big models are not a product in and of themselves. Inference is just a new type of compute. I'm confident that in two or three years, every product will have inference capabilities integrated into the experience, and models will become less and less distinctive from one another. What most products need from a model is a pretty short list: the ability to make tool calls well, accurate recall, and the ability to follow directions without wavering (whether or not those directions are baked into the weights or provided in a system prompt). That covers 95% of inference utility in products. We're nearly there, and I believe these capabilities will fit on small models. Because of this though, I predict hardware demand will stay high despite demand for "hosted" inference dropping. Unless there's some regulatory shenanigans that step in to say otherwise.
- NitpickLawyer 2mo ago> But I also think the demand for "fast/cheap/good-enough" models is just about to take off. There's a sort of "revelation" I had in ~early '24 when I used a 7B local model with a library called Guidance (initially out of MS, then the team moved) to create a flow where the model would receive pseudocode for tests, first write the tests, and once I approved then started writing code until the tests passed. This was before "thinking" models, and yet using that library I was able to "guide" the model in the required "prompt / instruct" context such that it was working towards completion, and I saw the first things like we see now in the thinking traces "oh, test x doesn't pass because blah, I need to..." and so on. Anyway, the revelation was "even if the models never improve, I'll have years of fun finding out all the ways I can use these things". And, obviously, the models improved a lot since then. But I think that revelation can still be applied, as a sort of "truism". We have, right now, access to things that 10-20 years ago would be considered magic. We are still finding ways of cobbling together systems with glue, duct tape and prayers and find new things they can do. I think the "good-enough" stage has come not just for API models (cheap, fast, etc) but for local as well. Even if slower, even if clunkier, but they are good enough for a set of ever increasing tasks, and what's more it's incredibly fun to work with them.
- LoveMistral 2mo agoSame. Mistral 7b has been more than I ever needed for text for years now. Unless you must 1-shot with no harness it’s the same amount of power, maybe more because the big “good” models make too many assumptions and tend to become rigid. Mistral 7b can do anything, and it’s basically instant even on an M3
- frigidwalnut 2mo agoSounds interesting. Can you give more details on your workflow and what tasks you use it for?
- LoveMistral 2mo agoCode, creative writing, email summaries, automated email replies, and I prefill my invoice notes and daily updates for work. Actually built a full invoicing product for that, using it too. I use Mistral 7b and LlamaIndexTS on Node, I run it on a MacBook M3 and on a Linux server with only 8GB VRAM (old gaming PC). Basically flawless, runs very fast and I don’t even know what paying for “tokens” is :)
- agcat 2mo agoI like the analogy on ways to make small model useful.
- zatkin 2mo agoMaybe I'm being super reductive here, but operating small models at the core of your business kind of moves the needle from making external API calls (against frontier models) to running internal API calls (against your locally-run models). It seems like if we want local models to take off, it will need to become easier to run local models for cheap. I'm thinking like reducing the barrier of entry for running "local models" in the cloud providers like DigitalOcean, AWS, etc.
- malfist 2mo agoYou should be glad to know digital ocean already offers this
- regularfry 2mo agoIn theory so does AWS, but the Bedrock model selection is badly in need of a refresh.
- spl757 2mo agoI only run local models and I don't give them access to much externally. I don't do anything serious with it, but it comes in handy and I know that they can do so much more. I'm on a meager RTX 3060 12GB and a GTX 1660 Ti with 6GB for some extra vram space. When I first started playing with local models, I was really impressed with what I was able to achieve locally. That's great, but the thing that worries me is that many companies have billions invested in the AI bubble. It's around 1.5 trillion last time I looked. It's all circular spending between the companies building out the infrastructure, and the models. None of it is profitable. They will want to recoup that 1.5 trillion from consumers, which means using online-only pay-as-you-go cloud models. They will inevitably see that people using capable local AI are "lost customers" and they will try to kill the ability to locally host AI or somehow enshitify it enough to make paying a subscription more palatable. I'm not saying I believe that will happen, I'm just worried that it will. Is anyone else worried about that as well?
- mlnj 2mo agoI am very excited that more makers will come up with fast memory for consumers rather than enterprise. Companies can only pre order so much RAM. At some point there will be a surplus of fast memory and even in a crash the current generation of SLMs are bounced to be plenty to build a lot of intelligence at home.
- NickNaraghi 2mo ago> Across his various startups, Peter has seen two kinds of work: > 1. the "IQ 180" work. some mad scientist genius type comes up with some crazy solution you've never thought of. > 2. the "token spewer" work. being ultra responsive, pushing the ball forward across dozens of different fronts. Interesting comp to pg's Maker's Schedule, Manager's Schedule https://www.paulgraham.com/makersschedule.html https://www.paulgraham.com/makersschedule.html I'm curious about not only which of these roles models will fill, but also how they will empower us to be in the mode we prefer.
- kridsdale1 2mo agoI conceive of this as Protoss vs Zerg. I’ve had a lot of success in my career following a Zerg strategy.
- leoh 2mo agoZerg rush
- drxzcl 2mo agoWe call it Ravenclaw vs Hufflepuff. I guess it's a generation thing.
- theshrike79 2mo agoThat's been my goal too, know a bit of everything - enough to know it exists and maybe Know A Guy I can ask if I need to know something more specific. Now in many cases the "Guy" is an LLM. I know enough to know when the LLM is bullshitting me and when I need to consult an actual human.
- charv 2mo ago> July 2009 > Thanks to Sam Altman, … for reading drafts of this. Oh how times have changed!
- satvikpendem 2mo agoHave they? Some say he's still reading those drafts, along with everything else on the Internet.
- wxw 2mo ago100% agreed. Small, cheap, and hosted models. Luna (and open weight models and others) is ridiculously cheap @ $0.2/$1.2, easily accessible, and more than good enough for basic use cases (e.g. summarization, simple tool calling, etc.).
- marius_ 2mo agoI wouldn't call $0.2/$1.2 "ridiculously cheap"
- hartator 2mo agoI have trouble seeing the points of using less capable models. I just want the smartest, best, and most capable models. It feels smaller models for speed and cost are just transitions towards better hardware allowing the very best model.
- trvz 2mo agoFirst, smaller models are fun for hackers: you can run them locally, or run them faster. Second, when cloud models become unavailable or otherwise deteriorate, these will be all you have. May as well prepare.
- breezybottom 2mo agoIf you're hacking a US-based entity, using a high-performance Chinese model through a VPN is probably safe enough. I doubt a local model is going to be sufficiently smart to hack any major company.
- trvz 2mo agoYou misunderstood what I was referring to by “hacker” there.
- polotics 2mo agoCan you define your use of the word 'smartest' here just in case some of us don't quite know what you mean?
- shafyy 2mo agoSome reasons: - Smaller models will always be cheaper - Smaller models will always use less energy, therefore better for the environment It's a bit like saying you always want the fastest and best car; Sure, you can have it if you keep paying for it. But a small car will also get you from A to B, will use less gas and will be much cheaper.
- tartuffe78 2mo agoCost is the point
- jmtulloss 2mo agoI forked my Big Serious Harness™ that models construction projects into a harness for building a vibe coded family assistant. I couldn't figure out how to make the toy operate at toy prices until Luna. Now you can vibe code all the little apps you might want for your fam for like $5 and operate it day to day for a few cents.
- throwaway63467 2mo agoI’m kind of cautiously excited for the next five to ten years, with these AI chips becoming incredibly fast and RAM capacities ramping up its in the cards that we’ll have chips like today’s ATMEL microprocessors that fit on a single board computer and can run small models locally, then all our gizmos can have local AI and I can have a truly intelligent home. Of course there will be a huge push to put all of it in the cloud but maybe we have a chance to take this technology home for good as it’s hard to imagine people will submit to this kind of surveillance required for AI home automation 24/7 (then again I might be wrong). Exciting times.
- mathgenius 2mo agoCan we stick one of these in something that looks and sounds like HAL 9000 ?
- verdverm 2mo agowith wireless tech, you can embody Ai in just about anything, waiting for that hacker post about turning their toaster into a thinking machine, I have a Anki vector I've been meaning to do this with (has camera, speakers, microphone, and screen built in)
- throwaway63467 2mo agoHard to decide between Star Trek computer voice, Glados or Jarvis, HAL would also be cool.
- dTal 2mo agoIs a "truly intelligent home" something I should want? None of the current generation of "smart" addons are true value adds - their entire purpose is data collection. The pretext is always absurdly thin. I just thought "I bet there's a wifi enabled microwave", googled it, and indeed, Samsung have released such a thing - you can control it with your voice! Wow! Never mind that you can't un/load food with your voice, or that you're never more than 10 feet from it in the kitchen anyway. Oh but there's an app. Of course there's an app. It "suggests recipes". Right. I think hoping for a locally hosted "smart" home is backwards. They have no other function than to invade your space. "Smart" objects are agents, and they don't work for you.
- possibilistic 2mo ago> Peter runs multiple companies. Beyond Segment, he's raised $100m+ for Charm Industrial, and just recently closed a Series A for Revoy. He's incredibly organized and efficient with his time. You can do this before an exit? Build and fundraise for multiple (3?) companies at the same time?
- zachthewf 2mo agoSegment had a $3B+ exit to Twilio back in 2020.
- mattmaroon 2mo agoThe demand for fast, cheap, good enough models has always been borderline infinite, it’s the supply that’s going to take off.
- kaiji1126 2mo ago[flagged]
- low_tech_punk 2mo agoThe tokens per second speed measurement is highly inflated nowadays because most of the tokens went into thinking. I wonder if there is a more realistic measurement for "effective speed", which accounts for thinking efficiency.
- ak_t 2mo agoMany benchmarks now measure the total cost or energy usage per completed task.
- deleted 2mo ago[deleted]
- stas4000 2mo ago[flagged]
- hnrprtlpdb 2mo agoWell said
- weinzierl 2mo agoSmall is relative. I'm looking for models that I can with run around 100 MiB mark (RAM just for the weights) to demo what you can do with this little memory. I know of SmolLM 2 which in Q4 is borderline regarding the size and rather dated. There is also TinyStories, which is also old and also focussed on children's stories. Is there anything newer in this category? Or should I try to distill something down to this size?
- laruss5 2mo ago[dead]
- dzonga 2mo agosmall models + a good application layer - are more than enough, good for routine business tasks. the application Layer i.e having a good graph RAG & connecting it up together is the missing piece for most.
- sroerick 2mo agoCan you elaborate on this?
- lantry 2mo agoThe model doesn't have to be smart if all it's doing is pushing a few different buttons. I don't have to be an automotive engineer to start my car and put it in drive.
- sroerick 2mo agoI mean - how specifically are you using knowledge graphs?
- dzonga 2mo agothere's this [0] - Zep. e.g for a lot of small business - they generate invoices - feed that into a RAG system. Then have an agent etc do follow ups. for agents then the smaller models are enough. [0]: https://help.getzep.com/graphiti/getting-started/overview https://help.getzep.com/graphiti/getting-started/overview
- yipinwong 2mo ago"Small models" nowadays work like someone who has IQ 100+ while SOTA ones are like 150, "relatively". Given sheer number of turns I can make with small models, I can do a lotta stufff - cheaper, and faster Harness makes differences: There have been many HN posts about how one made tiny models work better at certain tasks using harnesses. These "small" models with right context, and guidance, they work wonders. --- I've been saying Luna has been my go-to AI in previous comments and why Luna is still more compelling than GLM-5.3-flash. - https://news.ycombinator.com/item?id=49450353#49452248 https://news.ycombinator.com/item?id=49450353#49452248
- highfrequency 2mo ago> the "token spewer" work. being ultra responsive, pushing the ball forward across dozens of different fronts... ~95% of the work he does falls into bucket 2. It's hopping on calls. Nudging people. Blocking and tackling. This is a good insight broadly!
- deleted 2mo ago[deleted]
- pranav_tech26 2mo ago[flagged]
- yeasin-arafat 2mo ago[flagged]
- embedding-shape 2mo agoI love how "Small Models" apparently is "Model of unknown size but probably smaller than another model that we also don't know the size of".
- delis-thumbs-7e 2mo agoI was wondering the same, trying to find if OpenAI has started to release the ampumt of model parametres. Basically they seem to mean ”cheaper”.
- jgthvxevbc 2mo ago[flagged]
- Barbing 2mo agoThis is how your comment displays on my screen, perhaps a typo: >”@dang I really need an IP &/or account ban”
- RanginGfx 2mo ago[dead]
- Zigurd 2mo agoI recently had some relevant experience: for a couple of months now I've been experimenting with on device models to summarize feeds in a Bluesky client I am developing. The feature extracts topic areas, categorizes posts, and creates a summary under each topic. At first the results were hot garbage, and progress was slow. I hooked up the settings to download models from Hugging Face conveniently, so I could run experiments faster, and I massaged the prompts a bit. Last week this feature made a qualitative jump from science experiment to something I'd actually use. The fact that all runs on the device means I've got no variable costs associated with adding this to what will be, at best, a pretty low revenue product. I've tested it on trailing edge devices like an M1 Mac and a Pixel 8, and performance is very tolerable. The key is I'm not asking for open ended answers to open ended problems. When it proves to be useful it's not going to get less useful or more expensive. There are vast domains of uses for LLM models with similar characteristics and likely similar results.
- 2001zhaozhao 2mo agoA dream of mine is to be able to host a LLM-powered video game that I can host on a home server running a decent mid-range GPU like the RTX 5060, and the LLM is fast and intelligent enough to make for a fun game experience for a few dozen concurrent players. People can ask for features and they just get made and added to the game on the fly for the lobby to enjoy. The hosting costs would be manageable enough that I don't have to charge anything for the game. I think with one more year or so of small model progress, that might just be possible to accomplish.
- civvv 2mo ago[flagged]
- U4E4 2mo agoI’m doing this now. Homebrew DnD engine and mobile app chat interface. Rich 3d sim and haptics for the dice rolls. But otherwise no fancy graphics rendering. The juice is in the imaginative generate group story telling. Local real time STT on self hosted video chat, to convert the banter and side talk into next player action dice rolls. Qwen on an m3 ultra 96gb handles NPC, dm, bot party members, arc, consistency, history, memory, adventure beats. There’s even a mechanic for bards to compose an end-of-adventure ballad that compresses the shenanigans into verse. Bootstrapped it with fable, opus or sol 5.6 when I had extra usage to burn over the last month. Playtesting it solo as both DM and PC was fun too. Character consistency and originality is remarkably compelling. Upshot is that for the gaming hours that my friends have available, the social interpersonal gaming experience competes well for their AAA gaming time. There’s even some good async passive campaign continuation patterns that run in our signal thread to keep things going between sync sessions. And players have requested features directly in the session transcript that have been direct implemented.
- 0xdeadfa11 2mo agoI'm very interested in this. There already are a few commercial and open source solutions but none are of my liking. Are you going to publish yours somewhere? I'd love to play Cyberpunk 2020 with AI
- ittsel 2mo agoWatch reasoning tokens though. We tried a small reasoning model that burned ~2800 thinking tokens per call, 3x the cost of a cheaper non-reasoning one despite a better price sheet.
- bsramin 2mo ago[flagged]
- retinaros 2mo agosomehow I cant understand how luna is a step up. to me it feel dumber than 4.1. slower too
- teiferer 2mo agoA friend of mine told me earlier today that they had a discussion at work (a coding shop) about "downgrading" to luna from sol for cost reasons and that many were quite unhappy about this because they didn't want inferior tech to be forced upon them. Do they have a point? Is sol actually worth the extra cost? Especially if you ramp up the effort level?
- dude250711 2mo agoYes. If they don't like the cost then they should fire the "leader" who introduced the AI there to begin with.
- pseudosavant 2mo agoIt is a good question. Luna is definitely a very capable model. Much more capable than the top SOTA models from 12 months ago. It definitely isn't at the same level as Sol, but you get 20x the tokens for the cost, and it has a much faster tokens/second rate. If this is a cost conscious company where I'm going to get a fairly limited amount of Sol, or a nearly unlimited amount of Luna, I'm probably choosing Luna.
- ImaCake 2mo agoLuna on xhigh unlocks gh copilot for me. Sol, even discounted, is too expensive to use and is much slower. Luna on the other hand seems to be so cheap you don't have to think about cost.
- deaux 2mo ago> Much more capable than the top SOTA models from 12 months Confidently wrong here. It's absolutely not across the board "more capable" than GPT 5 or Opus 4.1 or even Gemini 2.5 Pro. It's potentially better at certain specific tasks, mostly agentic coding implementation work. I.e. tool calling and usage of bash. It's worse at a large range of other tasks.
- usef- 2mo agoLuna as a doer, with a smarter model planning, can be a good compromise. Using sol for everything can be expensive without much gain, as a lot of steps don't need that sort of intelligence.
- bartleeanderson 2mo agoNobody is questioning what is meant by small? When you started mentioning frontier models that don't run locally, I just go TB;DR "Too big, didn't read"
- daquisu 2mo agoI read the full post. It is more about cheap models with good enough performance rather than small models. "Local" was not mentioned at all
- fitsumbelay 2mo agousing small local models - with a little bit of extra work - for the first time over the past few days was _really_ illuminating and inspired similar thoughts about how far you can practically get with so little. column of zap emojis, mane ...
- verdverm 2mo agoI only use open models now, I really think the era of open models is upon us, big or small, but I also agree small models reached the point where you don't have hand hold them with qwen 3.8 27B
- anuptalwalkar 2mo agoI kind of agree with your assessment. Running models not just on local, but cheap and lightweight frameworks will drive the next phase. Not trying to plug, but I do't know any other way. I wrote a piece couple of days ago on small models and memory usage on the edge devices- https://polign.com/blog-edge-agent-memory https://polign.com/blog-edge-agent-memory and https://news.ycombinator.com/item?id=49450816 https://news.ycombinator.com/item?id=49450816 closing on the same problem.
- nullbio 2mo agoThe future is self-hosted models trained on your own reasoning traces autonomously, as you sleep, using QLoRA and whatever else.
- dgunay 2mo agoLuna max is suitable for like 90% of the kinds of code changes I want to make. I only find myself actually reaching for a Sol or Fable tier model if the problem is very complex. If you're willing to build the guardrails and do some extra planning, Luna is very capable.
- 1saadcodes 2mo agoThe cost difference is pretty dang nice. Going from roughly a dollar to $0.10 for the same kind of task makes it so that products that didn't make financial sense before become possible
- caruasdo 2mo agoThat's why the market has our solutions but economics.
- a13n 2mo agoRegarding the Pareto frontier and related benchmarks, I have a hard time taking anything seriously that claims that Opus is anywhere near the intelligence of Fable. Are there any benchmarks that haven't just been benchmaxxed that more accurately represent actual usage?
- kakugawa 2mo agoFrontierCode is prob the closest. [1] It's closed source (so no direct benchmaxxing), and it was calibrated by 20+ open source maintainers. It shows Opus 5 (medium), beating out the other reasoning levels by a large margin. i.e. Opus 5 w/ higher reasoning levels actually reduces performance. [2] However, you'll have to gauge for yourself how closely their tasks resemble your tasks. 1/ https://cognition.com/blog/frontier-code https://cognition.com/blog/frontier-code 2/ https://cognition.com/frontiercode https://cognition.com/frontiercode
- zmmmmm 2mo agoThe "good enough" concept is interesting because of how systematically people over estimate it. So often, things that are lower quality but thought to be "good enough" turn out to be either not good enough or not worth it compared to just using the higher quality "thing". I will believe that smaller models have hit that bar empirically when I see them in production. At the moment, even frontier models are stuck in most of the scenarios I am seeing for high value tasks at the "not good enough" gate - so small models are not even close to being on the scene there yet.
- andsoitis 2mo ago> One thing a few investors I've talked with have mentioned: "It's weird we're not seeing more consumer AI companies. Why is that?" What would consumer AI company even be? The frontier labs have declared they will eat everything and they have a head start. Best bet would to be a contrarian and build products and services that people actually want or need. Fine to be AI powered or augmented, but consumer companies do the hard part of understanding specific consumer needs and wants and pursuing that.
- shostack 2mo agoYeah, I think if we reframed it as "why is there no consumer AWS?" It would make more sense. AI is a utility that can abstract code to such a high level it is indiscernible from natural language.
- nonethewiser 2mo agoThere is a thin line. Lots of AI powered things are just workflows you could implement with claude code and some skills. Or even just a prompt on the web chat. But I think these big technological swings sometimes take a long time to shake out. Society is still adapting to the internet. I think there is more opportunity for consumer application of AI. Im surprised we dont see more models in game. Small models that do more purpose driven things not "this npc is just a chatbot with a personality in its system prompt." I mean we literally have systems in games called "AI" and we dont think these much more powerful AIs have a place?
- abustamam 2mo agoI don't know much about game development but given how intensive some AI models can be i feel like adding LLMs to games can have performance issues, whereas in-game AI for computer controlled characters works in-enginer.
- chillfox 2mo agoThe average game dev cycle is about 7 years these days, so expect to wait some more years before you see llms used much in games.
- dev_awesome 2mo agohow effective are the small model?
- senectus1 2mo agonot sure how this is really about model size. I'm keenly interested in seeing super small (like 10's or 100's of mb) special purpose models starting to come into their own. that might actually be transformational globally not just in rich countries.
- nocodeexportcom 2mo agoNice!!
- butterisgood 2mo agoI don’t care about the cost if the results are weak. I did a bunch of work with Sol only to find Opus 5 spot a bunch of bugs. And it was correct. The results weren’t as good as they should have been for Sol. How am I going to trust Luna?
- robinduckett 2mo ago100 t/s of crap is still crap.
- Kim_Bruning 2mo agoIt's kinda fun that people get to experience what the 8-bit era was like!
- scottgk 2mo agoLuna is definitely "good enough" for so, so many use cases. And that price point... Goes to show that open source = cheap, closed source = expensive is wrong-headed. Closed source is going to beat open source on price point due to scale (try buying a few GPUs to run your own model on...) and labs innovating across the full inference stack. That leaves the value prop of open source as privacy / customizability.
- abustamam 2mo agoAs someone who uses a lot of opus, I'm curious what changes to my workflow I need to make to make small models work. My workflow currently is for a large feature I'd explain the feature and how I envision it technically and run the grill me skill where it'd ask me questions that I may not have thought of, then it'll write a spec and execute from there. It spawns an orchestrator so supposedly its supposed to use haiku for mechanical things but im not confident that works. How much of this can be replaced with small models, and what changes to my workflow do i need to make?
- Computer0 2mo agoI’d imagine it’s more focused on individual methods and how they connect
- pornel 2mo agoOpus is good at deciding what to do beyond literal interpretation of the prompt (like checking assumptions before coding). Smaller models need to be told not only what to do, but how to behave. oh-my-pi's Advisor feature can fill a lot of this gap (a smaller model told to reflect on its own output catches when it's doing a rush job and skips instructions).
- kokessch 2mo ago[flagged]
- kokessch 2mo ago[flagged]
- techdecoded 2mo ago[dead]
- MASNeo 2mo agoWe mostly use specialized small models because larger are to expensive/slow and are prone to hallucinating quite a bit. Not sure how this is a surprise, seems more like a best practice.
- whiplash451 2mo agoIs there evidence that small models hallucinate less than big ones?
- jillesvangurp 2mo agoWe've been testing and developing a few openai based applications. Relatively simple stuff. I topped up my api credit with 10$ a few months ago. I just checked, we burned something like 61 cents over this period. About 126 requests over the last month. That's just us testing but it shows me that this stuff is relatively cheap. The reason we don't have a lot of requests is that most of this stuff pretty much works on the first try. This is not hard to build. I think AI is an underused capability in applications. People come up with all these big plans to build agents. But even a simple button to "calculate a color palette that takes into account these symbolic names the user entered" is a simple prompt that comes back quickly and doesn't require a lot of reasoning. We use that with Luna and we give it some more context. Image based stuff is more expensive but still quite reasonable. We do some things with Sol that work on images. It's not fast but the result is worth waiting for. These are features that save the user time and feel a bit magical. They just click a button, wait 20-30 seconds and then magic happens. Beats them clicking around for half an hour. Local model hosting would force me into higher cost and overhead for less intelligence. That's only a good tradeoff if you are really going to use lots of tokens or you need this for other reasons (e.g. data security). But otherwise you just end up paying for underutilized GPU capacity.
- magimas 2mo agoI think in general these smaller applications of AI are seriously overlooked even though they will actually add lots of economic value for companies and lots of quality-of-life features for users.
- paganel 2mo ago> Most of the "human tokens" at companies today are spent this way It sucks that we got to this point when it comes to our civilization. At least guys like this one here can be happy with their AI-sourced money.
- gpiechnik2 2mo agoI hope that one day it will be possible to use models as libraries on the backend (without having to host them somewhere/use a provider)
- incrudible 2mo agoWhere are these 180 IQ models? Even Fable needs guidance so it actually produces output I consider on par with what I could have written myself in a domain I am familiar with, and I am certainly not at the frontier in those domains.
- svantana 2mo agoThe "IQ 180" label seems to be referring to the (human) CTO of their startup.
- wispend26 2mo ago[flagged]
- claud_ia 2mo ago[flagged]
- davidguetta 2mo agonice now do the same for robotics, because itnernet trained vlm + action head was really a dumb take
- cbeach 2mo ago[flagged]
- pranav_tech26 2mo ago[flagged]
- ib33 2mo agoI have been using Sol Medium and it works great. I switched to 5.6 Luna Medium after reading this and I am back to feeling angry and gaslighted. One time, it said "You were right; I corrected the regressions and committed them." It was still not fixed. I do admit not hitting the five hour limit in two prompts is nice. However, I wonder if becoming an angry person because of cheap models is worth it.
- hnub7akvk6 2mo agoDidn't know that
- ozereray1 2mo ago[flagged]
- Myrmornis 2mo agoIs there a resource somewhere that tells me, "Given that you have X Gb RAM on your laptop, these are the current local models you should consider trying and here's how to configure them to leave enough memory for other applications on your machine."?
- blueferret 2mo agoSeveral exist actually. Try whichllm.app or fitmyllm.com.
- rcktmrtn 2mo ago> https://www.whichllm.app/ https://www.whichllm.app/ - Linux, general use case, balance - 16 GB RAM - 10 GB VRAM Recommendation: Kimi-K3 This checks out.
- sandos 2mo agoIt says Kimi-K3 for very set of parameters I put in! 64 or 128GB RAM, 6 or 64GB of VRAM.... Not sus at all.
- tencentshill 2mo agoRoleplay, windows, 16gb/4gb Kimi-K3 I think the slop site is broken or compromised...
- rurban 2mo agowhichllm sucks. It only recommends gguf models, and llama.cpp is not supported. fitmyllm is much better
- crossroadsguy 2mo agoIt gives astronomically optimistic results for my 16GB M1 Pro :) By the way own exploration sort of led me to qwen3.5:9b for the best case scenario balanced model considering almost 10-11GB of RAM is almost always gone anyway. Even with aggressive app quitting.
- MantisShrimp90 2mo agoI think technically sound its just that the frontier models are just sooo in the news that al the money is around intelligence and big breaks in intelligence. Once the economics change and we see less bit model deployments there will be much more work optimizing models at the pareto front. Also yes people need to be able to sustainably run these things to start running a business. Building a business on the model pricing today is as smart as building a retirement plan on a gambling schedule
- maestroquirk 2mo agoWe've been running a small finetuned VLM for OCR and yeah... cost is like 1/3rd of the usual vision APIs. Small models are kinda the whole product for us
- MaxikCZ 2mo agoWhich vlm are you using? Ive found a reddit thread listing like 20 of them, but its hard to find any concrete info on what is worth even tryin.
- n4pw01f 2mo agoI’m getting surprisingly good results on GEMMA4 2 bit, running on a Nvidia nano and raspberry pi
- ewwefwef 2mo agoWhere i can download this luna model? ;D
- conikeec 2mo agoI have been primarily using Codex (high/xhigh) for reasoning/exploring , but at times trivial and repetitive work was getting delegated as well. Like Calvin stated, we built a simple dispatch service that worked with any harness (claude, codex, etc) that could delegate to the cheaper model family trivially. After reading this post, I cleanup the project and decided to open source it. Post: https://modiqo.github.io/spewer/ https://modiqo.github.io/spewer/ Code is here:https://github.com/modiqo/spewer https://github.com/modiqo/spewer