4 ms·
It's not, its just too much noise to see. We are getting a lot of great models out of China in particular (Yi, Qwen, InternLM, ChatGLM), and some good continua
by brucethemoose2 3y ago
It's not, its just too much noise to see.
We are getting a lot of great models out of China in particular (Yi, Qwen, InternLM, ChatGLM), and some good continuations like Solar.
Lots of amazing papers on architectures and long context are coming out.
Backends are going crazy. Outlines is shoving constrained generation everywhere, and Lorax is a LLM revelation as far as I'm concerned.
But you won't hear about any of this on Twitter/HN. Pretty much the only thing people tweet about is vllm/llama.cpp and llama/mistral, but there's a lot more out there than that.
- Nick87633 3y agoWhere do you like to keep up to date on these? Arxiv preprints, or some other place?
- markab21 3y agoFor Llama-based progress - Reddit - /r/LocalLlama has been my top source of info, although it's been getting a little more noisy lately. I also hang out on a few Discord servers: - Nous Research - TogetherAI / Fireworks / Openrouter - LangChain - TheBloke AI - Mistral AI These, along with a couple of newsletters, basically keep a pulse on things.
- brucethemoose2 3y agoLots of interesting information is so fragmented in niche Discords. For instance, KoboldAI on merging and RP models in general, and some other niches. Llama-Index. VoltaML and some others in regards to SD optimization. I could go on and on, and know only a tiny fraction of the useful AI discords. And yeah, /r/LocalLlama seems to be getting noisier. TBH I just follow people and discuss stuff on huggingface directly now. Its not great, but at least its not discord.
- cyanydeez 3y agosurprised someone doesnt just build an AI aggregator for this type of thing, seems like a real valueable product.
- DANmode 3y agoThose rooms move too fast, and often are segregated good/better/best (meaning the deeper you want to go on a topic, the "harder" it is, politically and labor-wise, to get invited to the server).
- brucethemoose2 3y agoThey have! And posted them on HN! Some are pretty good! Check out this little curated nugget: https://llm-tracker.info/ https://llm-tracker.info/ I used to follow one with a UI that resembled HN itself, but now I can't find it in my bookmarks, lol.
- TeMPOraL 3y agoSpeaking of hard skills: how does one just hang out on a Discord server in any useful fashion? I lost the ability to deal with group chats when I started working full-time - there's no way I can focus on the job and keep track of conversations happening on some IRC or Discord. I wonder what the trick is to use those things as source of information, other than "be a teenager, student, or a sysadmin or otherwise someone with lots of spare time at work", which is what I realized communities I used to be part of consist of.
- brucethemoose2 3y agoThat's pretty much the trick. Discord is so time inefficient, its almost hilarious. For every incredible conversation between experts you observe, you have to weed through 200 times as much filler.
- sanxiyn 3y agoI follow https://www.reddit.com/r/LocalLLaMA/ https://www.reddit.com/r/LocalLLaMA/
- ipaddr 3y agoSubmissions on hn are welcome.
- brucethemoose2 3y agoI have submitted some in the past. Others are submitting them! And I upvote every one I like in /new. But HNers don't really seem interested unless its llama.cpp or Mistral, and I don't want to spam. I can't say I blame them either, there is a lot of insane crypto-like fraud in the LLM/GenAI space. I watch the space like a hawk... and I couldn't even tell you how to filter it, it's a combination of self-training from experience and just downloading and testing stuff myself.
- tmaly 3y agoI see a ton of papers on X related to this space.
- cyanydeez 3y agosubmissions << public interest
- rightbyte 3y agoI did try make a submission of what I thought was a "underreported" LLM two months ago. https://news.ycombinator.com/item?id=38505986 https://news.ycombinator.com/item?id=38505986 Zero interest for some reason. Edit: Deepseek coder has 4 submissions to HN with almost zero interest.
- leereeves 3y agoVery interesting, but the comment you replied to was specifically asking about OpenAI.
- brucethemoose2 3y agoYeah I misinterpreted it open ai as "open source ai" TBH I do not follow OpenAI much. I like my personal models local, and my workplace likes their models local as well.
- alchemist1e9 3y agoLoRAX [0] does sound super helpful and so I’d be curious if there are some good examples of people applying it. What are some current working deployments where one has 100s or 1000s of LoRA fine tuned models? I guess I can make up stuff that makes sense, so that’s not really what I’m asking, I’m interested in learning about any known deployments and example setups. [0] https://github.com/predibase/lorax https://github.com/predibase/lorax
- brucethemoose2 3y agoThere aren't really any I know of, because its brand new and everyone just uses vllm :P No one knows about it! Which is ridiculous because batched requests with loras is mind blowing! Just like many other awesome backends like InternLM's backend, LiteLLM, Outline's VLLM fork, Aphroidte, exllamav2 batching servers and and such. Heck, a lot of trainers don't even publish the loras they merge into base models. Personally we are waiting on the integration with constrained grammar before swapping to Lorax. Then I am going to add exl2 quantization support myself... I hope.
- semmulder 3y agoFYI, vLLM also just added experimental multi-lora support: https://github.com/vllm-project/vllm/releases/tag/v0.3.0 https://github.com/vllm-project/vllm/releases/tag/v0.3.0 Also check out the new prefix caching, I see huge potential for batch processing purposes there!
- brucethemoose2 3y agoMissed this, thanks. Everything is moving so fast!
- tgaddair 3y agoYes, we (LoRAX devs) saw that (we know the author pretty well). It's a useful addition, though quite a bit simpler than our level of support for multi-LoRA inference. We're planning on doing a more comprehensive comparison soon, now that it's officially out. I will say that if you want to explore the forefront of this multi-LoRA inference, definitely worth giving LoRAX a look. We just added support for per-request model merging (https://predibase.github.io/lorax/guides/merging_adapters/ https://predibase.github.io/lorax/guides/merging_adapters/) as an example, and are planning on continuing to double down on this idea of combining adapters in some pretty unique ways.
- ffiirree 3y agoI'm no expert but I assumed Falcon getting released was big news. But I haven't heard anything since. Is that one also pretty good?
- brucethemoose2 3y agoUnfortunately no. Falcon is pretty much worse than other models at the same parameter count.