6 ms·
Show HN: We made glhf.chat – run almost any open-source LLM, including 405B
Try it out! https://glhf.chat/ https://glhf.chat/
Hey HN!
We’ve been working for the past few months on a website to let you easily run (almost) any open-source LLM on autoscaling GPU clusters. It’s free for now while we figure out how to price it, but we expect to be cheaper than most GPU offerings since we can run the models multi-tenant.
Unlike Together AI, Fireworks, etc, we’ll run any model that the open-source vLLM project supports: we don’t have a hardcoded list. If you want a specific model or finetune, you don’t have to ask us for it: you can just paste the Hugging Face link in and it’ll work (as long as vLLM supports the base model architecture, we’ll run anything up to ~640GB of VRAM, give or take a little for some overhead buffer).
Large models will take a few minutes to boot, but if a bunch of people are trying to use the same model, it might already be loaded and not need boot time at all. The Llama-3-70b finetunes are especially nice, since they’re basically souped-up versions of the 8b finetunes a lot of people like to run locally but don’t have the VRAM for. We’re expecting the Llama-3.1 finetunes to be pretty great too once they start getting released.
There are some caveats for now — for example, while we support the Deepseek V2 architecture, we actually can only run their smaller “Lite” models due to some underlying NVLink limitations (though we’re working on it). But for the most part if vLLM supports it, we should too!
We figured Llama-3.1-405B Launch Day was a good day to launch ourselves too — let us know in the comments if there’s anything you want us to support, or if you run into any issues. I know it’s not “local” Llama, but, well, that’s a lot of GPUs…
- ec109685 2y agoPretty bait and switchy to let the person sign up and then demand an invite code.
- deleted 2y ago[deleted]
- brianjking 2y agoIn fairness, the invite code is at the top of this post. What concerns me more is the lack of any privacy policy or anything about training on inputs/outputs and much more. The lack of a system prompt seems weird too. How does this compare to OpenRouter.ai? https://openrouter.ai/models/meta-llama/llama-3.1-405b-instruct https://openrouter.ai/models/meta-llama/llama-3.1-405b-instr...
- reissbaker 2y agoIf you want a custom system prompt, try our API! (Click on your profile picture in the corner, then "API settings"). Yeah we should add a privacy policy / ToS: good point.
- latchkey 2y agoand a contact us link. (let's talk about access to GPUs with a lot more vram and no nvlink issues... mi300x, email in my profile)
- reissbaker 2y agoThere is a contact us page! Linked from the profile dropdown. I'll reach out to you though.
- latchkey 2y agoAh. I haven't logged in. =)
- reissbaker 2y agoSorry, the invite code was at the top of the post! (It was "405B" without the quotes.) I just disabled the invite code system entirely, since you weren't the only one who missed it; anyone should be able to sign up now. Definitely didn't mean to bait and switch, we want you to use it!
- imiric 2y agoCongrats on the launch! Your service looks interesting, but I think you need to be more transparent about your infrastructure. To which "inference providers" do you proxy to, and when? Who is hosting the GPU clusters? Also, a privacy policy and ToS document are pretty important, even at this stage.
- brianjking 2y agoyeah, I called this out too. How this compares to OpenRouter is something I'd like to know as well. https://openrouter.ai/models/meta-llama/llama-3.1-405b-instruct https://openrouter.ai/models/meta-llama/llama-3.1-405b-instr...
- adr1an 2y agoiirc openrouter doesn't support any vLLM or huggingface pasted link.
- reissbaker 2y agoGood callout! Currently: 1. If Together.ai has the model, we proxy to them, since they're faster than us. We might switch to Fireworks for the Llama-3.1 models since they offer them at lower cost; in general that's why I didn't specify the inference providers, since we'll probably change and optimize that quite a bit. (Also would be interesting to try using Groq, since they're so fast.) 2. If the model isn't hosted somewhere else, e.g. a lot of the Llama 3 finetunes, we run the models on our own GPU clusters hosted in Fly.io. This will probably change in the future as well, since some models would really benefit from NVLink (which Fly doesn't support currently).
- kcb 2y ago1) Give access to expensive hardware for free 2) ... 3) Profit
- android521 2y agoDo you have api access ? Most people needs to build their own apps with api and will choose you if you have cheaper hosting/api price
- billycao 2y agoHey there! I'm Billy, the other dev working on glhf.chat :) We do have API support! We expose an OpenAI compatible API. You can see details when logged in at https://glhf.chat/users/settings/api https://glhf.chat/users/settings/api Just like our web UI it supports feeding in any huggingface user/repo. (Also available via the user menu) Let us know if you have any questions/feedback!
- lxe 2y agoWhy AWQ instead of exl2?
- reissbaker 2y agoUnfortunately vLLM (the open source inference engine we're built on) doesn't support exl2. That being said I want to add additional backends so we can support exl2 as well!
- Tepix 2y agoLooks promising! However, no privacy policy is a total show stopper.
- billycao 2y ago2nd dev Billy here. Great point. Right now we don't log or store any chat messages for the API (only what models people are choosing to run). We do store messages for the web UI chat history and only share it with inference providers (currently together.ai) per request for popular models, but I know some hand-waved details from an HN comment doesn't suffice. We'll get on that ASAP. :)
- shmatt 2y agoattempt 1 with llama 3.1 405b what version llama are you I am an instance of Llama 2, an artificial intelligence model developed by Meta AI. how many parameters I am a Llama 2 model with 7 billion parameters. Attempt 2 which llama are you exactly I am a variant of the LLaMA model, specifically LLaMA 1.5B. Here's a brief overview of the different LLaMA models: LLaMA 7B: This is the largest and most powerful LLaMA model, with 7 billion parameters. It's the most advanced and capable model in the LLaMA family. LLaMA 3B: This model has 3 billion parameters and is a smaller but still very capable version of the LLaMA model. LLaMA 1.5B: This is the model I'm based on, with 1.5 billion parameters. It's a smaller and more efficient version of the LLaMA model, but still very knowledgeable and capable. LLaMA 650M: This is the smallest LLaMA model, with 650 million parameters. It's a more lightweight and efficient version of the model, but still useful for many tasks. is 3.1 this bad at hallucinations? or are you just serving something else This just keeps on giving when does your training data end " My training data is a massive corpus of text that was available to me up to a certain point in time. My training data is cut off at December 2021."
- Onawa 2y agoPeople continually asking the models what their abilities and knowledge are demonstrates a disconnect in the capabilities of the raw models and what people are expecting them to be capable of. Many of the answers that models give in regards to their capabilities are actually from the system prompts provided from the chat interface providers.
- Zambyte 2y agoI asked mistral-nemo:12b-instruct-2407-q8_0 > Who are you? In my own machine without a system prompt and it said: > I am a language model developed through a collaboration between Mistral AI and NVIDIA. Often times this information will be included in the instruct fine tuning. The same from qwen2:7b-instruct-q8_0: > I am a large language model created by Alibaba Cloud. I'm called Qwen. All I did was change the model and re-run the prompt in Open WebUI, so if there was a system prompt, it would have been used for both.
- reissbaker 2y ago
- langcss 2y agoA quick play and the UX is great. If you are allowing anyone to use 405B by API then you beat Groq to that :) they have limited the access. Probably need to know more about you. Main concerns are "can this be used for production" and "will you be around in a year" and "will you likely have 429s if you get too popular?" and of course likely pricing.
- billycao 2y agoThanks for the feedback! We're working on fleshing out ToS, privacy policy, and company specifics, but just to answer your first question, I'm Billy Cao, an ex-Google eng, and Matt Baker is ex-Airbnb, ex-Meta. Re: concerns, our infra will scale relatively well (several qps per model, probably), but we're still in the stages of fleshing things out and getting feedback. :) Feel free to drop us a line at hi@glhf.chat if you wanted to chat specifics! - Billy
- langcss 2y agoThanks Billy! Nothing specific... just wanted to get the feel. Out of interest are you bare metal or using AWS or something? Or both?
- billycao 2y agoAppreciate the feedback! We currently use fly.io as our cloud GPU provider, but we're actively investigating other providers due to various limitations (like NVLink support).
- sweca 2y agoIf you guys can beat Together AI pricing on LlaMa 3.1 405B, you will be my favourite AI provider
- campers 2y agoI noticed Fireworks.ai is only $3/million compare to $5 for together.ai for Llama3 405b And in Google Vertex its free at the moment!
- sweca 2y agoThanks for the recommendation! Fireworks pricing looks amazing still. $3 for input AND output, vs $3 and $15 for the comparable Claude 3.5 Sonnet from Anthropic.
- SuperHeavy256 2y agoI tried loading Mistal's Mamba Codestral. Didn't work. Got the error: "Application error: a client-side exception has occurred (see the browser console for more information)."
- SuperHeavy256 2y agoLink: https://huggingface.co/mistralai/mamba-codestral-7B-v0.1 https://huggingface.co/mistralai/mamba-codestral-7B-v0.1
- billycao 2y agoHey there! We currently use vllm under the hood and vllm doesn't support Codestral (yet). We're working on expanding our model support. Hence (almost) any model. Thanks for testing! :) https://github.com/vllm-project/vllm/issues/6479 https://github.com/vllm-project/vllm/issues/6479 - Billy :)
- Fantastic9881 2y agoFantastic. Thanks.
- vlugorilla 2y agoPlease, accept cryptocurrency payments! I've found very few LLM providers that can be paid with crypto, and most of them are of bad quality.
- azinman2 2y agoI’m curious your motivation. To me this is the easiest way for scams/bad actors to generate a lot of not-good-for-society content.
- codetrotter 2y agoAnd the US Dollar in cash is the easiest way for drug dealers to get payment for the drugs they are pushing on the street. Yet, the answer to that is not to demand that society goes cash-less. Money will always be used by people you disagree with. And you should not try to use the money itself as a way of limiting what people do. For if you do, one day it could be you that finds himself unable to spend his money the way he wants to because someone else, more powerful than you, disagrees with how you spend that money. Using money that is yours should never be illegal. Prosecute the bad people for the bad thing they did in the first place to get the money / the bad thing that someone paid them money for.
- mathnmusic 2y ago> you should not try to use the money itself as a way of limiting what people do Are you saying that Anti money laundering rules should be scrapped?
- herbst 2y agoIf they don't work, are fully ineffective and generally only catch small fishes I guess we could scrap them.
- newaccount74 2y ago> Prosecute the bad people for the bad thing they did in the first place How do you expect to do that, without following the money? If someone shows up with a suitcase full of money, it's extremely unlikely that they earned this money in a legal way and paid taxes. Same thing with crypto. Theoretically it's possible that people use it for legal means, in practice 99% of people do not.
- tardismechanic 2y agoIn case you were wondering, it supposedly stands for Good Luck Have Fun. It's in the title which is not immediately visible on mobile.
- imafish 2y agoAny gamer would know this by heart ;)
- renewiltord 2y agoIt worked very well. Llama-3.1-405B is really impressive! But then the whole thing stopped working, as more people started using I suppose. Can't wait for the paid offering.
- tspng 2y agoCongratulations on the launch! Unfortunately, I got an error signing up: "You have reached your limit of 500 users. If you need more users, please use a Production instance."
- ssl-3 2y agoMore worryingly: It presents this message after it has collected my personal login credentials.
- jdthedisciple 2y agoSame here, I guess they are using a free trial for the identity provider which is limited to 500 users.
- reissbaker 2y agoWhoops, sorry, we were indeed using the Clerk free tier (which claimed that we had a cap of 10k users, but apparently not?). Fixing that now.
- tspng 2y agoAFAIK 500 is the hard limit for dev instances, 10k is the limit of the free tier for production environments. Btw, there is a stray semicolon on the signup page after the cardbox ;)
- reissbaker 2y agoYes, we have sadly learned this the hard way... Also we can't actually migrate users automatically from dev to prod. T_T Signups are re-enabled, and I'm writing a script now to re-associate everyone's old chat history. Sorry for the launch day struggles!
- billycao 2y agoThis is now fixed, and signups should be working again! Had a misconfiguration with our auth provider, even though we were on the paid plan. Thanks for testing! :) - Billy
- jdthedisciple 2y agoProbably a really dumb question and I missed it somehow, but who is paying for this if it's free?
- reissbaker 2y agoCurrently my credit card is paying for it :P We weren't really sure how to price it, so we're using the beta period for now to figure out what mix of models people are using and trying to figure out reasonable pricing based on that, and also ironing out various bugs and sharp edges. Then we'll start charging for it; personally I'd prefer to have it be usage-based pricing rather than the monthly subscriptions that ChatGPT and Claude use, so that you can treat it more like API access for those companies and don't have to worry about message caps.
- solarkraft 2y agoI’m bit curious: How much are you investing and how set are you on making it profitable (vs. just a cool playground)? It currently reads to me like you’re just having fun with it at the moment.
- reissbaker 2y agoIt's relatively cheap to run at the moment, so I haven't had to invest too much! We're serious about making it work, though, which includes getting it to be profitable.
- nunobrito 2y agoRegistration limit: "You have reached your limit of 500 users. If you need more users, please use a Production instance."
- billycao 2y agoWhoops! We had a hiccup with our auth provider and have just pushed the fix. Sign up should be working again! Thanks for testing! :) - Billy
- GTP 2y agoI just tried opening the link, and got a 500 error. Too many connections from hn?
- billycao 2y agoMight have been a transient error as we were deploying some auth provider fixes. Should be good to go now! :) - Billy
- tcsenpai 2y agoGiving 500 error here, is it me?
- tcsenpai 2y agoNvm it self healed
- akadeb 2y agoCould you clarify what running models multi-tenant means?
- billycao 2y agoBorrowing Matt's words from our reddit thread: It means that we can spin up a single model server and use it for multiple people, effectively splitting the cost. Whereas if you try to rent the GPUs yourself on something like Runpod, you'll end up paying much more since you're the only person using the model. - Billy
- swalsh 2y agovLLM is nice because you can have multiple streams. I believe the tech is called batched attention, but working from memory so I might be wrong. When scaling up inference, this feature is super key. This is the difference between 70 tokens a second and 500 tokens a second using the same compute. My assumption is they're calling this ability multitenant.
- tcsenpai 2y agoThat's really amazing. How can you afford to run it for free? Tbh I'd pay more willingly this than perplexity pro
- billycao 2y agoThanks a bunch! For launch/beta, we're paying for the GPU time ourselves. :) Quoting Matt below: "We weren't really sure how to price it, so we're using the beta period for now to figure out what mix of models people are using and trying to figure out reasonable pricing based on that, and also ironing out various bugs and sharp edges. Then we'll start charging for it; personally I'd prefer to have it be usage-based pricing rather than the monthly subscriptions that ChatGPT and Claude use, so that you can treat it more like API access for those companies and don't have to worry about message caps." Open to feedback here! :) - Billy
- tcsenpai 2y agoThanks for the info! I think implementing both an usage based version for example on the API access and a flat fee for let's say the webapp can be a way to go, to increase flexibility, but is a personal opinion. I like Perplexity model which is like that, I can use the webapp for daily quick searches and use the API for more programmatical access with a more granular price control. Of course then is up to you, anyway I find this to be a great way to do it.
- deleted 2y ago[deleted]
- codetrotter 2y agoI was chatting with the 405B model and got this after a few messages: Application error: a server-side exception has occurred (see the server logs for more information). Digest: 1138364229
- jcuenod 2y agoSeems like setting max_tokens crashes your endpoint.
- codetrotter 2y agoAaaand it's gone :( I only get 503 Error response now when I try to load the page.