11 ms·
Llama-3.3-70B-Instruct
- LorenDB 2y agoHopefully this lands on Groq soon!
- 7thpower 2y agoYes, with a high TPM. I see them unboxing all the racks, which is exciting. I would move over to Groq in a New York minute if I could get enough tokens.
- neilellis 2y agoI'm wondering if they are ever actually going to go live (i.e. paid for and reasonable limits), we've been waiting since June!
- jazzyjackson 2y agoForgive me, what slang is "unboxing all the racks" ?
- 7thpower 2y agoThey’re posting images to their socials of them receiving their new racks.
- jazzyjackson 2y agoO so not slang at all then my bad :')
- jpgvm 2y agohttps://twitter.com/sundeep/status/1865094783234142240 https://twitter.com/sundeep/status/1865094783234142240 Enjoy! Disclaimer: I work at Groq.
- sergiotapia 2y agoI work at a YC company, and we would love to use Groq but our limits are so low at one million tokens per day. Could you help us get a better rate limit/token limit? Let me know if I should reach out via email, thank you!
- jpgvm 2y agoI suggest joining the Discord and enquiring there or depending on what you need emailing sales@.
- paxys 2y agoBenchmarks - https://www.reddit.com/r/LocalLLaMA/comments/1h85ld5/comment/m0qauyg/ https://www.reddit.com/r/LocalLLaMA/comments/1h85ld5/comment... Seems to perform on par with or slightly better than Llama 3.2 405B, which is crazy impressive. Edit: According to Zuck (https://www.instagram.com/p/DDPm9gqv2cW/ https://www.instagram.com/p/DDPm9gqv2cW/) this is the last release in the Llama 3 series, and we'll see Llama 4 in 2025. Hype!!
- 85392_school 2y agoFYI, due to Llama's naming scheme, there is no such thing as Llama 3.2 405B. 8B/70B/405B models are either Llama 3, 3.1, or 3.3 (except for 405B which wasn't initially released), while Llama 3.2 only contains 1B, 3B, 11B (vision), and 90B (vision) models. It's a bit confusing.
- paxys 2y agoAh, so I guess the comparison is to Llama 3.1 405B.
- freedomben 2y agoStill very impressive. Llama team is absolutely killing it right now, and the openness makes them the most important player IMHO
- yieldcrv 2y agoyeah I use Llama 3.2 3B and I'm blown away but also wrestled with this mentally. Meta both improves the technology or inference, while also trapping themselves alongside every other person training models to always update the training set every few months, so it knows what its talking about with relevant current events
- blueboo 2y agoIt could be worse. It could’ve been Llama 3.1 (New)
- Lerc 2y ago
- ulam2 2y agoNo base model? disappointed.
- monkmartinez 2y agointeresting comment... what are you doing with base models? Are you a "finetuner"? I have been trying my hand with finetunes on instruct models and the results have been ok, but not awesome. I have a base model downloading now to give that a proper shot.
- eldenring 2y agoIt is probably the same base model as Llama 3.0. They mention postraining improvements.
- paxys 2y agoThe base model is Llama 3.1 70B
- LorenDB 2y agoSeems to be more or less on par with GPT-4o across many benchmarks: https://x.com/Ahmad_Al_Dahle/status/1865071436630778109 https://x.com/Ahmad_Al_Dahle/status/1865071436630778109
- rvnx 2y agoExcept it is 25x cheaper, available offline, can be uncensored / unaligned, fine-tuneable and backupable. Sad day for OpenAI. Great for humanity.
- stainablesteel 2y agozuck is really on his redemption arc, he's out-doing himself
- jadbox 2y agoWould anyone be willing to compress this down to maybe 14b-20b for us on peasant 16gb rigs?
- orangepanda 2y agoOr maybe even middle class plebeian 24gb rigs?
- Narciss 2y agoThis is massive, really cool of meta to open source it
- dartos 2y agoHate to be one of those, but it’s not open source. It’s just regular old freeware. You can’t built llama yourself and it’s license contains a (admittedly generous) commercial usage restriction.
- fabiospampinato 2y agoTo be fair if they released detailed instructions and datasets on how to rebuild llama (considering that there's some randomness in the process) you still probably wouldn't be able to build it, like who has the resources? And if you had the resources you probably _still_ probably wouldn't _want_ to rebuild it yourself, it seems awfully expensive when you could instead spend those resources elsewhere. Fair point about the license, people have different definitions for what "open source" means.
- do_not_redeem 2y agoThat's true for most people for ordinary software too. How many people actually build Linux or Chromium from source? Building Chromium takes more RAM and HD space than most people even have. Yet the world gets immense value from the few who do. I wouldn't want to live in a world where WebKit and Chromium were closed source. You can run a Chromium fork without having to build it yourself. And compute costs will come down over time.
- fabiospampinato 2y agoLinux and Chromium seem at the edge of the current scale of "ordinary" open-source software. I think perhaps one should also take into account how much money would be needed to be able to build the thing in reasonable time. Building Chromium sounds awful, but I'm not sure I'd really need to buy another computer for that. If I did I'm sure I wouldn't need to spend billions on it, most probably not even millions. For LLaMa I definitely don't have the computer to build it, I definitely don't have the money to buy the computer, even if I won the lottery tomorrow I'm pretty sure I wouldn't have enough money to buy the hardware, even if I had enough money to buy the hardware I'm still not sure I could actually buy it in reasonable time, nvidia may be backlogged for a while, even if I already had all the hardware I probably wouldn't want to retrain llama, and even if I wanted to retrain it the process is probably going to take weeks if not months at best. Like I think it's one of those things where the difference in magnitude creates a difference in kind, one can't quite meaningfully compare LLaMa with the Calculator app that Ubuntu ships with.
- profsummergig 2y agoPlease help me understand something. I've been out of the loop with HuggingFace models. What can you do with these models? 1. Can you download them and run them on your Laptop via JupyterLab? 2. What benefits does that get you? 3. Can you update them regularly (with new data on the internet, e.g.)? 4. Can you finetune them for a specific use case (e.g. GeoSpatial data)? 5. How difficult and time-consuming (person-hours) is it to finetune a model? (If HuggingFace has answers to these questions, please point me to the URL. HuggingFace, to me, seems like the early days of GitHub. A small number were heavy users, but the rest were left scratching their heads and wondering how to use it.) Granted it's a newbie question, but answers will be beneficial to a lot of us out there.
- mhh__ 2y agoYes (don't know about JupypterLab), skip, not really, yes, quite irritating so just pay someone else to do it.
- profsummergig 2y agoThanks! Succinct and complete.
- joshhart 2y agoHi, Yes you can. The community creates quantized variants of these that can run on consumer GPUs. A 4-bit quantization of LLAMA 70b works pretty well on Macbook pros, the neural engine with unified CPU memory is quite solid for these. GPUs is a bit tougher because consumer GPU RAM is still kinda small. You can also fine-tune them. There are lot of frameworks like unsloth that make this easier. https://github.com/unslothai/unsloth https://github.com/unslothai/unsloth . Fine-tuning can be pretty tricky to get right, you need to be aware of things like learning rates, but there are good resources on the internet where a lot of hobbyists have gotten things working. You do not need a PhD in ML to accomplish this. You will, however, need data that you can represent textually. Source: Director of Engineering for model serving at Databricks.
- profsummergig 2y ago
- hrpnk 2y agoSeems that a bunch of quantized models are already uploaded to ollama: https://ollama.com/library/llama3.3/tags https://ollama.com/library/llama3.3/tags
- henry2023 2y agoI'm building a PC just to run inference on this and the QwQ 32B models. Any suggestions on RAM and GPU I should get?
- griomnib 2y agoI have an A6000, it’s about the most affordable for 48 GB VRAM (you can find for a little under $5k sometimes), which is roughly minimum to run a quantized 70b. System RAM doesn’t really matter, but I have 128GB anyway as RAM is pretty cheap.
- paxys 2y agoWhy not 2 x 4090? Will be cheaper than A6000 if you can manage to find them at msrp, and will perform a lot better.
- griomnib 2y agoMy time is worth a lot of money and 2x 4090 is more work, so it’s net more expensive in real terms.
- BoorishBears 2y agoFor both inference and training I haven't seen any modern LLM stack take more time for multiple GPUs/tensor parallelism I would take 1 RTX 6000 Ada, but if you mean the pre-Ada 6000, 2x4090 is faster for minimal hassle for most common usecases
- griomnib 2y agoI mean the newest ones. I only do LLM inference, whereas my training load is all DistilBERT models and the A6000 is a beast at cranking those out. Also by “time” I mean my time setting up the machine and doing sys admin. Single card is less hassle.
- 2y ago
- hubraumhugo 2y agoMeta continues to overdeliver. Their goal from the start was to target and disrupt OpenAI/Anthropic with a scorched earth approach by releasing powerful open models. The big winners: we developers.
- ben30 2y agoThis reminds me of Steve Jobs's famous comment to Dropbox about storage being 'a feature, not a product.' Zuckerberg - by open-sourcing these powerful models, he's effectively commoditising AI while Meta's real business model remains centred around their social platforms. They can leverage these models to enhance Facebook and Instagram's services while simultaneously benefiting from the community improvements and attention. It's not about selling AI; it's about using AI to strengthen their core business. By making it open, they get the benefits of widespread adoption and development without needing to monetise the models directly.
- lolinder 2y agoAlso don't underestimate the value they're getting from making more overtures to the developer community. It could be a coincidence, but it's only since they started releasing these models that I started noticing people on HN calling them "Meta", and attitudes towards them have been far more positive of late than usual. Good will isn't worth as much as cheap moderation automation and fancy features, but it's worth something.
- ecocentrik 2y agoIt would be strange if they didn't also use these models to generate much more sophisticated models of their user's interests and hyper-targeted advertising that always looks and feels like trusted friend recommendations for the exact product that's been missing from your life.
- lolinder 2y agoThere's the old HN! I knew it was around here somewhere!
- sangnoir 2y ago> It would be strange if they didn't also use these models to generate much more sophisticated models of their user's interests and generate the hyper targeted advertising You have the sequence reversed as Meta already created ad targeting models. Meta was forced to scale its AI competence for ad targeting when Apple sent out a privacy update that destroyed tracking-based ad-serving and tanked Meta's share price by deleting billions in revenue for Meta over many quarters. Now that Meta has this skill as a core-competence, they are creating new models for public release. Why they are doing so is debatable[2], but I imagine the cost is marginal since they already had the GPU clusters, talent and know-how for survival purposes. 1. https://www.businessinsider.com/metas-bet-on-ai-has-saved-its-ad-business-from-apples-privacy-update-2023-5 https://www.businessinsider.com/metas-bet-on-ai-has-saved-it... 2. I suspect Zuckerberg is not enthused by the idea of a future AI Apple-analog unilaterally shutting him out of the market. Having your net worth cut in half by a press-release has got to hurt.
- theanonymousone 2y agoI'm "tracking" the price of if 1M tokens in OpenRouter and it is decreasing every few refreshes. It's funny: https://openrouter.ai/meta-llama/llama-3.3-70b-instruct https://openrouter.ai/meta-llama/llama-3.3-70b-instruct
- philipkiely 2y agoJust spent a few minutes this morning spinning up a H100 model server and trying an FP8 quantized version (including kv cache quantization) to fit it on 2 H100s -- speed and quality looking promising. I'm excited to see if the better instruction following benchmarks improves function calling / agentic capabilities.
- bnchrch 2y agoOpen Sourcing Llama is one of the best example and roll out of "Commoditize Your Complement" in memory. Link to Gwern's "Laws of Tech: Commoditize Your Complement" for those who havent heard of this strategy before https://gwern.net/complement https://gwern.net/complement
- bazmattaz 2y agoThat was so interesting. Thanks for sharing
- theanonymousone 2y agoGiven the comments saying it's performance seems comparable to 4o/4o-mini, is it safe to say that GPT-4 performance can be achieved with less than 100B parameters,in contrary to what previously was thought?
- adt 2y agoModel card: https://github.com/meta-llama/llama-models/blob/main/models/llama3_3/MODEL_CARD.md https://github.com/meta-llama/llama-models/blob/main/models/... On the Models Table: https://lifearchitect.ai/models-table/ https://lifearchitect.ai/models-table/
- andy_ppp 2y agoHow many tokens per second can I get on an M4 Max with 128gb of RAM?
- deleted 2y ago[deleted]
- variety8675 2y agoI get about 10
- freediver 2y agoDoes unexpectedly well on our benchmark: https://help.kagi.com/kagi/ai/llm-benchmark.html https://help.kagi.com/kagi/ai/llm-benchmark.html Will dive into it more, but this is impressive.
- stavros 2y agoI asked it: > I have a sorcerer character on D&D 5e and I've reached level 6. What do I get? It confabulated a bunch of stuff. I also asked GPT-4, it confabulated a bit. Claude was spot on.
- danielhanchen 2y agoI uploaded 4bit bitsandbytes, GGUFs and original 16bit weights to https://huggingface.co/unsloth https://huggingface.co/unsloth for those interested! You can also finetune Llama 3.3 70B in under 48GB of VRAM and 2x faster and use 70% less memory with Unsloth!
- ndr_ 2y agoIt's available on IBM WatsonX, but the Prompt Lab may still report "model unavailable". This is because of overeager guardrails. These can be turned off, but the German translation for this option is broken too: look for "KI-Guardrails auf" in the upper right.
- antirez 2y agoHot take after trying it a bit. I was not impressed with llama 3.2, but this one, well, it looks like we finally have a very very strong free LLM.
- kstrauser 2y agoI know this has been discussed before but it changes frequently: what’s the good “generic” Mac desktop client these days? I’d like to use Ollama and/or ChatGPT. Maybe Claude. Perhaps Perplexity, too. I primarily want to use AI chats in various apps, like typing “write a function to…” into whatever random editor I’m using at the moment. It doesn’t have to be a desktop app, either. If there’s a great PopClip plugin or Keyboard Maestro macro, or even something that works as a system service, that’s perfectly fine by me. MacMind is nifty, but that feels like a lot of money for something that’s a front end to someone else’s API. “Stop being a cheapskate” is a legitimate answer.
- cgarvis 2y agoChatGPT has a Mac app. If you want to run your own models, Ollama with Chital is great. https://github.com/sheshbabu/Chital https://github.com/sheshbabu/Chital
- kstrauser 2y agoThe ChatGPT app only has the nifty "works with any app" feature if you're on the Plus plan and doesn't support other models. Chital looks very nice and is admirably tiny (3.6MB unzipped!) but doesn't have the editor integration at all, as far as I can tell. MindMac is an example of an app that meets all the criteria, but for more than it seems like such an app should cost.
- anon373839 2y agoI like Open WebUI. It’s a web app that runs locally, not a native MacOS app. But it’s very nice. https://github.com/open-webui/open-webui https://github.com/open-webui/open-webui
- einr 2y agohttps://github.com/AugustDev/enchanted https://github.com/AugustDev/enchanted
- ppp999 2y agoWe need more uncensored models
- knighthack 2y agoGiven how censored the 3.2 model was, is I'm looking forward to the abliterated 3.3 version to see if there's any significant improvements there that can replace it.
- nxobject 2y agoI'm surprised that, out of all of the East Asian languages, they chose Thai to support: do they have a big office there? (I imagine compared to, say, Japanese or (some form of) Mandarin?)
- kordlessagain 2y agoSummary of discussion: https://claude.site/artifacts/635d6816-9f60-4545-aeed-54ba180cfd5e https://claude.site/artifacts/635d6816-9f60-4545-aeed-54ba18...