22 ms·
Large Enough
- Always42 2y agoI'm really glad these guys exist
- TIPSIO 2y agoThis race for the top model is getting wild. Everyone is claiming to one-up each with every version. My experience (benchmarks aside) Claude 3.5 Sonnet absolutely blows everything away. I'm not really sure how to even test/use Mistral or Llama for everyday use though.
- ldjkfkdsjnv 2y agoSonnet 3.5 to me still seems far ahead. Maybe not on the benchmarks, but in everyday life I am finding it renders the other models useless. Even still, this monthly progress across all companies is exciting to watch. Its very gratifying to see useful technology advance at this pace, it makes me excited to be alive.
- bugglebeetle 2y agoI’ve stopped using anything else as a coding assistant. It’s head and shoulders above GPT-4o on reasoning about code and correcting itself.
- shinycode 2y agoGiven we don’t know precisely what’s happening in the black box we can say that spec tech doesn’t give you the full picture of the experience … Apple style
- LrnByTeach 2y agoSuch a relief/contrast to the period between 2010 and 2020, when the top five Google, Apple, Facebook, Amazon, and Microsoft monopolized their own regions and refused to compete with any other player in new fields. Google : Search Facebook : social Apple : phones Amazon : shopping Microsoft : enterprise .. > Even still, this monthly progress across all companies is exciting to watch. Its very gratifying to see useful technology advance at this pace, it makes me excited to be alive.
- jack_pp 2y agoGoogle refused to compete with Apple in phones? Microsoft also competes in search, phones Microsoft, Amazon and Google compete in cloud too
- deleted 2y ago[deleted]
- satvikpendem 2y agoI stopped my ChatGPT subscription and subscribed instead to Claude, it's simply much better. But, it's hard to tell how much better day to day beyond my main use cases of coding. It is more that I felt ChatGPT felt degraded than Claude were much better. The hedonic treadmill runs deep.
- TIPSIO 2y agoHave you (or anyone) swapped on Cursor with Anthropic API Key? For coding assistant, it's on my to do list to try. Cursor needs some serious work on model selection clarity though so I keep putting off.
- freediver 2y agoI did it (fairly simple really) but found most of my (unsophisticated) coding these days to go through Aider [1] paired with Sonnet, for UX reasons mostly. It is easier to just prompt over the entire codebase, vs Cursor way of working with text selections. [1] https://aider.chat https://aider.chat
- kevinbluer 2y agoI believe Cursor allows for prompting over the entire codebase too: https://docs.cursor.com/chat/codebase https://docs.cursor.com/chat/codebase
- freediver 2y agoThat is chatting, but it will not change the code.
- lifty 2y agoThanks for this suggestion. If anyone has other suggestions for working with large code context windows and changing code workflows, I would love to hear about them.
- 2y ago
- maccard 2y agoAgree on Claude. I also feel like ChatGPT has gotten noticeably worse over the last few months.
- deleted 2y ago[deleted]
- coder543 2y ago> I'm not really sure how to even test/use Mistral or Llama for everyday use though. Both Mistral and Meta offer their own hosted versions of their models to try out. https://chat.mistral.ai https://chat.mistral.ai https://meta.ai https://meta.ai You have to sign into the first one to do anything at all, and you have to sign into the second one if you want access to the new, larger 405B model. Llama 3.1 is certainly going to be available through other platforms in a matter of days. Groq supposedly offered Llama 3.1 405B yesterday, but I never once got it to respond, and now it’s just gone from their website. Llama 3.1 70B does work there, but 405B is the one that’s supposed to be comparable to GPT-4o and the like.
- d13 2y agoGroq’s models are also heavily quantised so you won’t get the full experience there.
- espadrine 2y agometa.ai is inaccessible in a large portion of world territories, but the Llama 3.1 70B and 405B are also available in https://hf.co/chat https://hf.co/chat Additionally, all Llama 3.1 models are available in https://api.together.ai/playground/chat/meta-llama/Meta-Llama-3.1-70B-Instruct-Turbo https://api.together.ai/playground/chat/meta-llama/Meta-Llam... and in https://fireworks.ai/models/fireworks/llama-v3p1-405b-instruct https://fireworks.ai/models/fireworks/llama-v3p1-405b-instru... by logging in.
- J_Shelby_J 2y ago3.5 sonnet is the quality of the OG GPT-4, but mind blowingly fast. I need to cancel my chatgpt sub.
- layer8 2y ago> mind blowingly fast I would imagine this might change once enough users migrate to it.
- kridsdale3 2y agoEventually it comes down to who has deployed more silicon: AWS or Azure.
- Tepix 2y agoClaude is pretty great, but it's lacking the speech recognition and TTS, isn't it?
- connorgutman 2y agoCorrect. IMO the official Claude app is pretty garbage. Sonnet 3.5 API + Open-WebUI is amazing though and supports STT+TTS as well as a ton of other great features.
- machiaweliczny 2y agoBut projects are great in Sonnet, you just dump db schema some core file and you can figure stuff out quickly. I guess Aider is similar but i was lacking good history of chats and changes
- m3kw9 2y agoIt’s these kind of praise that makes me wonder if they are all paid to give glowing reviews, this is not my experience with sonnet at all. It absolutely does not blow away gpt4o.
- simonw 2y agoMy hunch is this comes down to personal prompting style. It's likely that your own style works more effectively with GPT-4o, while other people have styles that are more effective with Claude 3.5 Sonnet.
- athnak 2y agoI would add that the task is relevant too. I feel there’s not yet a model that is consistently better at everything. I still revert to plain old GPT-4 for direct translation of text into English that requires creative editing to fit a specific style. Of all the Claudes and GPTs, it’s the one that gives me the best output (to my taste). On the other hand, for categorisation tasks, depending on the subject and the desired output, GPT-4o and Claude 3.5 might perform better than the other interchangeably. The same applies to coding tasks. With complex prompts, however, it does seem that Claude 3.5 is better at getting more details right.
- skerit 2y agoI don't get it. My husband also swears by Clause Sonnet 3.5, but every time I use it, the output is considerably worse than GPT-4o
- Zealotux 2y agoI don't see how that's possible. I decided to give GPT-4o a second chance after reaching my daily use on Sonnet 3.5, after 10 prompts GTP-4o failed to give me what Claude did in a single prompt (game-related programming). And with fragments and projects on top of that, the UX is miles ahead of anything OpenAI offers right now.
- lostmsu 2y agoJust don't listen to anecdata, and use objective metrics instead: https://chat.lmsys.org/?leaderboard https://chat.lmsys.org/?leaderboard
- PhilippGille 2y agoYou might also want to look into other benchmarks: https://old.reddit.com/r/LocalLLaMA/comments/1ean2i6/the_final_straw_for_lmsys/ https://old.reddit.com/r/LocalLLaMA/comments/1ean2i6/the_fin...
- usaar333 2y agoGPT-4o being only 7 ELO above GPT-4o-mini suggests this is measuring something a lot different than "capabilities".
- Chilko 2y agoAnecdata seems quite valid for LLM comparison when trying to evaluate 'usefullness' for users. The lmsys chat leaderboard is literally just mass anecdata.
- lostmsu 2y agoYes, "mass anecdata" + blind collection is usually called "data".
- deleted 2y ago[deleted]
- harlanlewis 2y agoTo help keep track of the race, I put together a simple dashboard to visualize model/provider leaders in capability, throughput, and cost. Hope someone finds it useful! Google Sheet: https://docs.google.com/spreadsheets/d/1foc98Jtbi0-GUsNySddvL0b2a7EuVQw8MoaQlWaDT-w/edit?usp=sharing https://docs.google.com/spreadsheets/d/1foc98Jtbi0-GUsNySddv...
- hypron 2y agoNot my site, but check out https://artificialanalysis.ai https://artificialanalysis.ai
- harlanlewis 2y agoFamiliar! The Artificial Analysis Index is the metric models are sorted by in my sheet. But their data and presentation has some gaps. I made this sheet to get a glanceable landscape view comparing the three key dimensions I care about, and fill in the missing evals. AA only lists scores for a few increasingly-dated and problematic evals benchmarks. Not just my opinion, none of their listed metrics are in HuggingFace Leaderboard 2 (June 2024). That said I love the AA Index score because it provides a single normalized score that blends vibe-check qual (chatbot elo) with widely reported quant (MMLU, MT Bench). I wish it composed more contemporary evals, but don't have the rigor/attention to make my own score and am not aware of a better substitute.
- mountainriver 2y agoIt’s so weird LMsys doesn’t reflect that then. I find it funny how in threads like this everyone swears one model is better than another
- jorvi 2y agoWhoever will choose to finally release their model without neutering / censoring / alignment will win. There is gold in the streets, and no one seems to be willing to scoop it up.
- usaar333 2y agoI'd rank Claude 3.5 overall better. GPT-4o seems to have on par to better vision models, typescript, and math abilities. llama is on meta.ai
- Zambyte 2y agoI recommend using a UI that you can just use whatever models you want. OpenWebUI can use anything OpenAI compatible. I have mine hooked up to Groq and Mistral, in addition to my Ollama instance.
- fourneau 2y agoClaude 3.5 is a trusted developer partner that will work with you and outline what it’s thinking. It’s not always right but because it outlines its reasoning you too can reason about the problem and catch it. ChatGPT, for me, was a stack overflow solution dump. It gives me an answer that probably could work but it’s difficult for me to reason about why I want to do it that way. Truthfully this probably boils down to prompting but Claude’s out of the box experience is fantastic for development. Ultimately I just want to code, not be a prompt wizard.
- joegibbs 2y ago3.5 Sonnet is brilliant. I use it to write Unreal Engine C++ (which is quite dense and poorly documented) and it destroys Github Copilot and GPT4o. Copilot just has no idea at all except for very obvious next-line suggestions, GPT4o hallucinates a ton of functions, but Sonnet gets it perfect almost every time.
- bugglebeetle 2y agoI love how much AI is bringing competition (and thus innovation) back to tech. Feels like things were stagnant for 5-6 years prior because of the FAANG stranglehold on the industry. Love also that some of this disruption is coming at out of France (HuggingFace and Mistral), which Americans love to typecast as incapable of this.
- tikkun 2y agoLinks to chat with models that released this week: Large 2 - https://chat.mistral.ai/chat https://chat.mistral.ai/chat Llama 3.1 405b - https://www.llama2.ai/ https://www.llama2.ai/ I just tested Mistral Large 2 and Llama 3.1 405b on 5 prompts from my Claude history. I'd rank as: 1. Sonnet 3.5 2. Large 2 and Llama 405b (similar, no clear winner between the two) If you're using Claude, stick with it. My Claude wishlist: 1. Smarter (yes, it's the most intelligent, and yes, I wish it was far smarter still) 2. Longer context window (1M+) 3. Native audio input including tone understanding 4. Fewer refusals and less moralizing when refusing 5. Faster 6. More tokens in output
- drewnick 2y agoAll 3 models you ranked cannot get "how many r's are in strawberry?" correct. They all claim 2 r's unless you press them. With all the training data I'm surprised none of them fixed this yet.
- vorticalbox 2y agoI just tried llama 3.1 8 b this is its reply. According to multiple sources, including linguistic analysis and word breakdowns, there are 3 Rs in the word "strawberry".
- tikkun 2y agoWhen using a prompt that involves thinking first, all three get it correct. "Count how many rs are in the word strawberry. First, list each letter and indicate whether it's an r and tally as you go, and then give a count at the end." Llama 405b: correct Mistral Large 2: correct Claude 3.5 Sonnet: correct
- layer8 2y agoIt’s not impressive that one has to go to that length though.
- unshavedyak 2y ago
- Liquix 2y agoThese companies full of brilliant engineers are throwing millions of dollars in training costs to produce SOTA models that are... "on par with GPT-4o and Claude Opus"? And then the next 2.23% bump will cost another XX million? It seems increasingly apparent that we are reaching the limits of throwing more data at more GPUs; that an ARC prize level breakthrough is needed to move the needle any farther at this point.
- iknownthing 2y agoand even if there is another breakthrough all of these companies will implement it more or less simultaneously and they will remain in a dead heat
- llm_nerd 2y agoPresuming the breakthrough is openly shared. It remains surprising how transparent many of these companies are about new approaches that push the SoTa forward, and I suspect we're going to see a change. That companies won't reveal the secret sauce so readily. e.g. Almost the entire market relies upon Attention Is All You Need paper detailing transformers, and it would be an entirely different market if Google had held that as a trade secret.
- talldayo 2y agoGiven how absolutely pitiful the proprietary advancements in AI have been, I would posit we have little to worry about.
- jsheard 2y agoOTOH the companies who are sharing their breakthroughs openly aren't yet making any money, so something has to give. Their research is currently being bankrolled by investors who assume there will be returns eventually, and eventually can only be kicked down the road for so long.
- 2y ago
- rkwz 2y ago> A significant effort was also devoted to enhancing the model’s reasoning capabilities. One of the key focus areas during training was to minimize the model’s tendency to “hallucinate” or generate plausible-sounding but factually incorrect or irrelevant information. This was achieved by fine-tuning the model to be more cautious and discerning in its responses, ensuring that it provides reliable and accurate outputs. Is there a benchmark or something similar that compares this "quality" across different models?
- amilios 2y agoUnfortunately not, as it captures such a wide spectrum of use cases and scenarios. There are some benchmarks to measure this quality in specific settings, e.g. summarization, but AFAIK nothing general.
- rkwz 2y agoThanks, any ideas why it's not possible to build a generic eval for this? Since it's about asking a set of questions that's not public knowledge (or making stuff up) and check if the model says "I don't know"?
- moralestapia 2y agoNice, they finally got the memo that GPT4 exists and include it in their benchmarks.
- gavinray 2y ago"It's not the size that matters, but how you use it."
- epups 2y agoThe graphs seem to indicate their model trades blows with Llama 3.1 405B, which has more than 3x the number of tokens and (presumably) a much bigger compute budget. It's kind of baffling if this is confirmed. Apparently Llama 3.1 relied on artificial data, would be very curious about the type of data that Mistral uses.
- OldGreenYodaGPT 2y agoI still prefer ChatGPT-4o and use Claude if I have issues but never does any better
- jasonjmcghee 2y agoThis is super interesting to me. Claude Sonnet 3.5 outperforms GPT-4o by a significant margin on every one of my use cases. What do you use it for?
- breck 2y agoWhen I see this "© 2024 [Company Name], All rights reserved", it's a tell that the company does not understand how hopelessly behind they are about to be.
- crowcroft 2y agoCould you elaborate on this? Would love to understand what leads you to this conclusion.
- breck 2y agoE = T/A! [0] A faster evolving approach to AI is coming out this year that will smoke anyone who still uses the term "license" in regards to ideas [1]. [0] https://breckyunits.com/eta.html https://breckyunits.com/eta.html [1] https://breckyunits.com/freedom.html https://breckyunits.com/freedom.html
- christianqchung 2y agoSo it's made up?
- breck 2y agoI do what I say and I say what I do. https://github.com/breck7/breckyunits.com/blob/afe70ad66cfbb6946c5872d9e948012f1da8b236/treenotationPaper.scroll#L124 https://github.com/breck7/breckyunits.com/blob/afe70ad66cfbb...
- crowcroft 2y agoEven if we accept the premise, and the conclusion that this equation 'explains the triumph of open source and public domain software', it explains very little about the relative success of open source/public domain work vs. commercial closed source work. I think history indicates that some commercial works can also triumph.
- 2y ago
- deleted 2y ago[deleted]
- doctoboggan 2y agoThe question I (and I suspect most other HN readers) have is which model is best for coding? While I appreciate the advances in open weights models and all the competition from other companies, when it comes to my professional use I just want the best. Is that still GPT-4?
- tikkun 2y agoMy personal experience says Claude 3.5 Sonnet.
- stri8ed 2y agoThe benchmarks agree as well.
- kim0 2y agoI kinda trust https://aider.chat/docs/leaderboards/ https://aider.chat/docs/leaderboards/
- doctoboggan 2y agoHave you used the actual aider command line tool? That looks interesting, just wondering if it works well.
- ashenke 2y agoI tested it with my claude prompt history, the results are as good as Claude 3.5 Sonnet, but it's 2 or 3 times slower
- Tepix 2y agoJust in case you haven't RTFA. Mistral 2 is 123b.
- validAgent123 2y ago[flagged]
- rkwasny 2y agoAll evals we have are just far too easy! <1% difference is just noise/bad data We need to figure out how to measure intelligence that is greater than human.
- omneity 2y agoGive it problems most/all humans can't solve on their own, but that are easy to verify. Math problems being one of them, if only LLMs were good at pure math. Another possibility is graph problems. Haven't tested this much though.
- tonetegeatinst 2y agoWhat doe they mean by "single-node inference"? Do they mean inference done on a single machine?
- simonw 2y agoYes, albeit a really expensive one. Large models like GPT-4 are rumored to run inference on multiple machines because they don't fit in VRAM for even the most expensive GPUs. (I wouldn't be surprised if GPT-4o mini is small enough to fit on a single large instance though, would explain how they could drop the price so much.)
- bjornsing 2y agoYeah that’s how I read it. Probably means 8 x 80 GB GPUs.
- calibas 2y ago"Mistral Large 2 is equipped with enhanced function calling and retrieval skills and has undergone training to proficiently execute both parallel and sequential function calls, enabling it to serve as the power engine of complex business applications." Why does the chart below say the "Function Calling" accuracy is about 50%? Does that mean it fails half the time with complex operations?
- simonw 2y agoMistral forgot to say which benchmark they were using for that chart, without that information it's impossible to determine what it actually means.
- Me1000 2y agoRelatedly, what does "parallel" function calling mean in this context?
- simonw 2y agoThat's when the LLM can respond with multiple functions it wants you to call at once. You might send it: Location and population of Paris, France A parallel function calling LLM could return: { "role": "assistant", "content": "", "tool_calls": [ { "function": { "name": "get_city_coordinates", "arguments": "{\"city\": \"Paris\"}" } }, { "function": { "name": "get_city_population", "arguments": "{\"city\": \"Paris\"}" } } ] } Indicating that you should execute both of those functions and return the results to the LLM as part of the next prompt.
- Me1000 2y agoAh, thank you!
- huevosabio 2y agoThe non-commercial license is underwhelming. It seems to be competitive with Llama 3.1 405b but with a much more restrictive license. Given how the difference between these models is shrinking, I think you're better off using llama 405B to finetune the 70B on the specific use case. This would be different if it was a major leap in quality, but it doesn't seem to be. Very glad that there's a lot of competition at the top, though!
- RyanAdamas 2y agoPersonally, language diversity should be the last thing on the list. If we had optimized every software from the get-go for a dozen languages our forward progress would have been dead in the water.
- moffkalast 2y agoYou'd think so, but 3.5-turbo was multilingual from the get go and benefitted massively from it. If you want to position yourself as a global leader, then excluding 95% of the world who aren't English native speakers seems like a bad idea.
- RyanAdamas 2y agoYeah clearly, OpenAI is rocketing forward and beyond.
- moffkalast 2y agoConstant infighting and most of the competent people leaving will do that to a company. I mean more on a model performance level though. It's been shown that something trained in one language trains the model to be able to output it in any other language it knows. There's quality human data being left on the table otherwise. Besides, translation is one of the few tasks that language models are by far the best at if trained properly, so why not do something you can sell as a main feature?
- gpm 2y agoLanguage diversity means access to more training data, and you might also hope that by learning the same concept in multiple languages it does a better job of learning the underlying concept independent of the phrase structure... At least from a distance it seems like training a multilingual state of the art model might well be easier than a monolingual one.
- RyanAdamas 2y agoMultiple input and output processes in different languages has zero effect on associative learning and creative formulation in my estimations. We've already done studies that show there is no correlation between human intelligence and knowing multiple languages, after having to put up with decades of "Americans le dumb because..." and this is no different. The amount of discourse on a single topic has a limited degree of usability before redundancies appear. Such redundancies would necessarily increase the processing burden, which could actually limit the output potential for novel associations.
- wesleyyue 2y agoI'm building a ai coding assistant (https://double.bot https://double.bot) so I've tried pretty much all the frontier models. I added it this morning to play around with it and it's probably the worst model I've ever played with. Less coherent than 8B models. Worst case of benchmark hacking I've ever seen. example: https://x.com/WesleyYue/status/1816153964934750691 https://x.com/WesleyYue/status/1816153964934750691
- mpeg 2y agoto be fair that's quite a weird request (the initial one) – I feel a human would struggle to understand what you mean
- wesleyyue 2y agodefinitely not an articulate request, but the point of using these tools is to speed me up. The less the user has to articulate and the more it can infer correctly, the more helpful it is. Other frontier models don't have this problem. Llama 405B response would be exactly what I expect https://x.com/WesleyYue/status/1816157147413278811 https://x.com/WesleyYue/status/1816157147413278811
- mpeg 2y agoThat response is bad python though, I can't think of why you'd ever want a dict with Literal typed keys. Either use a TypedDict if you want the keys to be in a specific set, or, in your case since both the keys and the values are static you should really be using an Enum
- ijustlovemath 2y agoWhat was the expected outcome for you? AFAIK, Python doesn't have a const dictionary. Were you wanting it to refactor into a dataclass?
- wesleyyue 2y agoYes, there's a few things wrong: 1. If it assumes typescript, it should do `as const` in the first msg 2. If it is python, it should be something like https://x.com/WesleyYue/status/1816157147413278811 https://x.com/WesleyYue/status/1816157147413278811 which is what I wanted but I didn't want to bother with the typing.
- avereveard 2y agoimportant to note that this time around weights are available https://huggingface.co/mistralai/Mistral-Large-Instruct-2407 https://huggingface.co/mistralai/Mistral-Large-Instruct-2407
- ilaksh 2y agoHow does their API pricing compare to 4o and 3.5 Sonnet?
- rvnx 2y ago3 USD per 1M input tokens, so the same as 3.5 Sonnet but worse quality
- ThinkBeat 2y agoA side note about the ever increasing costs to advance the models. I feel certain that some branch of what may be connected to the NSA is running and advancing models that probably exceed what the open market provides today. Maybe they are running it on proprietary or semi proprietary hardware but if they dont, how much does the market no where various shipments of NVIDEA processors ends up? I imagine most intelligence agencies are in need of vast quantities. I presume is M$ announces new availability of AI compute it means they have received and put into production X Nvidiam, which might make it possible to guesstimate within some bounds how many. Same with other open market compute facilities. Is it likely that a significant share of NVIDEA processors are going to government / intelligent / fronts?
- teaearlgraycold 2y agohttps://www.youtube.com/watch?v=rvrZJ5C_Nwg https://www.youtube.com/watch?v=rvrZJ5C_Nwg
- modeless 2y agoThe name just makes me think of the screaming cowboy song. https://youtu.be/rvrZJ5C_Nwg?t=138 https://youtu.be/rvrZJ5C_Nwg?t=138
- nen-nomad 2y agoThe models are converging slowly. In the end, it will come down to the user experience and the "personality." I have been enjoying the new Claude Sonnet. It feels sharper than the others, even though it is not the highest-scoring one. One thing that `exponentialists` forget is that each step also requires exponentially more energy and resources.
- toomuchtodo 2y agoI have been paying for OpenAI since they started accepting payment, but to echo your comment, Claude is so good I am primarily relying on it now for LLM driven work and cancelled my OpenAI subscription. Genuine kudos to Mistral, they are a worthy competitor in the space against Goliaths. They make someone mediocre at writing code less so, so I can focus on higher value work.
- bilater 2y agoAnd a factor for Mistral typically is it will give you less refusals and can be uncensored. So if I have to guess any task that requires creative output could be better suited for this.
- thntk 2y agoAnyone know what caused the very big performance jump from Large1 to Large2 in just a few months? Besides, parameter redundancy seems evidenced. Front-tier models used to be 1.8T, then 405B, and now 123B. Would front-tier models in the future be <10B or even <1B, that would be a game changer.
- nuz 2y agoLots and lots of synthetic data from the bigger models training the smaller ones would be my guess.
- duchenne 2y agoCounter-intuitively, larger models are cheaper to train. However, smaller models are cheaper to serve. At first, everyone was focusing on training, so the models were much larger. Now, so many people are using AI everyday, so companies spend more on training smaller models to save on serving.
- deleted 2y ago[deleted]
- erichocean 2y agoI like Claude 3.5 Sonnet, but despite paying for a plan, I run out of tokens after about 10 minutes. Text only, I'm typing everything in myself. It's almost useless because I literally can't use it. Update: https://support.anthropic.com/en/articles/8325612-does-claude-pro-have-any-usage-limits https://support.anthropic.com/en/articles/8325612-does-claud... 45 messages per 5 hours is the limit for Pro users, less if Claude is wordy in its responses—which it always is. I hit that limit so fast when I'm investigating something. So annoying. They used to let you select another, worse model but I don't see that option anymore. le sigh
- mvdtnz 2y agoImagine bragging about 74% accuracy in any other field of software. You'd be laughed out of the room. But somehow it's accepted in "AI".
- kgeist 2y agoWell, we had close to 0% a few years ago (for general purpose AI). I think it's not bad...
- SebaSeba 2y agoSorry for the slightly off topic question, but can someone enlighten me which Claude model is more capable, Opus or Sonnet 3.5? I am confused because I see people fuzzing about Sonnet 3.5 being the best and yet somehow I seem to read again and again in factual texts and some benchmarks that Claude Opus is the most capable. Is there a simple answer to the question, what do I not understand? Please, thank you.
- platelminto 2y agoSonnet 3.5. Opus is the largest model, but of the Claude 3 family. Claude 3.5 is the newest family of models, with Sonnet being the middle sized 3.5 model - and also the only available one. Regardless, it's better than Opus (the largest Claude 3 one). Presumably, a Claude 3.5 Opus will come out at some point, and should be even better - but maybe they've found that increasing the size for this model family just isn't cost effective. Or doesn't improve things that much. I'm unsure if they've said anything about it recently.
- SebaSeba 2y agoThank you :)
- zamadatix 2y agoI think this image explains it best: https://www-cdn.anthropic.com/images/4zrzovbb/website/1f044104447e9db6b22db3a06e45d114f50f274e-2200x1174.png https://www-cdn.anthropic.com/images/4zrzovbb/website/1f0441... I.e. Opus is the largest and best model of each family but Sonnet is the first model of the 3.5 family and can beat 3's Opus in most tasks. When 3.5 Opus is released it will again outpace the 3.5 Sonnet model of the same family universally (in terms of capability) but until then it's a comparison of two different families without a universal guarantee, just a strong lean towards the newer model.
- SebaSeba 2y agoThank you for clearing this out to me :)
- htk 2y agois it possible to run Large 2 on ollama?
- mattsf1998 2y ago[flagged]
- novok 2y agoI kind of wonder why a lot of these places don't give "amateur" sized models anymore at around the 18B & 30B parameter sizes that you can run on a single 3090 or M2 Max at reasonable speeds and RAM requirements? It's all 7B, 70B, 400B sizing nowadays.
- TobTobXX 2y agoJust a few days ago, Mistral released a 12B model: https://mistral.ai/news/mistral-nemo/ https://mistral.ai/news/mistral-nemo/
- logicchains 2y agoBecause you can just quantise the 70B model to 3-4 bits and it'll perform better than a 30B model but be a similar size.
- novok 2y agoA 70B 4bit model does not fit in a 24GB VRAM card, 30B models are the sweet spot for that size of card at 20GB, with 4GB left for the system to still function.
- whisper_yb 2y agoEvery day a new model better than the previous one lol
- philip-b 2y agoDoes any one of the top models have access to the internet and googling things? I want an LLM to look things up and do casual research for me when I’m lazy.
- tikkun 2y agoI'd suggest using Perplexity.
- freediver 2y agoSharing PyLLMs [1] reasoning benchmark results for some of the recent models. Surprised by nemo (speed/quality) and mistral large is actually pretty good (but painfully slow). AnthropicProvider('claude-3-haiku-20240307') Median Latency: 1.61 | Aggregated speed: 122.50 | Accuracy: 44.44% MistralProvider('open-mistral-nemo') Median Latency: 1.37 | Aggregated speed: 100.37 | Accuracy: 51.85% OpenAIProvider('gpt-4o-mini') Median Latency: 2.13 | Aggregated speed: 67.59 | Accuracy: 59.26% MistralProvider('mistral-large-latest') Median Latency: 10.18 | Aggregated speed: 18.64 | Accuracy: 62.96% AnthropicProvider('claude-3-5-sonnet-20240620') Median Latency: 3.61 | Aggregated speed: 59.70 | Accuracy: 62.96% OpenAIProvider('gpt-4o') Median Latency: 3.25 | Aggregated speed: 53.75 | Accuracy: 74.07% | [1] https://github.com/kagisearch/pyllms https://github.com/kagisearch/pyllms
- zone411 2y agoImproves from 17.7 for Mistral Large to 20.0 on the NYT Connections benchmark.
- greenchair 2y agocan anyone explain why the % success rates are so different between these programming languages? is this a function of amount of training data available for each language or due to complexity of language or what?
- h1fra 2y agoThere are now more AI models than javascript framework!
- demarq 2y agoSuper looking forward to this. I tried Codestral and nothing came close. Not even slightly. It was the only LLM that consistently put out code for me that was runnable and idiomatic.
- rldjbpin 2y agothe way these models are being pushed, it seems like more of one-upping each other through iterative improvements than actual breakthroughs. these benchmarks are as good as random hardware ones apple or intel pushes to sell their stuff. in the real world, most people will end up with some modifications for their specific use case anyways. for those, i argue, we already have "capable enough" models for the job.
- xkgask 2y ago[dead]