7 ms·
GPT-4, without specialized training, beat a GPT-3.5 class model that cost $10M
- chintler 3y ago$10 Million(M), not $10 Billion(B).
- CharlesW 3y agoThank you, fixed! Also, direct link to paper: https://arxiv.org/pdf/2305.05862.pdf https://arxiv.org/pdf/2305.05862.pdf
- affgrff2 3y agoNot looking forward for the times when an AI costs as much as an aircraft carrier.
- moffkalast 3y agoAt least with an aircraft carrier you can make your money back by holding a small country for ransom, har har.
- MuffinFlavored 3y agoDoes anything currently beat GPT-4? I saw some comments here say to check out Claude. From what I can tell, Claude hasn't figure out yet how to do the whole "generate Python code and run it in a Juptyer notebook" for math yet.
- jxdxbx 3y agoClaude is better than GPT 4 for my uses, and was able to help me do some simple coding things that GPT 4 could not. It’s worth trying at least.
- monsieurbanana 3y agoIsn't that something you get from the infrastructure surrounding the llm? I thought the "running code" feature didn't need specific support from the llm, besides being able to output conforming json or code when asked to.
- MuffinFlavored 3y agoThe LLM (Claude) currently doesn't know to not hallucinate numbers and instead write code + run it (something ChatGPT used to do but they fixed it)
- avree 3y agoDoesn't seem like you are very informed on how LLMs work, but just so you know, there are many different versions of Claude, just like how ChatGPT can use different versions of GPT.
- simonw 3y agoThat's because the Claude web UI doesn't yet have the equivalent of the ChatGPT Code Interpreter tool (though they say they're working on it). That's not about the quality of the Claude 3 Opus model, which is the model which people think compares to or beats GPT-4. It's about the tooling that has been built for ChatGPT.
- fragmede 3y agoCode interpreter is pretty neat, because you can tell ChatGPT to write some code and to make sure the code works, and then it'll write you some bad code, realize it's bad, and then iterate on it until it gets to a place that it's happy with. (Maybe I should say passes its test rather than anthropomorphize ChatGPT as being "happy".)
- cosmojg 3y agoRight, like the other commenter suggested, that's an infrastructure-level thing, not a model-level thing. Given that you're talking about ChatGPT, I assume you aren't accessing GPT-3.5 or GPT-4 directly through the API but using the app or the interface provided at chat.openai.com. The magic that makes the kinds of interactions you're describing possible amounts to a bit of clever prompting sprinkled on top of some rather impressive frontend design and engineering. Correctly prompted, even Mistral-7B can write and run code in response to questions, and it's a model that can run on laptops from half a decade ago, with two or three orders of magnitude fewer parameters that GPT-4.
- rubymamis 3y agoA programming task where Mistral-large beats both GPT-4 and Claude Opus: https://sdk.vercel.ai/s/Thi7RDx9e8VOZo1Ee6We5 https://sdk.vercel.ai/s/Thi7RDx9e8VOZo1Ee6We5 (only Mistral got the current syntax) Although based on other tasks, overall, GPT-4 seems to be the best, but by a very small margin, so I cancelled my subscription. Although the native mobile app is really great.
- cosmojg 3y agoDo you prefer Mistral-Large or Claude-Opus?
- rubymamis 3y agoNot sure. Most of the time GPT-4 is better. Since I'm using Vercel AI playground[1], on almost every query I get a response from all models so it's easy to compare. [1] https://sdk.vercel.ai/ https://sdk.vercel.ai/
- fragmede 3y agoIs there a way to use Mistral-large with TTS and STT engines so you can converse with it like you can ChatGPT in the mobile app? it's really great on long drives for learning/talking about stuff, like a customized personal podcast.
- rubymamis 3y agoExactly, I absolutely love this feature. And many times the conversation is quite natural and fluid (with good internet connection). I think I'll build something like that myself (:
- geor9e 3y agoI don't know what system and user prompt you are testing with, but as one anecdote, Claude 3 Opus (and only Opus) consistently gives me better coding answers than GPT-4. Maybe it's the type of stuff I am doing or how I phrase things, who knows. I was using GPT-4 since the day it came out but haven't felt like going back so far.
- treprinum 3y agoClaude-2 in some tasks albeit it's a bit slower, Mistral on some tasks and it's a bit faster.
- yieldcrv 3y agoThe benchmark is Sora or whatever Open AI is working on right now or next, not trying to beat the model released a year ago and still failing so when looking at it that way, the real question is what do you need? all I need is Mixtral 7x8B Q5 in an 8,000 token context window, at the moment I think there are plenty of other people that can design their applications and problems around lower fidelity tools, or just pursue something else
- jasonjmcghee 3y agoClaude Opus (largest v3 model) consistently outperforms GPT-4 for me. Better at following prompts, _feels_ much better.
- thorum 3y agoI don’t know what the people who say Claude 3 is better than GPT-4 are using it for. It’s been consistently worse for everything I’ve thrown at it. Debugging a Python function this morning. Claude 3 Opus failed completely. GPT-4 found the bug, as well as two others I hadn’t even been looking for.
- simonw 3y agoI've had the opposite experience: coding prompts that GPT-4 makes mistakes on Claude 3 Opus gets right the first time. As always, your results will vary based on your personal prompting style. My style apparently works great with Opus. Here's one example: GPT-4 gave me code that was missing some async/await keywords: https://chat.openai.com/share/117fb1ad-6361-41e2-be59-110f3262594d https://chat.openai.com/share/117fb1ad-6361-41e2-be59-110f32... Claude 3 Opus with the same prompt got it right the first time: https://gist.github.com/simonw/2002e2b56a97053bd9302a34e0b83074 https://gist.github.com/simonw/2002e2b56a97053bd9302a34e0b83...
- brianjking 3y agoYeah, Opus has entirely taken over any code specific use for me over ChatGPT 4 or OpenAI GPT-4 API. Once Opus has the ability to run a code interpreter, it'll really be an exciting time.
- drexlspivey 3y agoAccording to Chatbot Arena where people vote on responses blindly and an ELO rating is determined for each LLM, gpt4 is on top slightly ahead of Claude 3 Opus https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboar...
- marviel 3y agoClaude is excellent for brainstorming, being a thought partner, general knowledge acquisition tasks, and creative writing. The one mixed-bag weak spot I've found is in coding -- It tends to make more "d'oh" mistakes while coding, but comes up with more creative solutions at the same time ¯\_(ツ)_/¯
- dragonwriter 3y ago> Claude hasn't figure out yet how to do the whole "generate Python code and run it in a Juptyer notebook" for math yet. GPT-4 didn't figure that out, either; that’s just tooling built around the model, not something the model “figures out”.
- jebarker 3y agoHow do they know GPT-4 received no specialized financial training?
- CharlesW 3y agoMeaning, they used the same generalized foundation model that all of us have access to, with no special fine-tuning, no retrieval-augmented generation, etc.
- jebarker 3y agoI don't understand your point. To me GPT-4 is not a foundation model, it's been highly tuned for the chat task. Nobody outside of OpenAI knows what that fine-tuning really involved. So it's impossible to say how much finance specific data it was trained on (in pre-training or fine-tuning) or whether finance specific tasks were involved in fine-tuning.
- CharlesW 3y ago> To me GPT-4 is not a foundation model… It is. https://www.adalovelaceinstitute.org/resource/foundation-models-explainer/ https://www.adalovelaceinstitute.org/resource/foundation-mod...
- deleted 3y ago[deleted]
- jebarker 3y agoWhat I was meaning was that ChatGPT is not a foundation model since it's been fine-tuned. Although the definition in the link is sufficiently broad you could choose to include it. I can't tell from the OpenAI docs whether it's possible to access GPT-4 without the ChatGPT fine-tuning. If so, that'd make this result more meaningful. Otherwise, I just don't think you can draw any great conclusions from this.
- 3y ago
- __loam 3y agoGPT-4 cost like $100m so I don't think this is surprising?
- rafaelero 3y agoA lot of organizations still think they should have their own [finetuned] model to provide a custom experience to their users, so that may come as a surprise for them.
- ShamelessC 3y agoScaling laws basically guarantee that a sufficiently larger general model will usually beat a smaller specialist model. The misunderstanding is perhaps acceptable but the headline here is essentially restating a well known property of deep learning.
- __loam 3y agoHow long ago was the Bitter Lesson written?
- mistrial9 3y agocontrarian view - how these models actually operate at runtime is not understood.. the formal research papers repeat that over and over again. Therefore, there will be new twists and turns as these models evolve. With current technology stacks, the "bitter lesson" is looking good, yes. Will it always be so? no way to know it.
- minimaxir 3y agoExtremely hot LLM take: You will often get better results with few-shot prompting (with good examples) on a modern LLM than with a finetuned LLM. Finetuning was the best option for weaker LLMs with lower context windows (e.g. the original GPT-3): both problems have been solved nowadays. The cost economics are much better with few-shot prompting to modern LLMs too: input tokens are super cheap (especially with the recently-released Claude Haiku), so giving a lot of examples per call will still end up cheaper than finetuning. Meanwhile, a finetuned ChatGPT costs 4-6x of normal ChatGPT usage.
- viksit 3y agocan you give a few pointers on articles or examples of this?
- minimaxir 3y agoA low-tech example to create a good blog post title for submission to Hacker News would be a system prompt like: You are an expert copywriter. Write five distinct blog post titles optimized for high clickthrough for Hacker News for the article the user provides. Your response must follow the style of these titles: - The ü/ü Conundrum - Why isn't preprint review being adopted? - Majority of web apps could just run on a single server - Weather Planning for Eclipse Day - PSChess – A chess engine in PostScript Then provide the blog post as the user message input. I just ran one of my blog posts (https://news.ycombinator.com/item?id=39495476 https://news.ycombinator.com/item?id=39495476) with the workflow through Claude Haiku and got this: Here are five distinct blog post titles optimized for high clickthrough on Hacker News for the article provided: 1. Tipping ChatGPT: Does Offering Monetary Incentives Improve AI Text Generation? 2. Quantifying the Impact of Incentives on Large Language Model Performance 3. Carrot or Stick? Exploring the Effects of Positive and Negative Prompts on ChatGPT 4. Gamifying AI: Using "Generation Golf" to Test ChatGPT's Ability to Follow Length Constraints 5. The Curious Case of ChatGPT's Motivations: Can an AI Be Incentivized Like Humans? Not bad titles, although more verbose than the 5 input examples I gave. I only gave 5 for simplicity: my main point is that you can give it a lot more than five and/or be more aggressive with constraints, like the blog post linked incidentially.
- ldjkfkdsjnv 3y agoFine tuning will disappear, no reason to invest so heavily in it. Also, most LLM code being written right now (prompts, RAG, etc) will be obsolete once the next model comes out. Anyone starting an LLM application startup is arguably wasting their time, wait until the next iteration is out. Then you will know whats possible.
- minimaxir 3y ago> Also, most LLM code being written right now (prompts, RAG, etc) will be obsolete once the next model comes out. Not true. Most prompt techniques that work on current modern LLM models will work on different or future models, although it will require a QA pass for any regressions.
- ldjkfkdsjnv 3y agoYeah they will work, but they will also be unnecessary. You will also bake all sorts of logic into your application that will be solved natively on the stronger model. If you believe in the scaling theory, then writing LLM applications is non sensical.
- deleted 3y ago[deleted]
- mlyle 3y ago> Yeah they will work, but they will also be unnecessary. You will also bake all sorts of logic into your application that will be solved natively on the stronger model. I think that a whole lot of what I do in prompt engineering is what's necessary to fully specify the output that I want. A newer model may be less finicky, so I have a higher chance of getting it to work on the first try (and it's more reliable afterwards), but it's hard for me to imagine it needing a whole lot less prompt.
- Xenoamorphous 3y agoHow will a more powerful model be a substitute for RAG, which is usually used with private data that won’t be present in any training dataset?
- hulium 3y agoThere is also the open source FinGPT, that is claimed to beat GPT4 in some benchmarks at a fine tuning cost of $17.25. https://github.com/AI4Finance-Foundation/FinGPT https://github.com/AI4Finance-Foundation/FinGPT
- potatoman22 3y agoOne major advantage of FinGPT or Bloomberg's LLM is that the embeddings produced by the model can be used for downstream prediction tasks. GPT-4 does not expose its embeddings so it cannot be used for this.
- bernawil 3y agosorry, noob here trying to make sense of this: you mean you can extract embeddings from the model file or that the embeddings are available in the repo and you can just use those files?
- potatoman22 3y agoKind of. You feed the LLM the input text for your prediction, you extract the activations of the final layer of the LLM (so the weights * the input of the previous layers), then use that activation vector, or embedding, as the input for a separate model. This separate model that uses the embedding can be any classifier or regression. A common use case for this is document classification.
- LASR 3y agoI lead AI teams at my company. I've advised leadership against any kind of training / fine-tuning anything. We're not in the business of training models. We will never be as good as OpenAI / Anthropic etc. Where the real value in applications is smarter prompting techniques and RAG. There is a lot of room at the bottom in doing "dumb" things and simply feeding models with the right context to deliver customer value.
- elforce002 3y agoThis. I work in a startup and told upper management we need to keep focusing on ML models that bring tangible benefits to our customers and then, try to integrate LLMs into their current flow instead of pivoting completely to LLMs. It seems they valued the input and now we're going for a hybrid approach.
- jnwatson 3y agoIt is trivial to fine tune these days. RAG is already irrelevant with large context windows.
- Xenoamorphous 3y ago> RAG is already irrelevant with large context windows Just last Friday I took the contents of the 2024 folder of one of the teams at the company I work for, for which we use RAG at the moment. I dumped the text index, concatenated it and used Google’s API to return the token count, to see if it would fit in Gemini’s 1M context window; turned out it was 5.7M tokens. And that’s less than 3 months worth of documents for that team. So yeah RAG is not dead yet, although I do question its usefulness, but that’s a separate topic.
- greenavocado 3y agoDid I read this correctly? You uploaded millions of words of your company's internal communications to Google?
- 3y ago
- thorum 3y agoNote that the benchmarks used for comparison are basically measuring the model’s ability to understand financial content. In other words, reading comprehension for English, just in a specific domain. It shouldn’t really be surprising that a strong generalist model performs well here. On the other hand, GPT-4 actually did worse on the NER task - labelling and tagging terms used in the text - vs their finetuned model. I assume the finetuned model was better at using the specific labels they were targeting.
- Rustwerks 3y agohttp://www.incompleteideas.net/IncIdeas/BitterLesson.html http://www.incompleteideas.net/IncIdeas/BitterLesson.html
- hallqv 3y agoThis discussion is so dumb - finetuning a base model costs ~$1 with LORA/QLORA and can yield same performance as gpt-4, but at 1/100 of the cost per token. What Bloomberg did for $10M was not finetuning..
- simonw 3y ago"finetuning a base model costs ~$1 with LORA/QLORA and can yield same performance as gpt-4, but at 1/100 of the cost per token" That's a big claim - can you back that up with any examples?
- Implicated 3y agoI had opened a new tab back when this comment was just a few minutes old in hopes that when I came back there was some really great blog post linked with the details on the sorcery.
- hallqv 3y agohttps://arxiv.org/pdf/2402.00841.pdf https://arxiv.org/pdf/2402.00841.pdf
- krasin 3y agoFinetuning LLMs is currently the most promising way for next-gen robotics. One of such works (PaLM-e) among other things measured the impact of finetuning on general purpose tasks: https://twitter.com/DannyDriess/status/1632904698108821505 https://twitter.com/DannyDriess/status/1632904698108821505 In short, an 8B model could degrade almost 10x after being finetuned on robotics tasks, while 500B model experiences a very minor degradation (~4%) and there's a hope that with a larger-sized model, it could become either zero or even negative (adding more experience improves general purpose reasoning). What I am saying is that while GPT-4 could beat a finetuned GPT-3.5 class model, I predict good things about finetuned GPT-4 class models, when they become practical outside of OpenAI/Google.
- TMWNN 3y ago>there's a hope that with a larger-sized model, it could become either zero or even negative (adding more experience improves general purpose reasoning). The best single work of fiction ever created about LLMs' capabilities (and, perhaps, dangers) is Colossus by Jones. Although I think the film is even better than the book, only the latter mentions how, despite being created specifically for US national defense, Colossus is also fed unrelated data including Shakespeare's sonnets, because its creators do not know if it could be important.
- paulmd 3y agoWhat good is a revolution without dancing?
- gradascent 3y agoInteresting! I would like to learn more about how AI is being applied to robotics. Do you have any suggestions for how to keep up with developments/ideas in this field?
- krasin 3y agoThese two links could be a good start: ALOHA-2: https://aloha-2.github.io/ https://aloha-2.github.io/ RT-X: https://robotics-transformer-x.github.io/ https://robotics-transformer-x.github.io/
- atleastoptimal 3y agoYeah, the equivalent is: would it be better for a quant firm to spend 200 thousand dollars giving a first-class specialist education to a guy with an IQ of 95, or just hiring a guy with an IQ of 150 straight out of college.
- kcorbitt 3y agoIMO it's possible to over-generalize from this datapoint (lol). While it's true that creating a general "finance" model that's stronger than GPT-4 is hard, training a task-specific model is much easier. Eg. "a model that's better than GPT-4 at answering finance-related questions": very hard. "A model that's better than GPT-4 at extracting forward-looking financial projections in a standard format": very easy. And in practice, most tasks people are using GPT-4 for in production are more like the latter than the former. (Disclaimer: building https://openpipe.ai https://openpipe.ai, which makes it super easy to productize this workflow).
- jonplackett 3y agoFYI it was GPT-3.5 Class not GPT3.5. A lot of models claim to be GPT3.5 class that clearly are not in the first place.
- deleted 3y ago[deleted]