5 ms·
OpenAI slashes the cost of using its AI with a "mini" model
- ilaksh 2y agoAnyone see benchmarks comparing this to Gemini Flash or gpt-3.5 or open models that can run on groq?
- xendipity 2y agoI'm not finding any direct sources from OpenAI, but here's this snippet from a Reuters article [1] > Priced at 15 cents per million input tokens and 60 cents per million output tokens, the GPT-4o mini is more than 60% cheaper than GPT-3.5 Turbo, OpenAI said. It currently outperforms the GPT-4 model on chat preferences and scored 82% on Massive Multitask Language Understanding (MMLU), OpenAI said. ... > The GPT-4o mini model's score compared with 77.9% for Google's Gemini Flash and 73.8% for Anthropic's Claude Haiku, according to OpenAI. For some more context: We don't know the size of 4o-mini but Mistral's just released NeMo 12B scores 68% on the MMLU. [2] [1]: https://www.reuters.com/technology/artificial-intelligence/openai-unveils-cheaper-small-ai-model-gpt-4o-mini-2024-07-18/ https://www.reuters.com/technology/artificial-intelligence/o... [2]: https://mistral.ai/news/mistral-nemo/ https://mistral.ai/news/mistral-nemo/
- pzo 2y agoAlso for some reference: Gemma 2 27B scored: 75.2 in MMLU LLama 3 70B scored: 79.5 in MMLU Haiku scored: 75.2 in MMLU GPT 3.5 scored: 70.0 in MMLU Based on pricing I see in openrouter.ai across different providers this seems like the cheapest model for this kind of performance. ref: [0] https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/Model_Card_Claude_3.pdf https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bb... [1] https://blog.google/technology/developers/google-gemma-2/ https://blog.google/technology/developers/google-gemma-2/
- unraveller 2y agohttps://artificialanalysis.ai https://artificialanalysis.ai (groq) Quality to Price graph suggests gpt3.5 was the worst now 4o-mini inched out all others of that lower league. It supposedly gets you flash/llama3-70B tier quality at around llama3-8B price.
- andrewmcwatters 2y agoI wonder how this compares to running an ollama server on a vps. Edit: I’m amazed by how offended some people are by such a simple question.
- ilaksh 2y agoI don't think there are any VPSs that can do that in a way that is even remotely performant or a good value compared to something like an LLM inference provider or serverless GPU. I would look into together.ai and RunPod for that type of thing. But let me know if you find something. I just don't think something tiny like phi-3 which could run on a VPS, although great for it's size, is at all comparable to this stuff in terms of ability.
- wkat4242 2y agoTrue, you could run it at home on a server though. My AI server takes about 60W idle and 300-350W while running a query in llama3. At a kWh price of 0.15€ that ends up at about 7-10€ a month if it's not loaded too heavily. Not bad IMO. The server could be more energy optimized though. But that would cost me also.
- j5155 2y agoHow much did the server hardware cost?
- wkat4242 2y agoAbout 100€ for the PC (some hardware was surplus) and 300€ for the GPU which was a nice 16gb model with HBM2. Was pretty nice for an educational project IMO. I much rather do something like this than spend money on a course.
- ilaksh 2y agoLlama3 8b is in no way equivalent to Gemini Flash or gpt-4o mini.
- minimaxir 2y agoThis appears to be part of an embargoed news blitz from a few news organizations (Verge and Bloomberg posted the same news at the same time), which is an interesting PR deviation from OpenAI posting it on their blog and having it go viral. The news isn't on their official blog at all currently.
- rmorey 2y agoi think they just messed up the embargo times, and suddenly the scoops all dropped before they were ready
- xendipity 2y agoYeah I found that interesting. Flubbed embargo times makes sense, but could it also be that they're letting the news organizations have first dibs to build a little goodwill with the industry?
- dustedcodes 2y agoIs OpenAI still making any changes to make ChatGPT better and more accurate and correct, or are they only focusing on making it cheaper by giving us weaker/dumber responses in a faster way nowadays? I have cancelled my subscription recently because didn't think that the GPT-4 was much better than what I get for free at Claude or Gemini.
- mritchie712 2y agoas someone building a product with their API, GPT-4 is already smarter than we need for our use case. Request latency is my #1 concern right now.
- JimDabell 2y agoIf request latency is your top concern and you don’t need it to be as smart as GPT-4, why are you using OpenAI and not a model hosted by Groq or similar?
- mritchie712 2y agoWe need function calling, which wasn't great in Groq when we last tried it, but they seem to have made progress on it. We'll be testing it again soon.
- voiper1 2y agogroq just released: llama3-groq-70b-8192-tool-use-preview llama3-groq-8b-8192-tool-use-preview Sonnet3.5 also has tool usage, and is slightly cheaper than 4o ($3 input vs $5). If you _only_ need tool usage, then 4o-mini or others will be cheaper... mistal's Mixtral 8x22B has function calling for $2/$6, which is a bit cheaper than 4o.
- d13 2y agoGroq’s rate limits are so bad they’re only good enough for one person using it very rarely with an under 4k context size. It’s just a marketing stunt at the moment.
- m3kw9 2y agoThis must mean there is a big chunk of people using open source models that are cheaper and they want a slice of that action
- Me1000 2y agoOr it means that the existing models are too large to be profitable even at scale.
- exitb 2y agoThere's market for small models with large context, for cases where there’s little need for reasoning (summarization, searching etc). 4o-mini is probably better and cheaper to run than 3.5.
- unraveller 2y agoIt means they don't want anyone releasing much stronger models then they do or they will undercut you by 1 magnitude and out perform you by 5%. Hardly a recipe for long-term consumer benefit.
- deleted 2y ago[deleted]
- OutOfHere 2y agoI now see gpt-4o-mini listed at https://platform.openai.com/docs/models/gpt-4o-mini https://platform.openai.com/docs/models/gpt-4o-mini Looking at https://openai.com/index/gpt-4o-mini-advancing-cost-efficient-intelligence/ https://openai.com/index/gpt-4o-mini-advancing-cost-efficien... , gpt-4o-mini is better than gpt-3.5 but worse than gpt-4o, as was expected. gpt-4o-mini is cheaper than both, however. Independent third-party performance benchmarks will help.
- guilamu 2y agoClaude 3.5 sonnet is so much better in every test I made (coding and everyday mondain stuffs), it beats me why anyone would choose chatgpt (I'm using free versions only).
- OutOfHere 2y ago> Claude 3.5 sonnet is so much better Good to know, although this is targeting cheaper API use for specific applications in which a second-tier model is sufficient. Note however that according to the LMSYS Leaderboard, GPT-4o rates slightly higher than Claude 3.5 Sonnet. > mondain stuffs Mundane, not mondain.
- guilamu 2y agoFor some reason I can't edit and fix the typo. Anyhow, thanks for the correction. No idea what the leaderboard says, it's just been night and day for me since Sonnet 3.5 got out. Maybe my use case is just what sonnet does best.
- deadbabe 2y agoThe only OpenAI product I care to hear about is ChatGPT-5
- laweijfmvo 2y agoIf "AI" tools are eventually going to have to be "free" (as in beer) to compete, I shudder to think of what companies like OpenAI will have to extract from users to please investors...
- Zambyte 2y agoTo combat those incentives, they should consider being non-profit and releasing their models as open.
- exe34 2y agothey could found a charity, with the aim of creating open AI models. they could call it OpenAI to draw attention that it's not a proprietary commercial technology.
- mritchie712 2y agoyeah, they'd probably need a board to see over it tho, sounds tricky, but doable.
- BoiledCabbage 2y agoThen they could have their largest benefactor first pledge $100 Million to fund them. Then after suddenly insist they are woefully falling behind google on developing the tech and insist the organization be handed over to his sole control to help them catch up. After being told he's wrong, he could then get upset and renege on his pledge after giving only 10% of his commitment - they'd be left scrambling running out of funds and realize that being dependent on donations at their scale isn't working and they need a way to earn a profit or partnership to self-fund. Oh and after pulling the pledged money, that other party could go buy a social media site and use it to whine about how they aren't a non-profit anymore... while knowing they're the cause of it.
- fragmede 2y ago
- Yusefmosiah 2y agoOpenAI’s strategy has been bizarre since at least last November, when they launched custom GPTs, then had the boardroom coup. Since the launch of Claude 3 Opus, and then Claude 3.5 Sonnet, they have been significantly behind Anthropic in terms of the general intelligence of their models. And instead of deploying something on par or better, they are making demos of video generation (Sora) or audio-to-audio models, not releasing anything. GPT-4o is quite bad at coding, often getting stuck in a loop, and “fixing” buggy code by rewriting it without any changes. GPT-4o is speculated to be a distillation of a larger model, and now GPT-4o-mini is an even dumber smaller model. But what’s the point? Who is actually using small/fast/cheap/dumb models in production apps? Most real apps require higher reliability than even the biggest/slowest/priciest/smartest models can provide today. For the use case of transformers that has taken off, aiding students and knowledge workers in one-off tasks like writing code and prose, most users want smarter, more reliable outputs, even at the expense of speed and cost. GPT-4o-mini seems like a move to increase margins, not make customers happier. That, like demoing products without launching them, is what big old slow corporations do, not how world-leading startups operate.
- sunaookami 2y agoSince Claude 3.5 Sonnet was released I can't go back to GPT anymore. It sounds too "robotic" and is overly verbose. It explains every little detail that I don't want to know and still is far worse than Claude. OpenAI has to really step up their game if they don't want to fall behind. In fact, GPT-4 got worse back in November, the best version is still the one from June 2023 but it's only available in the API.
- jmccarthy 2y agoSonnet is great, but also suggest exploiting custom instructions in the ChatGPT UI. Here's a snippet from mine: Extremely concise, formal. As short as possible. Assume I am an industry expert in any topic we discuss. Answer assuming I have the highest level of intellect possible, and do not require explication regardless of the sophistication of the topic. In cases where one approach among many is superior, offer an opinionated argument in favor of that approach.
- machiaweliczny 2y agoI think they just want to maintain lead with data/task collections and that’s why having cheapest model wins