13 ms·

OpenAI's new open-source model is basically Phi-5

403 points by emschwartz 1y ago

NitpickLawyer 1y ago
Yeah, makes sense. Good observations regarding the benchmark vs. vibes in general, and I didn't know / made the connection between the lead of phi models going to oAI and gpt-oss. Could very well be a similar exercise + their "new" prompt level adherence (system > developer > user). In all the traces I've seen of refusals the model "quotes" the policy quite religiously. Similar thing was announced for gpt5. I think the mention of the "horny people" is warranted, they are an important part of the open models (and first to explore the idea of "identities / personas" for LLMs, AFAIK). Plenty of fine-tuning bits of know-how trickled from there to the "common knowledge". There's a thing that I would have liked to be explored, perhaps. The idea that companies might actually want what -oss offers. While the local llm communities might want freedom and a horny assistant, businesses absolutely do not want that. And in fact they spend a lot of effort into implementing (sometimes less than ideal) guardrails, to keep the models on track. For very easy usecases like support chatbots and the like, businesses will always prefer something that errs on the side of less than useful but "safe", rather than have the bot start going crazy with sex/slurs/insults/etc. I do have a problem with this section though: > Really open weight, not open source, because the weights are freely available but the training data and code is not. This is factually incorrect. The -oss models are by definition open source. Apache2.0 is open source (I think even the purists agree with this). The requirement of sharing "training data and code" is absolutely not a prerequisite for being open source (and historically it was never required. The craze surrounding LLMs suddenly made this a thing. It's not). Here's the definition of source in "open source": > "Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files. Well, for LLMs the weights are the "preffered form of making modifications". The labs themselves modify models the same as you are allowed to by the license! They might use more advanced tools, or better datasets, but in the end the definition still holds. And you get all the other stuff, like the right to modify, re-release, etc. I'd really wish people would stop proliferating this open weight nonsense. Models released under open source licenses are open source. gpt-oss, qwens and mistrals (apache2.0), deepseeks(MIT), etc. Models released under non open source licenses also exist, and they're not open source because the licenses under which they're released aren't. LLamas, gemmas, etc.
- jononor 1y ago
  No the preferred way of making modifications is the weights _together_ with training (or fine tuning) scripts, and the entire evaluation pipeline to measure performance. And the data required to support all of this. When someone joins your data science team your would give them all this code and data. Not just the weights and say - the weights are the source, modify that to improve the model, I look forward to see your MR next week. EDIT: Heck, sometimes the way to make improvements (modifications) is just to improve the data, and not touch the training code at all. It is often one of the most powerful ways. You still need training code though, and evaluation to measure the impact.
  - NitpickLawyer 1y ago
    The license gives you the right to modify the weights, how you do the modification is up to you. The rest is in the realm of IP, know-how, etc. Apples and oranges.
    - imiric 1y ago
      Having the right to modify one part of the product is not the same as having the right to modify the entire product. Labeling such projects as open source in the full spirit of the definition is disingenuous. This is similar to the approach taken by some video game studios: release the source code under a permissive license, but not the game assets. Which is better than a proprietary license, but it still presents a hurdle for the final product to be built from source. The open weights approach is much more user hostile, however. Proprietary game assets can at least be purchased, and the final product can be built. With open weights, this is not possible. Nobody can realistically build the same model or similar models from weights alone. They can use the weights and self-host the prebuilt model, but not create revisions of it, which is the whole point of open source. Weights are essentially the bytecode of language models. Sure, you can run and modify it with the right tools, but without the tools used to create it in the first place, the project is not much more useful than publishing binaries.
  - wizzwizz4 1y ago
    You also need the training data, so you can ensure you're not benchmarking on the training set, fine-tuning on the training set (overfitting with extra steps), or otherwise breaking things.
tarruda 1y ago
If a model is trained only on synthetic data, is it still possible it will output things like this? https://x.com/elder_plinius/status/1952958577867669892 https://x.com/elder_plinius/status/1952958577867669892
- LeoPanthera 1y ago
  By definition, a model can't "know" things that are not somewhere in its training set, unless it can use a tool to query external knowledge. The problem is that the size of the training set required for a good model is so large, that's really hard to make a good model without including almost all known written text available.
  - janalsncm 1y ago
    > all known written text available If phi5 is trained on synthetic data only then info on how to make drugs must be in the synthetic dataset.
  - eru 1y ago
    > By definition, a model can't "know" things that are not somewhere in its training set, unless it can use a tool to query external knowledge. Well, it could also make inferences. Like, it could find a new mathematical proof, even if that's never in the training set.
    - xwolfi 1y ago
      But how, it's not like it's thinking, it's just spitting the next likely token
      - createaccount99 1y ago
        That is an oversimplification, and not the whole truth. Read anthropic's blog posts if you want to learn more, or ask gpt5.
      - frotaur 1y ago
lifis 1y ago
Does anyone know how synthetic data is commonly generated? Do they just sample the model randomly starting from an empty state, perhaps with some filtering? Or do they somehow automatically generate prompts and if how? Do they have some feedback mechanism, e.g. do they maybe test the model while training and somehow generate data related to poorly performing tests?
- LeoPanthera 1y ago
  I don't know about Phi-5, but earlier versions of Phi were trained on stories written by larger models trained on real-world data. Since it's Microsoft, they probably used one of the OpenAI GPT series.
  - Mars008 1y ago
    > stories written by larger models trained on real-world data I suspect there are no larger models trained on pure real-world data. They all use a mix of real and generated.
- deleted 1y ago
  [deleted]
- janalsncm 1y ago
  It’s common to use rejection sampling: sample from the model and throw out the samples which fail some criteria like a verifiable answer or a judgement from a larger model.
- Mars008 1y ago
  One way of getting good random samples is to give model a random starting points. For example: "write a short story about PP doing GG in XX". Here PP, GG and XX are filled algorithmically from lists of persons, actions and locations. The problem is model's randomly generated output from the same prompt isn't actually that random. Changing the temperature parameter doesn't help much. But in general it's a big secret because the training data and techniques are the only difference between models as architecture is more or less settled.
- 1y ago
magicalhippo 1y ago
I've found good use of Phi-4 at home, and after a few tests of the GPT-OSS 20B version I'm quite impressed so far. Particularly one SQL question that has tripped every other model of similar or smaller size that I've tried, like Devstral 24B, Falcon 3 7B, Qwen2.5-coder 14B and Phi 4 14B. The question contains an key point which is obvious for most humans, and which all of the models I tried previously have failed to pick up on. GPT-OSS picked up on it, and made a reasonable assumption. It's also much more thorough at explaining code compared to the other models, again including details the others miss. Now if only I had a GPU that could run the whole thing...
- VladVladikoff 1y ago
  Can you share the question? Or are you intentionally trying to keep it out of the training data pool?
  - magicalhippo 1y ago
    Sadly no. I'd like to keep it untainted, but also because the tables involved are straight from my work, which is very much not OSS. I can however try to paraphrase it so you get the gist of it. The question asks to provide a SQL statement to update rows in table A based on related tables B and C, where table B is mentioned explicitly and C is implicit through the foreign keys provided in the context. The key point all previous models I've tested has missed, is that the rows in A are many-to-one with B, and so the update should take this into account. This is implicit from the foreign key context and not mentioned directly in the question. Think distributing pizza slices between a group of friends. All previous models has completely missed this part and just given each friend the whole pizza. GPT-OSS correctly identified this issue and flagged it in the response, but also included a sensible assumption of evenly dividing the pizza. I should note some of the previous models also missed the implicit connection to table C, and thus completely failed to do something sensible. But at least several of them figured this out. Of course I forgot to write that part down so can't say offhand which did what. As for the code, for example I've coded a Y combinator in Delphi, using intentionally terse non-descriptive names, and asked the models to explain how the code works and what it does. Most ~7B models and larger of the past year or so have managed to explain it fairly well. However GPT-OSS was much more thorough and provider a much better explanation, showing a significantly better "understanding" of the code. It was also the first model smaller than LLama 3 70B that I've tried that correctly identified it as a Y combinator.
diggan 1y ago
> for instance, they have broad general knowledge about science, but don’t know much about popular culture That seems like a good focus. Why learn details that can change within days of it being released? Instead, train the models to have good general knowledge, and be really good at using tools, and you won't have to re-train models from scratch just because some JS library now has a different API, instead the model goes out to fetch the latest APIs/gossip when needed.
- wmf 1y ago
  Yeah, it always seemed like a sad commentary on our world that AIs are devoting their weights to encyclopedic knowledge of Harry Potter, Pokemon, and Reddit trolling.
  - eru 1y ago
    Why? You gotta provide what your customers want. And it's far from sad that we have so many resources, we can give everyone a supercomputer in their pocket just to take selfies and talk about Pokemon. Why would our AIs be any different?
    - xwolfi 1y ago
      Because we really could use all that time, money and uranium to train them on complex problems we need to solve, rather than entertainment. But you're right, shareholders of OpenAI want profit, not progress, and they'll give us our intellectual Big Mac.
      - eru 1y ago
        You could say that about any piece of music or movie ever, too. Or any novel.
  - deleted 1y ago
    [deleted]
- eru 1y ago
  Why would anything change? You feed the model approximately all the text you have ever. And some things like 'popular culture of 2025' won't change, just because the calendar changed to 2026. Just like the popular culture of the 1980s is what it was, and won't change.
wmf 1y ago
I saw a bunch of people complaining on Twitter about how GPT-OSS can't be customized or has no soul and I noticed that none of them said what they were trying to accomplish. "The main use-case for fine-tuning small language models is for erotic role-play, and there’s a serious demand." Ah.
- kristopolous 1y ago
  Porn is always the frontier. It's a well-understood self-contained use-case without many externalities and simple business models. What more, with porn, the medium is the product probably more than the content. Having it on home-media in the 80s was the selling point. Getting it over the 1-900 phone lines or accessing it over the internet ... these were arguably the actual product. It might have been a driver of early smart phone adoption as well. Adult content is about an 80% consumption on handheld devices while the internet writ large is about 60%. Private tunable multi-media interaction on-demand is the product here. Also it's a unique offer. Role playing prohibited sexual acts can be done arguably victim free. There's a good fiction story there... "I thought I was talking to AI"
  - shortrounddev2 1y ago
    There's something Freudian about the idea that the more you can customize porn, the more popular it is. That, despite the impression that "all men want one thing", it turns out that men all want very different and very oddly specific things. Imbuing somrthing with a "magical" quality that doesnt exist is the origin of the term "fetish". Its not about the raw attractive preference for a particular hair color; its a belief in the POWER of that hair color.
    - kristopolous 1y ago
      oh it's wildly different. About 15 years ago I worked on a porn recommendation system. The idea is that you'd follow a number of sites based on likes and recommendations and you'd get an aggregated feed with interstitial ads. So I started with scraping and cross-reference, foaf, doing analysis. People's preferences are ... really complex. Without getting too lewd, let's say there's about 30-80 categories with non-marginal demand depending on how you want to slice it and some of them can stack so you get a combinatoric. In early user testing people wanted the niche and found the adventurous (of their particular kind) to be more compelling. And that was the unpredictable part. The majoritarian categories didn't have stickiness. Nor did these niches have high correlation. Someone could be into say, specific topic A (let's say feet), and correlating that with topic B (let's say leather) was a dice roll. The probabilities were almost universally < 10% unless you went into majoritarian categories (eg. fit people in their 20s). People want adventure on a reservation with a very well defined perimeter - one that is hard to map and different for every person. So the value-add proposition went away since it's now just a collection of niche sites again. Also, these days people have Reddit accounts reserved for porn where they do exactly this. So it was built after all.
pogue 1y ago
Is that true that most small language models are fine tuned for erotic role-play?
- int_19h 1y ago
  What you wrote is somewhat ambiguous, so allow me to rephrase. It is true that most fine-tunes of relatively small (which can mean anything up to 150B params, depending on who you ask!) LLMs are for uncensored roleplay purposes.
  - pogue 1y ago
    Sorry for the late reply. I've read a bit about this, but I don't know how it's done. I've seen some uncensored models using a process called "abliteration" [1] but I'm not sure if that's the same thing or not. I've never tried self training an AI model or anything but I'd be very curious about the process. Maybe this is something someone could do as a side hustle? [1] Uncensor any LLM with abliteration by Maxime Labonne https://huggingface.co/blog/mlabonne/abliteration https://huggingface.co/blog/mlabonne/abliteration
RandyOrion 1y ago
I mean, yeah. From the Table 9: Hallucination evaluations in GPT-OSS model card [1], GPT-OSS-20b/120b have accuracy of 0.067/0.168 and hallucination rate of 0.914/0.782 separately, while o4-mini has accuracy of 0.234 and hallucinate rate of 0.750. These numbers simply mean that GPT-OSS models have little real world knowledge, and they hallucinate hard. Note that little real world knowledge has always been a "feature" of the Phi-LLM series because of the "safety" (for large companies), or rather, "censorship" (for users) requirements. In addition, from Table 4: Hallucination evaluations in OpenAI o3 and o4-mini System Card [2], o3/o4-mini have accuracy of 0.49/0.20 and hallucination rate of 0.51/0.79. In summary, there is a significant real world knowledge gap between o3 and o4-mini, and another significant gap between o4-mini and GPT-OSS. Besides, the poor real world knowledge exhibited in GPT-OSS is aligned with the "feature" of Phi-LLM series. [1] https://cdn.openai.com/pdf/419b6906-9da6-406c-a19d-1bb078ac7637/oai_gpt-oss_model_card.pdf https://cdn.openai.com/pdf/419b6906-9da6-406c-a19d-1bb078ac7... [2] https://cdn.openai.com/pdf/2221c875-02dc-4789-800b-e7758f3722c1/o3-and-o4-mini-system-card.pdf https://cdn.openai.com/pdf/2221c875-02dc-4789-800b-e7758f372...
klooney 1y ago
> It’s not discussed publically very often, but the main use-case for fine-tuning small language models is for erotic role-play, and there’s a serious demand. Any small online community for people who run local models is at least 50% perverts. Amazing
- msgodel 1y ago
  Meh. For the first few decades consumer internet traffic was mostly porn. Stop freaking out and use the free effort people are willing to put into to solve technical problems.
dweinus 1y ago
Is it confirmed that synthetic data was used for gpt-oss training? I didn't pick up on that in the press release or see it elsewhere. Did I miss it or is Sean speculating that it is the case?
refulgentis 1y ago
This is really irresponsible, there's actual data on quality and this is just crazy to assert: "Any small online community for people who run local models is at least 50% perverts." --- I get what he's saying, there's def. communities where that's predominant, but it's simply not true in the vast majority of communities. Sort of like, in 1997, saying any small community on this here Internet thing is half perverts looking at porn It's extremely frustrating to try and reply to this because that which is asserted without evidence can't really be debunked by evidence. I find it especially shameful that he's dragging someone's name into this wildly histrionic review, in service of trying to find a sole person to attribute to, the sole attribute that led to whatever experience he had with it. The model is insanely good, wildly exceeded my expectations for local models, and will generate at least 18 months of sustainable value. I maintain a llama.cpp wrapper and this is quantum leap in quality. I despair that this will become a major source of people's opinions on it. We desperately need big companies actually investing here, Gemma ain't it, and pretending it doesn't work because ??? and then using it to create a corporate chickenshit narrative isn't exactly gonna help.
- ripped_britches 1y ago
  100%, what an insanely baseless article
KingOfCoders 1y ago
No training data, no open source. Don't fall for the company PR.
- Mars008 1y ago
  As long as it works who cares about training data. Obviously they can't open it for many reasons. License is one of them.
  - KingOfCoders 1y ago
    I don't care if it's open source or not. I care if people call something open source which it isn't. Do you care about binary blobs in the kernel? No. Are binary blobs in the kernel open source? No. But it is tedious to go through the same discussion every 10 years, with a relentless industry that wants to dupe people. If there wasn't a benefit in for them, they would not call it open source.
    - Mars008 1y ago
      For some reason people think of models as software and open source should have similar meaning. There are fundamental differences: 1) models aren't reproducible given everything, data, hardware, methodology. 2) they aren't even verifiable. i.e. given model and dataset it's impossible to say if model was trained on that data. 3) except for toys models are trained on copyrighted data. Some of it is private, like users' chats. 4) besides data there is a lot of human input after pretraining. This means given everything you have two options: 1) train similar model yourself 2) trust model provider. In software you can get script and run, or get code and compile it in exactly the same binaries. Naturally 'open source' has different meaning. Some are trying to monopolize it, like they know the 'truth'. Others simply ignore it. Eventually we'll settle on something.
      - jononor 1y ago
        A decent training pipeline will be able to reproduce models with equivalent aggregate performance (as measured by the evaluation metrics). And a high degree of similarity in behavior on specific inputs - but not identical. It will not be the same exact weights. But that is not a critical bar to reach. And may software builds also fails that bar.
wkat4242 1y ago
From the article: > For the same reason that Microsoft probably continued to train Phi-style models: safety. Releasing an open-source model is terrifying for a large organization. Once it’s out there, your name is associated with it forever, and thousands of researchers will be frantically trying to fine-tune it to remove the safety guardrails. I don't think this is really an issue in practice. Llama 2 and 3 were uncensored within a week. There's no bad press about this. What does give a company a bad reputation is crap models. The llama 4 disappointment hurt meta's AI reputation a lot more than some community uncensoring.
- teruakohatu 1y ago
  > researchers will be frantically trying to fine-tune it to remove the safety guardrails. It is a really weak excuse. They are more likely to take a reputation hit for having silly guardrails than for having someone to remove them. Imagine if Bill Gates decided not to release MS Paint in 1985 because someone could have drawn something offensive with it.
  - anshumankmr 1y ago
    Or imagine if Bill Gates decided to not release Comic Sans in the 90s cause someone could have written something offensive with it.... oh wait that wouldn't have been too bad (/S)
    - wkat4242 1y ago
      I like it. Just block anything that uses it, and you have cleaned up a big part of your internet and lost nothing of value.
  - umeshunni 1y ago
    > Imagine if Bill Gates decided not to release MS Paint in 1985 because someone could have drawn something offensive with it. Quite believable that this could have been the case in 2021.
- CjHuber 1y ago
anshumankmr 1y ago
The main aim of Phi3 mini was to be able to run on device, and it had tremendous speed and for a 128K context with 3B something params, its pretty damn good, I had used it for a project myself last year but ultimately we went with the Mistral's models who at the time had the best Open weights models.
egorfine 1y ago
> the main use-case for fine-tuning small language models is for erotic role-play Of course. This is indeed the fact. /s See that "/s"? I did write it here, but it's suspiciously absent in the original text. Almost makes you think it was typed in all seriousness.
- Sabinus 1y ago
  What do you think the main use case for fine tuning small language models is?
orbital-decay 1y ago
I gave it a random sci-fi novel and made it translate a chapter, which is something I do with all models. It refused to discuss minors in sexualized contexts. I was like W.T.F.?! and started bisecting the book, trying to find the piece that triggers this. Turns out there was some absolutely innocent, two sentence long romantic remark involving two secondary 17 years old characters in an unrelated place. Another issue is that it sometimes has occasional refusals and total meltdowns where it redacts entire paragraphs with placeholder characters, while just trying to casually talk with it about some routine life matters. That's ridiculous and makes that model garbage at any form of creative writing (including translation) or real life tasks other than math or coding. It has very poor knowledge for a 120B MoE. If you look at the "reasoning" it does, it actually mostly checks the request against the policy. I thought they must have spent most of their post-training hunting the wrongthink and dumbing the model down as a result, but I can see how the synthetic pretraining data can explain this.
- spacecadet 1y ago
  Its a public consumer facing model, not surprised. Go find an unaligned model that will better produce the content you seek...
  - bobsmooth 1y ago
    After running Dolphin Mistral on my own machine I won't trust censored models ever again.
    - krick 1y ago
      Recently I somehow wasn't using LLMs locally and relied mostly on ChatGPT for casual tasks. I think it was a little less than a year since I played with ollama, and I remember that my impression was that all recent popular models definitely aren't "uncensored" in a sense that some older modification of llama2 I used was, and all suck for prose-related tasks anyway. In fact, nothing but ChatGPT models seemed good enough for writing, but, of course, they refuse to talk about pretty much anything. Even DeepSeek is not great at writing, and it it much bigger than anything I ever ran locally. So, are there even good uncensored models now? Are they, like, really uncensored?
rynn 1y ago
Azure AI Foundry says: How was the model trained? The gpt-oss models were pretrained primarily using synthetic data along with some heavily filtered real code. The models were then post-trained using distillation (RLKD against o3/hailmary) and berry. For more details about the model training process, please refer to the documentation. Meaning the author was spot on.
jnmandal 1y ago
I think its likely that soon the "AI" aspect of the "models" get worse and worse with each iteration. This is because: 1. data pollution/dilution -- with each subsequent generation more and more LLM-produced content will make up the bulk of the web this means it either gets into get into the data set, which makes the model "spikey" or reduces the relevant knowledge available which dumbs the model down 2. LLMs were sort of a freak breakthrough in deep learning and while its remarkable, tuning them only makes them marginally better. Diminishing returns apply here. Its a new LLM but its still an LLM -- years old technology now However, despite these two realities, the total utility/productivity/societal gain from a product (note I didn't say model) like ChatGPT can still increase by orders of magnitude. That is because companies like OpenAI, and many, many, other technology corporations and startups are figuring out how to leverage LLMs to do stuff much more powerful than just answer questions from the corpus (ie: perform quality research, reevaluate itself, computer controls, etc, etc). Consider for example, flat screen displays were pioneered in like the 50's and arguably, they didn't become disruptively useful until the advent of smart phones. So yeah, the model may get sort of worse or maybe more brittle, but it almost doesn't matter if they are figuring out what to do with the model and making the model more useful for actual tasks. Sure its cute to talk to an AI persona and ask it questions but that is probably the least important aspect of these type of models. Microsoft word had Clippy, and yeah that was cool. But productivity gain came from word processing, the '.doc' filetype, filesharing, editing, etc. Clippy is just a meme now and that's a likely future scenario for the "chat" features in LLM products IMHO.

403 points by emschwartz 1y ago

Yeah, makes sense. Good observations regarding the benchmark vs. vibes in general, and I didn't know / made the connection between the lead of phi models going to oAI and gpt-oss. Could very well be a similar exercise + their "new" prompt level adherence (system > developer > user). In all the traces I've seen of refusals the model "quotes" the policy quite religiously. Similar thing was announced for gpt5. I think the mention of the "horny people" is warranted, they are an important part of the open models (and first to explore the idea of "identities / personas" for LLMs, AFAIK). Plenty of fine-tuning bits of know-how trickled from there to the "common knowledge". There's a thing that I would have liked to be explored, perhaps. The idea that companies might actually want what -oss offers. While the local llm communities might want freedom and a horny assistant, businesses absolutely do not want that. And in fact they spend a lot of effort into implementing (sometimes less than ideal) guardrails, to keep the models on track. For very easy usecases like support chatbots and the like, businesses will always prefer something that errs on the side of less than useful but "safe", rather than have the bot start going crazy with sex/slurs/insults/etc. I do have a problem with this section though: > Really open weight, not open source, because the weights are freely available but the training data and code is not. This is factually incorrect. The -oss models are by definition open source. Apache2.0 is open source (I think even the purists agree with this). The requirement of sharing "training data and code" is absolutely not a prerequisite for being open source (and historically it was never required. The craze surrounding LLMs suddenly made this a thing. It's not). Here's the definition of source in "open source": > "Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files. Well, for LLMs the weights are the "preffered form of making modifications". The labs themselves modify models the same as you are allowed to by the license! They might use more advanced tools, or better datasets, but in the end the definition still holds. And you get all the other stuff, like the right to modify, re-release, etc. I'd really wish people would stop proliferating this open weight nonsense. Models released under open source licenses are open source. gpt-oss, qwens and mistrals (apache2.0), deepseeks(MIT), etc. Models released under non open source licenses also exist, and they're not open source because the licenses under which they're released aren't. LLamas, gemmas, etc.

jononor 1y ago
No the preferred way of making modifications is the weights _together_ with training (or fine tuning) scripts, and the entire evaluation pipeline to measure performance. And the data required to support all of this. When someone joins your data science team your would give them all this code and data. Not just the weights and say - the weights are the source, modify that to improve the model, I look forward to see your MR next week. EDIT: Heck, sometimes the way to make improvements (modifications) is just to improve the data, and not touch the training code at all. It is often one of the most powerful ways. You still need training code though, and evaluation to measure the impact.
- NitpickLawyer 1y ago
  The license gives you the right to modify the weights, how you do the modification is up to you. The rest is in the realm of IP, know-how, etc. Apples and oranges.
  - imiric 1y ago
    Having the right to modify one part of the product is not the same as having the right to modify the entire product. Labeling such projects as open source in the full spirit of the definition is disingenuous. This is similar to the approach taken by some video game studios: release the source code under a permissive license, but not the game assets. Which is better than a proprietary license, but it still presents a hurdle for the final product to be built from source. The open weights approach is much more user hostile, however. Proprietary game assets can at least be purchased, and the final product can be built. With open weights, this is not possible. Nobody can realistically build the same model or similar models from weights alone. They can use the weights and self-host the prebuilt model, but not create revisions of it, which is the whole point of open source. Weights are essentially the bytecode of language models. Sure, you can run and modify it with the right tools, but without the tools used to create it in the first place, the project is not much more useful than publishing binaries.
- wizzwizz4 1y ago
  You also need the training data, so you can ensure you're not benchmarking on the training set, fine-tuning on the training set (overfitting with extra steps), or otherwise breaking things.

tarruda 1y ago

If a model is trained only on synthetic data, is it still possible it will output things like this? https://x.com/elder_plinius/status/1952958577867669892 https://x.com/elder_plinius/status/1952958577867669892

LeoPanthera 1y ago
By definition, a model can't "know" things that are not somewhere in its training set, unless it can use a tool to query external knowledge. The problem is that the size of the training set required for a good model is so large, that's really hard to make a good model without including almost all known written text available.
- janalsncm 1y ago
  > all known written text available If phi5 is trained on synthetic data only then info on how to make drugs must be in the synthetic dataset.
- eru 1y ago
  > By definition, a model can't "know" things that are not somewhere in its training set, unless it can use a tool to query external knowledge. Well, it could also make inferences. Like, it could find a new mathematical proof, even if that's never in the training set.
  - xwolfi 1y ago
    But how, it's not like it's thinking, it's just spitting the next likely token
    - createaccount99 1y ago
      That is an oversimplification, and not the whole truth. Read anthropic's blog posts if you want to learn more, or ask gpt5.
    - frotaur 1y ago

lifis 1y ago

Does anyone know how synthetic data is commonly generated? Do they just sample the model randomly starting from an empty state, perhaps with some filtering? Or do they somehow automatically generate prompts and if how? Do they have some feedback mechanism, e.g. do they maybe test the model while training and somehow generate data related to poorly performing tests?

LeoPanthera 1y ago
I don't know about Phi-5, but earlier versions of Phi were trained on stories written by larger models trained on real-world data. Since it's Microsoft, they probably used one of the OpenAI GPT series.
- Mars008 1y ago
  > stories written by larger models trained on real-world data I suspect there are no larger models trained on pure real-world data. They all use a mix of real and generated.
deleted 1y ago
[deleted]
janalsncm 1y ago
It’s common to use rejection sampling: sample from the model and throw out the samples which fail some criteria like a verifiable answer or a judgement from a larger model.
Mars008 1y ago
One way of getting good random samples is to give model a random starting points. For example: "write a short story about PP doing GG in XX". Here PP, GG and XX are filled algorithmically from lists of persons, actions and locations. The problem is model's randomly generated output from the same prompt isn't actually that random. Changing the temperature parameter doesn't help much. But in general it's a big secret because the training data and techniques are the only difference between models as architecture is more or less settled.
1y ago

magicalhippo 1y ago

I've found good use of Phi-4 at home, and after a few tests of the GPT-OSS 20B version I'm quite impressed so far. Particularly one SQL question that has tripped every other model of similar or smaller size that I've tried, like Devstral 24B, Falcon 3 7B, Qwen2.5-coder 14B and Phi 4 14B. The question contains an key point which is obvious for most humans, and which all of the models I tried previously have failed to pick up on. GPT-OSS picked up on it, and made a reasonable assumption. It's also much more thorough at explaining code compared to the other models, again including details the others miss. Now if only I had a GPU that could run the whole thing...

VladVladikoff 1y ago
Can you share the question? Or are you intentionally trying to keep it out of the training data pool?
- magicalhippo 1y ago
  Sadly no. I'd like to keep it untainted, but also because the tables involved are straight from my work, which is very much not OSS. I can however try to paraphrase it so you get the gist of it. The question asks to provide a SQL statement to update rows in table A based on related tables B and C, where table B is mentioned explicitly and C is implicit through the foreign keys provided in the context. The key point all previous models I've tested has missed, is that the rows in A are many-to-one with B, and so the update should take this into account. This is implicit from the foreign key context and not mentioned directly in the question. Think distributing pizza slices between a group of friends. All previous models has completely missed this part and just given each friend the whole pizza. GPT-OSS correctly identified this issue and flagged it in the response, but also included a sensible assumption of evenly dividing the pizza. I should note some of the previous models also missed the implicit connection to table C, and thus completely failed to do something sensible. But at least several of them figured this out. Of course I forgot to write that part down so can't say offhand which did what. As for the code, for example I've coded a Y combinator in Delphi, using intentionally terse non-descriptive names, and asked the models to explain how the code works and what it does. Most ~7B models and larger of the past year or so have managed to explain it fairly well. However GPT-OSS was much more thorough and provider a much better explanation, showing a significantly better "understanding" of the code. It was also the first model smaller than LLama 3 70B that I've tried that correctly identified it as a Y combinator.

diggan 1y ago

> for instance, they have broad general knowledge about science, but don’t know much about popular culture That seems like a good focus. Why learn details that can change within days of it being released? Instead, train the models to have good general knowledge, and be really good at using tools, and you won't have to re-train models from scratch just because some JS library now has a different API, instead the model goes out to fetch the latest APIs/gossip when needed.

wmf 1y ago
Yeah, it always seemed like a sad commentary on our world that AIs are devoting their weights to encyclopedic knowledge of Harry Potter, Pokemon, and Reddit trolling.
- eru 1y ago
  Why? You gotta provide what your customers want. And it's far from sad that we have so many resources, we can give everyone a supercomputer in their pocket just to take selfies and talk about Pokemon. Why would our AIs be any different?
  - xwolfi 1y ago
    Because we really could use all that time, money and uranium to train them on complex problems we need to solve, rather than entertainment. But you're right, shareholders of OpenAI want profit, not progress, and they'll give us our intellectual Big Mac.
    - eru 1y ago
      You could say that about any piece of music or movie ever, too. Or any novel.
- deleted 1y ago
  [deleted]
eru 1y ago
Why would anything change? You feed the model approximately all the text you have ever. And some things like 'popular culture of 2025' won't change, just because the calendar changed to 2026. Just like the popular culture of the 1980s is what it was, and won't change.

wmf 1y ago

I saw a bunch of people complaining on Twitter about how GPT-OSS can't be customized or has no soul and I noticed that none of them said what they were trying to accomplish. "The main use-case for fine-tuning small language models is for erotic role-play, and there’s a serious demand." Ah.

kristopolous 1y ago
Porn is always the frontier. It's a well-understood self-contained use-case without many externalities and simple business models. What more, with porn, the medium is the product probably more than the content. Having it on home-media in the 80s was the selling point. Getting it over the 1-900 phone lines or accessing it over the internet ... these were arguably the actual product. It might have been a driver of early smart phone adoption as well. Adult content is about an 80% consumption on handheld devices while the internet writ large is about 60%. Private tunable multi-media interaction on-demand is the product here. Also it's a unique offer. Role playing prohibited sexual acts can be done arguably victim free. There's a good fiction story there... "I thought I was talking to AI"
- shortrounddev2 1y ago
  There's something Freudian about the idea that the more you can customize porn, the more popular it is. That, despite the impression that "all men want one thing", it turns out that men all want very different and very oddly specific things. Imbuing somrthing with a "magical" quality that doesnt exist is the origin of the term "fetish". Its not about the raw attractive preference for a particular hair color; its a belief in the POWER of that hair color.
  - kristopolous 1y ago
    oh it's wildly different. About 15 years ago I worked on a porn recommendation system. The idea is that you'd follow a number of sites based on likes and recommendations and you'd get an aggregated feed with interstitial ads. So I started with scraping and cross-reference, foaf, doing analysis. People's preferences are ... really complex. Without getting too lewd, let's say there's about 30-80 categories with non-marginal demand depending on how you want to slice it and some of them can stack so you get a combinatoric. In early user testing people wanted the niche and found the adventurous (of their particular kind) to be more compelling. And that was the unpredictable part. The majoritarian categories didn't have stickiness. Nor did these niches have high correlation. Someone could be into say, specific topic A (let's say feet), and correlating that with topic B (let's say leather) was a dice roll. The probabilities were almost universally < 10% unless you went into majoritarian categories (eg. fit people in their 20s). People want adventure on a reservation with a very well defined perimeter - one that is hard to map and different for every person. So the value-add proposition went away since it's now just a collection of niche sites again. Also, these days people have Reddit accounts reserved for porn where they do exactly this. So it was built after all.

pogue 1y ago

Is that true that most small language models are fine tuned for erotic role-play?

int_19h 1y ago
What you wrote is somewhat ambiguous, so allow me to rephrase. It is true that most fine-tunes of relatively small (which can mean anything up to 150B params, depending on who you ask!) LLMs are for uncensored roleplay purposes.
- pogue 1y ago
  Sorry for the late reply. I've read a bit about this, but I don't know how it's done. I've seen some uncensored models using a process called "abliteration" [1] but I'm not sure if that's the same thing or not. I've never tried self training an AI model or anything but I'd be very curious about the process. Maybe this is something someone could do as a side hustle? [1] Uncensor any LLM with abliteration by Maxime Labonne https://huggingface.co/blog/mlabonne/abliteration https://huggingface.co/blog/mlabonne/abliteration

RandyOrion 1y ago

I mean, yeah. From the Table 9: Hallucination evaluations in GPT-OSS model card [1], GPT-OSS-20b/120b have accuracy of 0.067/0.168 and hallucination rate of 0.914/0.782 separately, while o4-mini has accuracy of 0.234 and hallucinate rate of 0.750. These numbers simply mean that GPT-OSS models have little real world knowledge, and they hallucinate hard. Note that little real world knowledge has always been a "feature" of the Phi-LLM series because of the "safety" (for large companies), or rather, "censorship" (for users) requirements. In addition, from Table 4: Hallucination evaluations in OpenAI o3 and o4-mini System Card [2], o3/o4-mini have accuracy of 0.49/0.20 and hallucination rate of 0.51/0.79. In summary, there is a significant real world knowledge gap between o3 and o4-mini, and another significant gap between o4-mini and GPT-OSS. Besides, the poor real world knowledge exhibited in GPT-OSS is aligned with the "feature" of Phi-LLM series. [1] https://cdn.openai.com/pdf/419b6906-9da6-406c-a19d-1bb078ac7637/oai_gpt-oss_model_card.pdf https://cdn.openai.com/pdf/419b6906-9da6-406c-a19d-1bb078ac7... [2] https://cdn.openai.com/pdf/2221c875-02dc-4789-800b-e7758f3722c1/o3-and-o4-mini-system-card.pdf https://cdn.openai.com/pdf/2221c875-02dc-4789-800b-e7758f372...

klooney 1y ago

> It’s not discussed publically very often, but the main use-case for fine-tuning small language models is for erotic role-play, and there’s a serious demand. Any small online community for people who run local models is at least 50% perverts. Amazing

msgodel 1y ago
Meh. For the first few decades consumer internet traffic was mostly porn. Stop freaking out and use the free effort people are willing to put into to solve technical problems.

dweinus 1y ago

Is it confirmed that synthetic data was used for gpt-oss training? I didn't pick up on that in the press release or see it elsewhere. Did I miss it or is Sean speculating that it is the case?

refulgentis 1y ago

This is really irresponsible, there's actual data on quality and this is just crazy to assert: "Any small online community for people who run local models is at least 50% perverts." --- I get what he's saying, there's def. communities where that's predominant, but it's simply not true in the vast majority of communities. Sort of like, in 1997, saying any small community on this here Internet thing is half perverts looking at porn It's extremely frustrating to try and reply to this because that which is asserted without evidence can't really be debunked by evidence. I find it especially shameful that he's dragging someone's name into this wildly histrionic review, in service of trying to find a sole person to attribute to, the sole attribute that led to whatever experience he had with it. The model is insanely good, wildly exceeded my expectations for local models, and will generate at least 18 months of sustainable value. I maintain a llama.cpp wrapper and this is quantum leap in quality. I despair that this will become a major source of people's opinions on it. We desperately need big companies actually investing here, Gemma ain't it, and pretending it doesn't work because ??? and then using it to create a corporate chickenshit narrative isn't exactly gonna help.

ripped_britches 1y ago
100%, what an insanely baseless article

KingOfCoders 1y ago

No training data, no open source. Don't fall for the company PR.

Mars008 1y ago
As long as it works who cares about training data. Obviously they can't open it for many reasons. License is one of them.
- KingOfCoders 1y ago
  I don't care if it's open source or not. I care if people call something open source which it isn't. Do you care about binary blobs in the kernel? No. Are binary blobs in the kernel open source? No. But it is tedious to go through the same discussion every 10 years, with a relentless industry that wants to dupe people. If there wasn't a benefit in for them, they would not call it open source.
  - Mars008 1y ago
    For some reason people think of models as software and open source should have similar meaning. There are fundamental differences: 1) models aren't reproducible given everything, data, hardware, methodology. 2) they aren't even verifiable. i.e. given model and dataset it's impossible to say if model was trained on that data. 3) except for toys models are trained on copyrighted data. Some of it is private, like users' chats. 4) besides data there is a lot of human input after pretraining. This means given everything you have two options: 1) train similar model yourself 2) trust model provider. In software you can get script and run, or get code and compile it in exactly the same binaries. Naturally 'open source' has different meaning. Some are trying to monopolize it, like they know the 'truth'. Others simply ignore it. Eventually we'll settle on something.
    - jononor 1y ago
      A decent training pipeline will be able to reproduce models with equivalent aggregate performance (as measured by the evaluation metrics). And a high degree of similarity in behavior on specific inputs - but not identical. It will not be the same exact weights. But that is not a critical bar to reach. And may software builds also fails that bar.

wkat4242 1y ago

From the article: > For the same reason that Microsoft probably continued to train Phi-style models: safety. Releasing an open-source model is terrifying for a large organization. Once it’s out there, your name is associated with it forever, and thousands of researchers will be frantically trying to fine-tune it to remove the safety guardrails. I don't think this is really an issue in practice. Llama 2 and 3 were uncensored within a week. There's no bad press about this. What does give a company a bad reputation is crap models. The llama 4 disappointment hurt meta's AI reputation a lot more than some community uncensoring.

teruakohatu 1y ago
> researchers will be frantically trying to fine-tune it to remove the safety guardrails. It is a really weak excuse. They are more likely to take a reputation hit for having silly guardrails than for having someone to remove them. Imagine if Bill Gates decided not to release MS Paint in 1985 because someone could have drawn something offensive with it.
- anshumankmr 1y ago
  Or imagine if Bill Gates decided to not release Comic Sans in the 90s cause someone could have written something offensive with it.... oh wait that wouldn't have been too bad (/S)
  - wkat4242 1y ago
    I like it. Just block anything that uses it, and you have cleaned up a big part of your internet and lost nothing of value.
- umeshunni 1y ago
  > Imagine if Bill Gates decided not to release MS Paint in 1985 because someone could have drawn something offensive with it. Quite believable that this could have been the case in 2021.
CjHuber 1y ago

anshumankmr 1y ago

The main aim of Phi3 mini was to be able to run on device, and it had tremendous speed and for a 128K context with 3B something params, its pretty damn good, I had used it for a project myself last year but ultimately we went with the Mistral's models who at the time had the best Open weights models.

egorfine 1y ago

> the main use-case for fine-tuning small language models is for erotic role-play Of course. This is indeed the fact. /s See that "/s"? I did write it here, but it's suspiciously absent in the original text. Almost makes you think it was typed in all seriousness.

Sabinus 1y ago
What do you think the main use case for fine tuning small language models is?

orbital-decay 1y ago

I gave it a random sci-fi novel and made it translate a chapter, which is something I do with all models. It refused to discuss minors in sexualized contexts. I was like W.T.F.?! and started bisecting the book, trying to find the piece that triggers this. Turns out there was some absolutely innocent, two sentence long romantic remark involving two secondary 17 years old characters in an unrelated place. Another issue is that it sometimes has occasional refusals and total meltdowns where it redacts entire paragraphs with placeholder characters, while just trying to casually talk with it about some routine life matters. That's ridiculous and makes that model garbage at any form of creative writing (including translation) or real life tasks other than math or coding. It has very poor knowledge for a 120B MoE. If you look at the "reasoning" it does, it actually mostly checks the request against the policy. I thought they must have spent most of their post-training hunting the wrongthink and dumbing the model down as a result, but I can see how the synthetic pretraining data can explain this.

spacecadet 1y ago
Its a public consumer facing model, not surprised. Go find an unaligned model that will better produce the content you seek...
- bobsmooth 1y ago
  After running Dolphin Mistral on my own machine I won't trust censored models ever again.
  - krick 1y ago
    Recently I somehow wasn't using LLMs locally and relied mostly on ChatGPT for casual tasks. I think it was a little less than a year since I played with ollama, and I remember that my impression was that all recent popular models definitely aren't "uncensored" in a sense that some older modification of llama2 I used was, and all suck for prose-related tasks anyway. In fact, nothing but ChatGPT models seemed good enough for writing, but, of course, they refuse to talk about pretty much anything. Even DeepSeek is not great at writing, and it it much bigger than anything I ever ran locally. So, are there even good uncensored models now? Are they, like, really uncensored?

rynn 1y ago

Azure AI Foundry says: How was the model trained? The gpt-oss models were pretrained primarily using synthetic data along with some heavily filtered real code. The models were then post-trained using distillation (RLKD against o3/hailmary) and berry. For more details about the model training process, please refer to the documentation. Meaning the author was spot on.

jnmandal 1y ago

I think its likely that soon the "AI" aspect of the "models" get worse and worse with each iteration. This is because: 1. data pollution/dilution -- with each subsequent generation more and more LLM-produced content will make up the bulk of the web this means it either gets into get into the data set, which makes the model "spikey" or reduces the relevant knowledge available which dumbs the model down 2. LLMs were sort of a freak breakthrough in deep learning and while its remarkable, tuning them only makes them marginally better. Diminishing returns apply here. Its a new LLM but its still an LLM -- years old technology now However, despite these two realities, the total utility/productivity/societal gain from a product (note I didn't say model) like ChatGPT can still increase by orders of magnitude. That is because companies like OpenAI, and many, many, other technology corporations and startups are figuring out how to leverage LLMs to do stuff much more powerful than just answer questions from the corpus (ie: perform quality research, reevaluate itself, computer controls, etc, etc). Consider for example, flat screen displays were pioneered in like the 50's and arguably, they didn't become disruptively useful until the advent of smart phones. So yeah, the model may get sort of worse or maybe more brittle, but it almost doesn't matter if they are figuring out what to do with the model and making the model more useful for actual tasks. Sure its cute to talk to an AI persona and ask it questions but that is probably the least important aspect of these type of models. Microsoft word had Clippy, and yeah that was cool. But productivity gain came from word processing, the '.doc' filetype, filesharing, editing, etc. Clippy is just a meme now and that's a likely future scenario for the "chat" features in LLM products IMHO.