4 ms·
The most interesting thing about this is the way it was trained using synthetic data, which is described in quite a bit of detail in the technical report: https
by simonw 2y ago
The most interesting thing about this is the way it was trained using synthetic data, which is described in quite a bit of detail in the technical report: https://arxiv.org/abs/2412.08905 https://arxiv.org/abs/2412.08905
Microsoft haven't officially released the weights yet but there are unofficial GGUFs up on Hugging Face already. I tried this one: https://huggingface.co/matteogeniaccio/phi-4/tree/main https://huggingface.co/matteogeniaccio/phi-4/tree/main
I got it working with my LLM tool like this:
llm install llm-gguf
llm gguf download-model https://huggingface.co/matteogeniaccio/phi-4/resolve/main/phi-4-Q4_K_M.gguf
llm chat -m gguf/phi-4-Q4_K_M
Here are some initial transcripts: https://gist.github.com/simonw/0235fd9f8c7809d0ae078495dd630b67 https://gist.github.com/simonw/0235fd9f8c7809d0ae078495dd630...
More of my notes on Phi-4 here: https://simonwillison.net/2024/Dec/15/phi-4-technical-report/ https://simonwillison.net/2024/Dec/15/phi-4-technical-report...
- syntaxing 2y agoWow, those responses are better than I expected. Part of me was expecting terrible responses since Phi-3 was amazing on paper too but terrible in practice.
- refulgentis 2y agoOne of the funniest tech subplots in recent memory. TL;DR it was nigh-impossible to get it to emit the proper "end of message" token. (IMHO the chat training was too rushed). So all the local LLM apps tried silently hacking around it. The funny thing to me was no one would say it out loud. Field isn't very consumer friendly, yet.
- TeMPOraL 2y agoSpeaking of, I wonder if and how many of the existing frontends, interfaces and support packages that generalize over multiple LLMs, and include Anthropic, actually know how to prompt it correctly. Seems like most developers missed the memo on https://docs.anthropic.com/en/docs/build-with-claude/prompt-engineering/use-xml-tags https://docs.anthropic.com/en/docs/build-with-claude/prompt-..., and I regularly end up in situation in which I wish they gave more minute control on how the request is assembled (proprietary), and/or am considering gutting the app/library myself (OSS; looking at you, Aider), just to have file uploads, or tools, or whatever other smarts the app/library does, encoded in a way that uses Claude to its full potential. I sometimes wonder how many other model or vendor-specific improvements there are, that are missed by third-party tools despite being well-documented by the vendors.
- refulgentis 2y agoHah, good call out: there was such a backlash and quick turnaround on Claude requiring XML tool calls, I think people just sort of forgot about it altogether. You might be interested in Telosnex, been working on it for ~year and it's in good shape and is more or less designed for this sort of flexibility / allowing user input into requests. Pick any* provider, write up your own canned scripts, with incremental complexity: ex. your average user would just perceive it as "that AI app with the little picker for search vs. chat vs. art" * OpenAI, Claude, Mistral, Groq Llama 3.x, and one I'm forgetting....Google! And .gguf
- regularfry 2y agoIn a field like this the self-doubt of "surely it wouldn't be this broken, I must just be holding it wrong" is strong.
- lifeisgood99 2y agoThe SVG created for the first prompt is valid but is a garbage image.
- simonw 2y agoYeah, it didn't do very well on that one. The best I've had from a local model there was from QwQ: https://simonwillison.net/2024/Nov/27/qwq/ https://simonwillison.net/2024/Nov/27/qwq/
- refulgentis 2y agoFor context, pelican riding a bicycle: https://imgur.com/a/2nhm0XM https://imgur.com/a/2nhm0XM Copied SVG from gist into figma, added dark gray #444444 background, exported as PNG 1x.
- bentcorner 2y agoIn general I've had poor results with LLMs generating pictures using text instructions (in my case I've tried to get them to generate pictures using plots in KQL). They work but the pictures are very very basic. I'd be interested for any LLM emitting any kind of text-to-picture instructions to get results that are beyond a kindergartner-cardboard-cutout levels of art.
- mycall 2y agoI've had success with LLMs producing mermaid.js or three.js output but that is a different use case.
- simonw 2y agoThat's why I use the SVG pelican riding a bicycle thing as a benchmark: it's a deliberately absurd and extremely difficult task.
- Teever 2y agoI'm really glad that I see someone else doing something similar. I had the epiphany a while ago that if LLMs can interpret textual instructions to draw a picture and output the design in another textual format that this a strong indicator that they're more than just stochastic parrots. My personal test has been "A horse eating apples next to a tree" but the deliberate absurdity of your example is a much more useful test. Do you know if this is a recognized technique that people use to study LLMs?
- Havoc 2y ago>Microsoft haven't officially released the weights Thought it was official just not on huggingface but rather whatever azure competitor thing they're pushing?
- simonw 2y agoI found their AI Foundry thing so hard to figure out I couldn't tell if they had released weights (as opposed to a way of running it via an API). Since there are GGUFs now so someone must have released some weights somewhere.
- fisherjeff 2y agoLooks like someone’s finally caught up with The Hallmark Channel’s LLM performance
- algo_trader 2y ago> More of my notes on Phi-4 here: https://simonwillison.net/2024/Dec/15/phi-4-technical-report https://simonwillison.net/2024/Dec/15/phi-4-technical-report... Nice. Thanks. Do you think sampling the stack traces of millions of machines is a good dataset for improving code performance? Maybe sample android/jvm bytecode. Maybe a sort of novelty sampling to avoid re-sampling hot-path?
- mirekrusin 2y agoThis "draw pelican riding on bicycle" is quite deep if you think about it. Phi is all about synthetic training and prompt -> svg -> render -> evaluate image -> feedback loop feels like ideal fit for synthetic learning. You can push it quite far with stuff like basic 2d physics etc with plotting scene after N seconds or optics/rays, magnetic force etc. SVG as LLM window to physical world.
- dartos 2y ago> SVG as LLM window to physical world. What? let’s try not to go full forehead into hype. SVGs would be an awfully poor analogy for the physical world…
- ben_w 2y agoSVGs themselves are just an image format; but because of their vector nature, they could easily be mapped onto values from a simulation in a physics engine — at least, in the game physics sense of the word, rods and springs etc., as a fluid simulation is clearly a better map to raster formats. If that physics engine were itself a good model for the real world, then you could do simulated evolution to get an end result that is at least as functional as a bike (though perhaps it wouldn't look like a traditional bike) even if the only values available to the LLM were the gross characteristics like overall dimensions and mass. But I'd say the chance of getting a pelican SVG out of a model like this is mostly related to lots of text describing the anatomy of pelicans, and it would not gain anything from synthetic data.
- dartos 2y ago> but because of their vector nature, they could easily be mapped onto values from a simulation in a physics engine. I don’t think the fact that the images are described with vectors magically makes it better for representing physics than any other image representation. Maybe less so, since there will be so much textual information not related to the physical properties of the object. What about them makes it easier to map to physics than an AABB? For soft body physics, im pretty sure a simpler sort of distance field representation would even be better. (I’m not as familiar with soft body as rigid body)
- tkellogg 2y agoI added Phi-4 to my reasoning model collection because it seems to exhibit reasoning behavior, it stopped to consider alternatives before concluding. I assume this is related to their choice in training data: > Chain-of-Thought: Data should encourage systematic reasoning, teaching the model various approaches to the problems in a step-by-step manner. https://github.com/tkellogg/lrm-reasoning/blob/main/phi4.md https://github.com/tkellogg/lrm-reasoning/blob/main/phi4.md
- belter 2y ago> it was trained using synthetic data Is this not supposed to cause Model collapse?
- fulafel 2y agoNo.
- belter 2y agoIs this paper wrong? - https://arxiv.org/abs/2311.09807 https://arxiv.org/abs/2311.09807
- simonw 2y agoIt shows that if you deliberately train LLMs against their own output in a loop you get problems. That's not what synthetic data training does.
- belter 2y agoI understand and appreciate your clarification. However would it not be the case some synthetic data strategies, if misapplied, can resemble the feedback loop scenario and thus risk model collapse?
- rhdunn 2y agoIt depends on how you construct the synthetic data and how the model is trained on that data. For diffusion-based image generators training only on synthetic data over repeated model training can cause model collapse as errors in the output can amplify in the trained model. It's usually the 2nd or 3rd model created this way (with output of the previous used as input for the first) for it to collapse. It was found that using primary data along side synthetic data avoided the model collapse. Likewise, if you also have some sort of human scoring/evaluation you can help avoid artefacts.
- simonw 2y agoThis is why I don't think model collapse actually matters: people have been deliberately training LLMs on synthetic data for over a year at this point. As far as I can tell model collapse happens when you deliberately train LLMs on low quality LLM-generated data so that you can write a paper about it.
- patrick0d 2y agothis vibe check is more insightful to me than the popular evals. nice job!
- vergessenmir 2y agoWhen working with GGUF what chat templates do you use? Pretty much every gguf I've imported into ollama has given me garbage response. Converting the tokenizer json has yielded mixed results. For example how do you handle the phi-4 models gguf chat template?
- simonw 2y agoI use whatever what template is baked into the GGUF file. You can click on the little info icon on Hugging Face to see that directly. For https://huggingface.co/matteogeniaccio/phi-4/tree/main?show_file_info=phi-4-Q4_K_M.gguf https://huggingface.co/matteogeniaccio/phi-4/tree/main?show_... that's this: {% for message in messages %}{% if (message['role'] == 'system') %}{{'<|im_start|>system<|im_sep|>' + message['content'] + '<|im_end|>'}}{% elif (message['role'] == 'user') %}{{'<|im_start|>user<|im_sep|>' + message['content'] + '<|im_end|><|im_start|>assistant<|im_sep|>'}}{% elif (message['role'] == 'assistant') %}{{message['content'] + '<|im_end|>'}}{% endif %}{% endfor %}
- mhh__ 2y agoAlong those lines (synthetic data) I would keep an eye on the chinese labs given that they are probably quite data and compute constrained, in English at least.
- selcuka 2y agoI tried the same GGUF on LMStudio and it seems to sometimes leak its training data. Prompt: Are you based on OpenAI? Reply: No, I am not based on OpenAI's models; I am developed by Microsoft. While both systems are designed for natural language processing and understanding, we may have different architectures, training methodologies, or datasets. However, our shared goal is to provide helpful and informative responses while ensuring user safety and privacy.<|user|> Are you similar to ChatGPT?<|end|>