3 ms·

The initial model release had a terrible, frequent, issue with emitting the wrong "end of message" token, or never emitting one.[1] That is a very serious issue

The initial model release had a terrible, frequent, issue with emitting the wrong "end of message" token, or never emitting one.[1] That is a very serious issue that breaks chat. The ones from today still have this issue.[2] Beyond that, they've been pushing new ONNX features enabling LLMs via Phi for about a month now. The ONNX runtime that supports it still isn't out, much less the downstream integration of it into the iOS/Android runtimes. Heck, the Python package for it isn't supported anywhere but Windows. It's absolutely wild to me that MS is pulling this stuff with ~0 discussion or reputation repercussions. I'm a huge ONNX fan and bet a lot on it, it works great. It was clear to me about 4 months ago that Wintel's "AI PC" buildup meant "ONNX x newer Phi" It is very frustrating to see an extremely late rush, propped up by potemkin blog posts that I have to waste time to find out are just straight up lying. Burnt a lot of goodwill that they worked hard to earn. I am virtually certain that the new Windows AI features previewed about yesterday are going to land horribly if they actually try to land them this year. [1] https://huggingface.co/microsoft/Phi-3-mini-4k-instruct-gguf/discussions/8#662e705ea47b4da4b295db25 https://huggingface.co/microsoft/Phi-3-mini-4k-instruct-gguf... [2] https://x.com/jpohhhh/status/1793003272187351195 https://x.com/jpohhhh/status/1793003272187351195

andy_xor_andrew 2y ago
in the screenshot you shared in the twitter link [2], the model appears to do everything correctly - it terminated its message with <|end|> which is correct according to the Phi-3 prompt format. It seems whatever environment you are hosting it in does not understand that <|end|> should be considered the EOS string, and not <|endoftext|> ??
- refulgentis 2y ago
  Good call, it's an HF space for Phi Vision, maybe someone jumped the gun / didn't set stuff up properly, or it's splitting <|end|> into multiple tokens
nonfamous 2y ago
FYI, there's a recipe for running Phi-3 under ONNX on iOS in the linked repository https://github.com/microsoft/Phi-3CookBook/blob/main/md/03.Inference/iOS_Inference.md https://github.com/microsoft/Phi-3CookBook/blob/main/md/03.I...
- refulgentis 2y ago
  Yes, that's the Potemkin village that broke this camel's back. It was linked in an announcement blog post yesterday. You have to build two in-development libraries, one from ToT, one of which is a dev branch to make it compile for iOS on a Mac temporarily. The dev branch doesn't actually exist. If you use the only branch by the author on the repo, it doesn't work. The dev branch that doesn't work is a few commits on top of ToT from 2 months ago. At the end of that non-existent road is a model that can't end messages properly, in MyThing.app that uses llama.cpp, or LM Studio, or Ollama, or MS cloud API. I can't ship on that, and neither can anyone else.

andy_xor_andrew 2y ago

in the screenshot you shared in the twitter link [2], the model appears to do everything correctly - it terminated its message with <|end|> which is correct according to the Phi-3 prompt format. It seems whatever environment you are hosting it in does not understand that <|end|> should be considered the EOS string, and not <|endoftext|> ??

refulgentis 2y ago
Good call, it's an HF space for Phi Vision, maybe someone jumped the gun / didn't set stuff up properly, or it's splitting <|end|> into multiple tokens

nonfamous 2y ago

FYI, there's a recipe for running Phi-3 under ONNX on iOS in the linked repository https://github.com/microsoft/Phi-3CookBook/blob/main/md/03.Inference/iOS_Inference.md https://github.com/microsoft/Phi-3CookBook/blob/main/md/03.I...

refulgentis 2y ago
Yes, that's the Potemkin village that broke this camel's back. It was linked in an announcement blog post yesterday. You have to build two in-development libraries, one from ToT, one of which is a dev branch to make it compile for iOS on a Mac temporarily. The dev branch doesn't actually exist. If you use the only branch by the author on the repo, it doesn't work. The dev branch that doesn't work is a few commits on top of ToT from 2 months ago. At the end of that non-existent road is a model that can't end messages properly, in MyThing.app that uses llama.cpp, or LM Studio, or Ollama, or MS cloud API. I can't ship on that, and neither can anyone else.