10 ms·
Phi-2: The surprising power of small language models
- etergri 3y agoAre these commercially usable? I believe not but I'd better ask
- deleted 3y ago[deleted]
- thro1 3y agoNo surprise - humans are (late version ;) of small language model too ! (as Bing Chat says: https://yanirseroussi.com/2023/04/21/remaining-relevant-as-a-small-language-model/ https://yanirseroussi.com/2023/04/21/remaining-relevant-as-a... )
- PoignardAzur 3y ago> With its compact size, Phi-2 is an ideal playground for researchers, including for exploration around mechanistic interpretability, safety improvements, or fine-tuning experimentation on a variety of tasks. I think this aspect is underrated. Research are pouring more and more effort into things like monosemanticity, for which small very powerful language models will probably be extremely helpful.
- dragganaitool 3y ago[dead]
- hexomancer 3y agoThe website seems to be down. Here is the cached version: https://webcache.googleusercontent.com/search?q=cache:r-GYHX2IwQYJ:https://www.microsoft.com/en-us/research/blog/phi-2-the-surprising-power-of-small-language-models/&hl=en&gl=ae https://webcache.googleusercontent.com/search?q=cache:r-GYHX...
- bratao 3y ago2.7B size with better performance than Mistral 7B is impressive!!
- reqo 3y agoSerious competition in the small model space recently. The main goal of models this small is to be deployed locally to phone/laptop (consumer electronics soon maybe?) I wonder if this will lead to a new generation of apps/UI, if it already has not. Edit: typo
- collaborative 3y agoCan't access the model link due to sone MS auth issue. Does anyone know how large (in GB) the model is? Can it be run locally or is it azure-only?
- sytelus 3y agoFile size on disk is ~10GB.
- armcat 3y agoInteresting, so they are using single precision (fp32), so 2.7B x 4 Bytes = ~10 GB. With CUDA overhead and room for context, you would need at least 12GB VRAM. They could use half precision and half that VRAM requirement and save costs for everyone involved. Maybe there is a performance reason why they use full precision.
- collaborative 3y agoYes, the reason I asked is that when I see SLM I got all excited thinking "finally a small model that fits in cheap hardware for simpler tasks"
- eightysixfour 3y agoModels at this size are regularly quantized to 5/4bit to reduce the size. While there is some degradation, it isn’t as substantial as you would expect.
- deleted 3y ago[deleted]
- sigmar 3y agoThat's just the parameters? So 32 bit parameters? This blogpost is incredibly misleading by putting Gemini nano-2's "size" as larger than Phi-2 (in Table 2 displaying only the number of parameters) and saying "Phi-2 matches or outperforms the recently-announced Google Gemini Nano 2, despite being smaller in size." Because Gemini nano parameters are 4 bit. So Gemini nano-2 is 1.6 GB (3.25/2) in size compared to Phi-2's 10GB
- duchenne 3y ago> The training for Phi-2 took 14 days on 96 A100 GPUs This would mean that it costs around ~30k USD to train. If training an LLM becomes cheaper than buying a car, it could democratize AI a lot.
- eternauta3k 3y agoYou don't need to train it again, Microsoft already did. Unless you want to develop a new one, then you also need the team of researchers/engineers.
- alecco 3y agoNote the model is trained on data generated by GPT-4. It's probably orders of magnitude more expensive to generate the data at current API prices. The whole point of these papers is that training data quality is key. I would much prefer for these companies to release the training data than the weights. But that will never happen. "We speculate that the creation of synthetic datasets will become, in the near future, an important technical skill and a central topic of research in AI."
- verdverm 3y agoThis sounds like the methodology from "Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes" i.e. master teaches apprentice or LLM trains SLM https://arxiv.org/abs/2305.02301 https://arxiv.org/abs/2305.02301 (May '23)
- eightysixfour 3y agoYes, I think we are seeing the beginning of a feedback loop where we can use current LLMs to generate better datasets at a scale large enough to create new LLMs. This is the positive feedback loop that I think is going to make the biggest difference in model quality over the next few years.
- digdugdirk 3y ago
- _lvbh 3y agoWe don’t have any real information on it but the benchmarks make me feel like some test data made it into the training set
- dafelst 3y agoThey address this in the document, citing both an effort to eliminate test data from the training set, as well as the usage proprietary, ostensibly private test sets for evaluation.
- YetAnotherNick 3y agoSomeone showed embedding similarity is not enough to find data leakage.
- monlockandkey 3y agoCan we download this model locally or is it Azure only?
- antimatter15 3y agoLooks like it is possible to download it locally, but as far as I can tell you have to manually copy all the various files from the Artifacts folder individually
- niutech 3y agoCould you post a magnet link?
- ShamelessC 3y agoWhat makes you think they have a magnet link?
- sowbug 3y ago"Could you...?" is not necessarily a question about one's ability. It is sometimes a colloquial request to do something, similar to "Would you mind...?"
- niutech 3y agoNever mind, it is already on HuggingFace: https://huggingface.co/microsoft/phi-2/tree/main https://huggingface.co/microsoft/phi-2/tree/main
- ofou 3y agoWhat is the context window?
- bionhoward 3y agoheck, microsoft seems sketchy AF, i reported em to the department of justice today, here's a link to the evidence of microsoft's anticompetitive conduct https://i.postimg.cc/MGqPvPz5/cartel-microsoft-microsoft-openai-microsoft-github-nvidia-cartel-customer-noncompete-clauses.png https://i.postimg.cc/MGqPvPz5/cartel-microsoft-microsoft-ope... just seems really serious and makes me not want to have anything to do with microsoft microsoft openai, microsoft github, or nvidia ever again. the timing seems suspicious since i sent the email to DOJ today, but im a nobody so hey, what do i know? tried a billion ways to remedy by contacting them directly before i sent that. you know it's a stressful workday when you're sweaty from making a google drawing! Ironically, Mistral cofounders just took down their similar clause. Google has nothing like this. Anthropic and Inflection both do. I'm sick of it, but I feel morally obligated to keep speaking out about this. Satya could just go into the codebase and delete it, but he didnt, I asked him to, multiple times. A WS also has such terms, really bad because they're in charge of Rust Language. Conflict of interest. TL;DR: I meekly suggest we boycott these companies because they are heavily and explicitly anti-competitive and this goes for all their businesses (Microsoft OpenAI and Microsoft GitHub) also it all runs on NVIDIA chips which have similar terms. Great job on Phi, but this is not for me!
- abeppu 3y agoIANAL, and I'm genuinely curious -- are these customer non-compete clauses actually illegal? These clauses do seem clearly anti-competitive, but is that enough? Your complaint mentions attempts to "acquire and maintain monopoly power" which seems like a stretch given that no company seems within reach of a "monopoly" ... which is why you're able to rattle off a list of companies in this space. Like, if they're not actually conspiring, but they're each trying to squash competition, and a likely result is that only a small number of rich organizations have the means (including user data) to continually improve LLMs ... is that competition-blocking a crime?
- ctoth 3y agoThere really isn't a respectful way I can say this. Please reach out and speak to a therapist or similar professional. This stream of consciousness where you talk about personally reaching out to random CEOs and expecting them to do things for you and being surprised when you don't get a response reminds me strongly of when someone close to me was going through a manic period of delusional breakdown. Please, talk to someone.
- echo_time 3y agoShame about the "research use only" limitation. That performance really puts local use in range for all sorts of devices - and with (allegedly) great performance! The future is bright/terrifying.
- dang 3y agoRelated article (via https://news.ycombinator.com/item?id=38620339 https://news.ycombinator.com/item?id=38620339, but we moved those comments hither): https://venturebeat.com/ai/microsoft-releases-phi-2-a-small-language-model-ai-that-outperforms-llama-2-mistral-7b/ https://venturebeat.com/ai/microsoft-releases-phi-2-a-small-...
- man4 3y ago"""""safety""""" score lmao
- deleted 3y ago[deleted]
- GolDDranks 3y agoGPT-3: 174B parameters. Phi-2: 2.7B parameters. Indeed, parameter-wise that is small, 65 times smaller. However... GPT-3: trained with 300B tokens. Phi-2: trained with 1400B tokens. The volume of training data, on the other hand, is around 5 times larger. For curiosity, just the other day I calculated that a human baby learns a language with around 30M "token-equivalents" of learning data. This sounds, to me, a reasonable argument for innatism, that is, human "architecture" is biologically geared for language acquisition and contains some strong "guides" or constraints that reduce the hypothesis space of "possible human languages". I wonder if language models are able to find similar architectures that would allow learning with less data.
- gjm11 3y agoThe baby, unlike the LLM, isn't learning just from a stream of text. Anything the LLM knows about the world that a lot of that text describes has been inferred from the structure of the text, but a baby learning (say) English learns what a car is by having someone point at a car and say "car", etc. I think this means (1) that there's a lot more than those 30M tokens of actual input and (2) the input is better structured for teaching the baby about the world as well as the language.
- lumost 3y agoThe LLM is also trained to predict next token given a sequence of "real" tokens. This means that any case where the LLM predicts tokens that don't look like the "real" token string is Out of Domain. Children learn in an iterative manner generating language, self-evaluating their own language, given validations from parents, and interactions with others. We've also cultivated a curriculum learning approach to make sure that students are always learning language. We really don't know how small of a corpus is required for an LLM to learn. Perhaps there is an optimal curriculum which could be generated that demonstrates progressively more complex language and knowledge tasks to keep gradients large throughout training. Or a larger LLM could generate new training examples based on the cross-entropy loss of a given batch.
- GolDDranks 3y agoOf course, you are totally correct about the baby not being limited to text input. It would be interesting to see in the future whether building "world models" through multimodal input would help scaffolding language ability with less data.
- peter_d_sherman 3y agoHere's a purely academic question about AI / Language Models: What is the smallest, tiniest amount of parameters that will result in a language model which understands how to add, multiply, subtract, divide, handle "if..then" type constructs, and perform loops? In other words, what is the smallest amount of parameters for a language model that will result in a Turing-complete (https://en.wikipedia.org/wiki/Turing_completeness https://en.wikipedia.org/wiki/Turing_completeness) AI LM -- regardless of how few words it would recognize? ? I submit that one to all of the (L)LM AI researchers in the world. Tiniest Turing-complete Language Model, please!
- tobbe2064 3y agoSuch a LLM model will never exist.
- renonce 3y agoGPT-1 with 100 million parameters is suffucient
- stavros 3y agoIt's interesting to see two replies to the grandfather, one saying this has existed for years, and one saying this will never exist.
- renonce 3y agoWondering if anyone actually care about this? Turing-complete LMs is an idea that I’ve had for years but I’m not sure if anyone would ever be interested in this. Anyone who knows about the academic publishing process?
- stavros 3y agoYep: https://arxiv.org/pdf/2301.13196.pdf https://arxiv.org/pdf/2301.13196.pdf
- regularfry 3y ago
- simonw 3y agoI was confused about whether or not they've released the weights for Phi-2 They HAVE released the weights, but you have to sign into Azure studio to get them. https://twitter.com/SebastienBubeck/status/1734801722831413312 https://twitter.com/SebastienBubeck/status/17348017228314133... has a screenshot: > to download phi-2 go to Azure AI Studio, find the phi-2 page and click on the "artifacts" tab. See picture.
- kragen 3y agopresumably someone will upload them to huggingface tho?
- simonw 3y agoThe license says you're not allowed to redistribute the weights, so it would have to be illicit.
- kragen 3y agothat might be a concern in the eu, due to sui generis database protection laws, but they probably aren't copyrightable in the us under the feist doctrine, so it's probably the license that's illicit https://en.wikipedia.org/wiki/Feist_Publications,_Inc.,_v._Rural_Telephone_Service_Co https://en.wikipedia.org/wiki/Feist_Publications,_Inc.,_v._R....
- nojvek 3y agoThey distributed phi1.5 to huggingface but I believe for phi-2 Microsoft is only adding it to Azure AI Studio in the hopes that it will make developers sign up for it. https://huggingface.co/microsoft https://huggingface.co/microsoft I wonder why Microsoft is choosing not to go where rest of OS folks hang out. huggingface is the github of ML and I was under impression Microsoft was going to eventually acquire huggingface since it's right up their alley.
- weberer 3y ago>I wonder why Microsoft is choosing not to go where rest of OS folks hang out. Is this rhetorical, or is it just your first time dealing with this awful company?
- niutech 3y agoIt is already published on HuggingFace: https://huggingface.co/microsoft/phi-2 https://huggingface.co/microsoft/phi-2
- nojvek 3y agoAmazing. Thank you Microsoft.