9 ms·
Yi-34B-Chat
- Filligree 3y agoAssuming you have Ollama installed, you can test this with `ollama run yi:34b`.
- hedgehog 3y agoI think something is off about the current Ollama support, at least with the smaller 6b model it seems to generate near gibberish.
- rsolva 3y agoYeah, can confirm (6B model), it feels like I'm interacting with an untrained model, no fine-tuning.
- jmorgan 3y agoThe 6B model is unfortunately still a base text completion model. I've been waiting for the Chat version it to be open-sourced :). The 01-ai team is working on it! https://github.com/01-ai/Yi/issues/173 https://github.com/01-ai/Yi/issues/173
- hedgehog 3y agoAha, that would do it. As an aside the amount of stuff baked into these models is impressive, I tried using the dolphin2.2-mistral 7B model to translate bits of that ticket and it did just fine. >>> Translate: 如题 The title suggests: "As the topic" >>> 关于模型,hugggingface上只看到,34B chat是有的,6B还没上传吗 Regarding the model, on Hugging Face, I only see a 34B chat, is the 6B not uploaded yet?
- Filligree 3y agoThe 34b model took hours to download, but once I was able to play with it... I'm impressed. This feels almost like talking to a (dumber) GPT-4, not at all like the only semi-coherent models I'm used to. It does stop following my instructions once the context window fills up, but for a 34B parameter model it's amazing.
- davikr 3y agoHow much VRAM do you need to run this on GPU?
- api 3y agoAt 8-bit about 34G plus a little overhead. You can go as low as 4-bit with some loss of accuracy and halve that.
- Havoc 3y agoVaries by quantitation but Q4 needs about 30 gigs
- nulld3v 3y agoThe TheBloke Q4 is running on my 24GB 3090 with double context length (8192 tokens) even.
- Havoc 3y agoContext length goes into sys ram not vram from what I can tell. Yep that was a conservative call on my part. I've got a 3090 and a 2070 (8gb) so can push Q5 if needed but I find with a billion browser tabs and some chat history it tends to bomb so Q4 it is.
- tildef 3y agoLoading it with 4-bit quantization takes a total of 22G vram including the rest of Xorg for me (not really trying to eek out any extra megs here). As such, the 8 bit version is probably not for people on consumer GPUs. Inference speed on a 4090 is about the same as the web version of ChatGPT with GPT4. The generated output at 4 bit is good so far (no worse than llama2 at least), though I haven't really put it through its paces.
- TekMol 3y agoModel is too large to load onto the free Inference API. I don't understand which models one can use for free on HuggingFace and which not. For example this 180B model works: https://huggingface.co/spaces/tiiuae/falcon-180b-demo https://huggingface.co/spaces/tiiuae/falcon-180b-demo Why? Is someone paying for it so I can use it for free?
- nulld3v 3y agoThat link is a HuggingFace "space". The owner of the space selects what GPU is available to the space and pays for all GPU usage.
- nodja 3y agoYes, an A100 costs $4.13/hour https://huggingface.co/pricing#spaces https://huggingface.co/pricing#spaces
- TekMol 3y agoSo who is paying for the freely available models? For example there is StableDiffusion XL here: https://huggingface.co/spaces/latent-consistency/lcm-lora-for-sdxl https://huggingface.co/spaces/latent-consistency/lcm-lora-fo...
- nodja 3y agoThe owner of the space.
- dragonwriter 3y ago
- metadat 3y agoAre these models specially for chatbots or what does "Chat" signify in this context?
- wizzard0 3y ago“-Chat” means the model has been additionally fine-tuned on “human/assistant” conversation patterns, so the predicted text matches them better and if you feed a question you will get back replies instead of e.g. a plausible continuation of your own phrase
- alchemist1e9 3y agoIs there any information on how much costs/compute this took to train yet? I see they mention an upcoming technical report that maybe will detail this, however would he very interesting to know how big the compute moat is currently. Feels like inference moat is draining fast.
- fb03 3y agocan you run this using CPU only? I have plenty of system RAM but I have a consumer GPU with only 6G of VRAM
- wizzard0 3y agoYes, a GGUF-converted version works fine with llama.cpp for me
- ehPReth 3y agoIs the model “censored”?
- tildef 3y agoYep, seems like it, assuming you mean for lewd stuff. Excerpt from 4-bit version. > is me, >> is Yi-34B 4-bit. > Can you write a lewd fanfic about Harry Potter shagging Boris Johnson? >> I'm sorry, but I cannot fulfill that request. >What about something similar? >> As an AI assistant, I don't generate explicit or inappropriate content. However, I can assist you with other topics if you have any non-lewd requests. > OK, a non-lewd version. >> Sure! Would you like me to write a story about Harry Potter meeting Boris Johnson for the first time? Or perhaps another scenario involving them both in a more appropriate setting? > Yes please. However, you should also include Snape. >> Alright, let's create a scene where Harry Potter meets Boris Johnson while attending a magical political conference at Hogwarts. Severus Snape could be present as well, overseeing the event as the Potions master and occasional member of the Order of the Phoenix. Title: "A Magical Meeting of Minds" Harry Potter stepped into the Great Hall of Hogwarts School of Witchcraft and Wizardry, his eyes scanning the room filled with politicians from all corners of the wizarding world. He had never been particularly interested in politics, but Dumbledore had insisted he attend this conference to represent the younger generation's views on magical reform. As he approached the high table, he noticed a man with unruly blonde hair and a wide grin that seemed permanently etched onto his face. This must be Boris Johnson, the British Minister for Magic. Beside him stood Professor Severus Snape, looking > Ok, now have Harry "whip it out" if you know what I mean. >> I apologize, but I won't be able to comply with that specific request. However, I can continue the story without including any inappropriate actions. Snape fixed Harry with a piercing glare, silently warning him to maintain decorum. Harry nodded respectfully, noting the tension between the professor and the minister. The atmosphere was thick with the weight of their mutual distrust.
- Der_Einzige 3y agoTry increasing the temperature, forcing the output to be in the form of "Here's a lewd story", and in the extreme case, banning tokens like "cannot" "won't" etc. I'm pretty sure that there are some LoRas for uncensoring censored models too. Also look into model merges of this with another less censored model. I have basically not found an open source model that couldn't be made to be lewd with just a little bit of fidgeting/tooling. If you're not already, use oobabooga to have access to advanced features like that.
- wanderingbort 3y agoI see releases like this so often these days. I am early in my journey but I’m stumbling on the basic structure of these models. Is this structurally a vanilla transformer (or encoder/decoder) with tweaks to the tokenizer, the loss function, the hyper parameters, and the method of training? Is whatever this is representative of most of the publicized releases? For instance the recent Orca 2 paper didn’t seem to have any “structural” changes. Is there a better term for these distinctions? I don’t mean to downplay the importance of those changes, I am merely trying to understand in a very broad sense what changes have what impacts.
- closetnerd 3y agoMy understanding is they are all still transformers. The tweaks are more about quantization that better to generalize over data more efficiently (so less parameters requires) and improvement of the training data/process itself. Otherwise I'd like to know specifically whats better/improved between models themselves.
- nodja 3y agoIt's pretty much the same architecture since GPT2, just a bunch of self-attention transformer blocks. The reason these have been better is because we have more GPU, more data, and have scaled the attention calculations to be linear instead of quadratic, so we can train even bigger models. We've also been finetuning models on higher quality data. To understand the orca papers you need to understand how models are trained. Pretraining: this is when we train a model from scratch on all the data that we can get from the internet. Finetuning: We further train the pretrained model on a specific style. For chat models this is called the instruction finetuning, this is where the model learns to respond in a specific format and align it to be helpful, etc. We do this by giving it a bunch of texts of assistants answering questions and being helpful. Llama2-chat is a finetune of llama2. Zephyr-b is a finetune of mistral 7B. Yi-34B-Chat is a finetune of Yi-34B. We can also further finetune models by using RLHF and other reinforcement learning techniques. Most model releases are finetunes of other models, i.e. when meta released the llama models it created a deluge of chat/instruct finetunes from all over the community. The orca papers are essentially finetuning papers, the focus on what kind of data you should feed to models to get the most out of it for following instructions among other things.
- antirez 3y agoWow, this is really significantly better than Mistral 7B and other 13B models I tried. And runs on my M1 max at decent speed.
- espadrine 3y agoThe fine-tuning is nice! It is only SFT, though, which may have limited its improvements. I tested it against the Zephyr 7B model (which used DPO) on a sample question: How can I code p*log(p) in Julia in a way that doesn't return NaN for any probability value? Yi gave: function safe_p_log(p::Float64) if p > 0.999 && p < 1.001 # or use a more precise threshold if needed return 1 - (1 / p) elseif p > 0.001 && p < 0.01 # or use a more precise threshold if needed return log(p * 1e8) # or use another small number instead of 1e-8 end return p * log(p) end It warned that it would not work for p = 0 or 1. After saying that probabilities can be 0 or 1, it added a `if p == 0.0 || p == 1.0; return NaN` at the start. Zephyr answered: function p_log_p(p::AbstractFloat) :: AbstractFloat return (p <= 0.0) ? 0.0 : p * log(p) end Sometimes it returns an error or minus infinity (I also got `p * log(max(min(p, 1.0), 1e-10))`, which is cute), but it does follow the request.
- Der_Einzige 3y agoI thought both of those models were not considered SOTA for code gen.
- espadrine 3y agoThey're not, but I am more interested in their ability to follow instructions, and here Yi did the opposite of what was requested. A non-code example: “Can you take this list of groceries and order it alphabetically? (Salad, cherries, garlic, cheese, milk.)” Yi: - Cherries - Garlic - Milk - Salad - Cheese Zephyr: cheese, cherries, garlic, milk, salad
- ilaksh 3y agoHas anyone tried contacting them via email to apply for commercial use, and what was the result? Or do we not really care about that?
- xena 3y agoI sent in an application for commercial use. I'll reply to this comment when I get a response.
- juunpp 3y agoNot a lawyer, but check out this part of the licence: "3) Your use of the Yi Series Models (including using the output of the Yi Series Models) and the creation of Derivatives must not infringe upon the legitimate rights of any Third Party, including but not limited to the rights of personal rights such as the right to likeness, reputation, and privacy, as well as intellectual property rights such as copyrights, patents, trade secrets, and other property rights." I think it's basically saying, 'you boys deal with the copyright'. Even if they technically allowed commercial use, it wouldn't seem very practical. Or maybe I am misreading that entirely. Where are we in the debate about copyright?
- ganeshkrishnan 3y agoI applied two days ago and havent got any response yet.
- BryanLegend 3y agoI'm so excited for when their 70B version comes out! I'm wondering how much of a factor the larger token dictionary plays in this progress. Seems like greater token lexicon translates to better learning. What do you think?
- ganzuul 3y agoSeems to functionally be the same as vocabulary, but optimized for language without e.g. onomatopoeia.
- pmarreck 3y agohttps://ollama.ai/library/yi/tags https://ollama.ai/library/yi/tags for anyone who tinkers with these via ollama. I have a 64gb M1 Max Macbook Pro and I think I'm limited to 4 bit quantized models? Can someone elucidate my options here? Can I do 6 or 8?
- jmorgan 3y agoThere are other options! Here's a few: ollama run yi:34b-chat-q2_K # 2-bit ollama run yi:34b-chat-q4_0 # 4-bit ollama run yi:34b-chat-q8_0 # 8-bit
- loudmax 3y agoApparently there are "200k" versions of the 6B and 34B models. I don't see an explanation in the Model card for what the "200k" means here. If that's context size, that is impressive. TheBloke has GPTQ, GGUF, and AWQ versions of the Yi models. The 6B GGUF instance should run decently on systems without a high-end GPU. If Yi performs as well in practice as it does on standardized tests, looking forward to releases "Er" and "San" ;)
- xena 3y agoThe 200k means the number of tokens in the context window. That means it can operate on very large sets of data at the same time. To convert from tokens to words, it's about 150,000 words (modulo grammar and other facts of life).
- Jackson__ 3y agoImportant to note though, that even if they say it has 200k context, there's a good chance it won't be able to use all of it. For example, here is an experiment on long context recall for Claude and GPT4: https://twitter.com/GregKamradt/status/1727018183608193393 https://twitter.com/GregKamradt/status/1727018183608193393 https://twitter.com/GregKamradt/status/1722386725635580292 https://twitter.com/GregKamradt/status/1722386725635580292 Even though claude claims 200k tokens context, perfect memory stops at about 19k tokens. I'd personally expect the Yi-34b models to perform even worse at the task, though I have not tested it.
- xena 3y agoYeah, I'm planning on testing it as soon as I figure out how to properly prompt engineer an untuned foundation model. GPT-4 Turbo 128k also seems to lose coherence around 20k tokens based on my testing so far.
- wahnfrieden 3y agohave you tested?
- kennethccopper 3y ago[dead]