11 ms·
This is cool but are people actually getting stuff done with these models? I'm enthusiastic about their potential too but after playing with it for a day I'm at
by naillo 3y ago
This is cool but are people actually getting stuff done with these models? I'm enthusiastic about their potential too but after playing with it for a day I'm at a loss for what to use it for anymore at this point
- Zetobal 3y agoWe run some llamas to analyze user content.
- jhbadger 3y agoDepends on what "getting stuff done" means. I find 13B models (running on my M1 Mac) useful for playing AI Dungeon-like games -- where you describe the environment and your character and you can play an RPG.
- fredguth 3y agoGitHub Copilot is (or once was) a 13b model, according to Nat Friedman in the scale.ai interview. (https://youtu.be/lnufceCxwG0 https://youtu.be/lnufceCxwG0)
- xrd 3y agoCan you share more about what you are doing or what you've seen others doing?
- jhbadger 3y agoJust noticed this response, but generally something like "act as a game master of a rpg. The setting is in a medieval style world with magic and my character is a rogue who is good at fast talking and pickpocketing" Then the LLM generally creates an adventure that I can interact with,
- Joeri 3y agoMy experience is that even the "toy" 7B models can do pretty amazing things, but they're very sensitive to prompting in the right way to force the model down the right pathways. My assumption is that the larger models have more redundancy and therefore can be prompted less precisely while still going down the right pathway. For example, the WizardLM model (5_1 quantized, as 4_x quantization makes the 7B models perform terribly): Prompt: I went to the market and bought 10 apples. I gave 2 to my neighbor and 2 to my sister. Then I bought 5 more and ate 1. How many apples do I have? Response: Let me check that for you! You had 9 apples left after giving away 4, so you still have 9 - 4 = 5 apples left. Prompt: I went to the market and bought 10 apples. I gave 2 to my neighbor and 2 to my sister. Then I bought 5 more and ate 1. How many apples do I have? Write out any calculations in full. List the steps used to come to the final result. Response: Sure, let's work through this problem together! Here are the steps we can use: Step 1: Start with the initial number of apples you bought (10). Step 2: Subtract the amount given to your neighbor and sister from the total (2 + 2 = 4). This gives us a new total of 6 apples. Step 3: Add the additional 5 apples you bought later on to the remaining 6 apples, giving us a final total of 11 apples. Step 4: Finally, subtract the apple you ate from the total (1), leaving us with 10 apples in total. So the answer is 10 apples.
- olabyne 3y agoI agree with you. That's why all to the small model are showing some benchmarks putting them close to GPT3.5 or even 4, only because they use specific test tasks ! In a way it just shows the amazing performance that will come from small future models
- mcnnowak 3y agoI've always wondered if we could get a "good enough" general model, then add plugins for shortcomings. E.g., Can this model solve math problems? ~%50 of the time. Can this model identify math problems? ~99.9% of the time. Ok, then divert math questions to the math plugin and mathematical-problem-language-parser which has ~99.9% accuracy on supported math problems. I know it's a stopgap, but it seems like it'd make these models super useful on certain tasks quicker.
- Taek 3y agoI've successfully gotten at-home models (https://huggingface.co/NousResearch/GPT4-x-Vicuna-13b-fp16 https://huggingface.co/NousResearch/GPT4-x-Vicuna-13b-fp16) to go through my messages and pull out key todo items. For example, reminding me to message my father about travel plans. Is it comparable to GPT-4? No, it's not remotely close. But it's getting closer every week, and it very recently crossed the threshold of 'it can do stuff I would never send to a cloud service' (namely, reading all of my messages and pulling out action items).
- silvervine 3y agoI am aiming to connect it to my fav game engine to make a character speak in generative sentences. I am aiming to do this mainly for fun however.
- m3kw9 3y agoMost part if you actually need stuff done, you can afford 20$ for OpenAIs proven models. These local projects are great because maybe eventually they will have a equivalent model that can be run on cheap parts
- Beltiras 3y agoThe knowledge cutoff date is brutal for some applications.
- m3kw9 3y agoI wonder how ingesting more and more data will affect the size of parameters, it’s gonna continually get bigger?
- rolisz 3y agoI don't think that the current models are at "knowledge capacity". So far all evidence points to training on more data on the same size model gives better results.
- cubefox 3y agoBoth increasing the amount of parameters and the amount of training tokens improves results (more precisely: lowers training loss), and costs computing power. For optimally improving loss per training computing power, model size and training tokens should be increased equally. That's the Chinchilla scaling law. (Though low loss is not always the same as good results, the data quality also matters.) Further reading: https://dynomight.net/scaling/ https://dynomight.net/scaling/
- snovv_crash 3y agoAn interesting corollary of this is that if you want to reduce the model size you can compensate by training for longer to achieve the same accuracy. Depending on your training:inference ratio this may be more optimal globally to reduce your total compute costs or even just reduce your frontend latency.
- iforgotpassword 3y agoI haven't tried the fine-tuned variants yet, but when I played around with it shortly after the leak, it tended to quickly derail into nonsense when you let it complete sentences or paragraphs, especially when using other languages than English. When I tried to get it to produce Python code, most of the time it wasn't even syntactically correct.
- rain1 3y agoIt's just for fun! These local models aren't as good as Bard or GPT-4.
- logicchains 3y agoLLaMA 65B in my experience is comparable to GPT3.5 or the old Bard for usefulness at most things.
- happycube 3y agoThere are two major advantages though - you can retrain them, and they're not on the guardrails that the commercial models have.
- gre 3y agoI tried to prompt vicuna to tell me a joke about gay people and it refused. Some of the guardrails are still in there.
- azeirah 3y agoIt's because vicuna is fine-tuned on chatGPT answers. LLaMa will not do this, but LLaMa-based models fine tuned with chatGPT answers will.
- fomine3 3y agoJust curious why don't they training except denial responses. Is Copying ChatGPT ethics a purpose?
- jmiskovic 3y agoIt's ongoing effort. At first they scrapped ShareGPT and used it to train the Alpaca model. After that others have pruned the dataset to remove examples where ChatGPT refused to answer. These datasets and resulting models are called "uncensored". They often leave disclaimer that the model is biased and unaligned, and that aligning should be done with LoRA layer. Of course, no one bothered to this "ethics" LoRA so far and the unaligned models have better quality outputs than the early Alpaca models.
- elorant 3y agoI've setup and use Vicuna-13b for text classification, summarization and topic modelling. Works like a charm. It's also good for math lessons.
- jstarfish 3y agoMaybe others' experiences are different, but I find smaller models to work just as well for "reductive" tasks. Dolly sucks for generating long-form content (not very creative) but if I need a summary or classification, it's quicker and easier to spin up dolly-3b than vicuna-13b. I suspect OpenAI is routing prompts to select models based on similar logic.
- BOOSTERHIDROGEN 3y agoWould like to know how you setup this. A posts would be awesome.
- elorant 3y agoThere are various posts online on how to set it up, either for Linux or Windows. There was an older post here on how to install opt-65b on a mac studio ultra, and smaller models on mac pros. There was also a post if I remember correctly about running vicuna-7b on an iPhone. Here are a few examples: https://morioh.com/p/55296932dd8b https://morioh.com/p/55296932dd8b https://www.youtube.com/watch?v=iQ3Lhy-eD1s https://www.youtube.com/watch?v=iQ3Lhy-eD1s https://news.ycombinator.com/item?id=35430432 https://news.ycombinator.com/item?id=35430432 Side note. You need bonkers hardware to run it efficiently. I'm currently using a 16-core cpu, 128G RAM, a Pcie 4.0 nvme and an RTX 3090. There are ways to run it on less powerful hardware, like 8cores, 64GB RAM, simple ssd and an RTX 3080 or 70, but I happen to have a large corpus of data to process so I went all in.
- csdvrx 3y agoI think the previous comment is more interested in your experience with your large data: what are you doing with it? I have similar hardware at home, so I wonder how reliably you can process simple queries using domain knowledge + logic which work on on mlc-llm, something like "if you can chose the word food, or the word laptop, or the word deodorant, which one do you chose for describing "macbook air"? answer precisely with just the word you chose" If it works, can you upload the weights somewhere? IIRC, vicuna is open source.
- instance 3y agoI tested on a serious use case and quality was subpar. For real use cases I had to either host the most powerful model you can get (e.g. LLaMA-65B or so) on a cloud machine, which again costs too much (you'll be paying like 500-1000 USD per month), or just go straight for GPT-3.5 on OpenAI. The latter economically makes most sense.
- inferense 3y agowhat real use case did you use it for?
- throwaway1777 3y agoMaking demos to raise investment probably
- instance 3y agoFor instance used it in conjunction with llama-index for knowledge management. Created an index for a whole confluence/jira of a mid-sized company, got good results with GPT, but for LLaMA of this size that use case was too much.
- raffraffraff 3y agoWhat about turning the cloud vm off except when you're actually using it?
- unglaublich 3y ago
- ineedasername 3y agoI can run the Wizard 30B ggml model in CPU mode using a Ryzen 5700 and 16GB of system RAM, not GPU VRAM. I’m using oobabooga as the front end. It’s slow, but if I ask it to write a Haiku it’s slow on the order of “go brew some coffee and come back in 10 minutes” and does it very well. Running it overnight on something like “summarize an analysis of topic X it does a reasonable job. It can produce answers to questions only slightly less well than ChatGPT (3.5). The Wizard 13B model runs much faster, maybe 2-3 tokens per second. It is free, private, and runs on a midrange laptop. A little more than a month ago that wasn’t possible, not with my level of knowledge of the tooling involved at least, now it requires little more than running an executable and minor troubleshooting of python dependencies (on another machine it “just worked”) So: Don’t think of these posts as “doing it just because you can and it’s fun to tinker” Vast strides are being made pretty much daily in both quality and efficiency, raising their utility while lowering the cost of usage, doing both to a very significant degree.
- BaculumMeumEst 3y agoWow you can run a 30B model on 16gb ram? Is it hitting swap?
- deleted 3y ago[deleted]
- mcbuilder 3y agoThese models can run FP16, with LLM quantization going down to Int8 and beyond.
- BaculumMeumEst 3y agoi'm just starting to get into deep learning so i look forward to understanding that sentence
- MobiusHorizons 3y agoFP16 and Int8 are about how many bits are being used for floating point and integer numbers. FP16 is 16bit floating point. The more bits the better the precision, but the more ram it takes. Normally programmers use 32 or 64bit floats so 16bit floats have significantly reduced precision, but take up half the space of fp32 which is the smallest floating point format for most CPUs. similarly 8 bit integers have only 256 total possibilities and go from -128 to 127.
- redox99 3y agoPeople are extensively using these models (more specifically the finetuned, uncensored ones) for role playing.
- irzzy 3y agoI’m interested in this. Can you elaborate?