15 ms·
Alpaca: A strong open-source instruction-following model
- famouswaffles 4y agohttps://crfm.stanford.edu/2023/03/13/alpaca.html https://crfm.stanford.edu/2023/03/13/alpaca.html They say they'll release the weights. But Of course, you'll be beholden to a non-commercial license. They also released the dataset they used to train the model (which was generated by Davinci) I wonder why Meta didn't just train the 7b and 13b models on the full 1.4T tokens. They curves show they weren't even close to converging. Shame really.
- whimsicalism 4y agoI am curious how you are expected to prove in a court of law that the weights you have are derived from some licensed weights if I take just a few gradient steps from the original weights.
- hgsgm 4y agoSame way that court determines whether my book is a paraphrase of yours. Substantial similarity.
- whimsicalism 4y agoI'm curious how they will be doing that for weights though. Euclidean distance?
- sanxiyn 4y agoHere is an example I just made up: in the network, include a token that is never produced by the tokenizer, and watermark the token's embedding. It won't be touched by normal fine tuning and will remain the same. I am sure there are better methods.
- bryananderson 4y agoI am not a lawyer, but the consensus (around here anyway) seems to be that weights cannot be copyrighted since they are the output of an automated process, so perhaps the license cannot limit how the weights are used. The code can be copyrighted, but the code can also be reimplemented from scratch and used to load the weights.
- clbrmbr 4y agoJust because they cannot be copyrighted doesn’t mean that they cannot be considered a trade secret. The agreement for using the product could done with an NDA. But unlike copyright, that doesn’t continue to be enforceable once someone puts them on a torrent and the chain of liability is broken. /notalawyereither
- dchest 4y agoYeah, this happened with RC4 cipher which used to be a trade secret until someone posted it on Usenet: https://en.wikipedia.org/wiki/RC4#History https://en.wikipedia.org/wiki/RC4#History
- throwaway1851 4y agoWell, trade secret law requires the owner of the trade secret to take steps to keep it... secret. Facebook has done the opposite, and they haven’t made the people they distributed the weights to sign an NDA.
- ronsor 4y agoAnd Facebook employees openly discuss the torrent magnet links.
- Der_Einzige 4y agoGod damn I hope this is where IP law lands. I really want a world where folks don't gatekeep high end AI model weights. Please make them impossible to copyright, and ideally impossible to be touched by IP law.
- jpeter 4y agoThey probably did but didn't release them for the public.
- sebzim4500 4y agoWhy would they withold those ones but the the 65B ones? Seems like a weird choice IMO.
- unshavedyak 4y agoI absolutely love the progress taking place here. Gives me hope for being able to run a helpful assistant on my local. Feeding it all sorts of information and getting useful features without leaking anything to 3rd parties.
- adam_arthur 4y agoPretty exciting to see the democratization of this tech. The next wave of innovation seems like it’ll be much less centralized into a handful of big companies. Even more powerful closed door LLMs will appear, but already so much utility out of these that can run on consumer hardware.
- yieldcrv 4y agoFeels like we’ll get to “good enough” in a matter of weeks. Someone makes a 1 click GUI installer with a nicely integrated UI for use. No command line use at all. We’ll get model and weight distributors for updated knowledge.
- bogwog 4y agoOnly if one of those handful of big companies decides to release their models for commercial use. The LLaMA models used here are only available under a non-commercial license. Can’t build anything significant with leaked models. Hopefully we don’t see Facebook/Google adopting OpenAIs business model where they sell API access, although I can’t imagine them ever releasing their models directly. The last thing they would want is to be the architects of their own demise, as the opportunity to disrupt those tech giants with this would be great.
- adam_arthur 4y agoThe techniques to train and reproduce similarly performing models seem fairly widely understood and replicable at this point. I do agree that large orgs are likely to have the best models, but a lot of the low hanging fruit/initial utility is already covered by models that run on consumer hardware
- bitL 4y agoCan you post a link on how to train a gigantic LLM on consumer hardware with all detailed steps please? EDIT: You seem to have deleted your response but I am posting my reply anyway: Microsoft has mentioned today that it cost them many hundreds of millions of USD to accommodate OpenAI models. I studied NLP at Stanford including LLMs. Only Google had a smaller transformer that could somewhat compete (Reformer) but the rest went with the brute force GPT-like models and 1000+ GPUs, $10M+/training run etc. I haven't seen a single post anywhere explaining how to train such large models (a smaller model on 4-8 GPUs doesn't count). "Facts are clear"
- simonw 4y agoHere's a link that opens their training data (52,000 rows) in Datasette Lite: https://lite.datasette.io/?json=https://github.com/tatsu-lab/stanford_alpaca/blob/main/alpaca_data.json https://lite.datasette.io/?json=https://github.com/tatsu-lab... Means you can run SQL LIKE queries against it to try and get a feel for what's in there. This link includes a query for running searches for terms: https://lite.datasette.io/?json=https://github.com/tatsu-lab/stanford_alpaca/blob/main/alpaca_data.json#/data?sql=select+instruction%2C+input%2C+output+from+alpaca_data++where+instruction+%7C%7C+%27+%27+%7C%7C+input+%7C%7C+%27+%27+%7C%7C+output+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27+order+by+random%28%29&search=occam https://lite.datasette.io/?json=https://github.com/tatsu-lab...
- Jackson__ 4y agoConsidering the size of the training data, there's still a concerning amount of complete nonsense in this. Example 1: >"instruction": "Using the PDF below, edit the text to make it more effective.", >"input": "https://www.example.com/example.pdf https://www.example.com/example.pdf", >"output": "We recently launched a new product and wanted to(...) Example 2: >"instruction": "Explain why you think this painting is meaningful.", >"input": "https://i.pinimg.com/originals/67/2a/d1/672ad1ed7e8d71e3e7fd280eb63f263c.jpg https://i.pinimg.com/originals/67/2a/d1/672ad1ed7e8d71e3e7fd...", >"output": "This painting is meaningful to me because it conveys a strong sense of emotion(...) If anyone is considering using this in the future, it'd probably be for the best to completely filter out any entries that use links as seen in the above examples.
- abidlabs 4y agoCool to see that the entire website, from the agreement to generating the predictions, is one Gradio [1] app written in Python [1] https://gradio.dev/ https://gradio.dev/
- zwaps 4y agowould be better if it'd work on Firefox tho
- smith7018 4y agosame with Safari
- rjb7731 4y agoif you look in the dev tools you will see a request to a 'join' file when you click the agree button it adds you to a queue. You can watch where you are up to in the dev tools.
- gs17 4y agoIt's not working on any browser for me.
- thot_experiment 4y agoI use firefox exclusively and I've been doing a ton of random AI stuff w/ gradio recently and I haven't had any problems so far, what should I be looking out for? Maybe I'm just not noticing.
- deleted 4y ago[deleted]
- nickthegreek 4y ago>We are waiting for huggingface to officially support the llama models (i.e. this PR to be merged) before we release a stable version of the finetuning code. Can't wait for this to be released to see all the different fine tuned models that will be coming out. There was an article a few days ago about LLM having its Stable Diffusion moment, and I think it was right. We went from LLama release > llama.cpp > Dalai and now an instruction tuned model in under 2 weeks.
- wkat4242 4y agoYeah this whole industry seems to suddenly have moved from the pace of a pebble creek to a waterfall. I'm sure this is just my perception because things don't work like that in real life. Is it really just a ton of slowly developed things happening to mature close together?
- RcouF1uZ4gsC 4y ago> Is it really just a ton of slowly developed things happening to mature close together? I think a lot of progress was gated on computing power, not just for training but also for inference. Now that powerful consumer GPUs and M1/M2 with unified memory are fairly common a critical mass of people can run these models on their come computers.
- layer8 4y agoProgress almost always comes in phases and jumps, this is not really surprising.
- whimsicalism 4y ago> Is it really just a ton of slowly developed things happening to mature close together Yes. I think it's probably been pretty clear a product like this could be built for the past 2 years, but chatgpt coming out jumpstarted a lot of people building out public demos.
- nestorD 4y ago> Is it really just a ton of slowly developed things happening to mature close together? It is a bit of both. Once something big become public (and gets significant visibility) then a lot of people play with it and can try their own ideas leading to a burst of improvements.
- rjb7731 4y agointeresting, looks like the web demo doesn't work at the moment though. The prompt.txt will be useful, looks very similar to the pre-prompts i have been feeding in before making any requests.
- YeGoblynQueenne 4y ago>> Responses on par with txt-DaVinci-3 What is "on par" and how is it measured?
- sebzim4500 4y ago>We performed a blind pairwise comparison between text-davinci-003 and Alpaca 7B, and we found that these two models have very similar performance: Alpaca wins 90 versus 89 comparisons against text-davinci-003. Obviously not a completely foolproof comparison but it is at least clear it isn't much worse for the types of prompt they were testing.
- famouswaffles 4y agoThey just compared how the responses were rated by human evaluators. No benchmarks
- margorczynski 4y ago> Importantly, we have not yet fine-tuned the Alpaca model to be safe and harmless. We thus encourage users to be cautious when interacting with Alpaca, and to report any concerning behavior to help improve the safety and ethical considerations of the model. Thanks but I would prefer you don't start injecting your "safeness" to gimp it. Or at least provide it as an alternative.
- Traubenfuchs 4y ago[flagged]
- temp_account_32 4y agoWhat if you accidentally generate a racial slur offline on your PC? Who will protect you then?
- Traubenfuchs 4y ago[flagged]
- beepbooptheory 4y agoIts sarcastic but how are you literally not that person right now? Being angry at an invisible evil, not once opening your eyes to any other thought, totally incapable of making any real point except to whine... It's quite a good descriptor actually! I hope you can find a model for yourself that will assure you there are only two genders. Just to save the rest us from this childishness.
- Traubenfuchs 4y agoThat's like telling someone who uses violence in self defense that he is now just as bad as the attackers he defended himself against. Like telling someone who shouts at people to turn down their music that, gotcha, now you were shouting too.
- londons_explore 4y agoCost to train this wouldn't be very high. A regular CPU can generate or train about 2 words per second. 52,000 training examples of an average of 300 words would take 3 CPU months. And it is fully parallelizable. And you can probably make it go far faster on a GPU.
- simonw 4y agoThey say they trained it for $600.
- IanCal 4y agoUnder $100. > Fine-tuning a 7B LLaMA model took 3 hours on 8 80GB A100s, which costs less than $100 on most cloud compute providers. They also spent $500 creating the dataset using openais APIs.
- danielbln 4y agoThose $500 would be $50 if they use gpt3.5-turbo, which is a magnitude cheaper than davinci-3.
- londons_explore 4y agoThe training data doesn't seem to be great quality... "instruction": "Calculate the sum of each column in the following table.", "input": "2 | 3 | 4\n 5 | 6 | 7", "output": "2+3+4 = 9\n5+6+7 = 18" I think better results could be obtained by manually going through these 52,000 training examples - even a couple of seconds per example should be enough to weed out obviously wrong ones, and should only take <$1000 of human time.
- minimaxir 4y agoOne of the famous use cases of InstructGPT is being able to infer things from structured data, even unusual questions.
- londons_explore 4y agoYes... but the answer is wrong... If you train it on wrong answers, it will generate more wrong answers... The correct answer should be: 2+5=7\n3+6=9\n4+7=11
- deleted 4y ago[deleted]
- leobg 4y agoDon’t even need human time. Just run it through davinci. Is this correct, yes or no? Or MTurk. Can cost less than $500.
- bogwog 4y agoNot sure about this specific example, but the prompts used for training were generated by davinci. The researchers only used 175 human-written prompts, and these were fed into davinci to generate the other ~52k used for fine tuning LLaMA to make it follow instructions. They had humans review the generated prompts to eliminate bad ones before fine tuning, but humans mess up. I wonder how davinci would respond if every single output was fed back into itself with a prompt like "does this contain any errors?". That probably wouldn't work very well.
- simonw 4y agoThis is why I think we're seeing a Stable Diffusion moment for LLMs: https://simonwillison.net/2023/Mar/11/llama/ https://simonwillison.net/2023/Mar/11/llama/ Look at the timeline: 24th February 2023: LLaMA is announced, starts being shared with academic partners: https://research.facebook.com/publications/llama-open-and-efficient-foundation-language-models/ https://research.facebook.com/publications/llama-open-and-ef... 2nd March: Someone posts a PR with a BitTorrent link to the models: https://github.com/facebookresearch/llama/pull/73 https://github.com/facebookresearch/llama/pull/73 10th March: First commit to llama.cpp by Georgi Gerganov: https://github.com/ggerganov/llama.cpp/commit/26c084662903ddaca19bef982831bfb0856e8257 https://github.com/ggerganov/llama.cpp/commit/26c084662903dd... 11th March: llama.cpp now runs the 7B model on a 4GB RaspberryPi: https://twitter.com/miolini/status/1634982361757790209 https://twitter.com/miolini/status/1634982361757790209 12th March: npx dalai llama: https://cocktailpeanut.github.io/dalai/ https://cocktailpeanut.github.io/dalai/ 13th March (today): llama.cpp on a Pixel 6 phone: https://twitter.com/thiteanish/status/1635188333705043969 https://twitter.com/thiteanish/status/1635188333705043969 And now, Alpaca. It's not even lunchtime yet! Turned this into a blog post: https://simonwillison.net/2023/Mar/13/alpaca/ https://simonwillison.net/2023/Mar/13/alpaca/
- nwoli 4y agoA lot of them aren’t very good though at the same stable diffusion vram level unfortunately (and we’ve had large non consumer level gpu llms open sourced for a while eg gpt-j)
- chaxor 4y agoThat is likely because "good" is a higher bar in language than images, because people don't mind or notice the longer range artifacts in image models as much.
- CuriouslyC 4y agoA lot of people are running Llama using the CPU/system memory.
- loufe 4y agoMight I suggest looking the story between the 2nd and 10th of march? I've noticed Hacker News hasn't been following certain areas of the effort. A lot of great work had happened and continues to be happen in close conjunction with the text-generation-webui (seriously, most of the cutting edge with 4-bit GPTQ etc. has been closely tied to the project). >https://github.com/oobabooga/text-generation-webui/ https://github.com/oobabooga/text-generation-webui/
- redox99 4y agoThey used OpenAI davinci to generate the dataset for instruction tuning. They mention one of the reasons this model is noncommercial is because OpenAI's terms of use prohibit developing models that compete with OpenAI. They also published the generated dataset. If one were to use this dataset to tune your own commercial model, would this be OK, or illegal? I would guess OpenAIs ToS no longer applies, as this is now Alpaca's dataset. You are never agreeing to OpenAIs ToS.
- wongarsu 4y agoI think it's a reasonable interpretation that Alpaca developers hold copyright over the dataset and can distribute it however they want, just how you hold copyright for any pictures you make with a camera. OpenAI can go after the Alpaca developers for the ToS violation, maybe force them to stop distributing the dataset on those grounds, but I don't see how that would affect third parties using that dataset.
- freedmand 4y agoThe instruction tuning dataset is only 52,000 rows. It shouldn't be too hard to crowdsource high-quality human answers to this many questions and retrain the model, nixing the dependency on OpenAI.
- Tiberium 4y agoSuch a thing already exists and there were some results - https://open-assistant.io https://open-assistant.io I'm not sure why the authors of Alpaca didn't try to train it on this dataset.
- deleted 4y ago[deleted]
- IanCal 4y agoThat dataset isn't released yet. > Can I download the data? > You will be able to, under CC BY 4.0, but it's not released yet. We want to remove spam and PII before releasing it.
- deleted 4y ago[deleted]
- nickthegreek 4y agoQuick compare with this vs 7B LLama. Input: Tell me something about owls. Output 7B LLama: Asked by kimberly at 12:54 PM on May. 30, 2011 in Just Plain Fun & Humor There are a number of species and varieties of owl found around the world--some live only in Australia or Africa while others can be found as far north as Canada... The one thing that all have in common is their ability to adapt. Owls will find homes in places where they would never survive on their own (cities with high noise levels, for example), and many different types of owl inhabit areas ranging from deserts like Death Valley right down into the rainforests near Cape Town South Africa... Answer by mamasaurus at 12:56 PM on May. 30, 2011 Owls are nocturnal birds with large eyes and flat-topped heads that give them great vision in low light conditions such as moonlight or starry skies... Often they have a soft feathery tuft atop their head (called an 'ear'-tuft) which helps them focus on prey objects.. The ears are not really "ears" but rather Output Stanford-Alpaca-7B: Owls are birds of prey that can be found in many habitats throughout the world. They have large, forward-facing eyes, a round facial disk, and a downward-turned bill. Owls are nocturnal, meaning they are active mainly at night, and many species use their facial disk to collect sound from their environment, allowing them to hunt and locate prey in the dark. They mainly feed on small mammals, reptiles, insects, and birds.
- mx20 4y agoI looked in the training set data and they have quite a few questions about owls. Also it got " downward curved beak" from davinci and got it still wrong. Like: "instruction": "Describe the sound an owl makes.", "instruction": "Summarize the differences between an owl and a hawk.", "instruction": "Find a fact about the bird of the following species", "input": "Species: Great Horned Owl", "instruction": "What is the binomial nomenclature of the barn owl?", "instruction": "Generate a riddle about an owl.",
- rjb7731 4y agoWith the 7B none-instruction tuned you have to really pre-prompt it to get good answers to things so like: SYSTEM: You are an AI research assistant. You use a tone that is technical and scientific. USER: Hello, who are you? ASSISTANT: Hi! I am an AI research assistant. How can I help you today? USER: Can you tell me something about owls.
- rjb7731 4y agoThe inference on the gradio demo seems pretty slow, about 250 seconds for a request. Maybe I am too used to the 4-bit quant version now ha!
- sebzim4500 4y agoI'm sure it's partially the HN hug of death.
- bilsbie 4y agoWhat do they mean by instruction? Is it just regular LLM?
- isaacfung 4y agoLLM just predicts the next token given the previous tokens(this can be trained without manual labelling by humans). Instruct GPT and ChatGPT use reinforcement learning from human feedback to align the model with human intents so it understands instructions. https://huggingface.co/blog/rlhf https://huggingface.co/blog/rlhf
- bilsbie 4y agoThanks. So what does the output look like without rlhf?
- simonw 4y agoThis comment has a useful comparison between the two: https://news.ycombinator.com/item?id=35140447 https://news.ycombinator.com/item?id=35140447
- valine 4y agoIt can look like anything. Sometimes it will answer your questions, other times it will continue the question like its the one asking. I've also seen it randomly output footers and copyright like it just got to the end of a webpage. Its makes sense when you think about how the training data is random text on the internet. Sometimes the most likely next token is the end of a webpage after an unanswered question.
- sanxiyn 4y agoNote that Alpaca is NOT using RLHF. It explicitly states it used supervised finetuning.
- est 4y agoIt says > We train the Alpaca model on 52K instruction-following demonstrations generated in the style of self-instruct using text-davinci-003 Which leads to self-instruct https://github.com/yizhongw/self-instruct https://github.com/yizhongw/self-instruct From a glimpse they used a LM to classify instructions & train the model which IMHO is very similar to RLHF
- marban 4y agoWill it run Doom => Will it run LLaMA
- all2 4y agoWill LLaMA run DOOM.
- 29athrowaway 4y agoLooking forward for one called guanaco.
- ar-jan 4y agoAs the llama's wild and free-roaming cousin it had better be permissively licensed.
- Imnimo 4y ago>Alpaca cannot be used for [...] entertainment. Good luck with that.
- ugjka 4y agoJust some lip service
- wkat4242 4y agoIs this only for the 7B model? The 65B would be much nicer O:-)
- cerol 4y agoCan someone provide a guide on how run LLaMA on a fairly average CPU/Nvidia GPU?
- isoprophlex 4y agoCheck the git repo for llama.cpp. Model weights in torrent magnet:?xt=urn:btih:b8287ebfa04f879b048d4d4404108cf3e8014352 Runs slowly but without trouble on an intel i7 with 32 gb ram.
- eulers_secret 4y agoAnother great option is https://github.com/oobabooga/text-generation-webui https://github.com/oobabooga/text-generation-webui The 7B model will run without changes on a 3080. The 13B 4-bit model also runs on a 3080. This Reddit post has the instructions I followed: https://old.reddit.com/r/LocalLLaMA/comments/11o6o3f/how_to_install_llama_8bit_and_4bit/ https://old.reddit.com/r/LocalLLaMA/comments/11o6o3f/how_to_...
- superkuh 4y agoI've played a lot with davinci 3 ($25 of credits worth) and it can do some impressive rhyming and interpretation of concepts as emoji sequences. From the 3 times I've interacted with this fine tuned llama 7B it is clear it cannot do that. I've also run the "vanilla" 7B, 13B, and 30B on my home computer with llama.cpp modified for interactive "chat" mode with various pre-prompt and these can't do it either. I have no love for openai/microsoft but their text completion davinci 3 really is incredible. I don't think these littler models have quite reached the point of emergence for some functions despite fine tuning.
- valine 4y ago7B parameters is next to nothing when compared to gpt3. If 7B works as well as it does here, A fine tuned 65B model could very easily achieve chatGPT level performance.
- SteveDR 4y agoI thought ChatGPT is only 20B parameters to begin with? (Source https://www.forbes.com/sites/forbestechcouncil/2023/02/17/is-bigger-better-why-the-chatgpt-vs-gpt-3-vs-gpt-4-battle-is-just-a-family-chat/amp/ https://www.forbes.com/sites/forbestechcouncil/2023/02/17/is...)
- sebzim4500 4y agoI haven't seen anything official from OpenAI confirming that ChatGPT has fewer than 175B parameters, although it is a reasonable guess if you read between the lines of their statements. Given the author of that article is a CEO of an 'AI Ad Optimization Platform' I think that number is speculative at best.
- stavros 4y agoChatGPT is fine-tuned InstructGPT, which has 1.3B parameters, if I'm not mistaken. Reference for the former: https://www.technologyreview.com/2023/03/03/1069311/inside-story-oral-history-how-chatgpt-built-openai/ https://www.technologyreview.com/2023/03/03/1069311/inside-s...
- siliconc0w 4y agoI wonder when we'll get to a point that it'll be against the OpenAI TOS to 'fine-tune' your own model with their model's outputs since it really lets you cut down on the hardware requirements to bring up a competitor.
- O__________O 4y ago>> Second, the instruction data is based OpenAI’s text-davinci-003, whose terms of use prohibit developing models that compete with OpenAI. https://crfm.stanford.edu/2023/03/13/alpaca.html https://crfm.stanford.edu/2023/03/13/alpaca.html
- avmich 4y agoWith so much of texts being written by computers, can we have help on the reading side too? My emails are checked for spam and filtered, ok, but I'd still like to learn what are the important parts of the messages which pass through. Do we now have solutions for that already?
- simonw 4y agoLanguage models like LLaMA (and GPT-3, ChatGPT etc) are really good at exactly this. They can summarize text, they can extract key points into bullets, they can figure out common topics and so on.
- dougmwne 4y agoWaiting to be able to feed a LLM a transcript of an ongoing meeting, then ping me when my attention is needed with the context summarized.
- mirekrusin 4y agoWhy bother with pingig? Write answer, text-to-speech it and fake-voice it so it sounds like you. Don't forget to wrap it with supervisor AI so you can replicate minions 100x+ times.
- technerder 4y agoIs there anyway to use Google's Coral TPU PCIe Accelerator[1]'s to increase the inference performance of GPT models like this and Llama? [1] https://coral.ai/products/pcie-accelerator/ https://coral.ai/products/pcie-accelerator/
- tempaccount420 4y agoSupposedly TPUs are better than GPUs, but everyone other than Google uses GPUs for models.
- kossTKR 4y agoA Tangent, but how long before we will see half the population having relationships with their AI assistants like in the sci-fi movie "Her". Add cutting edge voice synthesis and maybe a realistic video stream in a few years and we're there. Japan already has problems with the concept of "2d Waifus" and lack of intimacy between the younger generation. Maybe the downfall will not just be climate catastrophe but hyper isolated people living alone with their ultra realistic bot friends and family without any desire to experience the ups and downs of actual social experience. Extrapolate a bit and you wont know what's real outside of direct experience. Things are getting crazy!
- antondd 4y agoOnce we have models that can generate perfect audio and video + better AR/VR tech, it’s game over.
- simonw 4y agoThat's been happening already with Replika: https://www.abc.net.au/news/science/2023-03-01/replika-users-fell-in-love-with-their-ai-chatbot-companion/102028196 https://www.abc.net.au/news/science/2023-03-01/replika-users...
- bentcorner 4y ago> A Tangent, but how long before we will see half the population having relationships with their AI assistants like in the sci-fi movie "Her". I don't know about half, but some people are already having relationships: https://replika.ai/ https://replika.ai/ > Maybe the downfall will not just be climate catastrophe but hyper isolated people living alone with their ultra realistic bot friends and family without any desire to experience the ups and downs of actual social experience. I think the danger is that bots are not necessarily ultra realistic, at least on an emotional level - they can be 100% subservient and loyal to you. Also - we already chide parents for letting their kids grow up stuck to a device. Imagine if children could actually have an imaginary friend? Would that AI share the same culture and values as your family? I suppose there could be some upsides but this is very uncharted territory.
- earleybird 4y ago
- LeoPanthera 4y agoI'm in my 40s, so I'm acutely aware of the risk of turning into a "get off my lawn" curmudgeon, so I will only say this: I am glad that I saved a dump of Wikipedia a few months ago, because soon it will probably be impossible to distinguish human-written content from AI.
- pmoriarty 4y agoIf it's indistinguishable, then how does it matter who/what it was written by?
- LeoPanthera 4y agoBecause I am far less likely to trust encyclopedic content written by an AI, which can lie so convincingly that it does not know that it is lying. I have even seen it generating fictional source references that are hard to verify. I just don't trust it.
- pmoriarty 4y agoIt takes more than merely being human to earn my trust.
- jxy 4y agoMaybe we just need a better attribution, so people can decide on the level of objective confidence. If you live in the US, it is already difficult to avoid seeing and listening to real people lie publicly, which may take journalists a long time to verify. Put it in another way, we might just see late night talk show hosts start parodying $next_gen_llm instead of real people, if this $next_gen_llm started affecting people in a more profound way.
- aortega 4y ago>can lie so convincingly that it does not know that it is lying. A human can also lie convincingly. The problem with AI is that you can do it massively.
- mellosouls 4y ago
- rjb7731 4y agoSeems not too bad, you can tell its been trained on shorter responses. Example #1 Input: explain a neural network in simple terms Alpaca-Output: A neural network is a type of machine learning algorithm that is made up of layers of interconnected nodes (aka neurons). It is inspired by the way the brain works and uses interconnected nodes to learn patterns in data and make predictions. Neural networks can be used to perform various tasks such as classification, regression, and speech/image recognition. Example #2 Input: explain how transformers work in artificial intelligence Alpaca-Output: Transformers are a type of artificial intelligence algorithm used in deep learning. They are responsible for taking input data and converting it into a different data representation, called the "output space". Transformers do this by using a series of mathematical operations, allowing them to map input data onto a new, desired output. This allows the artificial intelligence system to perform tasks such as natural language processing and image recognition. Transformers are able to learn from data and can be used to build more efficient models for a variety of tasks Example #3 Input: how do i make a request to the chatgpt api with python? Alpaca-Output: The API can be accessed using the Python language. You can use requests library to make API requests, for example: import requests # URL of the chatgpt API url = 'https://api.chatgpt.com/v1/messages https://api.chatgpt.com/v1/messages' # Post a new message data = { 'text': 'Hello World' } # Send the request and get the response response = requests.post(url, data=data) print(response.text)
- simonw 4y agoLove that hallucinated ChatGPT API example there.
- paper_good 4y agoI tired myself, but it seems Alpaca can't provide contextual conversation. So it's just very simple Q&A. This limit makes Alpaca far away from ChatGPT ability.
- musicale 4y agoInteresting to see Alpaca readily adopt the persona of a Stanford student for the congratulatory letter. I expect that AI-generated essays submitted by actual students are fairly convincing to their professors as well.
- zzleeper 4y ago> which is the largest city in Tanzania. (It was the capital until 1974, when it was replaced by Dodoma.) Find it funny that even the humans got it wrong. Dodoma only became the official capital in 1996.
- deleted 4y ago[deleted]
- spindle 4y agoI'm not sure about the name. In my experience, alpacas are much harder to train than llamas. They are smaller, though.
- nl 4y agoNote that the OpenAssistant project has a 100k+ multilingual instruction tuning dataset they are in the process of releasing. Stats from March 9 are 50K English prompts and 40K Spanish.
- meghan_rain 4y agoWow that was quick, they only started in what, December?
- nl 4y agoYeah amazingly fast. I haven't looked at the quality but some comments in the discord seemed to indicate it was decent quality but had a lot of coding-type prompts.
- haolez 4y agoDo we have some studies on the impact of extreme quantization on these models? In my understanding, the efforts to make them run in commodity hardware involves the quantization of the weights to 8 bits or even 4 bits. Is the model still reliable after this?
- O__________O 4y agoYes, intro on the topic if anyone is interested, include topic of benchmarking degradation: - https://huggingface.co/blog/hf-bitsandbytes-integration https://huggingface.co/blog/hf-bitsandbytes-integration
- haolez 4y agoFantastic article. Just a heads up for anyone reading this reply.
- vishal0123 4y agoI am surprised that they were able to launch this in stanford's domain. They clearly broke TOS of both Facebook and OpenAI, and even admitted doing that. I would be happy if the research decides to ignore openAI and facebook's useless restrictions.
- make3 4y agodoing that would make it less likely that they would publish a model next time
- charcircuit 4y agoWhat part of Facebook's tos did they break?
- Ensheng 4y agoWhy 175 tasks? I am interested in understanding the considerations, criteria, rules or guidelines that were used to develop these tasks. Specifically, I would like to know more about the thought process behind the task construction,and the factors that were taken into account when selecting the tasks.
- russellbeattie 4y agoHaving played with the 7B model and seen the resulting gibberish it produces, this is definitely impressive.
- naasking 4y agoCool, they should add MathPrompter support too: https://paperswithcode.com/paper/mathprompter-mathematical-reasoning-using https://paperswithcode.com/paper/mathprompter-mathematical-r...
- ar9av 4y agoWith 4-bit quantization you could run something that compares to text-davinci-003 on a Raspberry Pi or smartphone. What a time to be alive.
- ar9av 4y agoWith 8-bit this should fit on a 3060 12GB, which is pretty affordable right now. If this works as well as they state it's going to be amazing.
- frozenwind 4y agoFrom what I understood they used 175 seed tasks as an example for da-vinci which then generated the 52K instructions used to fine-tune LLama. What I don't understand is how did they achieve giving this context to GPT, since afaik the context window is just 2048 tokens? It seems that it's not fine-tuning, but few-shot.
- leodriesch 4y agoThey show an example of this under "Training Recipe". They prompt Davinci to give them instruction following examples, 20 at a time. And that a lot of times. The context window is also 4000 tokens, at least that is the maximum amount you can pick in the OpenAI playground.
- kumar12 4y agorfrefrefer
- biggerChris 4y ago[dead]
- akrymski 4y agoIs LLAMA actually better than Flan T5?
- marat666 4y agoJust wow!
- eternalban 4y agoSelf Instruct: https://arxiv.org/pdf/2212.10560.pdf https://arxiv.org/pdf/2212.10560.pdf https://github.com/yizhongw/self-instruct https://github.com/yizhongw/self-instruct
- meghan_rain 4y agotldr?
- clasto 4y ago[dead]