9 ms·
OpenOrca: open source dataset and instruct-tuned LLMs
- hnpobdo 3y ago[dead]
- brucethemoose2 3y agoAll GPT4 and GPT 3.5 data. This seems to be working well in other finetunes, but the lack of anything other than OpenAI output is still really bizzare to me. The error rate will surely be high, especially with GPT 3.5
- az226 3y agoOnce I’m used to GPT4 output and once in a while try and see if GPT3.5 can also get it right, it fails miserably. Every single time. I’m sure it’s fine for some things, simple things.
- Tostino 3y agoIt takes work, and breaking your problem down into very simple and clear tasks. It is possible to get decent data transformations from gpt 3.5 though. I generally start with gpt4 when prototyping an idea, and then after it's working consistently, I'll ask gpt4 to break the instructions down into smaller pieces. Even if it takes two or three runs through GPT 3.5 to get the full output transformed with what GPT4 could do in a single pass, it's still cheaper...
- npsomaratna 3y agoThat's interesting. Do you mean that you ask GPT-4 to break down it's own prompt into smaller/simpler pieces?
- Tostino 3y agoYes, and then I iterate on it until it is clear and simple enough for weaker models to actually follow consistently.
- bart_spoon 3y agoI believe they were referring to GPT-3.5, but yes, you can get it to be pretty successful at more complex tasks this way. You can also manually break the problem down yourself too, if it’s a standard workflow. So for example, rather than ask it to translate some text, summarize it, and classify it in a single prompt, doing it as 3 separate prompts that feed into one another is far more likely to be successful.
- jrflowers 3y ago> All GPT4 and GPT 3.5 data. It’s super funny that by saying “You can’t use GPT to create training data for competing language models” OpenAI convinced a bunch of folks that GPT would be Super Good at producing training data for making competing language models. It’s like “Do NOT use these prions to feed competing cattle, we would HATE IT if you did that”
- ahalbert 3y agoWhen will the model be available?
- drippingfist69 3y agoIt says mid July
- loudmax 3y agoAlso looking forward to Stable Diffusion's new SDXL model in "mid-July": https://stability.ai/blog/sdxl-09-stable-diffusion https://stability.ai/blog/sdxl-09-stable-diffusion I hope both of these deliver on their promises. They're exciting developments.
- zekrioca 3y agoWhy would they use the same name as the Gnome Orca project [1]? [1] https://help.gnome.org/users/orca/stable/introduction.html.en https://help.gnome.org/users/orca/stable/introduction.html.e...
- arjvik 3y agoBecause at this point, nearly every name is taken. Orca Screen Reader and Orca AI are quite easy to disambiguate
- throwaway290 3y agoBullshit, the scope of possible names is practically infinite. Even if actual words and sensible letter permutations run out, you can start borrowing from outside of software and have much less chance of confusion. Nike, Adidas, NYC, Rolex. The industry is different and there is no commerce involved so no grounds for trademark violation. There are two reasons to collide with another OSS project: basic laziness to do a quick google search before you settle on a name or desire to benefit from preexisting search traffic.
- orra 3y ago> you can start borrowing from outside of software and have much less chance of confusion. Nike, Adidas, NYC, Rolex. This is objectively not true. > The industry is different and there is no commerce involved so no grounds for trademark violation. Nike, Inc. have US trademarks in Nice class 9: 97095855 & 97096366. Rolex Watch U.S.A., Inc. have a class 9 and class 42 US trademark: 97655284. adidas AG have a class 9 EU trademark: 006703086. etc, etc. Besides, these brands are so well known that I'm certain you'd be challenged even if it was a different trademark class.
- throwaway290 3y agoTrademark is not a globally reserved word. If you are not doing commerce in relevant area that there can be confusion (or you don't imitate the logo), you are free to use it. This is basic free speech. Besides Orca is a registered trademark used by multiple class 9 businesses, now what.
- hklgny 3y agoI’m super bummed that we didn’t name this “Free Willy”
- koboll 3y agoI think that might evoke something... undesirable
- TX81Z 3y agoSpeak for yourself.
- snthpy 3y agoGot the movie reference but took me a moment to catch the second meaning. How about "Free Will-e"? That's less likely to evoke sexual connotations and references another (actually somewhat relevant) movie.
- bobse 3y agoChange your name! https://help.gnome.org/users/orca/stable/introduction.html.en https://help.gnome.org/users/orca/stable/introduction.html.e...
- aroo 3y agoOpenOrca's goal is to provide an opensource replica to Microsoft's Orca 13B model, so changing the name makes no sense.
- nullc 3y ago"We expect to release OpenOrca-LLaMA-13b in mid-July 2023." :( Personally I've found that announcing things ahead of availability hurts the impact, because the real announcement is old news and doesn't get seen and the pre-announcement loses people because there is nothing to do with it.
- chankstein38 3y agoIt's like a trailer or preview of a song or something, I want to listen to the song immediately and will have forgotten all about it by the time your overhyped single release has happened.
- nullc 3y agoIt's also an invitation for third parties to trip you up. "Gee, if we send a legal threat now we can probably block the release of this" vs "welp, it's already out, can't put the horse back in the barn all we could hope to do is immolate our goodwill by suing a researcher".
- nullc 3y agoCalled it. It's gone now.
- SCUSKU 3y agoI really enjoyed the tenor of this post. It's great to see cordial acknowledgement of the contributors efforts, and with such glowing language :)
- Havoc 3y agoI’m guessing there is still no consensus on legality of using openai to train a model?
- letitgo12345 3y agoProbably legal if you use third party collected data (as you personally haven't agreed to OAIs ToS and AI generations can't be copyrighted so aren't owned by OAI) but I guess corps are too wary to risk a court battle. Directly using OAI to train a model for commercial use that competes with OAI is a violation of their ToS though
- ilaksh 3y agoRight so that's why I am not sure I understand why there are so many "Open source" efforts that use GPT-4 when 75+% of people interested in open source want to use it in a commercial effort. Makes me think that quite a lot are just ignoring that part of the service terms. Then it makes more sense to keep using GPT-4 for open source models -- if you just decide you are going to ignore that part.
- letitgo12345 3y agoIf you're releasing open source models but not profiting from them, how are you competing with OAI. I don't think that's a violation of ToS.
- weinzierl 3y agoModel Size Compute Estimate 7b 1k GPU-Hours 13b 2k GPU-Hours 30/33b 4k-6k GPU-Hours 40b 8k-10k GPU-Hours 65b 10k-15k GPU-Hours What would that mean in USD?
- heyitsguay 3y agoAbsolute best case in the cloud for the kind of GPUs this needs? ~$1/GPU/hr, but maybe up to $5/GPU/hr depending on provider and configuration. But companies or other organizations with extra capacity on their in-house hardware might also be able to just run their training script for a while, at which point the cost is more like electricity + opportunity cost.
- weinzierl 3y agoThat sounds cheap. Can I really train a 13b model from scratch for just USD 2000?
- vessenes 3y agoNope. Salesforce just announced their heavily trained 7B model cost them $150k at Google. What you can do is fine-tune an open 7B model for a few thousand dollars, and that's the plan for these folks.
- pk-protect-ai 3y agoThis is full training right? Not fine tuning. Fine tuning must be cheap like those mentioned $2k ...
- SparkyMcUnicorn 3y agoDepending on how much fine-tuning you're doing, it can be free (colab) or a few bucks.
- loudmax 3y ago
- vessenes 3y agoIt makes a lot of economic sense to use existing functional LLMS for data extension and augmentation. But, I find myself skeptical and deeply tired already of what I see as a major failure mode of relying on ChatGPT for alignment instruction: "As an AI model, I cannot.." If I were training a model, I would excise with extreme justice any data like this from the training set. As the developer of a very high-powered tool, I may well wish to limit its use in many contexts. But, I never wish to limit the tool's usefulness ahead of time. To my knowledge we only have Vicuna-uncensored in the wild that's taken this approach, and right in the name I see either misdirection or misunderstanding or poor branding on the benefits. It's not really about whether your private LLM will sext with you, (although you should definitely be able to do such a thing with your own LLM if you like), it's whether you've preemptively lobotomized your tool in accordance with someone else's take on what a safe consumer-oriented final output should be. I just don't accept this sort of constraint from my other software tools, and I begrudge it in my hardware tools, and I remain a little surprised that most people training these models don't mind it.
- spacebanana7 3y ago> As the developer of a very high-powered tool, I may well wish to limit its use in many contexts. But, I never wish to limit the tool's usefulness ahead of time. Exactly, content moderation is largely an application layer problem not a foundation layer one. Imagine the problems of MySQL trying to perform content moderation for Facebook.
- vessenes 3y agoYes, great analogy!
- pksebben 3y ago(the year is 2048. The camera pans across an office at Quantico, which is eerily serene. A messenger knocks on an important-looking door with a plaque that reads 'DIRECTOR') Director: Come in Messenger: Message from the Tulsa field office, sir. They're reporting that they've found a sex trafficking ring, but they're not sure what to do about it. Director: Not sure? Arrest them, obviously. What's the problem? Messenger: Well, they can't seem to secure a warrant. Some technical issue with the system. Director: I know we migrated to a new system recently. Let's see if we can get this sorted. (Director thwacks at the keyboard briefly) Computer: Your request for "Child Sex Trafficking Warrant" has been found to contain content marked "Not Safe For Work". This violation has been reported. Director: What the hell. Messenger: Yeah, we tried to email you about it but the filters dropped the message. That's why they sent me. Director: I'll deal with this. Let me make a call. (Director picks up phone and dials) Director: Hello? Hi, Paul. Yeah, we're having some issues with the new warrant system.... No, it's doing everything as advertised... yes, it's a lot faster and we've managed to lay off a ton of our data staff. The problem is with getting warrants; Me and my guys have been trying to get one but it keeps getting rejected... Oh, you know, some sex trafficking ring in Tulsa.... Hello? Phone: Your call cannot be completed as spoken. Our automated systems have detected content related to sex trafficking. This incident will be reported. Director: God Damnit. (as the director holds the phone trembling in frustration, the power goes out and they are enveloped in darkness in the windowless room. Roll credits)
- artemonster 3y agoPlease recommend a good tutorial/book/video on modern LLMs and NNs in general, for programmers and technical people. Where you get the idea of how it works. Tried googling with dozens of queries and it just sucks, a lot of hand-wavy articles for lay people or some paid courses.
- boredemployee 3y agoYes. I have a fomo everytime I see people discussing it in depth and I can't understand what they're talking about.
- un_ess 3y agowould https://course.fast.ai/ https://course.fast.ai/ be of assistance ?
- loudmax 3y agoI recommend the Huggingface courses: https://huggingface.co/course https://huggingface.co/course The courses are geared toward writing Python applications around these models. They're fairly hands on, so it would still be a good thing to complement them by reading papers or watching videos on fundamental principles of AI and ML.
- gAI 3y agoThank you for this - I've been meaning to finally take the dive into hugging face but hadn't looked into how.
- pk-protect-ai 3y agoJust to say thank you for your efforts! It is kinda very sad that MS hid the Orca, I wanted to test it from the start on. Now I found out that guanaco 30b has the spatial comprehension to some limited extent though this is the best 30b model so far. Can you add this model for the list of your training? And it would be very nice as well if someone would sponsor the MPT-7b (better then LLaMA-13b in most of the cases) and RWKV (I would really love to see how it will perform after such tuning). The RWKV should be the cheapest to tune, isn't it?
- SparkyMcUnicorn 3y agoGuanaco is a LLaMA fine-tune, so you'll almost definitely be able to add that on top of the LLaMA results if you wanted.