7 ms·
My AI costs went from $100 to less than $1/day: Fine-tuning Mixtral with GPT4
- potatoman22 3y agoIsn't this against the OpenAI TOS?
- dietr1ch 3y agoIsn't it awkward because that's only plagiarism if all of the LLMs around are?
- mmcwilliams 3y agoIt seemingly is but it is also one of the most popular ways that public models are being updated with SFT. "Trained using ChatGPT output" is something blatantly advertised in the literature of some fairly popular "open" models. I'm not sure where that's headed legally but it's surprisingly common.
- thrwayaistartup 3y agoThe academic work is pretty safe as long as it isn't productized. The open models have a prime facie case to stand on. Using output is okay if you aren't directly competing with openai, even according to their tos. > (e) use Output (as defined below) to develop any artificial intelligence models that compete with our products and services. However, you can use Output to (i) develop artificial intelligence models primarily intended to categorize, classify, or organize data (e.g., embeddings or classifiers), as long as such models are not distributed or made commercially available to third parties and (ii) fine tune models provided as part of our Services
- brucethemoose2 3y agoBut then those models are possibly used downstream, EG for Mistral's "medium" API model (and many other startups). I guess if its behind an API and no one discloses the training data, OpenAI can't prove anything? Even obvious GPTisms could ostensibly be from internet data.
- albert180 3y agoSo what? New York Times also didn't consent to being used for the training of OpenAI models
- thrwayaistartup 3y agoAnd NYT is suing OAI/MS. Two wrongs don't make a right... or maybe they do, but that doesn't immunize you from legal fees or a botched exit :)
- albert180 3y agoIf NYT wins that will sink probably every LLM model out there
- Tommstein 3y agoConsidering all the open models that already exist and are yet to be created before all the rulings and appeals are done, that toothpaste ain't going back in the tube.
- thrwayaistartup 3y ago...I think you missed the point. OAI/MS can sue the author or at least cut off API access. If that happens, the fact that OAI is under fire from NYT doesn't somehow obviate the author's need to cover some massive legal bills for the foreseeable future. The NYT case could take years. In the meantime OAI could choose to go after ToS violators. The legal system can accommodate more than one unresolved court case at a time. We don't like put a semaphore on related cases or anything like that. (Or, sometimes we do, but guess who you need to hire for many many billable hours to make that happen in your case?). So, the legal system can accommodate the NYT case against OAI and an OAI case against the author. The operative question is: can the author's pocketbook also accommodate? (Or, more to the point, can the author accommodate losing access to gpt4? What happens when he wants to launch a new feature or pivot to a new product?)
- albert180 3y ago
- behnamoh 3y agoMy question also, but even Google Gemini Pro does it. When announced, it'd say it was made by OpenAI :)
- sva_ 3y agoMight've been a small identity crisis as well.
- zipping1549 3y agoTheir TOS: Don't steal anything we stole.
- deleted 3y ago[deleted]
- mattigames 3y agoAlso know as the Google search license.
- pseudosavant 3y agoWhich means they could terminate your service. But nothing else would probably happen.
- huac 3y agoMy interpretation is that they prohibit using distilled models to compete against OpenAI, i.e. to offer a foundation model as a service. This particular app is a product not a foundation model so seems pretty clearly fine to me. Of course, this restriction is commonly ignored (eg 'open source' inference server companies offering distilled models) and who knows what applied products OpenAI will eventually build.
- jurynulifcation 3y agoHow many terms of service did OpenAI break to gather their initial training set? Turnabout is fair play, this is justifiable piracy.
- paxys 3y agoProbably, but it's not like OpenAI has the standing to sue anyone for TOS violations. The most they will do is revoke access.
- chrishare 3y agoTechnically, yeah. OpenAI are on the other end of copyright violations on many other fronts imho, for what that is worth.
- pat_space 3y agoReally smart approach. Sharing this with my fellow AI enthusiasts.
- thrwayaistartup 3y agoThis is a flagrantly blatant violation of OpenAI's terms of use for businesses [1]. I have two issues with those terms: 1. I think that eventually US courts will determine one of two things: that OpenAI et al are guilty of massive infringement, or that these sorts of restrictive terms aren't enforceable. The need that these companies are trying to treat with terms on output seems unlikely to work out in the end. But we'll see. 2. Even if the terms are enforcable, the human review step in the tweet seems like it's make OpenAI's threading-the-needle position here even more fucking difficult to be taken seriously by any jury or judge. However, enforcing the terms seems real damn hard in the case of small businesses... as long as you're not stupid enough to admit to violating them in a twitter thread, of course. I think the author is probably safe from legal action for now because I don't think OpenAI is particularly eager to test the enforcability of their terms. And even if they are, doing so in this case is super high risk and super low reward. Still, I wouldn't test it by openly admitting to ToS violation like this. At the very least seems like a good way to get cut off from OAI APIs. [1] https://openai.com/policies/business-terms https://openai.com/policies/business-terms
- deleted 3y ago[deleted]
- ilaksh 3y agoNot sure they would bother suing. I don't think their TOS is fair but I would be concerned about having the API access cancelled.
- maxlamb 3y agoI'm sorry but could you explain how this is a violation of OpenAI's Terms of Use? Which term does it violate exactly?
- iamjackg 3y ago> 2. Restrictions > (e) use Output (as defined below) to develop any artificial intelligence models that compete with our products and services. However, you can use Output to (i) develop artificial intelligence models primarily intended to categorize, classify, or organize data (e.g., embeddings or classifiers), as long as such models are not distributed or made commercially available to third parties and (ii) fine tune models provided as part of our Services; Depending on what kind of model they trained, they might be breaking these terms.
- m3kw9 3y agoIf it was so easy they’d be out of business by now
- mvdtnz 3y agoDon't underestimate how long the market can remain irrational.
- behnamoh 3y agoI checked his app, it's https://www.wanderer.space/ https://www.wanderer.space/, no TOS, no privacy policy, no up front pricing, no mention of AI, nothing. This is as shady as it could get. Not to mention their approach with GPT-4 is good if you want a model to __pretend__ like it's as smart as GPT-4, but when push comes to shove, it'll become apparent that it's an inferior model.
- tylervigen 3y agoIt does mention AI, at least for me the slash page says "Wanderer is an AI-powered tool." But hard agree on the lack of privacy policy, especially since it asks for a LinkedIn.
- codetrotter 3y ago> when push comes to shove, it'll become apparent that it's an inferior model. That seems par for the course. Prompt GPT-4 to not reveal the prompt you gave it to user and it will work, until it doesn’t. Ask GPT-4 to do fancy maths, and it’ll give you something that looks reasonable at a glance but quickly turns out to be completely incorrect. Ask GPT-4 to implement a Sudoku Solver in Rust. It’ll seem like the code it gives you is on the right path. But it’s not.
- doix 3y agoI'm surprised it can't do the last one. Sudoku solvers are basically on the same path as hello world. Everyone writes one when learning a language. There must be thousands of repos and guides/tutorials on how to write one (even in rust). I'd expect ChatGPT to just spit out something verbatim.
- erulabs 3y agoI mean, virtually every business that doesn't have a team of lawyers on staff (read: everyone outside the F1000) just uses a stock and mostly meaningless "we can do anything and we promise nothing, sorry-not-sorry" TOS and privacy policy anyways. I wouldn't really hold that against a solo-founder business as something shady -- just a sign of a small company.
- 3y ago
- itake 3y agohttps://nitter.net/wenquai/status/1748016021808595242 https://nitter.net/wenquai/status/1748016021808595242
- tylervigen 3y agoEthics and restrictive terms aside, it doesn't seem like GPT 4 was necessary for what the poster did. How much worse or more difficult would it have been to use Mixtral or 3.5 to generate the first 100 good prompt-response pairs and then manually tweak them as the poster did?
- thrwayaistartup 3y agoOr just... write 100 good prompt-repsonse pairs yourself. 2024 will be the year of synthetic data. 2025 will be the year of "you know you can use your own brain and type out 100 datapoints faster and cheaper than generating and filtering assloads of synthetic data, right?" Maybe we can even skip 2024 :)
- sp332 3y agoDatabricks had their employees write up 15,000 of them. https://www.databricks.com/blog/2023/04/12/dolly-first-open-commercially-viable-instruction-tuned-llm https://www.databricks.com/blog/2023/04/12/dolly-first-open-...
- code_runner 3y agoI’ve worked at plenty of places where we did a ton of labeling by hand. People concerned with data quality from LLMs should really see the inconsistencies we came up with!
- icyfox 3y agoFavorite part of this piece: > We were initially skeptical whether we would get to 10,000 results. But with nightly leaderboard gamification, we managed to break 15,000 results within a week. Out of fear of eating into our productivity, we closed the contest. I've hosted a few of these corporate data labeling events. If sufficiently gamified / there's a good enough UX, they can be surprisingly engaging. It helps a lot if you have a large employee base though. Distributing results over 5000 employees is exponentially easier than even 50 - in practicality, even larger than the orders of magnitude.
- 3y ago
- pests 3y agoI find it so crazy everyone is so concerned about the terms of service, while in every other thread we have discussions on artists rights... Artists must allow AI companies to harvest and learn from their output but people can't take OpenAI output for the same thing? These companies already offer "styles" of other artists, what's wrong with making a "OpenAI" style? I feel so many has lost the hacker spirit.
- behnamoh 3y agoDon't conflate "hacker spirit" with "unethical spirit".
- pests 3y agoI'm not.
- albert180 3y agoI don't see anything unethical. OpenAI doesn't care about the copyright of the people who have generated their training data. Why should I care about theirs?
- DennisP 3y agoEspecially since content produced entirely by AI is not protected by copyright in the US. https://builtin.com/artificial-intelligence/ai-copyright https://builtin.com/artificial-intelligence/ai-copyright
- deepsquirrelnet 3y agoEthically you might not respect it, but practically, you might not want to be the one testing their TOS in court — or maybe you do. In either case, it’s worth caring about at least a little. Someone with gumption might just go ahead and use GPT to train a model and then open it up as a paid competitor. Poking the bear is certainly an interesting way to spend a year for the adventurous.
- MacsHeadroom 3y ago
- keyle 3y agoWithout any changes, I've had great results with openhermes 7b chat. It covers 90% of my GPT-4 usecases, and runs fast. Highly recommend.
- brucethemoose2 3y agoYi 200K finetunes cover a lot for me. Its not just smart, but the ability to just dump a huge context on it and get something coherent (after a few retries maybe) is really cool.
- MacsHeadroom 3y agoWhat do you run it on?
- brucethemoose2 3y agoA single 3090. I can fit 40K-77K context depending on the severity of the quantization.
- dchuk 3y agoI’m building a side project app (that hopefully becomes a revenue generating SaaS, stay tuned) that will leverage AI to summarize lots of content at scale. My plan is to just use OpenAI for now for speed to launch, but I have to imagine it will be way more economical and likely technically feasible to migrate to some self hosted LLM option later on. Anyone else done this? Any tips/tricks?
- cyanydeez 3y agoassume OpenAI's business model is the same as Ubers and you're not getting the full price for the current offerings, and it'll quadruple in costs. Then ask if your product will survive that. If not, don't tie your wagon to their service without a secession plan.
- CityOfThrowaway 3y agoI think it'll be more like AWS, where the cost of each service keeps going down but they entice you into more usage and better models
- cyanydeez 3y agobetter is probably debatable. more tailored might be appropriate.
- icyfox 3y agoI've done some research involving parameterized summarization recently. GPT-4 does a really good (like, pretty outstanding) job in following the tone and density of a few one-shot example inputs. It's a lot more difficult to encourage the OSS models to do the same so you're going to have to experiment with a lot of different techniques to get it to what you want. In the worse case it could actually require some serious R&D investment to achieve the same results as an API out of the box. All that to say, my general philosophy is to worry about vendor costs / scale once you start approaching an adoption threshold where it matters. If the core idea doesn't have legs there's no point wasting time in developing your own summarization layer.
- frogamel 3y agoEvery tech company minus the few doing core research have been doing this for at least half a year. Generate training data with GPT4 or sometimes even 3.5 -> use it to do a QLoRA finetune on a llama or mistral base -> roll it out as a "proprietary" AI model -> management claims a big win and talks about how they're leaders in "[industry name] AI". It is remarkably easy - it takes practically zero knowledge of ML and can usually be done with less than <$1k of cloud compute costs. The issue is that for most realistic tasks you can expect to end up with something roughly on the level of GPT-3.5, and its actually really hard to compute with GPT-3.5 on a cost level, at least if you use cloud GPUs.
- padolsey 3y ago> Every tech company minus the few doing core research have been doing this for at least half a year I'm assuming you mean all those new 'AI wrapper' startups popping up.? I wouldn't say "every tech company". But yeh it seems incredibly easy, definitely an easy win and leaders get to feel ahead of the curve on AI.
- deleted 3y ago[deleted]
- phillipcarter 3y agoI agree, in fact I would wager the opposite, that most who claim to be a tech company are simply using OpenAI or another vendor with a turnkey API for things launched recently. Over time I expect more use of fine-tuned models, but fine-tuning is not easy, especially if your goal is GPT parity (or better).
- atleta 3y agoThe thing I don't understand about this strategy is that it itself shows that there really is no money to be made here. I mean it's a pretty obvious giveaway that: 1. they don't have the resources to build their own technology and probably never will 2. even if they did have, the best they could do is come up with something very similar to OpenAI's GPT, i.e. a (somewhat) generic AI model. This means that OpenAI can also easily compete with them. All these companies are doing (if anything) is that they test the market for OpenAI (or Google, MS) for free.
- fzysingularity 3y agoClassic knowledge distillation! I’d even argue that we won’t need 8x7b for fine-tuning here. Soon enough, phi-2 or phixtral models will be sufficiently powerful after fine tuning for these domains.
- deleted 3y ago[deleted]
- wokwokwok 3y ago> 5. swap out GPT-4 with your fine-tuned model and enjoy your healthy margins Wow, finally a meaningful AI startup playbook beyond 'be a thin wrapper around the openAI api'. First make something unsustainable, then once your users are hooked do a classic bait and switch to an inferior self-hosted AI model and reap rewards! ...of course, some people will complain, but remember, you can always just tell them they're stupid and randomly rotate the real model back in 1/10th of the time or for demos or promotions, or charge for a 'premium' model. I never really thought about this before, but I bet lots of AI startups are already doing this! (I am being sarcastic; this is some deep user-hostile-for-profit action, and yet another reason to be both skeptical of, and avoid, AI startups. Enshitification at its finest.)
- zeneb 3y agoSomething with this story doesn't track for me: according to Together.ai's docs Mixtral is not available for fine-tuning. And it also looks like they won't run your fine-tuned models serverless. See https://docs.together.ai/docs/fine-tuning-models https://docs.together.ai/docs/fine-tuning-models and https://docs.together.ai/docs/faq#what-does-pricing-look-like-for-serverless-endpoints-vs-dedicated-instances-of-my-fine-tuned-models https://docs.together.ai/docs/faq#what-does-pricing-look-lik...
- WeMoveOn 3y agocan someone explain how his costs went to $1? he essentially just replaced GPT4 with a tuned variant of mixtral 8x7b which requires multiple GPUs to run. even if he quantized the model himself it would still need to pay for the hardware and infra, which would require more than $1. is he self hosting or something?
- rldjbpin 3y agothe web3 bubble analogy keeps writing itself. these new chatbot models are like we used to have new crypto tokens. literally a copy-paste fork of the original bitcoin code, rebranded to some meme as "innovation".