4 ms·
I don't agree it's cheap. For generation at fairly small scale, sure, but generation is just the party trick. The real power for my use case lies in how much be
by specproc 4y ago
I don't agree it's cheap. For generation at fairly small scale, sure, but generation is just the party trick. The real power for my use case lies in how much better it seems to do at traditional NLP tasks than an out-of-the-box model, with no further fiddling and faffing required.
Say I've got a corpus of ~1m documents, each of 10+ paragraphs and I want to run quote extraction on them (it does this beautifully), vectorise them for similarity search, whatever. This gets pretty expensive pretty fast.
- rattray 4y agoHow much would that example cost, for a sense of scale?
- avibhu 4y agoTangential: you can finetune something like flan-ul2 to do quote extraction using examples generated from chatgpt. If you have a good enough GPU, it should help cut down costs significantly
- specproc 4y agoNice, that sounds like it's worth exploring. Much appreciated. Again though, it's the zero-effort part that's appealing. I'm on a very small team and getting that to close to the same standard will take time for a ham-fisted clod like myself. Worth giving a shot all the same though, thanks again.
- pfdietz 4y agoIt's interesting what you can do with ChatGPT with few shot learning. It generalizes at the drop of a hat, often correctly.
- leobg 4y agoThe zero shot ability is convenient. But for tasks that you need to get done millions of times, I’d much rather spend $10 on GPU compute and maybe a day of training data generation to train a T5 which I then “own”. Also, running your own specialized model locally can be much faster than using someone’s API.
- specproc 4y agoSure, purely a time issue for me. I'm not the most skilled in this area, and I've got a load of core stuff I need to keep on top of. I think we're not far off having something equivalent that can be pulled from Huggingface and run on a near consumer grade GPU. For now, I'll hang tight and see how things progress. Don't disagree.
- leobg 4y agoMaybe one day you’ll be able to tell ChatGPT what kind of model you need and it’ll automatically select the right architecture, gather the training data, and commission the training using the cheapest and/or fastest provider. :)
- winddude 4y agoDon't they have in the ToS you aren't allowed to use outputs for training downstream? Which is a little ridiculous, considering it's ToS. But yea, they cheap cost and lack of training is making me a take a long hard look at how I'm implementing more traditional NLP solutions.
- swyx 4y ago> Don't they have in the ToS you aren't allowed to use outputs for training downstream? you mean this? "Data submitted through the API is no longer used for service improvements (including model training) unless the organization opts in" https://openai.com/blog/introducing-chatgpt-and-whisper-apis https://openai.com/blog/introducing-chatgpt-and-whisper-apis
- winddude 4y agowas referring to "(iii) use output from the Services to develop models that compete with OpenAI; (iv) except as permitted through the API, use any automated or programmatic method to extract data or output from the Services, including scraping, web harvesting, or web data extraction;" ~ https://openai.com/policies/terms-of-use https://openai.com/policies/terms-of-use I think I missed the exception for API, how ever not sure where they are, but seems to be fine based on alpaca. Also interesting they are so hard on web scraping and and extraction, lol. But wow, that is a poorly worded paragraph.
- hooande 4y agoI do this. It works.
- icelancer 4y agoCan you elaborate? Did some brief Google searching but had issues putting it together. We have thousands of documents and data stores we'd like to parse using GPT-3.5 (or the new ChatGPT API) and have been thinking of pretraining to cut things down. Thank you!
- hooande 4y agocontact me at the email in my profile
- Ultimatt 4y agoI suspect the author doesnt realise one request with hardly anything returned is many hundreds if not thousands of "tokens". It adds up very fast. Just some debug effort on a nonsense demo learning project cost $5 in a couple of hours. For maybe a hundred or so requests.
- manmal 4y agoI used it for dozens of requests yesterday and that amounted to less than 7 cents. I used MacGPT for that.
- Ultimatt 4y agoSo 7 cents for dozens of requests is only about 1/10th what I was saying. So could be I have the old API, but even that 7 cents for 10s of requests is not cheap compared to executing a model yourself at scale.
- manmal 4y agoIt's cheap enough to not bother about the price even for casual use.
- carlosdp 4y agoThat's straight up not true, unless that "demo learning project" is feeding GPT the entire Bible or something. I have a project that uses davinci-003 (not even the cheaper ChatGPT API) like crazy and I don't come close to paying more than $30-120/month. With the ChatGPT API, it'll be 10x less...
- Ultimatt 4y agoYou're right it was $3.74 https://ibb.co/gRLQBZ4 https://ibb.co/gRLQBZ4 Returning under 300 characters and the prompt sent was about 10 words in length.
- pharke 4y agoYou could have saved some money by writing tests. How much text were you sending at a time? I’ve been summarizing multiple 500 word chunks per query in my app as well as generating embeddings and haven’t broken $10 over the course of a couple weeks.
- manmal 4y agoI’d wager it could cost anywhere between 1-10k to do that, which is a considerable amount of money. Might still be worth it though? If the alternative is mechanical turk, that would probably cost x1000-10000? Are there any ML alternatives that reliably produce useful results?
- MacsHeadroom 4y agoIt would cost less than $50 of compute to finetune an existing open source LLM to do the job for another $50 of computer. OpenAI's APIs are incredibly expensive compared to powerful open source LLMs which run on very accessible hardware. As one data point, LLaMA-13B beats GPT-3 175B in benchmarks, runs on a single 8GB VRAM consumer GPU, and takes only 24GB of VRAM to fine tune. (Though this particular model can't be used for commercial purposes.)
- manmal 4y agoI actually read about LLaMA after posting the above comment, and I do agree with you. You probably can't use it OOTB right now because LLaMA prompts seem harder to get right, but seeing the rate of progress, that will probably be solved in a few weeks.
- celestialcheese 4y ago1000x this. Entity extraction from unstructured text with zero/few-shot is fantastic. I've got a use case where I need to extract model numbers from text - these LLMs are so good at it with very little work.
- Oras 4y agoBe careful with NER, while it is very good, it is not perfect. Example, I tried to extract skills from a job posting. ChatGPT did well, but there were skills missing. It is good to find some entities but then you need to extend the labeling manually.
- celestialcheese 4y agoFor sure. If perfect accuracy is important, it's still good to do sampling and human review to figure out accuracy rate and decide if further checking is required. But it still beats the pants off accuracy of other methods for the amount of work required. That said, with fine-tuning, `davinci-003` is _excellent_ at the types of entity extraction you're describing.
- dandiep 4y agoThere is no such thing as davinci-003 per se. There is text-davinci-003 which you can't fine tune, and davinci which you can.
- rolisz 4y agoSure, but SOTA on NER is around 90% for things like Names and Places. For skills, a Spacy model will get around 60-70%. And training a Spacy model takes a bit of fiddling.
- Oras 4y agoCheckout Argilla for annotation. You can use vectors to speed up the annotations and you can also start with zero-shot feedback to improve the training as you go https://www.argilla.io/ https://www.argilla.io/
- andix 4y agoWhat’s the alternative? Hiring humans to do the job for you? Probably much more expensive.
- coolspot 4y agoA cheaper alternative is fine-tuning more specialized open-source model and hosting it on your own hardware.
- 00117 4y agoWhat happens when you hit a guardrail because there was a politically-charged or socially-condoned phrase? What if the documents are potentially sensationalist news articles or Wikipedia articles with potentially-questionable topics?
- sp332 4y agoIsn't there a limit on the input length? Like you couldn't put a whole research paper into it?
- dandiep 4y agoYes, however, it can generate a training dataset to fine tune a smaller model really cheaply, which can then scale.
- solomonyardley 4y agoHow expensive, if you would actually run the numbers for your use case?