9 ms·
GPT-J: GPT-3 Democratized
- likecarter 5y agoIronic that we have to create open source versions of things from “OpenAI”…
- imvetri 5y agoHaha
- smnrchrds 5y agoThere is open and there is open. OpenAI is more like the latter, similar to OpenVMS. EDIT: Or OpenWindows desktop environment.
- Judgmentality 5y agoSo not really open, and just bullshit marketing.
- meowkit 5y agoClosedAI doesn't really have the same ring to it...
- ampdepolymerase 5y agoPretty much, it is like "OpenBionics" where everything is closed sourced.
- throwaway2048 5y agoopen as in "open your wallet"
- rorykoehler 5y agoDupe https://news.ycombinator.com/item?id=27727009 https://news.ycombinator.com/item?id=27727009
- Kelamir 5y agoIs it a dupe though if it's a blog post, as opposed to the earlier Github submission?
- rorykoehler 5y agoDebatable but then the title should be better
- Tenoke 5y agoYeah, twice in one day (and 3+ times in a month) is a bit much and this post adds little new except some oddities like singling kubernetes in particular as 'the technology that helped train GPT-3'.
- polynomial 5y agoAs the article notes, "There’s quite a similarity between cloud native and Artificial Intelligence."
- zitterbewegung 5y agoI run GPT-J on a Titan RTX where I am writing a novel with it. To make it generate about 20k tokens or two pages of content takes a few minutes . I would say the output is comparable to other language models quality and so forth. Note that refinement or transfer learning doesn’t apply anymore it’s more like using a zero shot classifier or in other words you have to craft the input like Siri or wolfram alpha but expect text back instead It runs on about 15gb of VRAM and https://www.eleuther.ai/ https://www.eleuther.ai/ released it under the Apache license. I use this endpoint written by kinoc using FastAPI https://gist.github.com/kinoc/f3225092092e07b843e3a2798f7b3986 https://gist.github.com/kinoc/f3225092092e07b843e3a2798f7b39... which is released under the MIT licence.
- mmastrac 5y agoCan I run it on an RTX-3090? Where would I find information on how to do it? EDIT: Reply below pointed out that the gist linked above specifically mentions 3090 at the top.
- Tenoke 5y agoThe link mentions Titan and 3090 specifically so I'd guess yes.
- gurchik 5y ago> I am writing a novel with it. Could you say more?
- coolspot 5y agoSpam Amazon Kindle store with GPT-generated $.99 novel books.
- zitterbewegung 5y agoNovel will be a stream of consciousness of a person that imitates a ;light novel.
- c7DJTLrn 5y ago
- mhuffman 5y agoI was told that GPT-3 was too much power for mere mortals (without a paid subscription!) so what terror is this going to bring upon us?
- robbedpeter 5y agoTransparent and competent automated content moderation, maybe, easily available for anyone to run their own communities by their own standards. Once matured, you can easily envision people sharing policies and templates, or providing moderation as a service, for any sort of social text interaction.
- EamonnMR 5y agoI was just thinking about this today. How many (and which?) ML books would one need to read to be able to throw comments into a model like this and have it decide if they're acceptable vs unacceptable? How practical would it be to build a frequently re-training model that could serve as autopilot for a small forum or subreddit moderation? And how much would that serve to reinforce existing filter bubbles and further divide the internet?
- robbedpeter 5y ago0 books. The huggingface hosted gpt-neo 125 model is capable of doing basic question/answer analysis to a sufficient level for most moderation. No finetuning needed if you have it work out yes or no zero shot answers to things like "is this text bullying, excessively negative, political, sexist..." etc, and then iterate over cases and log results. Modify your prompts to handle edge cases and it would probably be capable of replacing 80% of human moderation.
- EamonnMR 5y agoHmm, I'll still need to scrounge up enough context to understand most of that, but thanks.
- loosetypes 5y agoDoes anyone else take fault with the increasingly commonplace usage of the word democratize as applied to technology? For every valid case, I see others that make my head hurt. Is it just a buzzword for telling a story with emotional appeal to users and investors? Making something available to someone who didn’t have it earlier isn’t democratizing. And ignoring future considerations is just lazy. If guns were invented today they’d probably touted as democratizing violence.
- mkl95 5y agoI had a History teacher who told me democracy often doesn't really mean anything. I've been thinking about it ever since. "Free GPT-3" would probably be better.
- loosetypes 5y agoInteresting. I guess my off the cuff, unexamined criteria for democratizing is whether it’s a gating/enabling function for the occurrence of one party’s objectives as directly decided by the wider community. So something like GoFundMe or crowdfunding, sure. Although doesn’t necessarily need to be monetary. But I’d be interested in hearing counteropinions.
- jfengel 5y agoThat's pretty much exactly what "democratizing" means, at least etymologically. It brings power (kratia) to the people (demos). "Democracy" refers to a political system, but "democratizing" has pretty much always meant to open things up more widely. Example: "The State wishes to democratize instruction by its "French instruction" and the standard must inevitably be the lowering of the very standard it sets up." -- 1894 https://www.google.com/books/edition/Education_from_a_National_Standpoint/P9YKAQAAIAAJ?hl=en&gbpv=0 https://www.google.com/books/edition/Education_from_a_Nation... And yeah -- guns do democratize violence. That's precisely why they were invented. A Google search for "guns democratize violence" turns up several hits.
- hekec 5y agoLove this! Could anyone recommend a tutorial how to run this? I need to generate answers to 1000 questions for my app. Still waiting for GPT-3 invite for months
- Kelamir 5y agoIf you don't need to host it, you could use it online https://6b.eleuther.ai/ https://6b.eleuther.ai/ https://bellard.org/textsynth/ https://bellard.org/textsynth/
- captainmuon 5y agoHow much resources would it take to train something like this? Let's say in RTX-3090 months, or in $ running on AWS (or whoever offers TPUs)? In the link it says training FLOPs: 1.5e22, a RTX 3090 has 285 TFLOPS/sec for tensor operations. If you can actually calculate it like that, 1000 GPUs would take 20 months. https://www.google.com/search?q=1.5e22+%2F+%281000+*+285e9+%2F+s%29+in+months https://www.google.com/search?q=1.5e22+%2F+%281000+*+285e9+%... Working with grid computing at CERN, I've had access to some pretty big computing resources myself, but the scale of this is mind-boggling...