5 ms·

I run GPT-J on a Titan RTX where I am writing a novel with it. To make it generate about 20k tokens or two pages of content takes a few minutes . I would say th

I run GPT-J on a Titan RTX where I am writing a novel with it. To make it generate about 20k tokens or two pages of content takes a few minutes . I would say the output is comparable to other language models quality and so forth. Note that refinement or transfer learning doesn’t apply anymore it’s more like using a zero shot classifier or in other words you have to craft the input like Siri or wolfram alpha but expect text back instead It runs on about 15gb of VRAM and https://www.eleuther.ai/ https://www.eleuther.ai/ released it under the Apache license. I use this endpoint written by kinoc using FastAPI https://gist.github.com/kinoc/f3225092092e07b843e3a2798f7b3986 https://gist.github.com/kinoc/f3225092092e07b843e3a2798f7b39... which is released under the MIT licence.

mmastrac 5y ago
Can I run it on an RTX-3090? Where would I find information on how to do it? EDIT: Reply below pointed out that the gist linked above specifically mentions 3090 at the top.
- Tenoke 5y ago
  The link mentions Titan and 3090 specifically so I'd guess yes.
gurchik 5y ago
> I am writing a novel with it. Could you say more?
- coolspot 5y ago
  Spam Amazon Kindle store with GPT-generated $.99 novel books.
- zitterbewegung 5y ago
  Novel will be a stream of consciousness of a person that imitates a ;light novel.
c7DJTLrn 5y ago
That seems ridiculously slow. How is machine learning supposed to scale like this?
- retrac 5y ago
  I mean there's two ways. 1. Just wait a few years for better hardware. Not even joking on that one. The computational requirements of most modern GAN and transformer-based approaches, like the StyleGAN image generation I was playing around with recently, would have been daunting even on the largest supercomputers just 10 or 15 years ago and before that it was science fiction. 2. Find more shortcuts. We all knew neural nets could learn any function all along, in principle. If you have infinite time and space, some of the simplest multilayer feedback networks can learn anything learnable. I don't think anyone really expected to find such relatively efficient algorithms for that, like we have in the last few years, though.
- zitterbewegung 5y ago
  I’m only running one Titan RTX which is slower than a 3090 and you can parallelize the system by buying multiple graphics card or by increasing the amount of VRAM used on the endpoint and buying an Ampere .
drusepth 5y ago
It sounds like we're using very similar setups for writing. :) I've primarily been using GPT-3 (and burning through millions of tokens) so I've been experimenting with GPT-J more lately and I've found it makes significantly more basic logic errors (e.g. mixing up pronouns, "forgetting" characters, using new characters), which makes me lean more towards constantly regenerating text versus revising it, and I feel like I'm backed into a box of having to babysit ~100-200 tokens at a time instead of generating significantly more at a time like I do with GPT-3. I also built a quick tool that lets me adjust how much context I'm using in the prompt and generate 2-3 side-by-side completions to pick and choose from (just to speed up the flow of `click best suggestion` --> `keep generating from there`), but I haven't integrated GPT-J yet since it just feels... lower quality (it feels similar to ~Babbage, IMO). But being comparatively* free, I'm still excited about GPT-J. Do you have any tips or processes you've found to make it spit out higher-quality text? 20k tokens at a time is quite a lot -- do you also have problems with winding paths / staying on a general "plot"? Would love to hear any suggestions you have, because I'd sure love to move off GPT-3 to something comparable in quality!
- deleted 5y ago
  [deleted]
- zitterbewegung 5y ago
  I didn't experiment that much to get the difference between Babbage and GPTJ but that sounds similar than what I experienced. I am not sure what your novel is going to look like but due to the limitations between GPT-3 / GPTJ etc.. I plan on it being something like a stream of consciousness. I thought of babysitting lower amounts of tokens but I was getting okay quality with a few thousand but when I looked into it I saw that in the middle usually there was some weird change. I know that there is a GPTJ setting (and I believe GPT-3 has this) where you can see the best of X tries. Clicking best suggestion sounds like a good idea of a tool but it might be better just to recursively store all outputs. I am interested in the tool you are using are you able to share it? The writing process I am using is that there was a YouTube reviewer that used a dataset of Light Novels and utilized a refined version of GPT-2 on that to create a title of some light novels and then basically cherry picked them / did analysis / had his friends do a Turing test. I would assume that using GPT-3 / GPTJ could do the same thing but doing a zero shot with a 10-20 token limitation. (EDIT GPT-3 using Davinci and the Two-sentence Horror Story would allow you to generate interacting story titles and then from there you can use GPT-3 / GPTJ to do the rest.) (also if anyone is interested in the novel I am writing email me at zitterbewegung at gmail dot com).
la6471 5y ago
However the fact that it is trained with 6B parameters compared to GPT-3’s 175B parameters indicate open source GPT still has a lot to catch on. GodSpeed and full speed !!!

mmastrac 5y ago

Can I run it on an RTX-3090? Where would I find information on how to do it? EDIT: Reply below pointed out that the gist linked above specifically mentions 3090 at the top.

Tenoke 5y ago
The link mentions Titan and 3090 specifically so I'd guess yes.

gurchik 5y ago

> I am writing a novel with it. Could you say more?

coolspot 5y ago
Spam Amazon Kindle store with GPT-generated $.99 novel books.
zitterbewegung 5y ago
Novel will be a stream of consciousness of a person that imitates a ;light novel.

c7DJTLrn 5y ago

That seems ridiculously slow. How is machine learning supposed to scale like this?

retrac 5y ago
I mean there's two ways. 1. Just wait a few years for better hardware. Not even joking on that one. The computational requirements of most modern GAN and transformer-based approaches, like the StyleGAN image generation I was playing around with recently, would have been daunting even on the largest supercomputers just 10 or 15 years ago and before that it was science fiction. 2. Find more shortcuts. We all knew neural nets could learn any function all along, in principle. If you have infinite time and space, some of the simplest multilayer feedback networks can learn anything learnable. I don't think anyone really expected to find such relatively efficient algorithms for that, like we have in the last few years, though.
zitterbewegung 5y ago
I’m only running one Titan RTX which is slower than a 3090 and you can parallelize the system by buying multiple graphics card or by increasing the amount of VRAM used on the endpoint and buying an Ampere .

drusepth 5y ago

It sounds like we're using very similar setups for writing. :) I've primarily been using GPT-3 (and burning through millions of tokens) so I've been experimenting with GPT-J more lately and I've found it makes significantly more basic logic errors (e.g. mixing up pronouns, "forgetting" characters, using new characters), which makes me lean more towards constantly regenerating text versus revising it, and I feel like I'm backed into a box of having to babysit ~100-200 tokens at a time instead of generating significantly more at a time like I do with GPT-3. I also built a quick tool that lets me adjust how much context I'm using in the prompt and generate 2-3 side-by-side completions to pick and choose from (just to speed up the flow of `click best suggestion` --> `keep generating from there`), but I haven't integrated GPT-J yet since it just feels... lower quality (it feels similar to ~Babbage, IMO). But being comparatively* free, I'm still excited about GPT-J. Do you have any tips or processes you've found to make it spit out higher-quality text? 20k tokens at a time is quite a lot -- do you also have problems with winding paths / staying on a general "plot"? Would love to hear any suggestions you have, because I'd sure love to move off GPT-3 to something comparable in quality!

deleted 5y ago
[deleted]
zitterbewegung 5y ago
I didn't experiment that much to get the difference between Babbage and GPTJ but that sounds similar than what I experienced. I am not sure what your novel is going to look like but due to the limitations between GPT-3 / GPTJ etc.. I plan on it being something like a stream of consciousness. I thought of babysitting lower amounts of tokens but I was getting okay quality with a few thousand but when I looked into it I saw that in the middle usually there was some weird change. I know that there is a GPTJ setting (and I believe GPT-3 has this) where you can see the best of X tries. Clicking best suggestion sounds like a good idea of a tool but it might be better just to recursively store all outputs. I am interested in the tool you are using are you able to share it? The writing process I am using is that there was a YouTube reviewer that used a dataset of Light Novels and utilized a refined version of GPT-2 on that to create a title of some light novels and then basically cherry picked them / did analysis / had his friends do a Turing test. I would assume that using GPT-3 / GPTJ could do the same thing but doing a zero shot with a 10-20 token limitation. (EDIT GPT-3 using Davinci and the Two-sentence Horror Story would allow you to generate interacting story titles and then from there you can use GPT-3 / GPTJ to do the rest.) (also if anyone is interested in the novel I am writing email me at zitterbewegung at gmail dot com).

la6471 5y ago

However the fact that it is trained with 6B parameters compared to GPT-3’s 175B parameters indicate open source GPT still has a lot to catch on. GodSpeed and full speed !!!