3 ms·

Wait, it’s just vanilla gpt-j? No fine tuning? “Back in my day, we had to train our own models..” already sounds anachronistic. Nicely polished. Looks like b

Wait, it’s just vanilla gpt-j? No fine tuning? “Back in my day, we had to train our own models..” already sounds anachronistic. Nicely polished. Looks like bmk (nabla theta) was right that arxiv was an impactful addition to The Pile. I bet that’s where J got its knowledge in this case.

thesephist 5y ago
Yep! No fine tuning. Here's the prompt I use for the description (from source, https://github.com/thesephist/modelexicon/blob/main/src/main.oak#L50-L54 https://github.com/thesephist/modelexicon/blob/main/src/main... ) --- Proceedings of Deep Learning Advancements Conference, list of accepted deep learning models 1. [StyleGAN] StyleGAN is a generative adversarial network for style transfer between artworks. It uses a traditional GAN architecture and is trained on a dataset of 150,000 traditional and modern art. StyleGAN shows improved style transfer performance while reducing computational complexity. 2. [GPT-2] GPT-2 is a decoder-only transformer model trained on WebText, OpenAI\'s proprietary clean text corpus based on Wikipedia, Google News, Reddit, and others comprising a 2TB dataset for autoregressive training. GPT-2 demonstrates state-of-the-art performance on several language modeling and conversational tasks. 3. [$MODELNAME]
- sillysaurusx 5y ago
  That’s awesome! How’d you get such great code usage examples out of J? It almost seems like the code is properly related to the names. GAN code seems to look like GAN code. But I’m not sure.
  - thesephist 5y ago
    The code generated is most definitely related to the names/descriptions! To do this, I have to first generate the description then generate the code _from the descriptions_. The downside of this is that I can't parallelize text generation, but the upside is the code feels much more realistic. Here's the prompt I used (from that same file): The idea here was to give the model a prompt that felt like a tutorial or some kind, and try to minimize non-Python non-ML-y code. --- $MODEL_DESCRIPTION_FROM_EARLIER Let\'s use this model. The basic use case takes only a few lines of Python to run the inference. Here are the first few lines. ```python

thesephist 5y ago

Yep! No fine tuning. Here's the prompt I use for the description (from source, https://github.com/thesephist/modelexicon/blob/main/src/main.oak#L50-L54 https://github.com/thesephist/modelexicon/blob/main/src/main... ) --- Proceedings of Deep Learning Advancements Conference, list of accepted deep learning models 1. [StyleGAN] StyleGAN is a generative adversarial network for style transfer between artworks. It uses a traditional GAN architecture and is trained on a dataset of 150,000 traditional and modern art. StyleGAN shows improved style transfer performance while reducing computational complexity. 2. [GPT-2] GPT-2 is a decoder-only transformer model trained on WebText, OpenAI\'s proprietary clean text corpus based on Wikipedia, Google News, Reddit, and others comprising a 2TB dataset for autoregressive training. GPT-2 demonstrates state-of-the-art performance on several language modeling and conversational tasks. 3. [$MODELNAME]

sillysaurusx 5y ago
That’s awesome! How’d you get such great code usage examples out of J? It almost seems like the code is properly related to the names. GAN code seems to look like GAN code. But I’m not sure.
- thesephist 5y ago
  The code generated is most definitely related to the names/descriptions! To do this, I have to first generate the description then generate the code _from the descriptions_. The downside of this is that I can't parallelize text generation, but the upside is the code feels much more realistic. Here's the prompt I used (from that same file): The idea here was to give the model a prompt that felt like a tutorial or some kind, and try to minimize non-Python non-ML-y code. --- $MODEL_DESCRIPTION_FROM_EARLIER Let\'s use this model. The basic use case takes only a few lines of Python to run the inference. Here are the first few lines. ```python