5 ms·

What frustrates me about Stable Diffusion is there doesn't seem to be any documentation as to what artists or vocabulary it understands. Generally people say "l

What frustrates me about Stable Diffusion is there doesn't seem to be any documentation as to what artists or vocabulary it understands. Generally people say "look at existing prompts or use various prompt generators" but that doesn't really solve the problem. I don't want to just look at what other people have randomly discovered; I want to know what the program really knows.

deleted 4y ago
[deleted]
pwinnski 4y ago
It's the nature of ML models--nobody is 100% sure what it understands until they try something and get results. It was given a lot of tagged data: 600 million captioned images from LAION-5B. So if you want to know what it might support, you could try any one of the captions from those 600 million images.
- jhbadger 4y ago
  But why isn't the list of words from those captions available anywhere (at least as far as I can tell)? There may be 600 million captions, but the number of unique words would probably be 10 or 20 thousand at most, completely feasible to browse or grep.
  - spijdar 4y ago
    I don't think the underlying model is word based, but character based. You could download the caption data for LAION and grep that, but it's not strictly 1:1 with what SD was trained against.
    - nl 4y ago
      No, it's word based. The vocabulary is here: https://huggingface.co/openai/clip-vit-base-patch32/resolve/main/vocab.json https://huggingface.co/openai/clip-vit-base-patch32/resolve/... It is contextual though, so words in different orders mean different things.
      - jhbadger 4y ago
        Thank you. This is really helpful. Yes, you don't know exactly how SD will respond, but for example you can grep celebrity names and can know whether SD has any chance of drawing a picture with them in it or not rather than just randomly guessing.
        pwinnski 4y ago
        It's a word list, so as I'm sure you've already figured out, you have to grep first and last names separately. For example, "jennifer" as a first name is token 19786, while "garner</w>" is token 20340. If you want "james garner" instead, looks like that's tokens 6963 and 20340. Except, since it's a word list, there's still no guarantee that either celebrity is necessarily represented until you try.
      - spijdar 4y ago
        Huh, interesting, I had just ... assumed CLIP's tokenizer was character based, like GPT's was. At least, I think GPT's is character based? Is there any reason it couldn't be character based, besides the (presumably very large) increase in resources needed to train and run inference? This is all way out of my league, but seems like you could get interesting results from this, since (by my caveman understanding) this hypothetical transformer could make some sense of words it had never seen before, so spelling variants or neologisms and such.
        nl 4y ago
        I started a proper reply but had to board a plane. It's actually a byte-pair encoded (BPE is better than character encoding but can do the things you mentioned) list of things that includes words. You can find common English suffixes in it listed separately too.
        spijdar 4y ago
        Thanks for the responses, I really appreciate the help. My only background with ML is playing with LSTMs and simple sequence-to-sequence models back before transformers, and the last few days I've been trying to deep dive as much as I can into the "state-of-the-art". I dislike treating the technology as a magical black box...
  - thedorkknight 4y ago
    I haven't downloaded the database myself, but I imagine if you did it wouldn't be too hard to get that data. Looks like you can get the torrent here https://laion.ai/blog/laion-400-open-dataset/ https://laion.ai/blog/laion-400-open-dataset/
  - bawolff 4y ago
    Its kind of a weird complaint. If i am having a conversation with someone, i wouldn't be concerned about knowing the set of all possible nouns.
    - jhbadger 4y ago
      SD isn't a person you can converse with. It's just a program trained on captions and can do no more than what's in them. It's like those old adventure games that would always complain "I don't that word" except even worse because SD will happily make a picture with words it doesn't know and not tell you.
      - bawolff 4y ago
        I think anyone who has both played an IF game and has played with stable diffusion knows there is a world of difference between the two. The main difference is that coming up with a word SD doesn't know that's not contrived is really difficult. In an IF game, you are constantly guessing the correct word.
  - moyix 4y ago
    You can download it here: https://github.com/rom1504/img2dataset/blob/main/dataset_examples/laion5B.md https://github.com/rom1504/img2dataset/blob/main/dataset_exa... You probably would want to stop after getting the metadata, unless you have 240TB available for the images :) More details and links to dataset explorers here: https://laion.ai/blog/laion-5b/ https://laion.ai/blog/laion-5b/
  - nl 4y ago
    The complete vocabulary is available here: https://huggingface.co/openai/clip-vit-base-patch32/resolve/main/vocab.json https://huggingface.co/openai/clip-vit-base-patch32/resolve/... It's a bit less than 50k words, but that includes space-padded duplicates.
  - pwinnski 4y ago
    49,407 tokens, many of which are not useful. It's an arduous process to narrow down, so this link decided to go in the other direction, working from zero up rather than 49,407 down.
spijdar 4y ago
The simple answer is that there is no clean cut list of artists that it "understands". The model has no explicitly programmed concept of artist or style -- just the CLIP based text encoding used to train the conditional autoencoding part of the denoiser network, trained on (AFAIK) caption data recorded with the image. So in practice asking for art "in the style of <x>" is sort of limiting the denoiser to statistical pathways resembling other images captioned "in the style of <x>". At least, that's my understanding. Still trying to grok ML and diffusion models.
- astrange 4y ago
  You can create (or discover) explicit vocabulary in the model using “textual inversion”, or train more into it using fine tuning.
smoldesu 4y ago
You can't really debug an AI. The dimensions of it's understanding are quite literally beyond human interpretation, which makes it both smarter/more efficient than humans while also extremely dumb and context-unaware. Most of our attempts at adding a 'memory' to AI has been a hack thus far, which is why all of these prompts consist of people force-feeding word salad down the AI's throat for generally reproducible results.
- the8472 4y ago
  They're debuggable, with effort. Here's the finance neuron in CLIP: https://microscope.openai.com/models/contrastive_4x/image_block_4_5_Add_6_0/1330 https://microscope.openai.com/models/contrastive_4x/image_bl...
- sdenton4 4y ago
  For memory, check out: 1) Retro, which is essentially attention over large databases, and fast as hell. 2) S4 Layers, explicitly designed for handling long dependencies. These are orthogonal approaches to memory, and both very effective at what they do.
novok 4y ago
You can by searching the training image set and their text tags essentially. https://laion-aesthetic.datasette.io/laion-aesthetic-6pls/images https://laion-aesthetic.datasette.io/laion-aesthetic-6pls/im...
dopidopHN 4y ago
Honest question, would a solid understanding of the open training data help? Having the art vocabulary down as well. In effect, knowing what is present and how it’s tagged so you can « invoke » it more readily in the prompt-result. Maybe I’m out of my depth. I know the corpus of tagged image used for training is enormous … but I still think that would help the user ( a prompt-crafter )
- jhbadger 4y ago
  I downloaded the vocab.json mentioned above and I think it helps. For example it explains why I can get SD to make pictures involving Einstein but not Feynman. Feynman simply isn't in the training set, but Einstein is.
  - dopidopHN 4y ago
    thanks for the follow up, that was my understanding but your example is telling. I will have to check myself. I tryed with minor success to have stable diffusion draw lesser know non-us personality. It always kind of work, but the palette is limited. For instance I tried charles de gaule and you get something that look like him. But he's depicted talking on a radio or waving his arm around like a politician. I tried to make him to grocery or play volley ball, it does not really work. While Michael jackson or Dennis Rodman get a way better treatment. edit : that vocab file is smaller than I thought. "De Gaule" is not in there but neither Einstein or feiman? I think I missing something. I Can find "obama", trump or Macron. Unclear about Michael Jackson. No beyonce or Denis Rodman. hmm weird, I had great result with all of them. Like.. recognizable details like tatoo or silly glasses.
otabdeveloper4 4y ago
"It" doesn't understand anything. It's just a very advanced madlibs engine based on a database of a billion of already known images.
- bawolff 4y ago
  Some people would probably argue that humans are essentially the same.
  - krapp 4y ago
    Whatever else this emergent "creative AI" phenomenon may or may not do, it's definitely touching nerves in people who still believe there's something ineffable and transcendent about the creative experience.
    - otabdeveloper4 4y ago
      Stable Diffusion is literally copy-pasting existing artwork. It's not creating anything new. If anything, this sort of "AI" only makes human ineffable creative experiences even more valuable, because without it there would be no training set and no "AI".
      - krapp 4y ago
        > Stable Diffusion is literally copy-pasting existing artwork. It's not creating anything new. Do you have a source for that? Somehow I doubt there would be this much interest in a tool that literally just copies existing artwork.
      - bawolff 4y ago
        Whether or not its creating something "new" is debatable, but its clearly not copy pasting. It is basing the artwork on a combination of other sources, nothing is created de novo, but that is very different than copy & pasting
      - int_19h 4y ago

deleted 4y ago

[deleted]

pwinnski 4y ago

It's the nature of ML models--nobody is 100% sure what it understands until they try something and get results. It was given a lot of tagged data: 600 million captioned images from LAION-5B. So if you want to know what it might support, you could try any one of the captions from those 600 million images.

jhbadger 4y ago
But why isn't the list of words from those captions available anywhere (at least as far as I can tell)? There may be 600 million captions, but the number of unique words would probably be 10 or 20 thousand at most, completely feasible to browse or grep.
- spijdar 4y ago
  I don't think the underlying model is word based, but character based. You could download the caption data for LAION and grep that, but it's not strictly 1:1 with what SD was trained against.
  - nl 4y ago
    No, it's word based. The vocabulary is here: https://huggingface.co/openai/clip-vit-base-patch32/resolve/main/vocab.json https://huggingface.co/openai/clip-vit-base-patch32/resolve/... It is contextual though, so words in different orders mean different things.
    - jhbadger 4y ago
      Thank you. This is really helpful. Yes, you don't know exactly how SD will respond, but for example you can grep celebrity names and can know whether SD has any chance of drawing a picture with them in it or not rather than just randomly guessing.
      - pwinnski 4y ago
        It's a word list, so as I'm sure you've already figured out, you have to grep first and last names separately. For example, "jennifer" as a first name is token 19786, while "garner</w>" is token 20340. If you want "james garner" instead, looks like that's tokens 6963 and 20340. Except, since it's a word list, there's still no guarantee that either celebrity is necessarily represented until you try.
    - spijdar 4y ago
      Huh, interesting, I had just ... assumed CLIP's tokenizer was character based, like GPT's was. At least, I think GPT's is character based? Is there any reason it couldn't be character based, besides the (presumably very large) increase in resources needed to train and run inference? This is all way out of my league, but seems like you could get interesting results from this, since (by my caveman understanding) this hypothetical transformer could make some sense of words it had never seen before, so spelling variants or neologisms and such.
      - nl 4y ago
        I started a proper reply but had to board a plane. It's actually a byte-pair encoded (BPE is better than character encoding but can do the things you mentioned) list of things that includes words. You can find common English suffixes in it listed separately too.
        spijdar 4y ago
        Thanks for the responses, I really appreciate the help. My only background with ML is playing with LSTMs and simple sequence-to-sequence models back before transformers, and the last few days I've been trying to deep dive as much as I can into the "state-of-the-art". I dislike treating the technology as a magical black box...
- thedorkknight 4y ago
  I haven't downloaded the database myself, but I imagine if you did it wouldn't be too hard to get that data. Looks like you can get the torrent here https://laion.ai/blog/laion-400-open-dataset/ https://laion.ai/blog/laion-400-open-dataset/
- bawolff 4y ago
  Its kind of a weird complaint. If i am having a conversation with someone, i wouldn't be concerned about knowing the set of all possible nouns.
  - jhbadger 4y ago
    SD isn't a person you can converse with. It's just a program trained on captions and can do no more than what's in them. It's like those old adventure games that would always complain "I don't that word" except even worse because SD will happily make a picture with words it doesn't know and not tell you.
    - bawolff 4y ago
      I think anyone who has both played an IF game and has played with stable diffusion knows there is a world of difference between the two. The main difference is that coming up with a word SD doesn't know that's not contrived is really difficult. In an IF game, you are constantly guessing the correct word.
- moyix 4y ago
  You can download it here: https://github.com/rom1504/img2dataset/blob/main/dataset_examples/laion5B.md https://github.com/rom1504/img2dataset/blob/main/dataset_exa... You probably would want to stop after getting the metadata, unless you have 240TB available for the images :) More details and links to dataset explorers here: https://laion.ai/blog/laion-5b/ https://laion.ai/blog/laion-5b/
- nl 4y ago
  The complete vocabulary is available here: https://huggingface.co/openai/clip-vit-base-patch32/resolve/main/vocab.json https://huggingface.co/openai/clip-vit-base-patch32/resolve/... It's a bit less than 50k words, but that includes space-padded duplicates.
- pwinnski 4y ago
  49,407 tokens, many of which are not useful. It's an arduous process to narrow down, so this link decided to go in the other direction, working from zero up rather than 49,407 down.

spijdar 4y ago

The simple answer is that there is no clean cut list of artists that it "understands". The model has no explicitly programmed concept of artist or style -- just the CLIP based text encoding used to train the conditional autoencoding part of the denoiser network, trained on (AFAIK) caption data recorded with the image. So in practice asking for art "in the style of <x>" is sort of limiting the denoiser to statistical pathways resembling other images captioned "in the style of <x>". At least, that's my understanding. Still trying to grok ML and diffusion models.

astrange 4y ago
You can create (or discover) explicit vocabulary in the model using “textual inversion”, or train more into it using fine tuning.

smoldesu 4y ago

You can't really debug an AI. The dimensions of it's understanding are quite literally beyond human interpretation, which makes it both smarter/more efficient than humans while also extremely dumb and context-unaware. Most of our attempts at adding a 'memory' to AI has been a hack thus far, which is why all of these prompts consist of people force-feeding word salad down the AI's throat for generally reproducible results.

the8472 4y ago
They're debuggable, with effort. Here's the finance neuron in CLIP: https://microscope.openai.com/models/contrastive_4x/image_block_4_5_Add_6_0/1330 https://microscope.openai.com/models/contrastive_4x/image_bl...
sdenton4 4y ago
For memory, check out: 1) Retro, which is essentially attention over large databases, and fast as hell. 2) S4 Layers, explicitly designed for handling long dependencies. These are orthogonal approaches to memory, and both very effective at what they do.

novok 4y ago

You can by searching the training image set and their text tags essentially. https://laion-aesthetic.datasette.io/laion-aesthetic-6pls/images https://laion-aesthetic.datasette.io/laion-aesthetic-6pls/im...

dopidopHN 4y ago

Honest question, would a solid understanding of the open training data help? Having the art vocabulary down as well. In effect, knowing what is present and how it’s tagged so you can « invoke » it more readily in the prompt-result. Maybe I’m out of my depth. I know the corpus of tagged image used for training is enormous … but I still think that would help the user ( a prompt-crafter )

jhbadger 4y ago
I downloaded the vocab.json mentioned above and I think it helps. For example it explains why I can get SD to make pictures involving Einstein but not Feynman. Feynman simply isn't in the training set, but Einstein is.
- dopidopHN 4y ago
  thanks for the follow up, that was my understanding but your example is telling. I will have to check myself. I tryed with minor success to have stable diffusion draw lesser know non-us personality. It always kind of work, but the palette is limited. For instance I tried charles de gaule and you get something that look like him. But he's depicted talking on a radio or waving his arm around like a politician. I tried to make him to grocery or play volley ball, it does not really work. While Michael jackson or Dennis Rodman get a way better treatment. edit : that vocab file is smaller than I thought. "De Gaule" is not in there but neither Einstein or feiman? I think I missing something. I Can find "obama", trump or Macron. Unclear about Michael Jackson. No beyonce or Denis Rodman. hmm weird, I had great result with all of them. Like.. recognizable details like tatoo or silly glasses.

otabdeveloper4 4y ago

"It" doesn't understand anything. It's just a very advanced madlibs engine based on a database of a billion of already known images.

bawolff 4y ago
Some people would probably argue that humans are essentially the same.
- krapp 4y ago
  Whatever else this emergent "creative AI" phenomenon may or may not do, it's definitely touching nerves in people who still believe there's something ineffable and transcendent about the creative experience.
  - otabdeveloper4 4y ago
    Stable Diffusion is literally copy-pasting existing artwork. It's not creating anything new. If anything, this sort of "AI" only makes human ineffable creative experiences even more valuable, because without it there would be no training set and no "AI".
    - krapp 4y ago
      > Stable Diffusion is literally copy-pasting existing artwork. It's not creating anything new. Do you have a source for that? Somehow I doubt there would be this much interest in a tool that literally just copies existing artwork.
    - bawolff 4y ago
      Whether or not its creating something "new" is debatable, but its clearly not copy pasting. It is basing the artwork on a combination of other sources, nothing is created de novo, but that is very different than copy & pasting
    - int_19h 4y ago