4 ms·

The other day someone was saying that DALL-E's work is not creative enough to be art, and now this article is saying it's not accurate enough. Should DALL-E pl

by HKH2 4y ago

The other day someone was saying that DALL-E's work is not creative enough to be art, and now this article is saying it's not accurate enough. Should DALL-E play around with interpretations or not?

Geonode 4y ago
Art is using the qualities of a medium to enrich your message. Dall-E and it's training data are a medium with qualities that we can use to enrich a text based message.
TheOtherHobbes 4y ago
It's not creative in the sense that it's clearly an automaton and has no awareness of culture, history, or context. It seems to operate like a huge multidimensional map of an image space with some verbal tags stuck into it rather randomly, and with style transfer applied as an operator to merge different features in the results. What it needs is an equivalent map for the concepts behind the signposts, and also a mapping between concepts and images. The concept map would be very hard to train because a concept space that includes all of culture and history would be huge and messy. A large concept map with a huge set of associations is one of the features of educated humans. But the concept mesh is grounded in subjective experience, so it would need an equivalent layer for that.
- visarga 4y ago
  > What it needs is an equivalent map for the concepts behind the signposts, It's the CLIP text and image encoders of DALL-E 2. > and also a mapping between concepts and images. It's the "prior" module. You guessed right but DALL-E 2 already has these functionalities in its design. https://i0.wp.com/bdtechtalks.com/wp-content/uploads/2022/04/DALL-E-2-architecture.jpg https://i0.wp.com/bdtechtalks.com/wp-content/uploads/2022/04...
  - astrange 4y ago
    It seems like it'd be easier to use if you could see the intermediate steps, i.e. what the CLIP encoder "thought" the text prompt meant, and how it "thought" to turn that into an image prior. That could be done by decoding both into text again. Also, if you could feed it some kind of "knowledge graph" of words and images rather than a sentence, you wouldn't be misunderstood the same way, but you'd presumably run into the limit of what its embeddings can contain.
- HKH2 4y ago
  I would argue that the risk-taking part of the creative process can also be indifferent to context. Highly impulsive processes need not be inclined to rationality. Yes there needs to be an evaluator to guide and contextualize the results, but when most of the mechanical parts can be done so quickly (often most of the effort of the creative process), that's bound to lead to far higher output and willingness to get creative work done. You're right that it in itself is not strictly creative because it has no intentionality, but I've no doubt it will lead to more creativity.
Dylan16807 4y ago
Being inaccurate does not imply being creative. There's some connection but they're largely separate.
- HKH2 4y ago
  Taking risks is a necessary part of the creative process. Making something less 'accurate' is one possible risk. Whether something is on a good path or not is part of the curation process.
  - Dylan16807 4y ago
    The way it's failing, and the way that gets triggered, isn't really a "taking risks" kind of failing. There's a lot of 'creativity' to showcase, but that doesn't seem to be what's causing these failures.
    - HKH2 4y ago
      As with typical art, time and energy are an investment. In this case the risk is taken by the person using it. I guess 'creative' will always have intentionality built into it. Maybe an imagination unconstrained by purpose (e.g. dreaming) is not creative either.
      - Dylan16807 4y ago
        > As with typical art, time and energy are an investment. In this case the risk is taken by the person using it. This happens with or without creativity. There are failure modes that showcase creativity, and there are failure modes that don't showcase creativity. These seem like they're largely the latter to me.

by HKH2 4y ago

The other day someone was saying that DALL-E's work is not creative enough to be art, and now this article is saying it's not accurate enough. Should DALL-E play around with interpretations or not?

Geonode 4y ago

Art is using the qualities of a medium to enrich your message. Dall-E and it's training data are a medium with qualities that we can use to enrich a text based message.

TheOtherHobbes 4y ago

It's not creative in the sense that it's clearly an automaton and has no awareness of culture, history, or context. It seems to operate like a huge multidimensional map of an image space with some verbal tags stuck into it rather randomly, and with style transfer applied as an operator to merge different features in the results. What it needs is an equivalent map for the concepts behind the signposts, and also a mapping between concepts and images. The concept map would be very hard to train because a concept space that includes all of culture and history would be huge and messy. A large concept map with a huge set of associations is one of the features of educated humans. But the concept mesh is grounded in subjective experience, so it would need an equivalent layer for that.

visarga 4y ago
> What it needs is an equivalent map for the concepts behind the signposts, It's the CLIP text and image encoders of DALL-E 2. > and also a mapping between concepts and images. It's the "prior" module. You guessed right but DALL-E 2 already has these functionalities in its design. https://i0.wp.com/bdtechtalks.com/wp-content/uploads/2022/04/DALL-E-2-architecture.jpg https://i0.wp.com/bdtechtalks.com/wp-content/uploads/2022/04...
- astrange 4y ago
  It seems like it'd be easier to use if you could see the intermediate steps, i.e. what the CLIP encoder "thought" the text prompt meant, and how it "thought" to turn that into an image prior. That could be done by decoding both into text again. Also, if you could feed it some kind of "knowledge graph" of words and images rather than a sentence, you wouldn't be misunderstood the same way, but you'd presumably run into the limit of what its embeddings can contain.
HKH2 4y ago
I would argue that the risk-taking part of the creative process can also be indifferent to context. Highly impulsive processes need not be inclined to rationality. Yes there needs to be an evaluator to guide and contextualize the results, but when most of the mechanical parts can be done so quickly (often most of the effort of the creative process), that's bound to lead to far higher output and willingness to get creative work done. You're right that it in itself is not strictly creative because it has no intentionality, but I've no doubt it will lead to more creativity.

Dylan16807 4y ago

Being inaccurate does not imply being creative. There's some connection but they're largely separate.

HKH2 4y ago
Taking risks is a necessary part of the creative process. Making something less 'accurate' is one possible risk. Whether something is on a good path or not is part of the curation process.
- Dylan16807 4y ago
  The way it's failing, and the way that gets triggered, isn't really a "taking risks" kind of failing. There's a lot of 'creativity' to showcase, but that doesn't seem to be what's causing these failures.
  - HKH2 4y ago
    As with typical art, time and energy are an investment. In this case the risk is taken by the person using it. I guess 'creative' will always have intentionality built into it. Maybe an imagination unconstrained by purpose (e.g. dreaming) is not creative either.
    - Dylan16807 4y ago
      > As with typical art, time and energy are an investment. In this case the risk is taken by the person using it. This happens with or without creativity. There are failure modes that showcase creativity, and there are failure modes that don't showcase creativity. These seem like they're largely the latter to me.