3 ms·

OK but... what's happening here is SD is capturing textures and some of their relationships. It clearly has no understanding of the objects it's generating. So

OK but... what's happening here is SD is capturing textures and some of their relationships. It clearly has no understanding of the objects it's generating. So the output is a kind of Dali-esque mushed up melted version of the original content. It's entertaining because it's simultaneously referential and heavily distorted in unexpected ways. You could use this a manual art technique, and it would be interesting-ish. I'm curious if it's capable of getting to the next stage, of understanding and distorting the actual design relationships in these objects to the point where it could deepfake catalog pages and they'd be indistinguishable from the real thing. I suspect there's quite a gap between that stage and this one.

notahacker 4y ago
Feels like you'd make more progress with discrete models working together. At the very simple level, Stable Diffusion is pretty terrible at words and fonts but a reasonable model of where [patterns of pixels that look like] text might fit, and how big should it be. Add a second step which recognises Stable-Diffusion generated pseudotext blocks and replaces them with GPT-generated text to the same prompt set in an actual font scaled to match StableDiffusion's attempted font and you'll more likely get something that passes the "zoom in and try to read it" test. Though there may not be very much correspondence between the images and text. A more complex arrangement adds in a model which chooses which does high level structure to fit a prompt (subprompts on suitable topics for images and text for each page), a 'house style' model to pick fonts and copy/paste stuff like the RadioShack logo direct from its source material, plain old StableDiffusion to draw lots of individual pictures "cassette player deluxe RadioShack 1970" and plain old GPT to write the text which is typeset to the 'house style' model's specification, and probably an "observer" model that forces new iterations of really bad pages. Great thing about this magazine creation process is it also tends to work better for humans than having one person do everything!
- uoaei 4y ago
  The analogy in my mind is the heyday of x86 ubiquity vs the coming revolution of many SoCs in one device. Huge, all-purpose, massive-capacity NNs are the norm for the current advances in SOTA. This is probably because of a certain limit on development complexity: designing a complex system of richly interacting parts is hard, so as long as development of these sorts of tools is manual, it will be much easier just to shove more compute at the problem with singular, massive models. Compare this to the all-encompassing architecture hegemony of x86. This makes it relatively easy to write software that people will be able to use on devices they already own, and this reduction in developer effort and complexity is enormous for enabling rapid growth in the number of possible software programs that can be created per unit time. As ML design becomes more automated and functional, new possibilities open up regarding breaking out sub-tasks to individual, hyper-specialized tools and combining them into a resilient and capable whole that is more than the sum of its parts. That is the true power of automatic differentiation frameworks: when your system can be end-to-end differentiable throughout many devices and specialized functions, and you can train parts as easily as you can train the whole, you begin to observe the creation -- the growth, really -- of a new kind of digital cognition machine.
- flemhans 4y ago
  So the first "prompt" would be "write a series of prompts for an AI to use as steps to make variations of this image, and connect it together afterwards"
SquibblesRedux 4y ago
"Looks like" and "resembles" are fairly orthogonal to "understands." Of one is looking for understanding, I would say Stable Diffusion is traveling in the wrong direction.
rjh29 4y ago
https://medium.com/dipchain/mario-klingemann-memories-of-passersby-i-c73f72675743 https://medium.com/dipchain/mario-klingemann-memories-of-pas...
orbital-decay 4y ago
Thing is, SD in particular is relatively small (860M parameters), and quantity can be somewhat transformed into quality in this case. As Google Parti landing page [0] conveniently demonstrates, more parameters with the same architecture yield more coherent output, including text and symbols. Given enough room (enough weights) in all parts of the model, starting from CLIP, it could even construct coherent text, as the compression ratio would be much less. However you'd need much beefier hardware to run it, and it might not be as efficient as having a better architecture or a dataset/training skewed towards symbols. [0] https://parti.research.google/ https://parti.research.google/

notahacker 4y ago

Feels like you'd make more progress with discrete models working together. At the very simple level, Stable Diffusion is pretty terrible at words and fonts but a reasonable model of where [patterns of pixels that look like] text might fit, and how big should it be. Add a second step which recognises Stable-Diffusion generated pseudotext blocks and replaces them with GPT-generated text to the same prompt set in an actual font scaled to match StableDiffusion's attempted font and you'll more likely get something that passes the "zoom in and try to read it" test. Though there may not be very much correspondence between the images and text. A more complex arrangement adds in a model which chooses which does high level structure to fit a prompt (subprompts on suitable topics for images and text for each page), a 'house style' model to pick fonts and copy/paste stuff like the RadioShack logo direct from its source material, plain old StableDiffusion to draw lots of individual pictures "cassette player deluxe RadioShack 1970" and plain old GPT to write the text which is typeset to the 'house style' model's specification, and probably an "observer" model that forces new iterations of really bad pages. Great thing about this magazine creation process is it also tends to work better for humans than having one person do everything!

uoaei 4y ago
The analogy in my mind is the heyday of x86 ubiquity vs the coming revolution of many SoCs in one device. Huge, all-purpose, massive-capacity NNs are the norm for the current advances in SOTA. This is probably because of a certain limit on development complexity: designing a complex system of richly interacting parts is hard, so as long as development of these sorts of tools is manual, it will be much easier just to shove more compute at the problem with singular, massive models. Compare this to the all-encompassing architecture hegemony of x86. This makes it relatively easy to write software that people will be able to use on devices they already own, and this reduction in developer effort and complexity is enormous for enabling rapid growth in the number of possible software programs that can be created per unit time. As ML design becomes more automated and functional, new possibilities open up regarding breaking out sub-tasks to individual, hyper-specialized tools and combining them into a resilient and capable whole that is more than the sum of its parts. That is the true power of automatic differentiation frameworks: when your system can be end-to-end differentiable throughout many devices and specialized functions, and you can train parts as easily as you can train the whole, you begin to observe the creation -- the growth, really -- of a new kind of digital cognition machine.
flemhans 4y ago
So the first "prompt" would be "write a series of prompts for an AI to use as steps to make variations of this image, and connect it together afterwards"

SquibblesRedux 4y ago

"Looks like" and "resembles" are fairly orthogonal to "understands." Of one is looking for understanding, I would say Stable Diffusion is traveling in the wrong direction.

rjh29 4y ago

https://medium.com/dipchain/mario-klingemann-memories-of-passersby-i-c73f72675743 https://medium.com/dipchain/mario-klingemann-memories-of-pas...

orbital-decay 4y ago

Thing is, SD in particular is relatively small (860M parameters), and quantity can be somewhat transformed into quality in this case. As Google Parti landing page [0] conveniently demonstrates, more parameters with the same architecture yield more coherent output, including text and symbols. Given enough room (enough weights) in all parts of the model, starting from CLIP, it could even construct coherent text, as the compression ratio would be much less. However you'd need much beefier hardware to run it, and it might not be as efficient as having a better architecture or a dataset/training skewed towards symbols. [0] https://parti.research.google/ https://parti.research.google/