4 ms·

Emu: Enhancing Image Generation Models Using Photogenic Needles in a Haystack

52 points by mfiguiere 3y ago

refulgentis 3y ago
Spectacular work, best paper I've read in a while, maybe since the DallE-2 paper. (and I've been kicking around since 2019). Easy to understand, really special results. Looks like a recipe for Midjourney.
kelseyfrog 3y ago
They managed to do the thing, but missed the point. The sample results feel straight out of Kinfolk[1] or Cereal[2]. They have that "over-curated" quality we associate with these publications. 1. https://www.kinfolk.com/ https://www.kinfolk.com/ 2. https://www.readcereal.com/ https://www.readcereal.com/
- solardev 3y ago
  I've never heard of these. What are they supposed to be? Just looks like random pictures of places and people?
  - kelseyfrog 3y ago
    They are two lifestyle magazines. I don't expect the HN crowd to be familiar with them, but they are exemplars of the style that Emu has been aligned with.
    - solardev 3y ago
      What exactly is a lifestyle magazine? I'm curious. Like that cereal site, it's just a bunch of unsorted pictures. What kind of lifestyle is it supposed to represent? Is it kinda like a Flickr for the analog world?
      - kelseyfrog 3y ago
        Fair question. Lifestyle magazines generally cover topics in travel, fashion, trends, foods, pop culture. They are one of the most popular types of magazines and their importance in generating demand is an important factor to consider. Readers of, say a cooking lifestyle magazine, might be looking for new recipes to try, while readers of a travel lifestyle magazine might be looking for ideas of where they would like to journey to next, or perhaps they simply subscribe for the joy they take in imagining themselves there. Some publications, as you've noticed, are aligned on other dimensions, but share other common features, like aesthetic. I chose the two sites as examples because they have a particular curation that matches the output of the model. I'm not a lifestyle magazine champion or anything, but I think HN in general likes poking out of their wheelhouse on occasion and this in particular I suspected would not be something in general they are familiar with - and what makes it have, hopefully, some value in a unique perspective.
        solardev 3y ago
        I see, thanks for the explanation! I knew there were magazines about activities (cooking and whatnot), but didn't realize there were some that go for "look" alone. Guess I'm not artsy enough :)
    - refulgentis 3y ago
      How is this the most upvoted thread? Yuck. Read the paper! It's not those magazines, and the story of the alignment and how they did is one of the most interesting parts.
      - totetsu 3y ago
        Don’t yuck their yum. Some of us see the other story here of the value laden use of the words ‘highly aesthetic’ and ‘visually appealing’. PP is questioning to what standard the model is benign aligned
        bawolff 3y ago
        I don't know, i think its pretty pretentious to nitpick what a "good" picture is, when its really irrelavent to the topic at hand and quite frankly not an interesting conversation since it is very subjective. The link isn't to an art contest.
        ryanklee 3y ago
        The paper is about generating art. The paper makes claims about the subjective value of the art. Seems central. If the art is not good, but commercial kitsch, then that matters. Now that software engineers are in the business of making software that generates art, get ready to have these conversations, pretentious or not, whatever is meant by that.
        kelseyfrog 3y ago
        That's right. The authors operationalized "highly aesthetic quality" and grounded it in composition, lighting, color and contrast, subject and background, and "additional subjective assessments." There's zero citations in this section related to accessing artistic quality, operationalization, or grounding. A LOT of programmers are going to hate this, but if you're doing soft-sciences work(which this is), you should be knowledgeable in that area and cite some sources that indicate you've at least considered what you're doing. The lack of citations in this area is an unknowing admittance of that ignorance. I don't care if they cite a paper just to disagree with it, but to not know that they don't know is simply unacceptable. This is basic undergrad material in other relevant fields.
- loeg 3y ago
  In what way did they miss the point?
- bawolff 3y ago
  Looking at the paper and then these websites they don't look that aligned to me. The paper's images are much more saturated and cutesy (which to be clear i dont consider better). Regardless, does it matter. I would assume you could chose any aethetic you like. The interesting bit is the training process not the aesthetic chosen.
ilaksh 3y ago
Is there an APi, GitHub, or web page or application where we can use this today? I think they said eventually it was going to be in a bunch of apps but not sure about right now.
zaptrem 3y ago
How is this any different from how SD was fine tuned on LAION Aesthetic after being pretrained on plain old LAION?

52 points by mfiguiere 3y ago

Spectacular work, best paper I've read in a while, maybe since the DallE-2 paper. (and I've been kicking around since 2019). Easy to understand, really special results. Looks like a recipe for Midjourney.

kelseyfrog 3y ago

They managed to do the thing, but missed the point. The sample results feel straight out of Kinfolk[1] or Cereal[2]. They have that "over-curated" quality we associate with these publications. 1. https://www.kinfolk.com/ https://www.kinfolk.com/ 2. https://www.readcereal.com/ https://www.readcereal.com/

solardev 3y ago
I've never heard of these. What are they supposed to be? Just looks like random pictures of places and people?
- kelseyfrog 3y ago
  They are two lifestyle magazines. I don't expect the HN crowd to be familiar with them, but they are exemplars of the style that Emu has been aligned with.
  - solardev 3y ago
    What exactly is a lifestyle magazine? I'm curious. Like that cereal site, it's just a bunch of unsorted pictures. What kind of lifestyle is it supposed to represent? Is it kinda like a Flickr for the analog world?
    - kelseyfrog 3y ago
      Fair question. Lifestyle magazines generally cover topics in travel, fashion, trends, foods, pop culture. They are one of the most popular types of magazines and their importance in generating demand is an important factor to consider. Readers of, say a cooking lifestyle magazine, might be looking for new recipes to try, while readers of a travel lifestyle magazine might be looking for ideas of where they would like to journey to next, or perhaps they simply subscribe for the joy they take in imagining themselves there. Some publications, as you've noticed, are aligned on other dimensions, but share other common features, like aesthetic. I chose the two sites as examples because they have a particular curation that matches the output of the model. I'm not a lifestyle magazine champion or anything, but I think HN in general likes poking out of their wheelhouse on occasion and this in particular I suspected would not be something in general they are familiar with - and what makes it have, hopefully, some value in a unique perspective.
      - solardev 3y ago
        I see, thanks for the explanation! I knew there were magazines about activities (cooking and whatnot), but didn't realize there were some that go for "look" alone. Guess I'm not artsy enough :)
  - refulgentis 3y ago
    How is this the most upvoted thread? Yuck. Read the paper! It's not those magazines, and the story of the alignment and how they did is one of the most interesting parts.
    - totetsu 3y ago
      Don’t yuck their yum. Some of us see the other story here of the value laden use of the words ‘highly aesthetic’ and ‘visually appealing’. PP is questioning to what standard the model is benign aligned
      - bawolff 3y ago
        I don't know, i think its pretty pretentious to nitpick what a "good" picture is, when its really irrelavent to the topic at hand and quite frankly not an interesting conversation since it is very subjective. The link isn't to an art contest.
        ryanklee 3y ago
        The paper is about generating art. The paper makes claims about the subjective value of the art. Seems central. If the art is not good, but commercial kitsch, then that matters. Now that software engineers are in the business of making software that generates art, get ready to have these conversations, pretentious or not, whatever is meant by that.
        kelseyfrog 3y ago
        That's right. The authors operationalized "highly aesthetic quality" and grounded it in composition, lighting, color and contrast, subject and background, and "additional subjective assessments." There's zero citations in this section related to accessing artistic quality, operationalization, or grounding. A LOT of programmers are going to hate this, but if you're doing soft-sciences work(which this is), you should be knowledgeable in that area and cite some sources that indicate you've at least considered what you're doing. The lack of citations in this area is an unknowing admittance of that ignorance. I don't care if they cite a paper just to disagree with it, but to not know that they don't know is simply unacceptable. This is basic undergrad material in other relevant fields.
loeg 3y ago
In what way did they miss the point?
bawolff 3y ago
Looking at the paper and then these websites they don't look that aligned to me. The paper's images are much more saturated and cutesy (which to be clear i dont consider better). Regardless, does it matter. I would assume you could chose any aethetic you like. The interesting bit is the training process not the aesthetic chosen.

ilaksh 3y ago

Is there an APi, GitHub, or web page or application where we can use this today? I think they said eventually it was going to be in a bunch of apps but not sure about right now.

zaptrem 3y ago

How is this any different from how SD was fine tuned on LAION Aesthetic after being pretrained on plain old LAION?