7 ms·

pix2pix-zero: Zero-Shot Image-to-Image Translation

115 points by blurbleblurble 4y ago

pavon 4y ago
Not my field so I don't have anything detailed to add, other than to say how fascinating it is to watch how quickly all these incremental improvements are occurring after the initial breakthrough. Also, props for not cherry picking results too much, both for honesty and because it is fun to see how these fail. Like the tree that grew limbs to fill in a watermark, or the sunset sky that turned into a lava field.
- lloeki 4y ago
  Interesting how the lava field version imprinted my mind. When going back to the original image I now see it as ambiguous - in the same way as a duck/rabbit multistable illusion - and can see a mountain range and sand instead of clouds.
  - MarcoZavala 4y ago
    [dead]
TekMol 4y ago
The "code" link takes you to a GitHub repo. But I don't know how to set up the needed environment. I wish authors of such tools would provide a Dockerfile to set up the environment needed to run their creations. Their setup section starts with conda env create -f environment.yml But that gives me bash: conda: command not found So I tried apt install conda and apt install python3-conda But nothing worked. Why is running software still such a tricky thing in 2023, where you have to read tutorials and tinker with manually setting up an environment? Guess I will wait until some SaaS company provides an API to use this on their machines then. Any ideas, which company might do this first?
- teruakohatu 4y ago
  > Why is running software still such a tricky thing in 2023, where you have to read tutorials and tinker with manually setting up an environment? I think this is unfair criticism. Conda is extremely common in data science, as common as Docker is in software development. While I agree docker is easier to use as a consumer, Conda automatically sets up the environment and a quick Google would show you how to install conda.
  - TekMol 4y ago
    I just said Dockerfile because when I say "shellscript", people think that is oldschool. A dockerfile shows me every step of what has to be done in an unmistakable way. I can easily turn it into a shellscript. The conda file (environment.yml, right?) does not tell me how to set up the environment. It contains a list of .. I don't know what. It starts with: channels: - pytorch - nvidia - defaults I don't know what to do with that. It does not look like computer code. > a quick Google would show you how to install conda No. It leads me to yet another tutorial with a bunch of manual steps: https://docs.conda.io/projects/conda/en/latest/user-guide/install/linux.html https://docs.conda.io/projects/conda/en/latest/user-guide/in... It even starts with decisions I shall make of which I know nothing: Download the installer: - Miniconda installer for Linux. - Anaconda installer for Linux. How do I know which one the software in question needs? How far down this rabbit hole is someone who just wants to run a software supposed to go?
    - extr 4y ago
      [flagged]
      - rowanG077 4y ago
        Conda is a special type of hell though. I have read the manual. Multiple tutorials. Yet I, not even once got conda to work for me.
    - IanCal 4y ago
amrrs 4y ago
Try this colab, if you want to try pix2pix video https://github.com/camenduru/pix2pix-video-colab https://github.com/camenduru/pix2pix-video-colab
- TekMol 4y ago
  Does something like this exist for the single image version? How hard is it to set up such a thing on HuggingFace?
- blurbleblurble 4y ago
  I think this uses instruct-pix2pix which is different! My understanding is that instruct-pix2pix requires a special fine tuned model, whereas this new pix2pix-zero thing is a new image to image technique that works without any need for additional training. All that to say here's a pix2pix-zero demo: https://huggingface.co/spaces/ysharma/pix2pix-zero-01 https://huggingface.co/spaces/ysharma/pix2pix-zero-01
  - TekMol 4y ago
    Trying to understand it ... It only lets me choose between "cat2dog" and dog2cat"? I thought I can add any prompt like "add glasses" or "change background to beach sunset"? What is the .pt file it creates? When I use the "Generate & Translate the SD image" button, I just get "Error" after a while of processing.
imranq 4y ago
I guess this involves training textual inversions which take quite a lot of computation vs other methods like instruct pix2pix which don't need to
- imranq 4y ago
  Actually this is wrong, after reading the paper: https://arxiv.org/pdf/2302.03027.pdf https://arxiv.org/pdf/2302.03027.pdf The main idea is to specify an edit direction which is a broad statement of what you want to do. The model then uses gpt-3 to generate text for that direction and then it calculates embeddings with CLIP and then the mean difference. The difference is used to guide the noise to make the final change. One pitfall seems like you can't specify very customized changes, but very cool approach
  - blurbleblurble 4y ago
    Is that difference the "inversion" mentioned?

Not my field so I don't have anything detailed to add, other than to say how fascinating it is to watch how quickly all these incremental improvements are occurring after the initial breakthrough. Also, props for not cherry picking results too much, both for honesty and because it is fun to see how these fail. Like the tree that grew limbs to fill in a watermark, or the sunset sky that turned into a lava field.

lloeki 4y ago
Interesting how the lava field version imprinted my mind. When going back to the original image I now see it as ambiguous - in the same way as a duck/rabbit multistable illusion - and can see a mountain range and sand instead of clouds.
- MarcoZavala 4y ago
  [dead]

TekMol 4y ago

The "code" link takes you to a GitHub repo. But I don't know how to set up the needed environment. I wish authors of such tools would provide a Dockerfile to set up the environment needed to run their creations. Their setup section starts with conda env create -f environment.yml But that gives me bash: conda: command not found So I tried apt install conda and apt install python3-conda But nothing worked. Why is running software still such a tricky thing in 2023, where you have to read tutorials and tinker with manually setting up an environment? Guess I will wait until some SaaS company provides an API to use this on their machines then. Any ideas, which company might do this first?

teruakohatu 4y ago
> Why is running software still such a tricky thing in 2023, where you have to read tutorials and tinker with manually setting up an environment? I think this is unfair criticism. Conda is extremely common in data science, as common as Docker is in software development. While I agree docker is easier to use as a consumer, Conda automatically sets up the environment and a quick Google would show you how to install conda.
- TekMol 4y ago
  I just said Dockerfile because when I say "shellscript", people think that is oldschool. A dockerfile shows me every step of what has to be done in an unmistakable way. I can easily turn it into a shellscript. The conda file (environment.yml, right?) does not tell me how to set up the environment. It contains a list of .. I don't know what. It starts with: channels: - pytorch - nvidia - defaults I don't know what to do with that. It does not look like computer code. > a quick Google would show you how to install conda No. It leads me to yet another tutorial with a bunch of manual steps: https://docs.conda.io/projects/conda/en/latest/user-guide/install/linux.html https://docs.conda.io/projects/conda/en/latest/user-guide/in... It even starts with decisions I shall make of which I know nothing: Download the installer: - Miniconda installer for Linux. - Anaconda installer for Linux. How do I know which one the software in question needs? How far down this rabbit hole is someone who just wants to run a software supposed to go?
  - extr 4y ago
    [flagged]
    - rowanG077 4y ago
      Conda is a special type of hell though. I have read the manual. Multiple tutorials. Yet I, not even once got conda to work for me.
  - IanCal 4y ago

amrrs 4y ago

Try this colab, if you want to try pix2pix video https://github.com/camenduru/pix2pix-video-colab https://github.com/camenduru/pix2pix-video-colab

TekMol 4y ago
Does something like this exist for the single image version? How hard is it to set up such a thing on HuggingFace?
blurbleblurble 4y ago
I think this uses instruct-pix2pix which is different! My understanding is that instruct-pix2pix requires a special fine tuned model, whereas this new pix2pix-zero thing is a new image to image technique that works without any need for additional training. All that to say here's a pix2pix-zero demo: https://huggingface.co/spaces/ysharma/pix2pix-zero-01 https://huggingface.co/spaces/ysharma/pix2pix-zero-01
- TekMol 4y ago
  Trying to understand it ... It only lets me choose between "cat2dog" and dog2cat"? I thought I can add any prompt like "add glasses" or "change background to beach sunset"? What is the .pt file it creates? When I use the "Generate & Translate the SD image" button, I just get "Error" after a while of processing.

imranq 4y ago

I guess this involves training textual inversions which take quite a lot of computation vs other methods like instruct pix2pix which don't need to

imranq 4y ago
Actually this is wrong, after reading the paper: https://arxiv.org/pdf/2302.03027.pdf https://arxiv.org/pdf/2302.03027.pdf The main idea is to specify an edit direction which is a broad statement of what you want to do. The model then uses gpt-3 to generate text for that direction and then it calculates embeddings with CLIP and then the mean difference. The difference is used to guide the noise to make the final change. One pitfall seems like you can't specify very customized changes, but very cool approach
- blurbleblurble 4y ago
  Is that difference the "inversion" mentioned?