3 ms·
Considering Stable Diffusion generates 3-channel (RGB) images, maybe it would be possible to train it on amplitude and phase data as two different channels?
by xnzakg 4y ago
Considering Stable Diffusion generates 3-channel (RGB) images, maybe it would be possible to train it on amplitude and phase data as two different channels?
- antognini 4y agoPeople have tried that, but the model essentially learns to discard the phase channel because it is too hard for it to learn any useful information from it.
- deleted 4y ago[deleted]
- techdragon 4y agoGot any citations... that sounds like a fascinating thing to read about.
- haykmartiros 4y agoWe took a look at encoding phase, but it is very chaotic and looks like Gaussian noise. The lack of spatial patterns is very hard for the model to generate. I think there are tons of promising avenues to improve quality though.
- dspig 4y agoPhase itself looks random, but what makes the sound blurry is that the phase doesn't line up like it should across frequencies at transients. Maybe something the model could grab hold of better is phase discontinuity (deviation from the expected phase based on the previous slices) or relative phase between peaks, encoded as colour? But the same thing could be done as a post-processing step, finding points where the spectrum is changing fast and resetting the phases to make a sharper transient.
- haykmartiros 4y agoThat makes a lot of sense, I would be keen to see attempts at that.