6 ms·
Friends don’t let friends train small diffusion models
- Scene_Cast2 4y agoIn my experience tinkering with plain vanilla neural networks, training quality is quite sensitive to the width of the first couple of base layers. In particular, using a rhombus / "diamond" shape works best. Here's my intuition. The output of the first layer is mostly a linear matrix transform. These linearities form the basis for the more complex features. If the network doesn't have enough of these base linear features, it can't compose the higher-level (and lower-dimensional) features later down the line.
- duped 4y agoYou might want to just make the first layer a DCT of the input vector which will split them into decorrelated components.
- wardedVibe 4y agoI see we are re-deriving conv nets
- duped 4y agoNot really, the first principle of not-too-bad ANNs is to use some kind of preprocessed, hopefully decorrellated feature vectors as the inputs. Conceptually that is way easier to implement with intuition and know-how over blindly training an arbitrary structure to do what you want. Ideally the training just derives non-intuitive relationships between these decorrelated input features to give you something useful.
- ausbah 4y agoby rhombus or diamond shaped do you mean the first layer being wider than the input layer - with each following hidden layer being smaller than the previous hidden layer?
- Scene_Cast2 4y agoBy rhombus, I meant something like input -> 20 -> 50 -> 200 -> 50 -> 20 -> output.
- raihansaputra 4y agoI'm still learning the basics of ML, but I assume the advantage of rhombus instead of just having 200*5 layers is training time? Would it have other advantages?
- draw_down 4y ago
- i_like_apis 4y agoHang on, is this thing actually generating halfway intelligible rap? Utterly amazing.
- svantana 4y agoAs I read it, this is just a spectrogram-->pcm converter (in speech known as a vocoder), which you can do decently without a model, just pure optimization. The hard part is to generate the spectrogram, which I assume they will tackle next.
- woodson 4y agoWell, there’s Griffin-Lim, but there is a reason most recent work uses a trained neural vocoder.
- jjoonathan 4y agoIs it halfway intelligible rap or fully intelligible ghost rap?
- moralestapia 4y agoI also have this question and it's a bit more complicated for me as I'm not a native speaker + the accent is heavily skewed because of the nusic style. So, are there some real words in there? Do the lyrics make sense? Or is it just "random" vocalizations that go along with the song?
- gwern 4y agoNo, he's still just experimenting with the audio representation. However, you certainly can generate halfway intelligible rap, and much more, if you check out https://openai.com/blog/jukebox/ https://openai.com/blog/jukebox/ (with GPT-3 lyrics) or consider other kinds of audio synthesis like https://speechbot.github.io/dgslm/ https://speechbot.github.io/dgslm/ (fully end to end).
- durnygbur 4y agolol, thisrapdoesntexist.com
- lostmsu 4y agoWhat's the size of the final model? What was the dataset? Training source code?
- WhyOhWhyQ 4y agoCan it be proved that any work generated with one of these models (i.e., music generation, visual art generation, writing generation) isn't blatantly stealing from something in the dataset? If the dataset is large enough, how can you know that you're not going to be sued at a later date for publishing work generated with one of these models?
- astrange 4y agoYou can't, which is why using general web scrapes are problematic. You're fine if you track the copyright status of your training set.
- leereeves 4y agoThe models do sometimes generate verbatim copies of their training data. Here's an example, Copilot regurgitating Quake code, including sweary comments: https://news.ycombinator.com/item?id=27710287 https://news.ycombinator.com/item?id=27710287
- smegsicle 4y agoof course you can't, see: the episode of seinfeld where elaine accidentally rips off a ziggy
- bitforger 4y agoWow! Super cool results. This is just begging to be text-guided with the lyrics. Then you could put in the lyrics and have the model generate a clip it thinks would go with the lyrics. :)
- wardedVibe 4y agoAny chance you'll post the code? Neat project
- forgingahead 4y agoThis blog is by the author of Tortoise-TTS (stylised as TorToiSe): https://github.com/neonbjb/tortoise-tts https://github.com/neonbjb/tortoise-tts Very cool work being done in ML these days, both inside large corps, and also by independent researchers/"hobbyists" (quotation marks because there is some really expert work being produced by them). And from yesterday, an open-source version of Google's Imagen by lucidrains: https://github.com/lucidrains/imagen-pytorch https://github.com/lucidrains/imagen-pytorch
- ausbah 4y ago>I got to thinking, though: what if there was some minimum bound to the number of channels for the base layer of a diffusion network? >To test out this theory, I needed a way to compress the output of the diffusion model. One idea that came to mind is using the concepts behind PixelShuffle. So I compressed the waveform by 16x using a 1D form of PixelShuffle, increased the base dimensionality to 256, and started another training run. I'm not familiar with the author's previous projects or the terminology here. Do they just mean he increased the size of the first hidden layer and (base dimensionality to 256) and compressed the output layer by 16?