6 ms·
I have reimplemented Stable Diffusion 3.5 from scratch in pure PyTorch
- squircle 1y agoAlthough I'm leaning heavily away from being passionate about software development, this is a cool project, and its freaken awesome how anyone can now reinvent the wheel from first principles.
- deleted 1y ago[deleted]
- albert_e 1y agoSounds like a great resources for learners. Just wondering aloud -- Is there a tutorial/explainer by any chance that a beginner could use to follow along and learn how this is done.
- an0malous 1y agofast.ai has a course on building Stable Diffusion: https://course.fast.ai/Lessons/part2.html https://course.fast.ai/Lessons/part2.html
- BinaryMachine 1y agoGreat resource Jeremy Howard is awesome, I have been waiting to take this course and follow along because anything older than a year in Deep Learning is already outdated. I hope they release a new version.
- whiplash451 1y agoI don’t think this is true. The fast.ai class covers a lot of fundamentals that are still valid and useful today.
- socalgal2 1y ago> If you ... are comfortable with building an SGD training loop from scratch in Python, being competitive in Kaggle competitions, using modern NLP and computer vision algorithms for practical problems, and working with PyTorch and fastai, then you will be ready to start the course. ...sigh... Yes, they tell you do their first course but I have no confidence that one course in this stuff will make me "comfortable with building an SGD training loop from scratch in Python, being competitive in Kaggle competitions, using modern NLP and computer vision algorithms for practical problems, and working with PyTorch and fastai"
- b0a04gl 1y ago[dead]
- reedlaw 1y agoI'm not sure what this means. If it means the Stable Diffusion 3.5 model, why is it fetching that here: https://github.com/yousef-rafat/miniDiffusion/blob/main/encoders/get_checkpoints.py#L29 https://github.com/yousef-rafat/miniDiffusion/blob/main/enco... The training dataset is very small, only including fashion-related pictures: https://github.com/yousef-rafat/miniDiffusion/tree/main/data/fashion_mnist/train https://github.com/yousef-rafat/miniDiffusion/tree/main/data...
- yousef_g 1y agoThe dataset is for trying out fine-tuning of the diffusion model. It's a reimplementation of SD3 by writing the code from scratch again, but the weights are taken from HuggingFace due to hardware constraints on my part.
- reedlaw 1y agoSo this implements SD3 inference and fine-tuning?
- jatins 1y ago> It's a reimplementation of SD3 by writing the code from scratch again, but the weights are taken from HuggingFace due to hardware constraints on my part. Could you clarify what you mean by this part -- if the weights are taken from HF then what's the implementation for?
- MoonGhost 1y agoMy guess the weights from HF are used as initial state for the model because full training is too expensive. Then small dataset is used to train in further for short time. Which is fine tuning. Together it shows that model is 1) compatible 2) trainable. In theory it can be trained from scratch on big dataset. I didn't look in the code yet so the questions are: 1) can it be trained in parallel? 2) resources required for training? Anyway, I may try to train it on limited specialized dataset...
- montebicyclelo 1y ago
- CamperBob2 1y agoAdd a Hugging Face Token in get_checkpoints.py before running the script. Can you be a bit more specific here? It's not clear what such a token is, what it takes to get one, or where it would be placed in get_checkpoints.py.
- einsteinx2 1y ago> what such a token is An API token from Hugging Face > what it takes to get one You generate them in your Hugging Face account > where it would be placed in get_checkpoints.py. Line 59 in the empty quotes where it says token = “”
- CamperBob2 1y agoAh, I see it now, thanks. That's the kind of thing that, stylistically speaking, it's good to define at the very top of the module.
- einsteinx2 1y agoAgreed. I’m not part of the project I just saw your comment and figured I’d try and help.
- Dwedit 1y agoLeaving off the "API" part from "API Token" causes confusion, since AI models tokenize all text into "tokens" before running the model. It's using the same word to describe two very different things.
- einsteinx2 1y agoYep totally. Fwiw I’m not part of the project I just saw the comment and figured I’d try and help.
- megamindbrian2 1y ago[dead]
- theturtle 1y agoCool. Can it still make images of Anne Hathaway leading a herd of blue giraffes on the Moon?
- IncreasePosts 1y agoSeems difficult, as there are no known portraits of Anne Hathaway
- liuliu 1y agoIf you are interested in this: Flux reference implementation is very minimalistic: https://github.com/black-forest-labs/flux/tree/main/src/flux https://github.com/black-forest-labs/flux/tree/main/src/flux The minRF project is very easy to start with training small diffusion models with rectified flow: https://github.com/cloneofsimo/minRF https://github.com/cloneofsimo/minRF Also, the reference implementation of SD 3.5 is actually minimalistic too: https://github.com/Stability-AI/sd3-ref https://github.com/Stability-AI/sd3-ref
- doctorpangloss 1y agoReference implementations are unmaintained and buggy. For example https://github.com/huggingface/transformers/issues/27961 https://github.com/huggingface/transformers/issues/27961 OpenAI's tokenizer for CLIP is buggy, it's a reference implementation, it isn't the one they used for training, and the problems with it go unsolved and get copied endlessly by other projects. What about Flux? They don't say it was used for training, it wasn't, there are bugs with it that break cudagraphs or similar that aren't that impactful. On the other hand, it uses CLIP reference, and CLIP reference is buggy, so this is buggy...
- BoredPositron 1y agoYou can disable clip l on flux without a loss in quality. You are also making an elephant out of a fly. CLIP is used everywhere.
- doctorpangloss 1y agoConsider another interpretation: CLIP L in Flux can be disabled without a loss in quality because the way it is used is buggy!
- BoredPositron 1y agooh lord.
- vergessenmir 1y agoIs there any notable properties of this implementation, are some parts slower, faster etc
- NoelJacob 1y agoSo, that's Stable Diffusion without license constraints, is it?
- Sharlin 1y agoNo, the inference/training algorithms, being math, are not copyrightable. OP just wrote another implementation. What's copyrighted are the models, which OP did not train from scratch (having neither the training material nor the compute to do that).
- echelon 1y agoWe should be specific when we say "models". The code outlining the network vs. the resultant weights. (Also vs. any training, inference, fine tuning, misc support code, etc.) The theoretical diagram of how the code networks and modules are connected is math. But an implementation of that in code is copyrightable. Afaik, the weights are still a grey area. Whereas code is code and is copyrightable. Weights are not produced by humans. They are the result of an automated process and are not afforded copyright protection. But this hasn't been tested in court. If OpenAI GPT 4o weights leak, I think the whole world could use it for free. You'd just have to write the code to run them yourself.
- bravesoul2 1y agoI use model architecture for the code/math and weights for the weights to avoid confusion! Then there are hyperparameters which are also needed to be known to use the weights with the model architecture.
- MoonGhost 1y ago> I use model architecture for the code/math Code is copyrightable and math is not. What about 'architecture'?
- vrighter 1y agowhich means he is still in full violation of their license
- caycep 1y agoHow usable is the original academic source available from Ludwig Maximilian University CompViz group?
- eapriv 1y agoI find it hilarious that “from scratch” now somehow means “in PyTorch”.
- monsieurbanana 1y agoIf any "from scratch" post doesn't start with linking to a Primitive Technology video, I'm closing the tab
- mkoubaa 1y agoUnless the author was raised by chimps I'm out
- 0cf8612b2e1e 1y agoNot fusing heavier elements from hydrogen? I’m out.
- chairmansteve 1y agoYeah. Should have done it in assembly.
- mardifoufs 1y agoPytorch is a pretty basic building block when you get to some degree of model complexity. It wouldn't really be interesting to implement autograd or some other things pytorch provides imo when the goal is to show a reimplantation of something as "higher" level as SD. It's similar to how I don't mind it when someone doesn't reimplement an OS, or a JavaScript engine when writing a web app from scratch. And there's been a recent surge in abstractions over pytorch, and even standalone packages for models that you are just expected to import and use as an API (which are very useful, don't get me wrong!). So it's nice to see an implementation that doesn't have 10 different dependencies that each abstract over something pytorch does.
- eapriv 1y ago> It wouldn't really be interesting Andrej Karpathy did exactly that, and I think it’s quite interesting.
- refulgentis 1y agoI'm embarrassed to ask: can someone elaborate on, say, what we have now that we didn't have before the repo existed? I have studiously avoided making models, though I've been adjacent to their output for years now... I think the root of my confusion is I kinda assumed there was already PyTorch based scripts for inference / training. (I assumed _at least_ inference scripts were released with models, and kinda figured fine-tuning / training ones were too) So then I'm not sure if I'm just looking at a clean room / dirty room rewrite of those. Or maybe everyone is using "PyTorch" but it's usually calling into CUDA/C/some proprietary thingy that is much harder to grok than a pure PyTorch impl? Anyways, these arent great guesses, so I'll stop myself here. :)
- _tqr3 1y agoStability AI, creators of Stable Diffusion models release their products under own Stability AI Community License which is not "free" like MIT license. You are not allowed to modify the weights in certain ways. This package is basically running the model (inference) and maybe fine tuning it using existing AI weights. A great way to learn but still could run into same licensing issue.
- refulgentis 1y agoYou can't finetune SD 3.5!? I thought the community license stuff was about keeping people from using it in prod and charging for it without Stability getting at least a small taste. This sucks. I haven't been keeping up with gooner squad on Civit, but I did have some understanding SD was less popular, but I thought it was just because 3.5 came far too long after Flux with too little, if any, quality increase to be worth building new scaffolding for.
- fc417fc802 1y ago> You can't finetune SD 3.5!? They don't want you finetuning it in specific ways that might make them look bad by association.
- djhn 1y ago
- hkon 1y agonow do it in minecraft
- ineedasername 1y agoWhen I think of SD 3.5 (or any version) I think of the portion that results from training, i.e., the weights. The code seems less important? I mean as far as output quality is concerned, or performance. But I'm honestly not sure, and not trying to judge these efforts on that basis.
- Dwedit 1y agoDoes using pure PyTorch improve performance on non-NVIDIA cards in any way? Or is PyTorch so highly optimized for CUDA that no other GPU vendors have a chance?
- 3abiton 1y agoIt seems to be the case, although pytorch rocm is coming around slowly. Very slowly, if you get it working that is.
- VeejayRampay 1y agoI believe pytorch works nicely with rocm, but I don't know if it's nicely to the point where it's "on par"
- chickenzzzzu 1y agoIt is possible to run ML workloads on for example AMD devices via Vulkan. With newer extensions like cooperative matrix, and maybe also in the future some scheduling magic exposed by the driver through a new extension, the remaining single digit percent gap CUDA has will evaporate.
- jwitthuhn 1y agoPytorch also runs great on apple silicon, though it is hard to directly compare because Apple's high end GPUs can't compute anywhere near as much as nvidia's high end stuff. e: I'll also add that pytorch does still have one oddity on apple silicon which is that it considers each tensor to be 'owned' by a particular device, either a cpu or gpu. Macs have unified memory but pytorch will still do a full copy when you 'move' data between the cpu and gpu because it just wasn't built for unified memory.
- brcmthrowaway 1y agoDoes pytorch work on AS out of the box? Or do you need some apple specific package
- thom 1y ago
- SV_BubbleTime 1y ago[flagged]
- godelski 1y agoself.q = nn.Linear(embed_size, embed_size, bias = False) self.k = nn.Linear(embed_size, embed_size, bias = False) self.v = nn.Linear(embed_size, embed_size, bias = False) Try self.qkv = nn.Linear(embed_size, 3*embed_size, bias = False) def forward(...): ... qkv = self.qkv(x)
- jszymborski 1y agoThis adds connections between the parameters of q, k, and v whereas the original doesn't, unless my very tired brain is missing something.
- smus 1y agoNope, they all depend on x and the same is true in this scenario
- godelski 1y agoIt is actually really common practice. It is a single linear layer so there's no connection intranodes. The reason to do this is because it is a bit less computationally intensive. tldr: linear layers have an associative property
- nothrowaways 1y agoPure pytorch?