6 ms·

A.I projects (and maybe all python projects in general) seem to always be ridiculously tedious, error-prone to get running, such that its a rare, celebratory t

• EamonnMR 4y ago
  Most Python projects aren't this tough. I suspect that they're using wonky libraries like Pandas, Numpy or some such that prioritize raw power over ease of installation.
  • CodeSgt 4y ago
    I've not touched Python in a couple years, but Pandas/NumPy used to be the defacto libs for anything to do with data science, are they considered "wonky" now?
    • machinekob 4y ago
      2be honest numpy is ez to install on all major platforms. In deep learning I'm almost never saw usage of pandas but deep learning models have problems with PyTorch as some projects just lock old PyTorch version that just dosent work on new/old python version.
    • gbear605 4y ago
      In my experience, they’re simultaneously important for data science and very annoying to install and manage.
      • Fomite 4y ago
        I mean, there are at least two different companies (Enthought and Continuum) that were founded on making the major scientific python packages easier to install.
    • version_five 4y ago
      Just to be clear, pandas and numpy are not the "wonky" libraries. They are, in my experience, basically two of the most easily installed and dependency managed libraries in python, given their ubiquity and maturity. Maybe there are machine configurations I'm not familiar with that they are not easily compatible, but I've never seen them cause issues. Usually it's cuda or other gpu stuff, or conflicts in less regularly maintained packages
• sp332 4y ago
  A bit, yeah. And it's been extra difficult to get it going on M1 Macs.
  • latchkey 4y ago
    lstein's fork [1] isn't that bad and the instructions are pretty easy to follow. It definitely requires some knowledge of how to install software via brew, but these are generally good to figure out anyway. [1] https://github.com/lstein/stable-diffusion/blob/main/docs/installation/INSTALL_MAC.md https://github.com/lstein/stable-diffusion/blob/main/docs/in...
    • dzink 4y ago
      If used to work but somehow was broken on M1Mac a few days ago. Not sure if fixed. Get an older branch if it doesn’t work for you.
      • latchkey 4y ago
        You might have had to rebuild the dependencies. I've been running the development branch, which has been working fine for me, but I've also rebuilt the dependencies a few times just to be certain. One protip is to use symlinks for the training files.
• somehnacct3757 4y ago
  These AI tools are going to be built into professional tools like Photoshop someday, but until then it's sort of a hackers paradise. I think Charl-e lets a new cohort of hackers play with the technology without becoming python command line warriors
  • robbomacrae 4y ago
    Christian Crantell has already made a Stable Diffusion plugin for photoshop[0]. Not 1st party mind, but given Photoshop's transforms already included I imagine they have people looking into it. [0]: https://christiancantrell.com/#ai-ml https://christiancantrell.com/#ai-ml
  • ies7 4y ago
    https://github.com/nousr/koi https://github.com/nousr/koi Stable diffusion already "happened" for Krita
• amelius 4y ago
  IT is in a package management crisis at the moment, and GPUs are not making things easier.
  • smlacy 4y ago
    What's the relationship between GPUs and package management?
    • hunkins 4y ago
      Most deep learning these days requires CUDA acceleration to enable GPU / TPU for the libraries (i.e. PyTorch, Tensorflow), which is an absolute nightmare to set-up.
      • thebruce87m 4y ago
        NVIDIA has a bunch of docker containers which make this slightly less painful. Or maybe it’s just a different kind of pain.
        • hunkins 4y ago
          These definitely help.. but still painful.
      • gpderetta 4y ago
        I wanted to try SD on my machine, but I just couldn't get CUDA to work. Mind, the the openvino based CPU implementation works just fine (pip install -r requirements.txt was sufficient) and given my CPU and GPU (a 10 core 10850k and and old 1070), there's isn't probably much to gain to switch to the GPU other than power usage.
      • 1024core 4y ago
        I guess the old "NVidia -vs- Linus" battle on kernel modules is still being fought....
• kragen 4y ago
  Cutting-edge software has always been like this. Web browsers were like this 30 years ago. Linux was like this 25 years ago. DNS and Unix were like this 40 years ago. AJAX and Comet (and really JS in general) was like this 20 years ago. Operating systems and high-level languages were like this 50 years ago. It takes a while for the rough edges to get sanded off. Most Python projects install with a matter of a single pip command.
  • userbinator 4y ago
    Back then, as in the early years ot the PC, software was mainly self-contained, single-executable extract-and-run. Somewhere along the way, they became so complex as to require special installation programs and the like. I'm not very familiar with AI and SD in particular, but from what I understand, this stuff is mostly-pure maths, so it shouldn't be a difficult thing to package and make portable. I know the models are rather large, but that's not really any additional complexity.
    • mgraczyk 4y ago
      The largeness of the model does add complexity, you can't really package it as part of the binary for example. It's not the case that useful software was ever self contained. If you recall trying to do anything online in the early to mid nineties, you'll remember how complicated it was to use almost any website and how much manual fiddling and configuration was involved to get online. Ignoring the internet, early games and graphical applications were a mess of settings and configuration. Even today you often have to tune graphics settings to get playable game performance on anything but top line hardware.
    • y42 4y ago
      >> Back then, as in the early years ot the PC, software was mainly self-contained, single-executable extract-and-run. Software in general? Yes. But of you tried to leave the path the manufacture prepared, then you entered a world of pain. I remember how difficult it was to connect my first smartphone around 20 years ago to my Windows PC using ActiveSync to achieve a synced calendar. Just one example of: there was no download and run solution for processes that seem simple today.
• mrtksn 4y ago
  Unfortunately this goes beyond A.I. stuff, it has become the state of software development and that's why people started shipping very large packages of everything so that you can have the exactly same environment so you can increase your chances that the code will run as expected. For A.I. stuff I actually don't judge, these scripts are written by people who specialise in other things than software engineering and they simply put together some code to run their algorithms and as a result they are poorly engineered in many aspects.
• deleted 4y ago
  [deleted]
• JimDabell 4y ago
  If you look at the code that is typically put out by non-software engineers (researchers, etc.), there seems to be a complete lack of software engineering knowledge (obviously). Sure, they can write code to do clever things, but put it into a package? What’s that? Run a linter? Never heard of it. Git? It’s that thing you use to save to the cloud, right? Documentation? Of course we published a paper! All respect due to them building something incredibly advanced, but you should view this as the place where the science has done its part and the software engineering is just getting started.
  • holoduke 4y ago
    I find the code of stable diffusion not bad at all. It's packaged nicely. It uses third-party libs where it should be. The algorithms are well laid out and reasonable easy to understand. This is not a project made by people without dev skills.
    • JimDabell 4y ago
      Really? I find the difference between the Stable Diffusion code and the code you find in a typical Python package to be night and day. Why is there precisely one test (that has nothing to do with the core functionality)? Why is the Git history full of things like “finish”, “correct merg”, “fix more”, “add code”? Where is the linting config? Why are there print statements everywhere? Why does non-UI code have UI code embedded in it? Why is there random code commented out? Why is there no consistency across the codebase? Why is everything written as if they’ve never seen a Python project before (comments that should be docstrings, docstrings that should be comments, print("WARNING: ") instead of using logging or warnings, underscores in CLI flags, no shebangs in scripts intended to be run from the command line…) Not all Python software gets all of this right, but it’s incredibly rare to have so many misses, even for hobbyist developers. Unless it’s code released by researchers, etc. It’s pretty typical in that context. > This is not a project made by people without dev skills. No, but it is a project made by people lacking software engineering skills, which is a distinction I drew in my earlier comment. Like I said, they can write clever code, but there’s a difference between bashing on code until it works and building it properly. This is the kind of codebase you get when you have people who have been writing code for a long time, but never in the context of a software engineering project alongside experienced software engineers they can learn from. Put them on a team like that and they’ll be forced to unlearn most of these bad habits fast because they’d never get their pull requests approved otherwise. I’m trying not to be harsh – I understand that this code is more of a code dump from researchers than a real software project, and they’ve done some incredibly clever things here – but if somebody is suggesting that Python projects in general are like this, it really should be pointed out that this is not in this slightest bit representative of a typical Python project.
• whywhywhywhy 4y ago
  Sorry but the Python ecosystem is an absolute joke, every single time I interact with it there is at least a 45 minute session of trying to get it into the correct state for all the things to co-exist and run. I can't believe people endure this stuff on a day to day basis, I dread it every time, the fact that different versions of packages can't co-exist and like installing something can downgrade my setuptools which then breaks my whole installation. Not even wrapping this all up within conda solves this stuff it just means you can burn the whole thing and start over easily. Maybe it's user error, but I never encounter these problems anywhere else.
  • yunwal 4y ago
    The issue is mostly 1) python appeals to a lot of people that work in development-adjacent industries (like AI). These people don’t usually have to care about packaging 2) Python has gone through many outdated forms of packaging 3) The zen of python seems to have encouraged everyone to install third party libraries for the smallest of tasks (implementing retries, formatting phone numbers, etc). These small packages often have only a few number of maintainers who end up dropping off the map. Modern python package management works pretty well, but there’s so much debt in the ecosystem I’m not sure when it’ll be better for end users.