10 ms·
pip install pytorch Environment broken Spend 4 hours fixing python environment pip install Pillow Something something incorrect cpu architecture for your Ma
by davedx 2y ago
pip install pytorch
Environment broken
Spend 4 hours fixing python environment
pip install Pillow
Something something incorrect cpu architecture for your Macbook
Spend another 4 hours reinstalling everything from scratch after nuking every single mention of python
pip install … oh time to go home!
- makapuf 2y agoMaybe pip should not work by default (but python -m venv then pip install should)
- avmich 2y agoLegends say there were times when you'd have a program.c file and just run cc program.c, and then could just execute the compiled result. Funny that programmer's job is highly automatable, yet we invent ourselves tons of intermediate layers which we absolutely have to deal with manually.
- EnergyAmy 2y agoAnd then you'd have to deal with wrong glibc versions or mysterious segfaults or undefined behavior or the the code assuming the wrong arch or ...
- KeplerBoy 2y agopython solves none of those issues. It just adds a myriad of ways those problems can get to you. All of a sudden you have people with C problems, who have no idea they're even using compiled dependencies.
- EnergyAmy 2y agoIn theory you're right, CPython is written in C and it could segfault or display undefined behavior. In practice, you're quite wrong. It's not really much of a counterargument to say that Python is good enough that you don't have to care what's under the hood, except when it breaks because C sucks so badly.
- KeplerBoy 2y agoI was specifically talking about python packages using C. You type "pip install" and god knows what's going to happen. It might pull a precompiled wheel, it might just compile and link some C or Fortran code, it might need external dependecies. It might install flawlessly and crash as soon as you try to run it. All bets are off. I never experienced CPython itself segfault, it's always due to some package.
- davedx 2y agoI actually did a small C project a couple of years ago, the spartan simplicity there can have its own pain too, like having to maintain a Makefile. LOL. It’s swings and roundabouts!
- makapuf 2y agoI agree simplicity is king. But you're comparing making a script using dependencies and tooling for those dependencies and a C program with no dependencies. You can download a simple python script and run it directly if it has no dependencies besides stdlib (which is way larger in python). That's why I love using bottle.py by example.
- avmich 2y agoAgree. But even with dependencies running "make" seems to be way simpler than having to install particular version of tools for a project, making venv and then picking versions of dependencies. The point is the same - we had it simpler and now, with all capabilities for automation, we have it more complex. Frankly, I suspect most of the efforts now are spent fighting non-essential complexities, like compatibilities, instead of solving the problem at hand. That means we create problems for ourselves faster than removing them.
- reportgunner 2y agoYou can do the same in python if you're not new to programming.
- jononor 2y agoSome Linux distros are moving that way, particularly for the included Python/pip version. My Arch Linux already does so some years, and I do not set it up myself - so I think it is default.
- sigmoid10 2y agoIf you're still doing ML locally in 2024 and also use an ARM macbook, you're asking for trouble.
- nicce 2y ago> ARM macbook Funnily, the only real competitor for Nvidias' GPUs are Macbooks with 128GB of RAM.
- spmurrayzzz 2y agoCan you expand on this a bit? My recent experiences with MLX have been really positive, so I'm curious what footguns you're alluding to here. (I don't do most of my work locally, but for smaller models its pretty convenient to work on my mbp).
- sigmoid10 2y agoMPS implementations generally lag behind CUDA kernels, especially for new and cutting edge stuff. Sure, if you're only running CPU inference or only want to use the GPU for simple or well established models, then things have gotten to the point where you can almost get the plug and play experience on Apple silicon. But if you're doing research level stuff and training your own models, the hassle is just not worth it once you see how convenient ML has become in the cloud. Especially since you don't really want to store large training datasets locally anyways.
- genevra 2y agoFor real
- deleted 2y ago[deleted]
- next_xibalba 2y agoDo people doing ML/DS not use conda anymore?
- buildbot 2y agoA lot do, personally, every single time I try to go back to conda/mamba whatever, I get some extremely weird C/C++ related linking bug - just recently, I ran into an issue where the environment was _almost_ completely isolated from the OS distro's C/C++ build infra, except for LD, which was apparently so old it was missing the vpdpbusd instruction (https://github.com/google/XNNPACK/issues/6389 https://github.com/google/XNNPACK/issues/6389). Except the thing was, that wouldn't happen when building outside of of the Conda environment. Very confusing. Standard virtualenvs are boring but nearly always work as expected in comparison. I'm an Applied Scientist vs. ML Engineer, if that matters.
- astromaniak 2y agoIt's probably easier to reinstall everything anew from time to time. Instead of fixing broken 18.04 just move to 22.04. Most tools should work, if you don't have huge codebase which requires old compiler... Conda.. it interfere with OS setup and has not always the best utils. Like ffmpeg is compiled with limited options, probably due to licensing.
- buildbot 2y agoI do all the time, and always have (in fact my first job was bare metal OS install automation), this was Rocky 9.4. New codebase, new compiler weird errors. I did actually reinstall and switch over to Ubuntu 24.04 after that issue lol.
- copperroof 2y agoIf they are they should stop. It causes so many entirely unnecessary issues. The conda developers are directly responsible for maybe a month of my wasted debugging time. At my last job one of our questions for helping debug client library issues was “are you using conda”. And if so we just would say we can’t help you. Luckily it was rare, but if conda was involved it was 100% conda fault somehow, and it was always a stupid decision they made that flew in the face of the rest of the python packaging community. Data scientist python issues are often caused by them not taking the 1-3 days it takes to fully understand their tool chain. It’s genuinely quite difficult to fuck up if you take the time once to learn how it all works, where your putbon binaries are on your system etc. Maybe not the case 5 years ago. But today it’s pretty simple.
- SushiHippie 2y agoCan recommend using conda, more specifically mambaforge/micromamba (no licensing issues when used at work). This works way better than pip, as it does more checks/dependency checking, so it does not break as easily as pip, though this makes it definitely way slower when installing something. It also supports updating your environment to the newest versions of all packages. It's no silver bullet and mixing it with pip leads to even more breakages, but there is pixi [0] which aims to support interop between pypi and conda packages [0] https://prefix.dev/ https://prefix.dev/
- tasuki 2y agoI had a bad experience with Conda: - If they're so good at dependency management, why is Conda installed through a magical shell script? - It's slow as molasses. - Choosing between Anaconda/Miniconda... When forced to use Python, I prefer Poetry, or just pip with freezing the dependencies. The Python people probably can't even imagine how great dependency management is in all the other languages...
- tedivm 2y agoI absolutely hate conda. I had to support a bunch of researchers who all used it and it was a nightmare.
- akkad33 2y agoMamba/micromamba solves the slowness problem of conda
- epoxia 2y agoTo add. Conda has parallelized downloads and is faster. Not as fast as mamba, but faster than previously. pr merged sep 2022 -> https://github.com/conda/conda/pull/11841 https://github.com/conda/conda/pull/11841
- SushiHippie 2y agoYeah, I agree, maybe I should have also mentioned the bad things about it, but after trying many different tools that's the one that I stuck with, as creating/destroying environments is a breeze once you got it working and the only time my environment broke was when I used pip in that environment. > The Python people probably can't even imagine how great dependency management is in all the other languages... Yep, I wish I could use another language at work. > Choosing between Anaconda/Miniconda... I went straight with mamba/micromamba as anaconda isn't open source.
- mysteria 2y agoI thought most ML engineers use their laptops as dumb terminals and just remote into a Linux GPU server.
- daemonologist 2y agoYeah, the workday there looks pretty similar though, except that installing pytorch and pillow is usually no problem. Today it was flash-attn I spent the afternoon on.
- ungamedplayer 2y agoIsn't this what containers are for. Someone somewhere gets it configured right and then you download and run pre setup container and add your job data ? Or am I looking at the problem wrong?
- rolisz 2y agoBut then how do you test out the latest model that came out from who knows where and has the weirdest dependencies and a super obscure command to install?
- fshbbdssbbgdd 2y agoJust email all your data to the author and ask them to run it for you.
- davedx 2y agoSpoiler: my main role isn’t ML engineer :) and that doesn’t sound like a bad idea at all
- jshbmllr 2y agoI do this... but air-gapped :(
- daemonologist 2y agoOof. At our company only CI/CD agents (and laptops) are allowed to access the internet, and that's bad enough.
- fragmede 2y agothat sounds very painful.
- rqtwteye 2y agoOh my! This hits home. We have some test scripts written in python. Every time I try to run them after a few months I spend a day fixing the environment, package dependencies and other random stuff. Python is pretty nice once it works, but managing the environment can be a pain.
- phaedrus 2y agoAs an amateur game engine developer, I morosely reflect my hobby seems to actually consist of endlessly chasing things that were broken by environment updates (OS, libraries, compiler, etc.) That is, most of the time I sit down to code I actually spend nuking and reinstalling things that (I thought) were previously working. Your comment makes me feel a little better that this is not merely some personal failing of focus, but happens in a professional setting too.
- ClimaxGravely 2y agoHappens in AAA too but we tend to have teams that shield everyone from that before they get to work. I ran a team like that for a couple years. For hobby stuff at home though I don't tend to hit those types of issues because my projects are pretty frozen dependency-wise. Do you really have OS updates break stuff for you often? I'm not sure I recall that happening on a home project in quite a while.
- davedx 2y agoOh god yes I remember trying to support old Android games several OS releases later… Impossible, I gave up! It’s why I still use react, their backcompat is amazing
- SJC_Hacker 2y agoDocker is your friend or pyenv at least
- navbaker 2y agoYes, I’ve switched from conda to a combination of dev containers and pyenv/pyenv-virtualenv on both my Linux and MacBook machines and couldn’t be happier
- __rito__ 2y agoJust use conda.
- SoftTalker 2y agoThen you get one of my favorites: NVIDIA-<something> has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.
- davedx 2y agoIirc I originally used conda because I couldn’t get faiss to work in venv, lol. That was a while ago though
- wszrfcbhujnikm 2y agoDocker fixes this!
- el_benhameen 2y ago> Environment broken >Something something incorrect cpu architecture for your Macbook I’m glad I have something in common with the smart people around here.
- davedx 2y agoYeah getting a good python environment setup is a very humbling experience
- mc10 2y agouv is a great drop-in replacement for pip: https://astral.sh/blog/uv https://astral.sh/blog/uv
- vergessenmir 2y agoWhy do ML, especially nowadays on a Mac, when you can do it on an Ubuntu based machine. Surely work can provide that?
- zxexz 2y agoWhy Ubuntu specifically? Not even being snarky. Calling out a specific distro, vs. the operating system itself. I've had more pain setting up ML environments with Ubuntu than a MacBook, personally - though pure Debian has been the easiest to get stable from scratch. Ubuntu usually screws me over one way or another after a month or so. I think I've spend a cumulative month of my life tracking down things related to changes inn netplan, cloud-init, etc. Not to mention Ubuntu Pro spam being incessant, as official policy of Canonical [0]. I first used the distro all the way back at Warty Warthog, and it was my daily driver from Feisty until ~Xenial. I think it was the Silicon Valley ad in the MotD that was the last straw for me. [0] https://bugs.launchpad.net/ubuntu/+source/ubuntu-meta/+bug/1930914 https://bugs.launchpad.net/ubuntu/+source/ubuntu-meta/+bug/1...
- 0x008 2y agoI can recommend to try poetry. It is a lot more succesful in resolving dependencies than pip. Although I think the UX of poetry is stupid and I do not agree with some design decisions, I have not had any dependency conflicts since I used it.
- globular-toast 2y agoYou could learn how to use Python. Just spend one of those 4 hours actually learning. Imagine just getting into a car and pressing controls until something happened. This wouldn't be allowed to happen in any other industry.
- sirlunchalot 2y agoCould you be a bit more specific about what you mean by "You could learn how to use python"? What resources would you recommend to learn how to work around problems the OP has? What basic procedures/resources can you recommend to "learn python"? I work as a software developer alongside my studies and often face the same problems as OP that I would like to avoid. Very grateful for any tips!
- globular-toast 2y agoBasically just use virtual environments via the venv module. The only thing you really need to know is that Python doesn't support having multiple versions of a package installed in the same environment. That means you need to get very familiar with creating (and destroying) environments. You don't need to know any of this if you just use tools that happen to be written in Python. But if you plan to write Python code then you do. It should be in Python books really, but they tend to skip over the boring stuff.
- d0mine 2y agoHow do you learn anything in the space of software engineering? In general, there are many different problems and even more solutions with different tradeoffs. To avoid spending hours on fixing broken environments after a single "pip install", I would make it easy to rollback to a known state. For example, recreate virtualenv from a lock requirements file stored in git: `pipenv sync` (or corresponding command using your preferred tool).
- mountainriver 2y agoOh wow, I’ve been a Python engineer for over a decade and getting dependencies right for machine learning has very little to do with Python and everything to do with c++/cuda
- loftyal 2y ago...and people criticise node's node_modules. At least you don't spend hours doing this
- RamblingCTO 2y agoBut you do because your local node_modules and upstream are out of sync and CI is broken. Happens at least once a month just before a release of course. I'd rather have my code failing locally than trying to debug what's out of sync on upstream.
- mountainriver 2y agoNot nearly as hard of a problem. Python does work just fine when it’s pure Python. The trouble comes with all the C/Cuda dependencies in machine learning
- posix_monad 2y agoPython's dominance is holding us back. We need a stack with a more principled approach to environments and native dependencies.
- llm_trw 2y agoHere's what getting PyTorch built reproducibly looks like: https://hpc.guix.info/blog/2021/09/whats-in-a-package/ https://hpc.guix.info/blog/2021/09/whats-in-a-package/ Since then the whole python ecosystem has gotten worse. We are building towers on quicksand. It's not about python, it's about people who don't care about dependencies.
- hkt 2y agoDependency management is just.. hard. It is one of the things where everything relies upon it but nobody thinks "hey, this is my responsibility to improve" so it is left to people who have the most motivation, academic posts, or grant funding. This is roughly the same problem that led to heartbleed for OpenSSL.
- manusachi 2y agoDo you know what other ecosystem comes closest to the existing in Python? I've heard good things about Julia. 13 years ago when I was trying to explore the field R seemed to be the most popular, but looks like not anymore. (I didn't get into the field, and do just a regular SWE, so I'm not aware of the trends). There is also a lot of development in Elixir ecosystem around the subject [1]. [1](https://dashbit.co/blog/elixir-ml-s1-2024-mlir-arrow-instructor https://dashbit.co/blog/elixir-ml-s1-2024-mlir-arrow-instruc...)
- posix_monad 2y agoI don't think the ML community has an appetite for learning a different language. There are Microsoft-backed F# bindings for Spark and Torch, but no one seems interested. And this is despite a pretty compelling proposition of lightweight syntax, strong typing, script-ability and great performance. The answer will probably be JavaScript. Everyone already knows the language - all that's missing is operator overloading and a few key bindings.
- ninkendo 2y agoI count at least a half dozen “just use X” replies to this comment, for at least a half dozen values of X, where X is some wrapper on top of pip or a replacement for pip or some virtual environment or some alternative to a virtual environment etc etc etc. Why is python dependency management so cancerously bad? Why are there so many “solutions” to this problem that seem to be out of date as soon as they exist? Are python engineers just bad, or? (Background: I never used python except for one time when I took a coursera ML course and was immediately assaulted with conda/miniconda/venv/pip/etc etc and immediately came away with a terrible impression of the ecosystem.)
- sjducb 2y agoTwo problems intersect: - You can’t have two versions of the same package in the namespace at the same time. - The Python ecosystem is very bad at backwards compatibility This means that you might require one package that requires foo below version 1.2 and another package that requires foo version 2 and above. There is no good solution to the above problem. This problem is amplified when lots of the packages were written by academics 10 years ago and are no longer maintained. The bad solutions are: 1) Have 2 venvs - not always possible and if you keep making venvs you’ll have loads of them. 2) Rewrite your code to only use one library 3) Update one of the libraries 4) Don’t care about the mismatch and cross your fingers that the old one will work with the newer library. Most of the tooling follows approach 1 or 4
- fragmede 2y agoDisk space is cheap, so where it's possible to have 2 (or more) venvs, that seems easiest. The problem with venv is that they don't automatically activate. I've been using a very simple wrapper around python to automatically activate venvs so I can just cd into the directory and do python foo.py and have it use the local venv. I threw it online at https://github.com/fragmede/python-wool/ https://github.com/fragmede/python-wool/
- sjducb 2y agoYou’re already managing a few hundred dependencies and their versions. Each venv roughly doubles the number of dependencies and they all have slightly different versions. Now your 15 venvs deep, and have over 3000 different package version combinations installed. Your job is to upgrade them right now because of a level 8 CVE
- llm_trw 2y agoJust use Linux. Then you only have to fight the nvidia drivers.
- htrp 2y agoUse standard cloud images > Something something incorrect cpu architecture for your Macbook
- KingMachiavelli 2y agoReally can't recommend Nix for Python stuff more, it's by far the best at managing dependencies that use FFI/native extensions. It can be a pain sometimes to port your existing Poetry/etc. project to work with nix2lang converters but really pays off.
- deleted 2y ago[deleted]
- drumttocs8 2y agoSo what I do in my free time, but getting paid for it?? Sign me up.