4 ms·
My experience working in a deep learning outfit for a few years has been that the software stack is incredibly immature and fragile. You're generally tied to p
by snek_case 5y ago
My experience working in a deep learning outfit for a few years has been that the software stack is incredibly immature and fragile.
You're generally tied to proprietary nvidia drivers that are a pain to install, and when they change version, it breaks software such as PyTorch, you're forced to update. PyTorch itself often has breaking changes. Then, most of the code is written in Python, often by academics who know and care very little about good software engineering practices. That also leads to fragile and breakable code. You'll often find yourself trying to install code that's just DOA because of multiple broken dependencies.
The quality of the code can be terrible, because academics are in a constant rush to publish the next thing, and tend to view code as throwaway. Worse yet, it's often written by students, and these students are taught by professors who view software engineering as an inferior, unscientific discipline. I've seen many github repos with no open source license, and no README, just a random code dump with no comments.
I'm personally not super enthusiastic by the idea of replacing a lot of code by machine learning just yet, because not only is the software ecosystem fragile, but the models can be as well. You're likely to introduce weird, unpredictable modes of failure. I can see that there's clearly a lot of people who dream of being able to basically create software without coding, and not have to deal with pesky software developers, but IMO, we're not there yet.
This guy talks about cloud-based machine learning solutions. You won't have to deal with drivers and general "IT" concerns there. The problem is that, besides being very expensive, these cloud-based solution will introduce extra layers of vendor lock-in. If you're training on the google cloud and using TPUs that you can't even physically buy/own, you might have a hard time porting your code to run without a TPU. If Google revokes your access for some reason, because they claim you've violated their TOS, you are fucked.
- jakear 5y agoThe trick is to get a disk that works; wire it up to boot up, run your job, then shut down, on whatever schedule you need; then never touch it for anything besides business logic changes. Brownie points are awarded for persisting the setup of the image in the form of a dockerfile, shell script, etc., but such documents are to be considered non-authoritative sources, and must be expected to break at a rate roughly proportional to their size.
- cjv 5y agoCan you name some breaking changes PyTorch made?
- snek_case 5y agoNot off-hand because it's been a while, but IIRC the names of some methods were changed, arguments were added/removed. I will say that was before version 1.0, so it may be more stable now. Last I saw the nvidia driver situation did not improve though. I wanted to come back to some old code, installed a new version of PyTorch. It complained that my GPU drivers were not recent enough, had to do this long tedious process that took over an hour to get newer drivers working.
- deleted 5y ago[deleted]
- zby 5y agoI have tried using Google Cloud - I spent a lot of time finding a location and getting the permissions to use GPUs. Then one time after I switched off the vm to economize on costs I could not switch it on again because of lack of resources. I don't know if I could migrate it somewhere else - but most probably not, because the GPUs setups are pretty unique. After a few weeks I gave up. I bought a laptop with nvidia. I switched off everything that I thought was to switch off. Then for 4 months I was constantly being reminded about something more to switch off. Now I still have a static IP address and I don't know how to switch it off because the VM it was attached to is deleted. By the way as a seasoned (and actually kind of retired) programmer I started learning ML by myself after deciding that I don't like the code I was getting from hired companies. It is a strange land with many software writing practicies surprising someone coming from different programming culture. I wish I could find a community of people who made a similar path - here is my blog: https://zby.github.io/ https://zby.github.io/.
- qcoh 5y agoTo add to that my current struggle: Most stuff is geared towards conda and terrible engineering practices (requirements.txt and not even locked versions). If you want to actually build and deploy something on actual hardware, you are in a world of hurt. And even for training the dependency situation is so ridiculous that most repos come with their own Dockerfile. I marvel at the cool machine learning demos but I'm kind of sick working on this stuff, tbh.
- snek_case 5y agoAlso, nvidia-docker...