This article doesn't tell me anything about using Docker for data science, just about using Docker in general. I have tried to use Docker for reproducible rese

This article doesn't tell me anything about using Docker for data science, just about using Docker in general. I have tried to use Docker for reproducible research, as a way of distributing ConceptNet with all its dependencies [1]. I am about to give up and stop supporting the Docker version. [1] https://github.com/commonsense/conceptnet5/wiki/Running-your-own-copy#running-conceptnet-using-docker https://github.com/commonsense/conceptnet5/wiki/Running-your... This was the process I needed to explain to researchers who wanted to use my stuff before Docker: 1. Get the right version of Python (which they can install, for example, from the official package repo of any supported version of Ubuntu). 2. Set up a Python virtualenv. 3. Install Git and PostgreSQL (also from standard packages). 4. Make sure they have enough disk space (but they'll at least get obvious error messages about it if they don't). 5. Configure PostgreSQL so that your local user can write to a database (this is the hardest step to describe, because PostgreSQL's auth configuration is bizarre). 6. Clone the repo. 7. Install the code into your virtualenv. 8. Run a script that downloads the data. 9. Run a script that builds the data and puts it in the database. (This takes up to five hours, because of PostgreSQL, but you get sorta-sensible output to watch while it's going on.) 10. Run the API on some arbitrary port. 11. Optionally, set up Nginx and uwsgi to run the API efficiently (but really nobody but me is going to do this part). 12. Optionally, extend the Python code to do something new that you wanted to do with it (this is the real reason people want to reproduce ConceptNet). - Now, here is the process after Docker: 1. Install Docker and Docker Compose. 2. No, not the ones in Ubuntu's repositories. Those are too old to work. Remove those. You need to set up a separate package archive with sufficiently new versions of Docker software. 3. Configure a Docker volume that's mapped onto a hard disk that has enough space available. (If you do this wrong, the build will fail and you will not find out why.) 4. Configure your user so that you can run Docker stuff and take over port 80. If you can't take over port 80, jump through some hoops. 5. Run the Docker build script. This will install the code, set up PostgreSQL, download the data, build the data, and put the data in the database. (The built data, of course, can't be distributed with the Docker image; that would be too large.) 6. At this point I tell people to wait five hours and then see if the API is up. If it is, it worked! But of course it didn't work, because the download got interrupted or you ran out of disk space or you had the wrong version of Docker or something. 7. Check your Docker logs. 8. Disregard all the irrelevant messages from upstream code, especially uwsgi, which is well known for yelling incoherently when it's working correctly. Somewhere around the middle of the logs, there might be a message about why your build failed. 9. Translate that message into a solution to the problem. (There is no guide to doing this, so at this point users have to send me an e-mail, a Git issue, or a Gitter message and I have to take the time to do tech support for them.) 10. Learn some new commands for administering Docker to fix the problem. 11. Delete your Docker volumes so that the PostgreSQL image knows it needs to reload the data, because Docker images don't actually have a concept of loading data and everything is a big hack. 12. Try again. 13. Wait five hours again. (Look, I can't just give you a tarball of what the PostgreSQL data directory should end up as; that's not portable. It needs to all go in through a COPY command.) 14. Now you can probably access the API. 15. Oh shit, you didn't just want the API, you wanted to be able to extend the code. Unfortunately, the code is running out of sight inside a Docker image with none of the programming tools you want. This is where you come and ask me for the non-Docker process. - Attempting to use Docker for data science has led to me doing unreasonable amounts of tech support, and has not made anything easier for anyone. I constantly run into the fact that Docker is designed for tiny little services and not designed for managing any amount of data. What's the real story about Docker for data science?