3 ms·
We use this in production every day. We've hit a couple bugs, but it's a surprisingly stable piece of software. We hit an issue with user namespaces, and it no
by sargun 9y ago
We use this in production every day. We've hit a couple bugs, but it's a surprisingly stable piece of software.
We hit an issue with user namespaces, and it not correctly chowning things when "exporting" them into Docker containers. I honestly forget how we solved this, but I believe it was just an upgrade.
We hit an issue with the first run of a Docker container on a system being extremely slow, to the point where downstream systems would give up, and retry. The issue was fixed by starting a container that leverages the driver at boot time, "warming" it up.
- etaioinshrdlu 9y agoIs it possible that the slowness is caused by disk speed of loading a large file from disk? I've observed the same thing but it seemed to be IO related, not CPU or GPU related.
- deleted 9y ago[deleted]
- tlb 9y agoIt's mainly the speed of creating a huge number of small files on disk. When unpacking a docker image into an AUFS file system, it has to create every file in the shadow system. A robust Ubuntu install has around 100000 files. On AWS with default EBS storage, this can take a few minutes. At OpenAI we pre-built AMI instances with file systems pre-unpacked.
- sipherhex 9y agoGiven the description of your workaround, you may be running into an issue of long GPU driver load times for the nvidia driver. If that is the root cause the recommended solution for this is nvidia-persistenced, though it seems you may have rolled your own equivalent. https://github.com/NVIDIA/nvidia-persistenced https://github.com/NVIDIA/nvidia-persistenced Docs here http://docs.nvidia.com/deploy/driver-persistence/index.html http://docs.nvidia.com/deploy/driver-persistence/index.html