4 ms·
Is it possible that the slowness is caused by disk speed of loading a large file from disk? I've observed the same thing but it seemed to be IO related, not CP
by etaioinshrdlu 9y ago
Is it possible that the slowness is caused by disk speed of loading a large file from disk?
I've observed the same thing but it seemed to be IO related, not CPU or GPU related.
- deleted 9y ago[deleted]
- tlb 9y agoIt's mainly the speed of creating a huge number of small files on disk. When unpacking a docker image into an AUFS file system, it has to create every file in the shadow system. A robust Ubuntu install has around 100000 files. On AWS with default EBS storage, this can take a few minutes. At OpenAI we pre-built AMI instances with file systems pre-unpacked.