4 ms·

> As with the original benchmark, I set up a Docker container containing the deep learning frameworks (based on cuDNN 6, the latest version of cuDNN natively su

by dsacco 9y ago

> As with the original benchmark, I set up a Docker container containing the deep learning frameworks (based on cuDNN 6, the latest version of cuDNN natively supported by the frameworks) that can be used to train each model independently. The Keras benchmark scripts run on the containers are based off of real world use cases of deep learning. Can someone who is knowledgeable about docker (and container performance in general) comment on how using a docker instance impacts benchmark performance? I understand containers are not virtual machines, but my preference would be for a benchmark run on bare metal without containerization involved. In particular, it’s not clear to me whether a container can take advantage of the same CPU and GPU instruction optimizations that a binary or compiled-from-source version can. For example, I’m curious if containers package redundant versions of software for convenience that are underoptimized when compared to versions packaged with the operating system. Is that a realistic concern? In other words, is there generally a meaningful downside to benchmarking with containers, and if so what is it? And equally importantly, are containers the way most people use deep learning frameworks? I have never installed Tensorflow or Keras using a container.

johncolanduoni 9y ago
Most binary distributions don’t have different sets of packages for different feature sets of 64-bit Intel processors. Most containers use the same exact binaries the given distribution would on bare metal. The processor won’t (and AFAIK can’t) report different feature sets for processes running in a Docker container, and most heavy optimized numerical libraries will choose to use things like AVX at runtime, not compile time.
- matt4077 9y ago
  Tensorflow does have compile-time optimisations, but it will warn if you're running a binary missing out on, for example, AVX when the hardware supports it.
simcop2387 9y ago
> it’s not clear to me whether a container can take advantage of the same CPU and GPU instruction optimizations that a binary or compiled-from-source version can. Since the container is running on the bare metal, with the same kernel and everything, just isolated, it absolutely can take advantage of these things. It'll require about the same amount of work to do the compilation for using any special instructions for performance as it would if it's on bare metal. > For example, I’m curious if containers package redundant versions of software for convenience that are underoptimized when compared to versions packaged with the operating system. Is that realistic concern? Maybe. It'll mean that there's extra data on disk and maybe in ram from loading other shared libraries which will mean some overhead, but unless there's some specific set of instructions or optimizations that would make a difference for the workload it should be basically the same.
- derefr 9y ago
  > unless there's some specific set of instructions or optimizations that would make a difference for the workload I think this was the parent's worry. Most HPC deployments have versions of libc, libm, etc. compiled with tuning flags specific to the microarchitecture of the system, to allow workloads deployed on them to squeeze as much performance as possible out of the system. I would worry that the versions of libraries in the container are just the generic packages that ship with the OS, compiled with -march=i686 or whatever the modern equivalent is, where they don't try to take advantage of e.g. AVX512 instructions if they're available.
  - malux85 9y ago
    Yes you’re 100% correct, but if you’re using a well tuned GPU Deep Learning pipeline the bottleneck is GPU compute and the CPU is always waiting on the GPU. It can be a bit tricky to optimise this though - some deep learning users do things like resizing images on the fly, or on the fly vectorising / scaling of input data in these cases the CPU can be the bottleneck and the optimisations above can help - but this is largely a symptom of a poor pipeline configuration in newbie shops, most serious places have optimised this away and the GPU is the bottleneck
  - 9y ago

by dsacco 9y ago

johncolanduoni 9y ago

Most binary distributions don’t have different sets of packages for different feature sets of 64-bit Intel processors. Most containers use the same exact binaries the given distribution would on bare metal. The processor won’t (and AFAIK can’t) report different feature sets for processes running in a Docker container, and most heavy optimized numerical libraries will choose to use things like AVX at runtime, not compile time.

matt4077 9y ago
Tensorflow does have compile-time optimisations, but it will warn if you're running a binary missing out on, for example, AVX when the hardware supports it.

simcop2387 9y ago

> it’s not clear to me whether a container can take advantage of the same CPU and GPU instruction optimizations that a binary or compiled-from-source version can. Since the container is running on the bare metal, with the same kernel and everything, just isolated, it absolutely can take advantage of these things. It'll require about the same amount of work to do the compilation for using any special instructions for performance as it would if it's on bare metal. > For example, I’m curious if containers package redundant versions of software for convenience that are underoptimized when compared to versions packaged with the operating system. Is that realistic concern? Maybe. It'll mean that there's extra data on disk and maybe in ram from loading other shared libraries which will mean some overhead, but unless there's some specific set of instructions or optimizations that would make a difference for the workload it should be basically the same.

derefr 9y ago
> unless there's some specific set of instructions or optimizations that would make a difference for the workload I think this was the parent's worry. Most HPC deployments have versions of libc, libm, etc. compiled with tuning flags specific to the microarchitecture of the system, to allow workloads deployed on them to squeeze as much performance as possible out of the system. I would worry that the versions of libraries in the container are just the generic packages that ship with the OS, compiled with -march=i686 or whatever the modern equivalent is, where they don't try to take advantage of e.g. AVX512 instructions if they're available.
- malux85 9y ago
  Yes you’re 100% correct, but if you’re using a well tuned GPU Deep Learning pipeline the bottleneck is GPU compute and the CPU is always waiting on the GPU. It can be a bit tricky to optimise this though - some deep learning users do things like resizing images on the fly, or on the fly vectorising / scaling of input data in these cases the CPU can be the bottleneck and the optimisations above can help - but this is largely a symptom of a poor pipeline configuration in newbie shops, most serious places have optimised this away and the GPU is the bottleneck
- 9y ago