4 ms·

A library I designed a few years ago (https://github.com/Netflix/vectorflow https://github.com/Netflix/vectorflow) is also much faster than pytorch/tensorflow i

A library I designed a few years ago (https://github.com/Netflix/vectorflow https://github.com/Netflix/vectorflow) is also much faster than pytorch/tensorflow in these cases. In "small" or "very sparse" setups, you're memory bound, not compute bound. TF and Pytorch are bad at that because they assume memory movements are worth it and do very little in-place operations. Different tools for different jobs.

teleforce 5y ago
Thanks for designing this tool, somehow it's missed my radar. Interestingly it is kind of given that D language performance is always fast. It is a shame that it's not being used more on data science since personally I think it's more natural and better fit to replace Python and C eco-system that's very popular now for data science. Heck, it even beats Fortran in its own turf for the BLAS benchmark several years back [1]. Just wondering why did you choose D when you write the Vectorflow library? [1]http://blog.mir.dlang.io/glas/benchmark/openblas/2016/09/23/glas-gemm-benchmark.html http://blog.mir.dlang.io/glas/benchmark/openblas/2016/09/23/...
chillee 5y ago
In "small" setups, you're actually more likely to be overhead-bound if anything (especially on CPU).
- mochomocha 5y ago
  Would you mind defining "overhead"?
  - spullara 5y ago
    Loading the model, preprocessing, post processing, i.e. things not even done in the NN.
    - mochomocha 5y ago
      If you're loading the model for every prediction, you're doing something really wrong. Pre-processing and post-processing are indeed usually trumping the core computation itself, but as you mentioned it's not done in the NN.
      - spullara 5y ago
        Could be something that is just one once per process or the like. But yeah, if you are doing multiple inferences it would be silly to load the model each time.
  - chillee 5y ago
    Anything that’s not an actual computational kernel (python interpreter, Pytorch dispatcher, etc.) See the flame graph here in the overhead section: https://horace.io/brrr_intro.html https://horace.io/brrr_intro.html
    - mochomocha 5y ago
      Right, all this overhead is very pytorch specific and doesn't have to be present in all cases. If you resolve everything at compile-time for a limited class of cases (which is what vectorflow does), it's possible to have a solution where none of this overhead exists at runtime (no dispatch, no indirections, no memory copies nor allocations...), neither during training nor serving. I think pytorch has a genius design and makes excellent tradeoffs. But it's not the perfect solution to every problem. Genericity vs specialization: hard to win on all fronts.
      - chillee 5y ago
        Yeah, my main point is that I don't think the issue is memory movement. PyTorch/Tensorflow do care a lot about memory movement, as memory movement doesn't stop being an issue with larger networks.
  - dragontamer 5y ago
    IMO, one of the big sources of overhead is the PCIe bus. Transferring data to-and-from the PCIe / GPU takes time. If the CPU is faster and can perform the calculation before the PCIe is done transferring, then there's no point even touching the GPU. PCIe is on the scale of ~5000 nanoseconds (20,000 CPU cycles assuming 4GHz). A PCIe write, then read could be ~40,000 CPU cycles or so, and it turns out that a lot of things can be done in that time before the GPU was even _NOTIFIED_ that there was work to do. CPUs can contact other CPU-cores within 50 to 500 nanoseconds or so, depending on the distance. A 64-core CPU could then have 64-cores * 18000 clocks == ~1-million CPU-clock cycles before the GPU gets any message at all.
    - westurner 5y ago
      There's no mention of GPUs, TPUs, or indeed QPUs in the memory hierarchy described by Wikipedia! Locality of reference ("Data locality") > Spatial and temporal locality usage : https://en.wikipedia.org/wiki/Locality_of_reference https://en.wikipedia.org/wiki/Locality_of_reference Memory hierarchy https://en.wikipedia.org/wiki/Memory_hierarchy https://en.wikipedia.org/wiki/Memory_hierarchy : > Most modern CPUs are so fast that for most program workloads, the bottleneck is the locality of reference of memory accesses and the efficiency of the caching and memory transfer between different levels of the hierarchy [citation needed]. As a result, the CPU spends much of its time idling, waiting for memory I/O to complete. This is sometimes called the space cost, as a larger memory object is more likely to overflow a small/fast level and require use of a larger/slower level. The resulting load on memory use is known as pressure (respectively register pressure, cache pressure, and (main) memory pressure). Terms for data being missing from a higher level and needing to be fetched from a lower level are, respectively: register spilling (due to register pressure: register to cache), cache miss (cache to main memory), and (hard) page fault (main memory to disk). Is it that PCIe is necessarily implied by the debuggable pipeline specified by the von Neumann architecture? https://en.wikipedia.org/wiki/Von_Neumann_architecture https://en.wikipedia.org/wiki/Von_Neumann_architecture Otherwise, computation within RAM avoids interconnect saturation. "Neuromorphic" computing, stateful RAM with operators mapped to particle interactions: Memristor > Derivative devices > memtransistor https://en.wikipedia.org/wiki/Memristor#Derivative_devices https://en.wikipedia.org/wiki/Memristor#Derivative_devices Quantum reservoir computing: https://en.wikipedia.org/wiki/Reservoir_computing#Quantum_reservoir_computing https://en.wikipedia.org/wiki/Reservoir_computing#Quantum_re... But these still need a faster and wider (and qubit) bus than PCIe, too: https://en.wikipedia.org/wiki/PCI_Express https://en.wikipedia.org/wiki/PCI_Express
  - dekhn 5y ago
    anything that prevents my code from running at one or more instruction per cycle.
generichuman 5y ago
Is this still used at Netflix? Looks like it's not actively developed anymore.
- mochomocha 5y ago
  It's not developed anymore. I still have a service in production using it: estimating the distribution of runtime of any container we launch, through joint conditional quantile regression (output of the net is p5, p10, ..., p95 of the runtime distribution). It's a good use case for it, because it needs to be low-latency (it's in the critical path when launching a container at Netflix) and it's using a lot of sparse features to do its job (hashed tokens of a lot of poorly structured metadata about the container and its runtime).

teleforce 5y ago

Thanks for designing this tool, somehow it's missed my radar. Interestingly it is kind of given that D language performance is always fast. It is a shame that it's not being used more on data science since personally I think it's more natural and better fit to replace Python and C eco-system that's very popular now for data science. Heck, it even beats Fortran in its own turf for the BLAS benchmark several years back [1]. Just wondering why did you choose D when you write the Vectorflow library? [1]http://blog.mir.dlang.io/glas/benchmark/openblas/2016/09/23/glas-gemm-benchmark.html http://blog.mir.dlang.io/glas/benchmark/openblas/2016/09/23/...

chillee 5y ago

In "small" setups, you're actually more likely to be overhead-bound if anything (especially on CPU).

mochomocha 5y ago
Would you mind defining "overhead"?
- spullara 5y ago
  Loading the model, preprocessing, post processing, i.e. things not even done in the NN.
  - mochomocha 5y ago
    If you're loading the model for every prediction, you're doing something really wrong. Pre-processing and post-processing are indeed usually trumping the core computation itself, but as you mentioned it's not done in the NN.
    - spullara 5y ago
      Could be something that is just one once per process or the like. But yeah, if you are doing multiple inferences it would be silly to load the model each time.
- chillee 5y ago
  Anything that’s not an actual computational kernel (python interpreter, Pytorch dispatcher, etc.) See the flame graph here in the overhead section: https://horace.io/brrr_intro.html https://horace.io/brrr_intro.html
  - mochomocha 5y ago
    Right, all this overhead is very pytorch specific and doesn't have to be present in all cases. If you resolve everything at compile-time for a limited class of cases (which is what vectorflow does), it's possible to have a solution where none of this overhead exists at runtime (no dispatch, no indirections, no memory copies nor allocations...), neither during training nor serving. I think pytorch has a genius design and makes excellent tradeoffs. But it's not the perfect solution to every problem. Genericity vs specialization: hard to win on all fronts.
    - chillee 5y ago
      Yeah, my main point is that I don't think the issue is memory movement. PyTorch/Tensorflow do care a lot about memory movement, as memory movement doesn't stop being an issue with larger networks.
- dragontamer 5y ago
  IMO, one of the big sources of overhead is the PCIe bus. Transferring data to-and-from the PCIe / GPU takes time. If the CPU is faster and can perform the calculation before the PCIe is done transferring, then there's no point even touching the GPU. PCIe is on the scale of ~5000 nanoseconds (20,000 CPU cycles assuming 4GHz). A PCIe write, then read could be ~40,000 CPU cycles or so, and it turns out that a lot of things can be done in that time before the GPU was even _NOTIFIED_ that there was work to do. CPUs can contact other CPU-cores within 50 to 500 nanoseconds or so, depending on the distance. A 64-core CPU could then have 64-cores * 18000 clocks == ~1-million CPU-clock cycles before the GPU gets any message at all.
  - westurner 5y ago
    There's no mention of GPUs, TPUs, or indeed QPUs in the memory hierarchy described by Wikipedia! Locality of reference ("Data locality") > Spatial and temporal locality usage : https://en.wikipedia.org/wiki/Locality_of_reference https://en.wikipedia.org/wiki/Locality_of_reference Memory hierarchy https://en.wikipedia.org/wiki/Memory_hierarchy https://en.wikipedia.org/wiki/Memory_hierarchy : > Most modern CPUs are so fast that for most program workloads, the bottleneck is the locality of reference of memory accesses and the efficiency of the caching and memory transfer between different levels of the hierarchy [citation needed]. As a result, the CPU spends much of its time idling, waiting for memory I/O to complete. This is sometimes called the space cost, as a larger memory object is more likely to overflow a small/fast level and require use of a larger/slower level. The resulting load on memory use is known as pressure (respectively register pressure, cache pressure, and (main) memory pressure). Terms for data being missing from a higher level and needing to be fetched from a lower level are, respectively: register spilling (due to register pressure: register to cache), cache miss (cache to main memory), and (hard) page fault (main memory to disk). Is it that PCIe is necessarily implied by the debuggable pipeline specified by the von Neumann architecture? https://en.wikipedia.org/wiki/Von_Neumann_architecture https://en.wikipedia.org/wiki/Von_Neumann_architecture Otherwise, computation within RAM avoids interconnect saturation. "Neuromorphic" computing, stateful RAM with operators mapped to particle interactions: Memristor > Derivative devices > memtransistor https://en.wikipedia.org/wiki/Memristor#Derivative_devices https://en.wikipedia.org/wiki/Memristor#Derivative_devices Quantum reservoir computing: https://en.wikipedia.org/wiki/Reservoir_computing#Quantum_reservoir_computing https://en.wikipedia.org/wiki/Reservoir_computing#Quantum_re... But these still need a faster and wider (and qubit) bus than PCIe, too: https://en.wikipedia.org/wiki/PCI_Express https://en.wikipedia.org/wiki/PCI_Express
- dekhn 5y ago
  anything that prevents my code from running at one or more instruction per cycle.

generichuman 5y ago

Is this still used at Netflix? Looks like it's not actively developed anymore.

mochomocha 5y ago
It's not developed anymore. I still have a service in production using it: estimating the distribution of runtime of any container we launch, through joint conditional quantile regression (output of the net is p5, p10, ..., p95 of the runtime distribution). It's a good use case for it, because it needs to be low-latency (it's in the critical path when launching a container at Netflix) and it's using a lot of sparse features to do its job (hashed tokens of a lot of poorly structured metadata about the container and its runtime).