3 ms·

Good overview. "Vectorized" is an old term that's been around since the early days of supercomputers and maybe before, not sure where it came from. Numba does a

Good overview. "Vectorized" is an old term that's been around since the early days of supercomputers and maybe before, not sure where it came from. Numba does a bunch of different things for code written to the Numpy API including CUDA acceleration. Certain machine learning frameworks like PyTorch and JAX also roughly follow the Numpy API because it is widely familiar and easy enough to work with. The kind of code that benefits from this kind of acceleration is hard to write yourself. A lot of workloads lean on linear algebra operations that are conceptually simple but complicated to implement with good performance, thus why all of this tooling isn't just a couple thousand lines of C. Good overview of matmul on CPU: https://gist.github.com/nadavrot/5b35d44e8ba3dd718e595e40184d03f0 https://gist.github.com/nadavrot/5b35d44e8ba3dd718e595e40184...

college_physics 4y ago
Cray supercomputers used to have special "vector" units that would perform operations on multiple scalars (eg 128 doubles) in parallel. A bit like gpu units. Any algorithm that could be cast in a form that benefited from this type of parallelism would be called vectorizable. Linear algebra obviously fits perfectly (but, depending on the problem, you might0 need to juggle the vector dimensions). Vectorizing code was fairly straightforward using the latter versions of fortran. It was all quite sweet and productive but could not provide the required hpc scaling so was eventually abandoned in favor of massively parallel designs.
- hedgehog 4y ago
  I think this is backwards, the early Crays and maybe other machines used essentially serial + aggressively pipelined CPUs while modern CPUs do actually execute operations in parallel via multiple ALUs but are from a programmer point of view kind of similar (fixed size vector).
  - college_physics 4y ago
    What is a vector unit was is simd etc can be confusing but at least some wikipedians are sure the early Crays had true vector processing (which may not be parallel processing according to your definition) https://en.m.wikipedia.org/wiki/Vector_processor https://en.m.wikipedia.org/wiki/Vector_processor
    - hedgehog 4y ago
      Yes, in the Cray design a vector instruction processes one item per clock so total latency increases with number of elements. Only parallel in the sense of pipelining being a form of parallelism. Something like an AVX-equipped Intel CPU processes all elements in parallel to deliver a result in an essentially constant number of cycles. Edit: There's a period write-up of the general Cray 1 design here: https://inst.eecs.berkeley.edu/~n252/sp07/Papers/Cray.pdf https://inst.eecs.berkeley.edu/~n252/sp07/Papers/Cray.pdf

college_physics 4y ago

Cray supercomputers used to have special "vector" units that would perform operations on multiple scalars (eg 128 doubles) in parallel. A bit like gpu units. Any algorithm that could be cast in a form that benefited from this type of parallelism would be called vectorizable. Linear algebra obviously fits perfectly (but, depending on the problem, you might0 need to juggle the vector dimensions). Vectorizing code was fairly straightforward using the latter versions of fortran. It was all quite sweet and productive but could not provide the required hpc scaling so was eventually abandoned in favor of massively parallel designs.

hedgehog 4y ago
I think this is backwards, the early Crays and maybe other machines used essentially serial + aggressively pipelined CPUs while modern CPUs do actually execute operations in parallel via multiple ALUs but are from a programmer point of view kind of similar (fixed size vector).
- college_physics 4y ago
  What is a vector unit was is simd etc can be confusing but at least some wikipedians are sure the early Crays had true vector processing (which may not be parallel processing according to your definition) https://en.m.wikipedia.org/wiki/Vector_processor https://en.m.wikipedia.org/wiki/Vector_processor
  - hedgehog 4y ago
    Yes, in the Cray design a vector instruction processes one item per clock so total latency increases with number of elements. Only parallel in the sense of pipelining being a form of parallelism. Something like an AVX-equipped Intel CPU processes all elements in parallel to deliver a result in an essentially constant number of cycles. Edit: There's a period write-up of the general Cray 1 design here: https://inst.eecs.berkeley.edu/~n252/sp07/Papers/Cray.pdf https://inst.eecs.berkeley.edu/~n252/sp07/Papers/Cray.pdf