4 ms·

Probably Larabee. Too early and too expensive for its time.

dragontamer 8y ago
The Japanese PEZY machine looks incredibly interresting. Too bad it is only used in that one super-computer. Its overall design sits somewhere between a GPU and CPU. Like a GPU, it supports 8-way hyperthreading (GPUs typically support 10. CPUs usually only support 2... maybe 4 for Larabee). Like a GPU, its programmed through an OpenCL-like language (I know you don't like that, but its how it works). However, the overall layout is very CPU-like as well: a "City" is composed of "Villages", which is composed of multiple "Processing Elements". Each PE is scalar like a normal CPU, although 8-way SMT. So its not like a GPU's SIMD or SIMT architecture in this regards. -------- See information here: https://en.wikichip.org/wiki/pezy/pezy-scx/pezy-sc https://en.wikichip.org/wiki/pezy/pezy-scx/pezy-sc It really is an interesting design that toes the line between CPU and GPU.
- nradov 8y ago
  Is PEZY still a going concern after the arrests?
twtw 8y ago
So that's why Xeon Phi has been so successful! /s Real talk, though. Why do you think larabee is the "right" hardware?
- hajile 8y ago
  I'd assume because of the extra serial processing power of a CPU. x86 isn't the right choice though. As the chips get smaller, all the x86 cruft starts to become a larger portion of each core. RISCV with 40-ish instructions plus a SIMD extension would probably be ideal. They've already proven that those designs use less space per core than ARM.
- vl 8y ago
  The truth is that for many types/sizes of models AVS2 or AVS512 in Xeons is as fast and GPUs.
  - frankchn 8y ago
    I am interested in seeing benchmarks regarding this. I believe this is true for models with truly huge embeddings, but otherwise if your models are even a little compute dense then GPUs are faster.
    - Cybiote 8y ago
      My experience is that you need the additional caveat of a streaming fairly homogeneous and highly parallelizable approach or the memory transfer, branching and communication overheads will eat away nearly all the GPU gains. I've also noticed that a lot of the time, people are comparing GPU to naive implementations (and sometimes, implementations written in dynamic languages) instead of to highly tuned BLAS or MKL implementations. For a large swathe of problem types, using the fastest math library will reduce the CPU/GPU gap to less than an order of magnitude or less.
      - twtw 8y ago
        > comparing GPU to naive implementations (and sometimes, implementations written in dynamic languages) instead of to highly tuned BLAS or MKL implementations In your experience, are these comparisons using naive GPU implementations as well? If so, that still seems like a valuable comparison.
      - shaklee3 8y ago
        No, they're not. Straight front the horse's mouth: https://software.intel.com/en-us/mkl/features/benchmarks https://software.intel.com/en-us/mkl/features/benchmarks In no case will you see it get close to 10TFLOPS. GPUs easily do this, and can approach 100TFLOPS with tensor cores.
        Cybiote 8y ago

The Japanese PEZY machine looks incredibly interresting. Too bad it is only used in that one super-computer. Its overall design sits somewhere between a GPU and CPU. Like a GPU, it supports 8-way hyperthreading (GPUs typically support 10. CPUs usually only support 2... maybe 4 for Larabee). Like a GPU, its programmed through an OpenCL-like language (I know you don't like that, but its how it works). However, the overall layout is very CPU-like as well: a "City" is composed of "Villages", which is composed of multiple "Processing Elements". Each PE is scalar like a normal CPU, although 8-way SMT. So its not like a GPU's SIMD or SIMT architecture in this regards. -------- See information here: https://en.wikichip.org/wiki/pezy/pezy-scx/pezy-sc https://en.wikichip.org/wiki/pezy/pezy-scx/pezy-sc It really is an interesting design that toes the line between CPU and GPU.

nradov 8y ago
Is PEZY still a going concern after the arrests?

twtw 8y ago

So that's why Xeon Phi has been so successful! /s Real talk, though. Why do you think larabee is the "right" hardware?

hajile 8y ago
I'd assume because of the extra serial processing power of a CPU. x86 isn't the right choice though. As the chips get smaller, all the x86 cruft starts to become a larger portion of each core. RISCV with 40-ish instructions plus a SIMD extension would probably be ideal. They've already proven that those designs use less space per core than ARM.
vl 8y ago
The truth is that for many types/sizes of models AVS2 or AVS512 in Xeons is as fast and GPUs.
- frankchn 8y ago
  I am interested in seeing benchmarks regarding this. I believe this is true for models with truly huge embeddings, but otherwise if your models are even a little compute dense then GPUs are faster.
  - Cybiote 8y ago
    My experience is that you need the additional caveat of a streaming fairly homogeneous and highly parallelizable approach or the memory transfer, branching and communication overheads will eat away nearly all the GPU gains. I've also noticed that a lot of the time, people are comparing GPU to naive implementations (and sometimes, implementations written in dynamic languages) instead of to highly tuned BLAS or MKL implementations. For a large swathe of problem types, using the fastest math library will reduce the CPU/GPU gap to less than an order of magnitude or less.
    - twtw 8y ago
      > comparing GPU to naive implementations (and sometimes, implementations written in dynamic languages) instead of to highly tuned BLAS or MKL implementations In your experience, are these comparisons using naive GPU implementations as well? If so, that still seems like a valuable comparison.
    - shaklee3 8y ago
      No, they're not. Straight front the horse's mouth: https://software.intel.com/en-us/mkl/features/benchmarks https://software.intel.com/en-us/mkl/features/benchmarks In no case will you see it get close to 10TFLOPS. GPUs easily do this, and can approach 100TFLOPS with tensor cores.
      - Cybiote 8y ago