8 ms·

Blogspam, kinda. Post just links to https://software.intel.com/sites/default/files/managed/69/78/319433-025.pdf https://software.intel.com/sites/default/files/m

Blogspam, kinda. Post just links to https://software.intel.com/sites/default/files/managed/69/78/319433-025.pdf https://software.intel.com/sites/default/files/managed/69/78... and says it mentions two new instructions: AVX512_4VNNIW (Vector instructions for deep learning enhanced word variable precision) and AVX512_4FMAPS (Vector instructions for deep learning floating-point single precision) On Intel's part, it seems kinda... late? It's like adding Bitcoin instructions when you already know everyone's racing to make Bitcoin ASICs. How could it beat dedicated hardware, or even GPUs, on ops/watt? Maybe it's intended for inference, not training, but that doesn't sound compelling either.

ninjin 10y ago
I am not so sure if it is "late". Sure, for training models it is far too late, we have semi-dedicated hardware for that at this point. But it is possible that it will become increasingly common to run these models on end-user systems and in that case these instructions could provide a bit of a speed boost.
valarauca1 10y ago
>>On Intel's part, it seems kinda... late? Not exactly AVX512 is used on the Xeon Phi which is Intel's Floating Co-Processor for Deep Learning/OpenCL tasks. But that being said, GPU's blow the Xeon Phi out of the water in terms of FLOPS, local RAM, and RAM bandwidth.
- astrodust 10y ago
  Isn't Phi better at dealing with branch-heavy code? GPUs absolutely kill on straight-forward compute, but they seem to flail helplessly if the logic is highly conditional. I've read that in some cases GPUs evaluate both branches of a condition and discard the unwanted results. A CPU doesn't do this.
  - valarauca1 10y ago
    >>Isn't Phi better at dealing with branch-heavy code? This maybe true. Generally code that is infinitely parallelize to be ran on a GPU isn't going to be branch heavy. There are no benchmarks to support this, but I would believe it is true. GPU's typically have 1 branching unit per 64-128threads. While the Phi has 1 branching unit per 2 threads. The real difference is GPU's use SIMT, while Phi uses MIMD. Each Phi thread is a POSIX thread. It can do its own thing, live its own life. While GPU's you just specify how parallel your work-group, thread block, warp, or wave front is (name depends on platform, in order OpenCL, CUDA, Nvida DX12, AMD DX12). >>I've read that in some cases GPUs evaluate both branches of a condition and discard the unwanted results. A CPU doesn't do this. Intel CPU's have done this since SSE4.
    - astrodust 10y ago
      If you mean branch prediction, then yes, it will end up inadvertently executing the wrong instructions and discard them, but it halts execution of the wrong branch at the first opportunity and reschedules to correct its mistake. A GPU will execute both branches to completion.
      - jasonwatkinspdx 10y ago
        No, he means the SSE predication instructions. The comparison instructions let you build a mask vector, and instructions like blendv use that mask vector to only impact a subset of elements. It's been a common feature on many RISC and VLIW cpu's over the years. It is in no way unique to GPU's.
        astrodust 10y ago
        Ah, thanks for clarifying.
    - gpderetta 10y ago
      > GPU's use SIMT I know very little of GPU architectures, but I tought that the last few generations of GPUs were straightforward SIMDs (i.e. all lanes are run in lockstep and divergence is handled at an higher level)
obmelvin 10y ago
As the other poster mentions, I assume their view is that these types of operations will become increasingly prevalent in consumer software applications. Lots will be powered by the cloud, but it would be useful if some inference can be done on a laptop CPU.
- Eridrus 10y ago
  "The cloud" still has to run the models on something, and as long as Intel roughly keeps up with the cost of specialised hardware for doing inference, many people will just keep doing inference on Intel CPUs, rather than bothering with specialised hardware.
  - obmelvin 10y ago
    That's true... I'd been assuming they'd use specialized hardware for inference when writing that, but totally failed to think how cost may make a CPU more desirable.
dsabanin 10y ago
Intel is also adding FPGAs[1] right next to their CPUs, which can possibly become a next big thing in HPC. [1] http://www.extremetech.com/extreme/184828-intel-unveils-new-xeon-chip-with-integrated-fpga-touts-20x-performance-boost http://www.extremetech.com/extreme/184828-intel-unveils-new-...
- yid 10y ago
  This aspect of their new chips is massively underrated. An FPGA is the future-proof solution here, not chip-level instructions for the soup-du-jour in machine learning. Edit: which is not to say that I'm not welcoming the new instructions with open arms...
  - greendragon 10y ago
    It will only really take off if they can get the user experience of getting a HDL program to the FPGA to the same level as getting a shader program to the GPU. Unless they can do all that in microcode though it's going to force them to open up some stacks of Altera's closed processes. I'm hopeful but there's a lot of proprietary baggage around FPGAs that I think have kept them from truly reaching their potential.
    - yid 10y ago
      > I'm hopeful but there's a lot of proprietary baggage around FPGAs that I think have kept them from truly reaching their potential. I don't really know much about this aspect, could you elaborate? I'm genuinely curious.
      - aseipp 10y ago
        Almost all FPGAs have entirely proprietary toolchains, from Verilog/HDL synthesis all the way down to bitstream loaders for programming your board. These tools are traditionally very poorly built as applications, scriptable only with things like TCL, terrible error messages, godawful user interfaces designed in 1999, etc etc.[1] This makes integrating with them a bit more "interesting" for doing things like build system integration, etc. Furthermore, FPGAs are inherently much more expensive for things like reprogramming, than just recompiling your application. Synthesis and place-and-route can take excessive amounts of time even for some kinds of small designs. (And for larger designs, you tend to do post-synthesis edits on the resulting gate netlists to fix problems if you can, in order to avoid the horrid turn around time of hours or days for your tools). The "flow" of debugging, synthesis, testing etc is all fundamentally different and has a substantially different kind of cost model. This means many kind of "dream" ideas of on-the-fly reconfiguration "Just-in-Time"-ing your logic cells, based on things like hot, active datapaths -- is a bit difficult for all but the simplest designs, due to the simple overhead of the toolchain and design phase. That said, some simple designs are still incredibly useful! What's more likely with these systems is you'll use, or develop, pre-baked sets of bitstreams to add in custom functionality for customers or yourselves on-demand, e.g. push out a whole fleet of Xeons, configure the cells of machines on the edge to do hardware VPN offload, configure backend machines that will run your databases with a different logic that your database engine can use to accelerate queries, etc. Whether or not that will actually pan out remains to be seen, IMO. Large enterprises can definitely get over the hurdles of shitty EDA tools and very long timescales (days) for things like synthesis, but small businesses that may still be in a datacenter will probably want to stick with commercial hardware accelerators. Note that I'm pretty new to FPGA-land, but this is my basic understanding and impressions, given some time with the tools. I do think FPGAs have not really hit their true potential, and I do think the shit tooling -- and frankly god awful learning materials -- has a bit to do with it, which drives away many people who could learn. On the other hand: FPGAs and hardware design is so fundamentally different for most software programmers, there are simply things that, I think, cannot so easily be abstracted over at all -- and you have to accept at some level that the design, costs, and thought process is just different. [1] The only actual open-source Verilog->Upload-to-board flow is IceStorm, here: http://www.clifford.at/icestorm/ http://www.clifford.at/icestorm/. This is only specifically for the iCE40 Lattice FPGAs; they are incredibly primitive and very puny compared to any modern Altera or Xilinx chip (only 8,000 LUTs!), but they can drive real designs, and it's all open source, at least. Just as an example, I've been learning using IceStorm, purely because it's so much less shitty. When I tried Lattice's iCE40 software they provided, I had to immediately fix 2 bash scripts that couldn't detect my 4.x version kernel was "Linux", and THEN sed /bin/sh to /bin/bash in 20 more scripts __which used bash features__, just so that the Verilog synthesis tool they shipped actually worked. What the fuck. Oh! Also, this was after I had to create a fake `eth0` device with a juked MAC address (an exact MAC being required for the licensing software), because systemd has different device names for ethernet interfaces based on the device (TBF, for a decent reason - hotplugging/OOO device attachment means you really want a deterministic naming scheme based on the card), but their tool can't handle that, it only expects eth0.
ericjang 10y ago
Not exactly late. There's no physical law that says a CPU can't beat a GPU for a certain family of operations, and in fact in practice, small toy networks with non-dense operations (for instance, if you are doing research on optimization algorithms for DNNs or Bayesian inference algorithms) tend to run faster on the CPU than GPU.
adevine 10y ago
I am not familiar with how deep learning algorithms would benefit from these instructions in a way different from how they can just use vector instructions optimized for graphics processing. So my question is, how are these "Vector instructions for deep learning enhanced word variable precision" different from other vector instruction sets?
- sbierwagen 10y ago
  Good question. It doesn't say! The PDF has absolutely no detail on what it actually does. Googling the instruction brings up people updating CPUID fields, but nothing to do with actually using it.
  - p1esk 10y ago
    I'm guessing it means you can choose the precision (word length) of the computations, eg 8 bits, 16 bits, 32 bits, etc. This probably only applies to integer ops, but if they could let us pack FP16 operands into 512 bit vectors, that would be awesome!
    - rspeer 10y ago
      I hope they're going even farther and considering minifloats. 8-bit integers are easy but limiting and unexciting. 8-bit floats are exactly what some machine learning needs. If you're updating your weights enough, precision can just be wasted computation. A 512-bit-wide computation could be updating 64 very coarse-grained weights at a time.
vegabook 10y ago
disingenuous to use a highly charged term like "blogspam" and then qualify it as "kinda". kettle/black. Why say "blogspam" if you don't mean it? And while I agree that the OP is not indepth, it is not blogspam. AVX512 is a very real, very very real, order of magnitude, performance improvement. To give you an idea, going from standard 64-bit to AVX-256 with MKL Numpy (under ipython) you will go from 6-7 seconds, to under 1 second, on an identical 2012+ processor, on this code: import numpy as np np.random.seed(1234) xx = np.random.rand(1000000).reshape(1000, 1000) %timeit np.linalg.eig(xx) The best way to test this is to run the above on bog-standard Ubuntu 16.04 (sudo apt-get install python(3)-pip / sudo pip(3) install numpy first), and then run it again after installing Anaconda Python. AVX512 will halve this timing again, and before you tell me that Cuda will do this much faster, I'll remind you that there is a point at which 4000 cuda stream processors operating on 64bit data can, in many algos, be a vastly too large window. Phi KL's 72 branching cores with 4 threads and 2x512-bit AVX each, can be a very strong competitor to the flatspace GPUs in nonlinear algos. There is clear blue sky between the scalar architectures and the massively parallel architectures, which has not been targeted yet and this is where KL fits in. After all, the GPUs are still, first and foremost, designed to rotate and translate massive-amounts of triangles in 3d shooters, a task for which a vast number of very dumb little cores is optimal. As much as Nvidia make like you to believe different, that is still their primary market and it doesn't always map well to everything. Between the current extremes of a small number of very flexible and clever cores (xeon/i7), and 4000 very stupid but numerous cores (GPU), there is a lot of room something in between. Intel isn't so much late to the party as attacking an obliquely different segment of the market, which may prove very interesting indeed.
wpietri 10y ago
> On Intel's part, it seems kinda... late? It's like adding Bitcoin instructions when you already know everyone's racing to make Bitcoin ASICs. Nah. You're thinking they have to be the best, that they have to win. That is true when the choice is buying an Intel CPU vs buying a GPU. But most developers will already have a late-model Intel CPUs around. For Intel, this is a defensive move. They're bending the performance curve for particular use cases. All this has to do is to keep a certain number of developers from bothering to go the extra mile to use special hardware and software. People working at Google's scale won't care. But there will be other people who try running something and say, "Current hardware's fine. We could do better with GPUs, but not so much better that it's worth the time and money." I'm sure it's also the start of a learning cycle for them. Some people told them their current processors were not enough, so they added this. They'll now hear why this isn't adequate either and adapt again if the business case is there.

ninjin 10y ago

I am not so sure if it is "late". Sure, for training models it is far too late, we have semi-dedicated hardware for that at this point. But it is possible that it will become increasingly common to run these models on end-user systems and in that case these instructions could provide a bit of a speed boost.

valarauca1 10y ago

>>On Intel's part, it seems kinda... late? Not exactly AVX512 is used on the Xeon Phi which is Intel's Floating Co-Processor for Deep Learning/OpenCL tasks. But that being said, GPU's blow the Xeon Phi out of the water in terms of FLOPS, local RAM, and RAM bandwidth.

astrodust 10y ago
Isn't Phi better at dealing with branch-heavy code? GPUs absolutely kill on straight-forward compute, but they seem to flail helplessly if the logic is highly conditional. I've read that in some cases GPUs evaluate both branches of a condition and discard the unwanted results. A CPU doesn't do this.
- valarauca1 10y ago
  >>Isn't Phi better at dealing with branch-heavy code? This maybe true. Generally code that is infinitely parallelize to be ran on a GPU isn't going to be branch heavy. There are no benchmarks to support this, but I would believe it is true. GPU's typically have 1 branching unit per 64-128threads. While the Phi has 1 branching unit per 2 threads. The real difference is GPU's use SIMT, while Phi uses MIMD. Each Phi thread is a POSIX thread. It can do its own thing, live its own life. While GPU's you just specify how parallel your work-group, thread block, warp, or wave front is (name depends on platform, in order OpenCL, CUDA, Nvida DX12, AMD DX12). >>I've read that in some cases GPUs evaluate both branches of a condition and discard the unwanted results. A CPU doesn't do this. Intel CPU's have done this since SSE4.
  - astrodust 10y ago
    If you mean branch prediction, then yes, it will end up inadvertently executing the wrong instructions and discard them, but it halts execution of the wrong branch at the first opportunity and reschedules to correct its mistake. A GPU will execute both branches to completion.
    - jasonwatkinspdx 10y ago
      No, he means the SSE predication instructions. The comparison instructions let you build a mask vector, and instructions like blendv use that mask vector to only impact a subset of elements. It's been a common feature on many RISC and VLIW cpu's over the years. It is in no way unique to GPU's.
      - astrodust 10y ago
        Ah, thanks for clarifying.
  - gpderetta 10y ago
    > GPU's use SIMT I know very little of GPU architectures, but I tought that the last few generations of GPUs were straightforward SIMDs (i.e. all lanes are run in lockstep and divergence is handled at an higher level)

obmelvin 10y ago

As the other poster mentions, I assume their view is that these types of operations will become increasingly prevalent in consumer software applications. Lots will be powered by the cloud, but it would be useful if some inference can be done on a laptop CPU.

Eridrus 10y ago
"The cloud" still has to run the models on something, and as long as Intel roughly keeps up with the cost of specialised hardware for doing inference, many people will just keep doing inference on Intel CPUs, rather than bothering with specialised hardware.
- obmelvin 10y ago
  That's true... I'd been assuming they'd use specialized hardware for inference when writing that, but totally failed to think how cost may make a CPU more desirable.

dsabanin 10y ago

Intel is also adding FPGAs[1] right next to their CPUs, which can possibly become a next big thing in HPC. [1] http://www.extremetech.com/extreme/184828-intel-unveils-new-xeon-chip-with-integrated-fpga-touts-20x-performance-boost http://www.extremetech.com/extreme/184828-intel-unveils-new-...

yid 10y ago
This aspect of their new chips is massively underrated. An FPGA is the future-proof solution here, not chip-level instructions for the soup-du-jour in machine learning. Edit: which is not to say that I'm not welcoming the new instructions with open arms...
- greendragon 10y ago
  It will only really take off if they can get the user experience of getting a HDL program to the FPGA to the same level as getting a shader program to the GPU. Unless they can do all that in microcode though it's going to force them to open up some stacks of Altera's closed processes. I'm hopeful but there's a lot of proprietary baggage around FPGAs that I think have kept them from truly reaching their potential.
  - yid 10y ago
    > I'm hopeful but there's a lot of proprietary baggage around FPGAs that I think have kept them from truly reaching their potential. I don't really know much about this aspect, could you elaborate? I'm genuinely curious.
    - aseipp 10y ago
      Almost all FPGAs have entirely proprietary toolchains, from Verilog/HDL synthesis all the way down to bitstream loaders for programming your board. These tools are traditionally very poorly built as applications, scriptable only with things like TCL, terrible error messages, godawful user interfaces designed in 1999, etc etc.[1] This makes integrating with them a bit more "interesting" for doing things like build system integration, etc. Furthermore, FPGAs are inherently much more expensive for things like reprogramming, than just recompiling your application. Synthesis and place-and-route can take excessive amounts of time even for some kinds of small designs. (And for larger designs, you tend to do post-synthesis edits on the resulting gate netlists to fix problems if you can, in order to avoid the horrid turn around time of hours or days for your tools). The "flow" of debugging, synthesis, testing etc is all fundamentally different and has a substantially different kind of cost model. This means many kind of "dream" ideas of on-the-fly reconfiguration "Just-in-Time"-ing your logic cells, based on things like hot, active datapaths -- is a bit difficult for all but the simplest designs, due to the simple overhead of the toolchain and design phase. That said, some simple designs are still incredibly useful! What's more likely with these systems is you'll use, or develop, pre-baked sets of bitstreams to add in custom functionality for customers or yourselves on-demand, e.g. push out a whole fleet of Xeons, configure the cells of machines on the edge to do hardware VPN offload, configure backend machines that will run your databases with a different logic that your database engine can use to accelerate queries, etc. Whether or not that will actually pan out remains to be seen, IMO. Large enterprises can definitely get over the hurdles of shitty EDA tools and very long timescales (days) for things like synthesis, but small businesses that may still be in a datacenter will probably want to stick with commercial hardware accelerators. Note that I'm pretty new to FPGA-land, but this is my basic understanding and impressions, given some time with the tools. I do think FPGAs have not really hit their true potential, and I do think the shit tooling -- and frankly god awful learning materials -- has a bit to do with it, which drives away many people who could learn. On the other hand: FPGAs and hardware design is so fundamentally different for most software programmers, there are simply things that, I think, cannot so easily be abstracted over at all -- and you have to accept at some level that the design, costs, and thought process is just different. [1] The only actual open-source Verilog->Upload-to-board flow is IceStorm, here: http://www.clifford.at/icestorm/ http://www.clifford.at/icestorm/. This is only specifically for the iCE40 Lattice FPGAs; they are incredibly primitive and very puny compared to any modern Altera or Xilinx chip (only 8,000 LUTs!), but they can drive real designs, and it's all open source, at least. Just as an example, I've been learning using IceStorm, purely because it's so much less shitty. When I tried Lattice's iCE40 software they provided, I had to immediately fix 2 bash scripts that couldn't detect my 4.x version kernel was "Linux", and THEN sed /bin/sh to /bin/bash in 20 more scripts __which used bash features__, just so that the Verilog synthesis tool they shipped actually worked. What the fuck. Oh! Also, this was after I had to create a fake `eth0` device with a juked MAC address (an exact MAC being required for the licensing software), because systemd has different device names for ethernet interfaces based on the device (TBF, for a decent reason - hotplugging/OOO device attachment means you really want a deterministic naming scheme based on the card), but their tool can't handle that, it only expects eth0.

ericjang 10y ago

Not exactly late. There's no physical law that says a CPU can't beat a GPU for a certain family of operations, and in fact in practice, small toy networks with non-dense operations (for instance, if you are doing research on optimization algorithms for DNNs or Bayesian inference algorithms) tend to run faster on the CPU than GPU.

adevine 10y ago

I am not familiar with how deep learning algorithms would benefit from these instructions in a way different from how they can just use vector instructions optimized for graphics processing. So my question is, how are these "Vector instructions for deep learning enhanced word variable precision" different from other vector instruction sets?

sbierwagen 10y ago
Good question. It doesn't say! The PDF has absolutely no detail on what it actually does. Googling the instruction brings up people updating CPUID fields, but nothing to do with actually using it.
- p1esk 10y ago
  I'm guessing it means you can choose the precision (word length) of the computations, eg 8 bits, 16 bits, 32 bits, etc. This probably only applies to integer ops, but if they could let us pack FP16 operands into 512 bit vectors, that would be awesome!
  - rspeer 10y ago
    I hope they're going even farther and considering minifloats. 8-bit integers are easy but limiting and unexciting. 8-bit floats are exactly what some machine learning needs. If you're updating your weights enough, precision can just be wasted computation. A 512-bit-wide computation could be updating 64 very coarse-grained weights at a time.

vegabook 10y ago

disingenuous to use a highly charged term like "blogspam" and then qualify it as "kinda". kettle/black. Why say "blogspam" if you don't mean it? And while I agree that the OP is not indepth, it is not blogspam. AVX512 is a very real, very very real, order of magnitude, performance improvement. To give you an idea, going from standard 64-bit to AVX-256 with MKL Numpy (under ipython) you will go from 6-7 seconds, to under 1 second, on an identical 2012+ processor, on this code: import numpy as np np.random.seed(1234) xx = np.random.rand(1000000).reshape(1000, 1000) %timeit np.linalg.eig(xx) The best way to test this is to run the above on bog-standard Ubuntu 16.04 (sudo apt-get install python(3)-pip / sudo pip(3) install numpy first), and then run it again after installing Anaconda Python. AVX512 will halve this timing again, and before you tell me that Cuda will do this much faster, I'll remind you that there is a point at which 4000 cuda stream processors operating on 64bit data can, in many algos, be a vastly too large window. Phi KL's 72 branching cores with 4 threads and 2x512-bit AVX each, can be a very strong competitor to the flatspace GPUs in nonlinear algos. There is clear blue sky between the scalar architectures and the massively parallel architectures, which has not been targeted yet and this is where KL fits in. After all, the GPUs are still, first and foremost, designed to rotate and translate massive-amounts of triangles in 3d shooters, a task for which a vast number of very dumb little cores is optimal. As much as Nvidia make like you to believe different, that is still their primary market and it doesn't always map well to everything. Between the current extremes of a small number of very flexible and clever cores (xeon/i7), and 4000 very stupid but numerous cores (GPU), there is a lot of room something in between. Intel isn't so much late to the party as attacking an obliquely different segment of the market, which may prove very interesting indeed.

wpietri 10y ago

> On Intel's part, it seems kinda... late? It's like adding Bitcoin instructions when you already know everyone's racing to make Bitcoin ASICs. Nah. You're thinking they have to be the best, that they have to win. That is true when the choice is buying an Intel CPU vs buying a GPU. But most developers will already have a late-model Intel CPUs around. For Intel, this is a defensive move. They're bending the performance curve for particular use cases. All this has to do is to keep a certain number of developers from bothering to go the extra mile to use special hardware and software. People working at Google's scale won't care. But there will be other people who try running something and say, "Current hardware's fine. We could do better with GPUs, but not so much better that it's worth the time and money." I'm sure it's also the start of a learning cycle for them. Some people told them their current processors were not enough, so they added this. They'll now hear why this isn't adequate either and adapt again if the business case is there.