11 ms·
Intel will add deep-learning instructions to its processors
- visarga 10y ago> Intel is hoping that the Xeon+FPGA package is enticing enough to convince enterprises to stick with x86, rather than moving over to a competing architecture (such as Nvidia’s Tesla GPGPUs ). Then let it prove its performance on a few deep learning benchmarks. If is so fast and accessible, we'll be impressed.
- pritambaral 10y agoHugged. Google cache link: http://webcache.googleusercontent.com/search?q=cache:lemire.me/blog/2016/10/14/intel-will-add-deep-learning-instructions-to-its-processors/&ie=utf-8&oe=utf-8&client=firefox-b-ab&gws_rd=cr&ei=yg0BWIaKIIzVvgSfr4uIDQ http://webcache.googleusercontent.com/search?q=cache:lemire....
- sbierwagen 10y agoBlogspam, kinda. Post just links to https://software.intel.com/sites/default/files/managed/69/78/319433-025.pdf https://software.intel.com/sites/default/files/managed/69/78... and says it mentions two new instructions: AVX512_4VNNIW (Vector instructions for deep learning enhanced word variable precision) and AVX512_4FMAPS (Vector instructions for deep learning floating-point single precision) On Intel's part, it seems kinda... late? It's like adding Bitcoin instructions when you already know everyone's racing to make Bitcoin ASICs. How could it beat dedicated hardware, or even GPUs, on ops/watt? Maybe it's intended for inference, not training, but that doesn't sound compelling either.
- ninjin 10y agoI am not so sure if it is "late". Sure, for training models it is far too late, we have semi-dedicated hardware for that at this point. But it is possible that it will become increasingly common to run these models on end-user systems and in that case these instructions could provide a bit of a speed boost.
- valarauca1 10y ago>>On Intel's part, it seems kinda... late? Not exactly AVX512 is used on the Xeon Phi which is Intel's Floating Co-Processor for Deep Learning/OpenCL tasks. But that being said, GPU's blow the Xeon Phi out of the water in terms of FLOPS, local RAM, and RAM bandwidth.
- astrodust 10y agoIsn't Phi better at dealing with branch-heavy code? GPUs absolutely kill on straight-forward compute, but they seem to flail helplessly if the logic is highly conditional. I've read that in some cases GPUs evaluate both branches of a condition and discard the unwanted results. A CPU doesn't do this.
- valarauca1 10y ago>>Isn't Phi better at dealing with branch-heavy code? This maybe true. Generally code that is infinitely parallelize to be ran on a GPU isn't going to be branch heavy. There are no benchmarks to support this, but I would believe it is true. GPU's typically have 1 branching unit per 64-128threads. While the Phi has 1 branching unit per 2 threads. The real difference is GPU's use SIMT, while Phi uses MIMD. Each Phi thread is a POSIX thread. It can do its own thing, live its own life. While GPU's you just specify how parallel your work-group, thread block, warp, or wave front is (name depends on platform, in order OpenCL, CUDA, Nvida DX12, AMD DX12). >>I've read that in some cases GPUs evaluate both branches of a condition and discard the unwanted results. A CPU doesn't do this. Intel CPU's have done this since SSE4.
- astrodust 10y agoIf you mean branch prediction, then yes, it will end up inadvertently executing the wrong instructions and discard them, but it halts execution of the wrong branch at the first opportunity and reschedules to correct its mistake. A GPU will execute both branches to completion.
- jasonwatkinspdx 10y ago
- obmelvin 10y agoAs the other poster mentions, I assume their view is that these types of operations will become increasingly prevalent in consumer software applications. Lots will be powered by the cloud, but it would be useful if some inference can be done on a laptop CPU.
- Eridrus 10y ago"The cloud" still has to run the models on something, and as long as Intel roughly keeps up with the cost of specialised hardware for doing inference, many people will just keep doing inference on Intel CPUs, rather than bothering with specialised hardware.
- obmelvin 10y agoThat's true... I'd been assuming they'd use specialized hardware for inference when writing that, but totally failed to think how cost may make a CPU more desirable.
- dsabanin 10y agoIntel is also adding FPGAs[1] right next to their CPUs, which can possibly become a next big thing in HPC. [1] http://www.extremetech.com/extreme/184828-intel-unveils-new-xeon-chip-with-integrated-fpga-touts-20x-performance-boost http://www.extremetech.com/extreme/184828-intel-unveils-new-...
- yid 10y agoThis aspect of their new chips is massively underrated. An FPGA is the future-proof solution here, not chip-level instructions for the soup-du-jour in machine learning. Edit: which is not to say that I'm not welcoming the new instructions with open arms...
- greendragon 10y agoIt will only really take off if they can get the user experience of getting a HDL program to the FPGA to the same level as getting a shader program to the GPU. Unless they can do all that in microcode though it's going to force them to open up some stacks of Altera's closed processes. I'm hopeful but there's a lot of proprietary baggage around FPGAs that I think have kept them from truly reaching their potential.
- yid 10y ago> I'm hopeful but there's a lot of proprietary baggage around FPGAs that I think have kept them from truly reaching their potential. I don't really know much about this aspect, could you elaborate? I'm genuinely curious.
- aseipp 10y agoAlmost all FPGAs have entirely proprietary toolchains, from Verilog/HDL synthesis all the way down to bitstream loaders for programming your board. These tools are traditionally very poorly built as applications, scriptable only with things like TCL, terrible error messages, godawful user interfaces designed in 1999, etc etc.[1] This makes integrating with them a bit more "interesting" for doing things like build system integration, etc. Furthermore, FPGAs are inherently much more expensive for things like reprogramming, than just recompiling your application. Synthesis and place-and-route can take excessive amounts of time even for some kinds of small designs. (And for larger designs, you tend to do post-synthesis edits on the resulting gate netlists to fix problems if you can, in order to avoid the horrid turn around time of hours or days for your tools). The "flow" of debugging, synthesis, testing etc is all fundamentally different and has a substantially different kind of cost model. This means many kind of "dream" ideas of on-the-fly reconfiguration "Just-in-Time"-ing your logic cells, based on things like hot, active datapaths -- is a bit difficult for all but the simplest designs, due to the simple overhead of the toolchain and design phase. That said, some simple designs are still incredibly useful! What's more likely with these systems is you'll use, or develop, pre-baked sets of bitstreams to add in custom functionality for customers or yourselves on-demand, e.g. push out a whole fleet of Xeons, configure the cells of machines on the edge to do hardware VPN offload, configure backend machines that will run your databases with a different logic that your database engine can use to accelerate queries, etc. Whether or not that will actually pan out remains to be seen, IMO. Large enterprises can definitely get over the hurdles of shitty EDA tools and very long timescales (days) for things like synthesis, but small businesses that may still be in a datacenter will probably want to stick with commercial hardware accelerators. Note that I'm pretty new to FPGA-land, but this is my basic understanding and impressions, given some time with the tools. I do think FPGAs have not really hit their true potential, and I do think the shit tooling -- and frankly god awful learning materials -- has a bit to do with it, which drives away many people who could learn. On the other hand: FPGAs and hardware design is so fundamentally different for most software programmers, there are simply things that, I think, cannot so easily be abstracted over at all -- and you have to accept at some level that the design, costs, and thought process is just different. [1] The only actual open-source Verilog->Upload-to-board flow is IceStorm, here: http://www.clifford.at/icestorm/ http://www.clifford.at/icestorm/. This is only specifically for the iCE40 Lattice FPGAs; they are incredibly primitive and very puny compared to any modern Altera or Xilinx chip (only 8,000 LUTs!), but they can drive real designs, and it's all open source, at least. Just as an example, I've been learning using IceStorm, purely because it's so much less shitty. When I tried Lattice's iCE40 software they provided, I had to immediately fix 2 bash scripts that couldn't detect my 4.x version kernel was "Linux", and THEN sed /bin/sh to /bin/bash in 20 more scripts __which used bash features__, just so that the Verilog synthesis tool they shipped actually worked. What the fuck. Oh! Also, this was after I had to create a fake `eth0` device with a juked MAC address (an exact MAC being required for the licensing software), because systemd has different device names for ethernet interfaces based on the device (TBF, for a decent reason - hotplugging/OOO device attachment means you really want a deterministic naming scheme based on the card), but their tool can't handle that, it only expects eth0.
- ericjang 10y agoNot exactly late. There's no physical law that says a CPU can't beat a GPU for a certain family of operations, and in fact in practice, small toy networks with non-dense operations (for instance, if you are doing research on optimization algorithms for DNNs or Bayesian inference algorithms) tend to run faster on the CPU than GPU.
- adevine 10y agoI am not familiar with how deep learning algorithms would benefit from these instructions in a way different from how they can just use vector instructions optimized for graphics processing. So my question is, how are these "Vector instructions for deep learning enhanced word variable precision" different from other vector instruction sets?
- sbierwagen 10y agoGood question. It doesn't say! The PDF has absolutely no detail on what it actually does. Googling the instruction brings up people updating CPUID fields, but nothing to do with actually using it.
- p1esk 10y agoI'm guessing it means you can choose the precision (word length) of the computations, eg 8 bits, 16 bits, 32 bits, etc. This probably only applies to integer ops, but if they could let us pack FP16 operands into 512 bit vectors, that would be awesome!
- rspeer 10y agoI hope they're going even farther and considering minifloats. 8-bit integers are easy but limiting and unexciting. 8-bit floats are exactly what some machine learning needs. If you're updating your weights enough, precision can just be wasted computation. A 512-bit-wide computation could be updating 64 very coarse-grained weights at a time.
- vegabook 10y agodisingenuous to use a highly charged term like "blogspam" and then qualify it as "kinda". kettle/black. Why say "blogspam" if you don't mean it? And while I agree that the OP is not indepth, it is not blogspam. AVX512 is a very real, very very real, order of magnitude, performance improvement. To give you an idea, going from standard 64-bit to AVX-256 with MKL Numpy (under ipython) you will go from 6-7 seconds, to under 1 second, on an identical 2012+ processor, on this code: import numpy as np np.random.seed(1234) xx = np.random.rand(1000000).reshape(1000, 1000) %timeit np.linalg.eig(xx) The best way to test this is to run the above on bog-standard Ubuntu 16.04 (sudo apt-get install python(3)-pip / sudo pip(3) install numpy first), and then run it again after installing Anaconda Python. AVX512 will halve this timing again, and before you tell me that Cuda will do this much faster, I'll remind you that there is a point at which 4000 cuda stream processors operating on 64bit data can, in many algos, be a vastly too large window. Phi KL's 72 branching cores with 4 threads and 2x512-bit AVX each, can be a very strong competitor to the flatspace GPUs in nonlinear algos. There is clear blue sky between the scalar architectures and the massively parallel architectures, which has not been targeted yet and this is where KL fits in. After all, the GPUs are still, first and foremost, designed to rotate and translate massive-amounts of triangles in 3d shooters, a task for which a vast number of very dumb little cores is optimal. As much as Nvidia make like you to believe different, that is still their primary market and it doesn't always map well to everything. Between the current extremes of a small number of very flexible and clever cores (xeon/i7), and 4000 very stupid but numerous cores (GPU), there is a lot of room something in between. Intel isn't so much late to the party as attacking an obliquely different segment of the market, which may prove very interesting indeed.
- wpietri 10y ago> On Intel's part, it seems kinda... late? It's like adding Bitcoin instructions when you already know everyone's racing to make Bitcoin ASICs. Nah. You're thinking they have to be the best, that they have to win. That is true when the choice is buying an Intel CPU vs buying a GPU. But most developers will already have a late-model Intel CPUs around. For Intel, this is a defensive move. They're bending the performance curve for particular use cases. All this has to do is to keep a certain number of developers from bothering to go the extra mile to use special hardware and software. People working at Google's scale won't care. But there will be other people who try running something and say, "Current hardware's fine. We could do better with GPUs, but not so much better that it's worth the time and money." I'm sure it's also the start of a learning cycle for them. Some people told them their current processors were not enough, so they added this. They'll now hear why this isn't adequate either and adapt again if the business case is there.
- joelthelion 10y agoSo Intel CPUs will be able to perform 16 operations at a time, when Nvidia can do thousands?
- sitkack 10y agoLatency
- j1vms 10y ago> Latency This. Also, trying to slow the loss of customers purchasing what are essentially Nvidia GPU-heavy systems with Nvidia CPUs tacked on to act as the controllers.
- wmu 10y agoOn Knights Landing CPU a single core is be able to perform 2 x 16 operations on single-precision numbers, as a core has two AVX512 execution units. Each core can run 4 thread, and KNL has 64 physical cores.
- antirez 10y agoThat's a good news! About that, two days ago I modified the implementation of neural networks inside Neural Redis in order to use AVX2. It was a pretty interesting experience and in the end after modifying most of it, the implementation is 2x faster compared to the vanilla C implementation (already optimized to be cache obvious). I never touched AVX or SSE in the past, so this was a great learning experience. In 30 minutes you can get 90% of it, but I think that to really do great stuff you need to also understand the relative cost of every AVX operation. There is an incredible user at Stack Overflow that replies to most AVX / SSE questions, if you check the AVX / SSE tag you'll find it easily. However I noticed that when there were many load/store operations to do, there was no particular gain. See for example this code: #ifdef USE_AVX __m256 es = _mm256_set1_ps(error_signal); int psteps = prevunits/8; for (int x = 0; x < psteps; x++) { __m256 outputs = _mm256_loadu_ps(o); __m256 gradients = _mm256_mul_ps(es,outputs); _mm256_storeu_ps(g,gradients); o += 8; g += 8; } k += 8*psteps; #endif What I do here is to calculate the gradient after I computed the error signal (error * derivative of the activation function). The code is equivalent to: for (; k < prevunits; k++) *g++ = error_signal*(*o++); Any hint about exploiting AVX at its max in this use case? Thanks. Ok probably this was more a thing for Stack Overflow, but too late, hitting enter.
- dfbrown 10y agoI would try unrolling 2-4 iterations of the loop. Multiple sequential loads isn't much slower than a single load, so batching your loads and stores together will let you do more arithmetic operations for each time you hit memory.
- YZF 10y agoIt might help to try and squeeze more work between the load and the store. E.g. do two of these in parallel (load/load/mul/mul/store/store). Think about latency vs. throughput, the load and the stores may have high latency but also high throughput so you need to be issuing more of them concurrently (without any dependencies between them). Just off the top of my head.
- happycube 10y agoNot holding my breath, since they haven't gotten AVX512 into desktops yet! (Kaby Lake might be worth something if they did... but nope, just more lie7's mostly...)
- unsignedqword 10y agoCool stuff, but unfortunately Intel has really delayed AVX512 instructions for their main consumer processors (ffs, it was supposed to hit on Skylake). It looks like we have another die shrink to go after Kaby Lake before we get that sweet ultra-wide SIMD: https://en.wikipedia.org/wiki/Cannonlake https://en.wikipedia.org/wiki/Cannonlake
- p1esk 10y agoSkylake Xeons are still on track to have AVX512, last I checked.
- jostmey 10y agoHow much will this help? Neural networks work so well on distributed computing systems because the operations run in parallel. With a neural network, you can leverage tens of thousands of separate computing cores. Top intel processors have around a dozen cores or so. What am I missing here?
- wmu 10y agoXeon Phi have got more cores, 50 or 60, and the cores share memory. Unlike GPUs memory space is uniform, there is no need to do explicit transfers to/from GPU's memory.
- arcanus 10y ago> Xeon Phi have got more cores, 50 or 60, and the cores share memory 60 cores. Furthermore, the cores have multiple (4) hardware threads per core. So on the latest Phi (Knights Landing) you can have several hundred threads (240) running concurrently without contention (in principle).
- steego 10y ago> What am I missing here? There may be a market sweet spot between dedicated high-performance machine learning that's ideal for experts who are willing to buy/lease dedicated hardware and regular developers who want to use simpler machine learning toolkits to solve easier problems. Toolsets like Julia, NumPy, R, etc would take advantage of the new instruction and basically give your average a speed boost for simply having an Intel PC.
- eb0la 10y agoThis reminds me when you had to buy crypto accelerator cards for your webservers (does anyone remember Soekris?)... The market stalled after Intel started adding support for crypto operations on-die. This looks like Intel is protecting his server chip business shooting at nVidia before they start selling Titan-X with other server chip (what chip - I don't know; but I bet there's a business plan on a spreadsheet somewhere). I guess training neural nets will a GPUs / custom chips task for some time; but as soon as you have developed the classifier / predictor you have to run it on whatever hardware you have and that means you might need less parallel computing power and it doesn't make (financial) sense buying GPUs for that.
- zenobit256 10y agoGreat, another extension to the ever-growing instruction x86 instruction set... I'd question why, but at this point, we're just going to keep bolting things on to an existing architecture. "Microcode it. Why not?"
- partycoder 10y agoYou can also try a binarized neural network. https://arxiv.org/abs/1602.02830 https://arxiv.org/abs/1602.02830
- deleted 10y ago[deleted]
- eveningcoffee 10y agoI did tests with Torch7 few days ago to compare it with OpenBLAS library on multiple cores against CUDNN. Single core difference was around 100x. Using multiple cores improved the situation but there still was about 15x difference that did not improve by adding more cores (I tested with up to Xeon 64 cores on a single machine). I did not test with Intel MKL library as I did not have time for this. I wonder how much these instructions would improve the situation and does anybody here have experience with Intel MKL library.
- AvenueIngres 10y agoWhat are the implications of this? Faster processing?
- gragas 10y agoTL;DR: You can basically do 8 operations at once (but of course, it's not that simple). Think about a normal 64-bit register: you can add and subtract 64-bit numbers, OR and AND them, etc. Now think about a 64-byte register. What could you do with that? Well, suppose you're looping through contiguous memory and changing each word AND the change that your making to each word is independent of the change you make to any other word. With a normal 64-bit register you'd have to do one operation on each word. But with a 64-byte "register", you can load 8 words into it and do just one operation for the same effect as applying that operation 8 times (once to each word). Thus, some code---vectorizable code---can be sped up nearly 8x.
- zbjornson 10y agoReally curious how Intel (others?) determined that these new instructions are for deep learning. The instructions are for permutation, packed multiply+low/high accumulate, and a sort of masked byte move. Are these super common in deep learning?
- p1esk 10y agoDeep learning is all about GEMM operations: matrix-matrix, or matrix-vector multiplications. And the values in these matrices are typically low precision (16 or even 8 bits is enough). So if you can pack many of such low precision values into vectors, and perform a dot product in parallel, you will get a pretty much linear speedup.
- schappim 10y agoThere is actually an undocumented neural net engine within the Intel chip on the Arduino 101.
- schappim 10y agoI guess it's documented now: http://www.general-vision.com/products/curieneurons/ http://www.general-vision.com/products/curieneurons/
- yongjik 10y agoApologies if I'm dumb, but did Intel actually tell exactly what operations these "deep learning" instructions do? I skimmed through the linked Intel manual, but it seems that it just defines two instruction family names (AVX512_4VNNIW and AVX512_4FMAPS) without actually saying what they do. * It almost feels like a marketing term, in the same way everything was "multimedia instructions" back in the 90s.
- iaw 10y agoIt's been a long time since I look at machine language but, if I understand correctly, the difference is the capacity of the operation. Instead of performing an addition operation on 1 or 2 bytes at a time the operation can be performed on 64 bytes at a time. Given that 'deep learning' requires a number of parallel simple operations expanding the volume of parallel operations in a single CPU instruction boost capacity. Seriously, just supposition until someone with better understanding chimes in.
- bradhe 10y agoheh, if this is the case then we're going to benefit in a lot of areas from this.
- denim_chicken 10y agoI hope they will be more useful than MPX.
- ktamiola 10y agoSounds good to me. Let's hope this is not just a marketing move.
- betolink 10y agoIs this the moment when Sarah Connor shows up to ruin the party?
- rosstex 10y agoCould someone explain how these are deep-learning optimized? All I see is that they're part of the 512-bit vector family that has previously existed.
- tomrod 10y agoWell. I guess I'm going to be using Intel!
- gragas 10y agoIsn't this basically just wider SIMD? Is this in anyway actually tailored specifically to deep learning?
- sunstone 10y ago"get me a beer' would be an appreciated instruction to add.
- thasaleni 10y agoExcuse my ignorance, but isn't "deep learning instructions" an oxymoron