4 ms·
> At this point, Intel doesn’t have bfloat16 implemented in any of its processors, so they used current AVX512 vector hardware present in its existing processor
by eatbitseveryday 7y ago
> At this point, Intel doesn’t have bfloat16 implemented in any of its processors, so they used current AVX512 vector hardware present in its existing processor to emulate the format and the requisite operations. According to the researchers, this resulted in “only a very slight performance tax.”
Why implement bfloat if you get just slightly less performance emulating it with AVX512, which already exists? Maybe it’s an “us too” claim?
- new_realist 7y agoHow would the researchers know how fast the native implementation would be? By making it architectural Intel can optimize it in future generations.
- jchw 7y agoThey did not specify what the tax was relative to. Maybe they meant relative to 32-bit float? AVX512 is expensive. I believe if you have an AVX512-heavy workload it can cause the processor to throttle.
- smitty1110 7y agoYes, there’s a bios setting to control this. It basically under clocks the core while AVX units are under load.
- celrod 7y agoTaking my 7980xe as an example: When it runs non-avx512 loads, I currently have it set to run at 4.1 GHz (all-core). When running avx-512 heavy loads, it instead runs at 3.6 GHz -- and tends to get much hotter (70-80C instead of 50-60C). 3.6 GHz is a mild overclock; Silicon Lottery reports 100% can achieve that speed for avx512 loads.[1] Running programs doing the same thing (eg, Hamiltonian Monte Carlo where the likelihood function has or has not been vectorized), the avx512 version is far faster than scalar, and routinely 50%+ faster than avx2. The avx512 instruction set itself also provides conveniences that make it easier to explicitly vectorize, even if most compilers don't take advantage of them on their own. Masking load and store operations in particular (they're better about masking to handle branches). On why avx512 vs a graphics card: I need double precision, and my code routinely has maximum widths smaller than the 32 or 64 a graphics card would want to computer in parallel. [1] https://siliconlottery.com/pages/statistics https://siliconlottery.com/pages/statistics
- m0zg 7y agoYeah, people tend to completely exaggerate the impact of throttling from AVX512. It's only an issue when you do short bursts of AVX512 and the rest is not AVX512. If you do math and your math can be done in AVX512, even with throttling it's going to be substantially faster. That it runs hotter doesn't concern me at all. Intel's claimed safe Tjunction is something like 105C. EEs tend to take the published component specifications seriously (e.g. your 1000v diode is guaranteed to withstand at least 1KV of reverse voltage), so I trust Intel when they say things are fine up to that temperature. Even beyond that it won't burn out, it'll just thermal throttle.
- magicalhippo 7y agoMaximum Tjunction for an STM32F303 (just happened to have datasheet open) is 150C, as is most other ICs I've seen. So is 105C just a very conservative number, compensating for the probe location, or are there process specific things which brings it down to 105C?
- userbinator 7y agoFrom what I understand, the newer very-high-density procsses are far more sensitive to voltage and temperature than the older larger ones.
- magicalhippo 7y agoMakes sense. The STM32G series, which still has 150C Tjmax, is ST's first 90nm MCU[1] so yeah. [1]: https://blog.st.com/stm32g0-mainstream-90-nm-mcu/ https://blog.st.com/stm32g0-mainstream-90-nm-mcu/
- marmaduke 7y agoWhat are you using to vectorize avx512 for HMC? Do you have a lot of element wise ops on big arrays? When running Stan (NUTS/HMC) on Xeon Phi, telling Eigen to use avx512 provided a noticeable speed up but I didn't look at the assembly to be sure.
- kccqzy 7y agoAll the newly added instructions, (VCVTNE2PS2BF16, VCVTNEPS2BF16, VDPBF16PS) to support BF16 are AVX512 instructions.
- kccqzy 7y agoI guess if it's easy enough to do with software, it won't be very difficult to implement with hardware either. So it doesn't really cost much additional die space.
- nagarajns 7y agoassssd
- justincormack 7y agoAs the exponent is the same size it is mostly a matter of truncating. However custom hardware could have ability to do more parallel 16 bit ops in avx mode. Presumably they are emulating by expanding to 32 bit in avx512 registers.
- zeckalpha 7y agoThere’s a tradeoff between neural net performance and CPU performance.
- Someone 7y agoThe reason to use float16 isn’t to make individual operations faster, its to fit more numbers in vector registers. For example, AVX512 vector registers are 512 bits. That’s 8 64-bit floats, 16 32-bit ones, or 32 16-bit ones. => A vector multiply using bfloat16 may not be much faster than one using float32, but it will do more multiplications.
- tomxor 7y ago> The reason to use float16 isn’t to make individual operations faster, its to fit more numbers vector registers. For example, AVX512 vector registers are 512 bits. That’s 8 64-bit floats, 16 32-bit ones, or 32 16-bit ones. I'm unclear on the advantage you are trying to explain. If both AVX and bfloat are SIMD instructions that cannot be the reason implementing bfloat is better. I'm expecting something like "bfloat16 is more specialised so it can have larger registers" or something? [Edit] Sorry re-reading your comment, I think you are trying to say that. The key part being: > fit more numbers vector registers (vs AVX i assume), so adding bfloat16 would provide more registers vs AVX with similar gate usage due to greater specialization?
- Someone 7y agoNot larger vector registers, smaller numbers, so that you can fit more numbers in a vector register without having to make the vector register larger. For CPU-bound algorithms, one would expect that bfloat16 in 512 bit vector registers would be about equal in speed to float32 in (hypothetical) 1024 bit vector registers. Also, for algorithms that are memory-bandwidth bound, halving the size of your numbers will (about) halve memory pressure.
- tomxor 7y ago> Not larger vector registers, smaller numbers, so that you can fit more numbers in a vector register without having to make the vector register larger. Sorry i'm pretty ignorant of AVX so trying to understand... is this because the smallest word size in AVX is 32bit? compared to bfloat16 is using twice the register space? Or rather with the same register space in bfloat16 you can have twice the numbers? (with no negative effects on convergence in NN due to exponent size.)