10 ms·
Parsing JSON faster with Intel AVX-512
- beached_whale 4y agoIf only intel wasnt dropping support for it on a lot of cpus
- nomel 4y agoWell, if it means more cores, it's almost certainly worth it, in the grand scheme of things.
- jrimbault 4y agoI'm using this comment as jumping point. What's cost/opportunity to optimizing for a specific platform/instruction set ? At what point is it worth doing, when isn't it worth doing ? AVX-512 strikes as something... "ephemeral".
- skavi 4y agowriting code for SIMD can get you absolutely massive performance improvements. Whether it's worth the added complexity depends on the situation. If your data is already arranged in a cache friendly way (SoA), it shouldn't be incredibly difficult to use SIMD intrinsics to optimize. I'd first take a look at what the compiler is already generating for you to see if manual intervention is worth it.
- throwaway92394 4y agoWell I mean this article is demoing a 28% improvement (if I did my math right) for json parsing. Sure AVX-512 is only applicable to specific workloads, and even many of those workloads the cost/opportunity of optimizing for AVX-512 might not be worth it. But there clearly ARE usecases that would benefit, and it might be worth it for more consumer applications to optimize for AVX-512 - but only if it can be used. The way I see it is that the benefit of optimizing for AVX-512 is far higher if it becomes normal for consumer CPUs to have it. A 28% improvement is pretty decent, but it's only worth implementing if enough people can utilize it.
- beached_whale 4y agoFor many maybe not, but when writing the foundations of software it is good to start fast. There are libraries that abstract various simd architectures now too. Simdjson has their own and there are ones lile KUMI
- jcranmer 4y agoIntel is not dropping support for it on a lot of CPUs. The only thing they've done is disable it in the hybrid Alder Lake cores, presumably because the E-cores couldn't support it (while the P-cores could), and they didn't want to deal with the headaches of ISA extensions being supported only on some cores in the system.
- beached_whale 4y agoAh, headlines foiled me. I read disabled in Alder lake all together.
- temac 4y agoIt is disabled in all consumer Alder lake (and I don't remember if there will be Xeon of that gen with P-core only -- IIRC Intel stop the AVX512 validation late on those cores, but it was still before it was formally finished, so probably not). At one point it worked with some Bios on P-core only chips or if you disabled E-core on hybrid ones, but with up-to-date Intel microcode it does not work anymore.
- Aardwolf 4y ago> Intel is not dropping support for it on a lot of CPUs. There are 0 current generation consumer CPUs of neither Intel nor AMD that have it > The only thing they've done is disable it in the hybrid Alder Lake cores Which happen to be all the current generation Intel CPUs
- coder543 4y ago> The only thing they've done is disable it in the hybrid Alder Lake cores That is incorrect. You can buy Alder Lake CPUs that only have one type of core (the i3 series only has P-cores, for example), and those do not support AVX-512 either. They're not "hybrid" in any way. Some of their motherboard partners initially allowed you to access AVX-512, but Intel has put a stop to this and the feature is disabled on all Alder Lake CPU SKUs, period. Newer Alder Lake chips have AVX-512 fused off in silicon, if the firmware blob disabling it isn’t enough: https://www.tomshardware.com/news/intel-nukes-alder-lake-avx-512-now-fuses-it-off-in-silicon https://www.tomshardware.com/news/intel-nukes-alder-lake-avx... > Intel is not dropping support for it on a lot of CPUs. That seems like a pretty questionable statement. Intel might keep AVX-512 around for Xeon, but it seems extremely dead on the consumer market. If Intel decides to bring it back for the next generation, that would be strange and very poor planning.
- timerol 4y ago> Of course, to get these new benefits, you need recent Intel processors with adequate AVX-512 support and, evidently, you also need relatively recent C++ processors. Some of the recent laptop-class Intel processors do not support AVX-512 but you should be fine if you rely on AWS and have big Intel nodes. What is meant by "relatively recent C++ processors"? Is that supposed to be "compilers"?
- deleted 4y ago[deleted]
- Narishma 4y agoIt's supposed to be Intel, not C++.
- NegativeLatency 4y ago"new" is relative since they've been out for almost 10 years: https://www.intel.com/content/www/us/en/developer/articles/technical/intel-avx-512-instructions.html https://www.intel.com/content/www/us/en/developer/articles/t...
- jeffbee 4y agoThis code uses VBMI2, which just came out quite recently.
- skavi 4y agoHopefully we'll see AVX-512 in Intel's little cores soon. Centaur's last CPU architecture proves that it is possible to implement the extension without a huge amount of area [0]. Once that happens, I expect we'll finally consistently see AVX-512 on new Intel processors. The masks really are a huge improvement to the design. AMD should be implementing AVX-512 on their own cores soon as well. Once Armv9 (with SVE2) becomes dominant, we'll pretty much be in a golden age of SIMD. [0]: https://chipsandcheese.com/2022/04/30/examining-centaur-chas-die-and-implementation-goals/ https://chipsandcheese.com/2022/04/30/examining-centaur-chas...
- dragontamer 4y ago> we'll pretty much be in a golden age of SIMD. We already are in the golden age of SIMD. NVidia and AMD GPUs are easier and easier to program through standard interfaces. Intel / AMD are pushing SIMD on a CPU, which is useful for sure, but always is going to be smaller in scope than a dedicated SIMD-processor like A100, 3060, AMD Vega, AMD 6800 xt and the like. SIMD-on-a-CPU is useful because you can perform SIMD over the L1 cache as communication (rather than traversing L1 -> L2 -> L3 -> DDR4 / PCIe -> GPU VRAM -> GPU Registers -> SIMD, and back). But if you have a large-scale operation that can work SIMD, the GPU-traversal absolutely works and is commonly done.
- skavi 4y agoGood point. Should have clarified I was referring to CPU SIMD.
- dragontamer 4y agoAVX2 is not as good as AVX512. But AVX2 still has vgather instructions, pshufb, and a few other useful tricks. AVX512 and ARM SVE2 bring the CPU up to parity with maybe 2010s-era GPUs or so (full gather/scatter, more permutation instructions, etc. etc.). But GPUs continued to evolve. Butterfly-shuffles are the generic any-to-any network building block, and are exposed in PTX (NVidia assembly) shfl.bfly, and AMD DPP (data-parallel primitives). Having a richer set of lane-to-lane shuffling (especially ready-to-use butterfly networks) would be best. It really is surprising how many problems require those rich-sets of data-movement instructions, or otherwise benefit from them. NEON and SVE had hard-coded data-movement for specific applications. The general-purpose instruction (pshufb) is kinda like permute/shfl from AMD/NVidia. A backwards-permute IIRC doesn't exist yet on CPU-side. And butterfly networks are the general-purpose solution, capable of implementing any arbitrary data-movement in just log(width) steps. (pshufb / permute instructions would be the full-sized butterfly network, but some cases might be "easier" and faster to execute with only a limited number of butterfly swaps, such as what inevitably comes up in sorting) -------- Still, all of these operations can be implemented in AVX2 (albeit slower / less efficiently). So its not like the "language" of AVX2 / AVX is incomplete... its just missing a few general-purpose instructions that could lead to better performance.
- PragmaticPulp 4y ago> Could we do better? Assuredly. There are many AVX-512 that we are not using yet. We do not use ternary Boolean operations (vpternlog). We are not using the new powerful shuffle functions (e.g., vpermt2b). We have an example of coevolution: better hardware requires new software which, in turn, makes the hardware shine. > Of course, to get these new benefits, you need recent Intel processors with adequate AVX-512 support AVX-512 support can be confusing because it’s often referred to as a single instruction set. AVX-512 is actually a large family of instructions that have different availability depending on the CPU. It’s not enough to say that a CPU has AVX-512 because it’s not a binary question. You have to know which AVX-512 instructions are supported on a particular CPU. Wikipedia has a partial chart of AVX-512 support by CPU: https://en.wikipedia.org/wiki/AVX-512#CPUs_with_AVX-512 https://en.wikipedia.org/wiki/AVX-512#CPUs_with_AVX-512 Note that some instructions that are available in one generation of CPU are can actually be unavailable (superseded, usually) in the next generation of the CPU. If you go deep enough into AVX-512 optimization, you essentially end up targeting a specific CPU for the code. This is not a big deal if you’re deploying software to 10,000 carefully controlled cloud servers with known specifications, but it makes general use and especially consumer use much harder.
- mikepurvis 4y agoAre there good libraries for doing runtime feature detection? Eg, include three versions of hot function X in the binary, and have it seamlessly insert the correct function pointer at startup? Or have the function contain multiple bodies and just JMP to the correct block of code? I know you can do this yourself, but last time I looked it was a heavily manual process— you had to basically define a plugin interface and dynamically load your selected implementation from a separate shared object. What are the barriers to having compilers able to be hinted into transparently generating multiple versions of key functions?
- colejohnson66 4y agoCheck out Agner Fog's vectorclass library: https://github.com/vectorclass/version2 https://github.com/vectorclass/version2
- 4y ago
- bfrog 4y agoAt what point is JSON not the right option? Surely when trying to do this sort of thing? At what point is it saner to use something like flatbuffers or capnproto style message encoding instead.
- vardump 4y agoSometimes you just don't have a choice when you need to interface with a third party data feed or software. Isn't it better to have all options open?
- avg_dev 4y agoGood thought. If you are coding in C++ maybe you can use some sort of binary serialization thing. Even in other languages if json parsing is a bottleneck it can possibly be optimized away through use of a binary wire format. That said vector operations available to programmers is always a welcome thing I’d say. And who knows how much production json parsing this library really does, it could be a ton. I’m torn. I’ve worked at shops where we aim over time to reduce response time while serving business logic and using statistical models that get iterated on. Even there I haven’t seen a blatant need for non-JSON rpc. But I know my experience doesn’t mirror everyone’s. And I like seeing and learning about instruction sets. I’m currently taking a course in parallel computing and I just used a avx2 for the first time in a toy program to subtract one vector from another in a single instruction which while not particularly useful is a window into more interesting things and is still SIMD. I think on the whole making json parsing for a large enough fraction of processors is probably a huge win for the environment. But who is parsing json in C++?
- ollien 4y ago> But who is parsing json in C++? Well, Facebook for one! Folly has lots of utilities for this (see folly::dynamic[1]). We make extensive use of this at my (non-Facebook) job. [1] https://github.com/facebook/folly/blob/master/folly/docs/Dynamic.md https://github.com/facebook/folly/blob/master/folly/docs/Dyn...
- refulgentis 4y agoProbably a lot of people. JSON is a pretty popular interchange format.
- pabs3 4y agoI wonder if simdjson is going to replace the native JSON parsing in various languages at some point.
- worewood 4y agoUsing specialized instructions not always turn into performance improvements. Processors are pretty smart these days and the generated u-ops may be the same
- incrudible 4y agoIn my experience, even some of the simplest loops that a compiler can trivially vectorize do get a noteworthy speedup from using SIMD instructions.
- Const-me 4y agoUpgrading from scalar to SSE vector, I often see an improvement by a large factor like 3-4. Sometimes, for smaller integer lanes or bytes, much more. When upgrading vectors from SSE to AVX, the speedup is very rarely by a factor of 2. More often it's within 30-70%. I never programmed AVX512, but I would expect for real-life code AVX512/AVX2 performance improvement to be much closer to AVX/SSE than to SSE/scalar.
- incrudible 4y agoThere is definitely an argument to be made regarding uops that just because the vector width of the interface doubled, the hardware resouces need not have doubled as well. AVX512 originally came from KNL accelerators, which were designed for more specific purposes.
- Const-me 4y agoI agree. And these resources are not even limited to execution units inside cores. Another one is bandwidth. On computers with dual-channel DDR4, aligned SSE vectors are delivered from main memory with a single transaction. For ideal scaling from SSE, AVX512 would need octa-channel DDR4, and 4x as much caches. Another one is waste heat. On modern processors, sustained performance is often limited by thermals. For ideal scaling, AVX512 would need twice as much cooling compared to AVX.
- ec109685 4y agoOne of the most “10,00 foot” unintuitive things is that a text format like json is faster parse than a binary protocol like protobufs.
- jxi 4y agoIs it actually faster?
- winrid 4y agoIf you apply the same optimizations to the binary protocol it will be much faster than JSON. Even comparing common deserializarion libraries like gson and thrift, thrift is faster despite being much older than protobufs.
- ec109685 4y agoGood point. I was comparing to protobufs, which is slower to parse than json. Other formats are faster.