5 ms·
The problem with JIT is not all information known at runtime is the correct information to optimize one. In finance the performance critical code path is often
by bluGill 4y ago
The problem with JIT is not all information known at runtime is the correct information to optimize one.
In finance the performance critical code path is often the one run least often. That is you have a if(unlikely_condition) {run_time_sensitive_trade();}. In this case you need to tell the compiler to ensure the CPU will have a pipeline stall because of a branch misprediction most of the time to ensure the time that counts the pipeline doesn't stall.
The above is a rare corner case for sure, but it is one of those weird exceptions you always need to keep in mind when trying to make any blanket rule.
- ridiculous_fish 4y agoHow does the compiler arrange for the CPU to mispredict the branch most of the time? I didn't think there were any knobs for the branch predictor other than static ones (e.g. backwards-jumps statically predicted as taken, or PowerPC branch hint bit).
- bluGill 4y agoCPUs have documentation, for this. I forget which one, but the one I did read it was as simple as the true case is assumed more common, and it is easy to arrange logic around that (I probably mis remember, but close enough to that). The common case also should be near the if in memory (inline code), so it is likely on the same cache line (or the next which is prefetched), while the other case is farther away and so you can stall the cache if the if goes that way.
- mikepurvis 4y agoC++20 is gaining the [[likely]] attribute for guiding the predictor in a portable way: http://eel.is/c++draft/dcl.attr.likelihood http://eel.is/c++draft/dcl.attr.likelihood Prior to this it was also possible, but required compiler-specific macros like GCC's __builtin_expect, which is used all over the kernel: https://github.com/torvalds/linux/search?q=__builtin_expect https://github.com/torvalds/linux/search?q=__builtin_expect
- wumpus 4y ago20 years of "progress" and hey, C++20 is gaining the likely attribute.
- astrange 4y agoThere are no ways to hint the branch predictor on some common CPUs. I think Intel says it's fully dynamic and has no static predictions. If you want to cause a pipeline stall, there's usually serializing instructions like cpuid/eieio.
- vgatherps 4y agoI believe intel used to always predict forward branches not taken and backward branches always taken if it didn’t have any history. Regardless, you can at least lay out code in a way that heavily favors one branch over the other, and potentially optimize for one branch (I.e. give one branch very expensive prelude/exit to make another branch very cheap). The last thing I’ve seen compilers greatly struggle with though…
- menaerus 4y agoTo my knowledge this does not have any direct impact to the CPU branch prediction mechanism. If that had been the case then we would at least have some x86-(64) instruction to manipulate with the BP. If I write a quick example such as https://godbolt.org/z/j7Y81j5fe https://godbolt.org/z/j7Y81j5fe, I also see no such instruction in the generated output. But what likely/unlikely mechanism can do is that it can serve as a hint to the compiler which the compiler can then use to generate more optimal code layout. For example, if we provide the compiler with likely/unlikely hints in our code, compiler will try to use that hint to stitch together the more probable code paths first. In theory this approach should result with better utilization of CPU instruction cache and thus it may lead to better performance.
- travisgriggs 4y agoThe problem with static compilation is not all information known at compile time is the correct information to optimize on. Assuming either source of data is the single source of truth for all optimizations is a fallacy. Use the right tool for the right job. Use all the tools if you can.
- kllrnohj 4y agoStatic compilers usually don't have to make such a tradeoff, though. They are free to spend arbitrarily long amounts of time optimizing all branches. And they often do exactly that. Static + LTO w/ PGO is pretty much the practical ideal. JITs don't offer much until you start adding dynamically loaded code where LTO just isn't possible anymore.
- pjmlp 4y agoTooling and real data sets. Most PGO for AOT scenarios suffers from tooling experience and not using data sets similar to production workflows.
- kllrnohj 4y agoPerhaps, but that's orthogonal. JITs don't just come with ideal production set sampling either after all. They only capture snippets, and rarely re-optimize already compiled functions in the face of changing workloads or new information. You can do crowd-sourced profiles (like Android does), but that's a completely independent set of infrastructure from JIT vs. AOT. You can feed that same profile to PGO for AOT. In fact, that's how Android uses it. They don't feed the profile to the JIT, they feed it to their offline, install-time or idle-maintenance AOT compiler.
- KMag 4y agoIf your hardware is designed to allow very lightweight profiling and tracing, then static + LTO w/ PGO can still be improved by runtime re-optimization. If designed properly, the runtime overhead can be brought arbitrarily low by increasing the sampling period.
- wumpus 4y ago> In finance Isn't low-latency trading only a subset of finance?
- bluGill 4y agoYes. And the part I mentioned is a subset of low latency. See the other reply.
- dahfizz 4y agoThe other issue with JIT is that it is unreliable. It optimizes code by making assumptions. If one of the assumptions is wrong, you pay a large latency penalty. In my field of finance, having reliably low latency is important. Being 15% faster on average, but every once in a while you will be really slow, is not something customers will go for.
- bluGill 4y agoI'm not in finance. I just remember one talk by a finance guy and it was a mind bender so I remember it.