4 ms·
There's a part which seems strange, where clang is used with "-O2" to generate code: $ clang -O2 -target bpf -Xclang -target-feature -Xclang +alu32 -c sub64.
by justinclift 8y ago
There's a part which seems strange, where clang is used with "-O2" to generate code:
$ clang -O2 -target bpf -Xclang -target-feature -Xclang +alu32 -c sub64.c -o - | llvm-objdump -S -
> Apparently the compiler decided it was better to operate on 64-bit registers and discard the upper 32 bits.
The workaround was to use the `volatile` keyword.
The problem kind of sounds like one of the LLVM optimisation passes made the change.
http://releases.llvm.org/8.0.0/docs/Passes.html#transform-passes http://releases.llvm.org/8.0.0/docs/Passes.html#transform-pa...
Wonder if disabling optimisations ("-O0") would have also worked?
- praseodym 8y agoLooks like it: $ clang -O2 -target bpf -Xclang -target-feature -Xclang +alu32 -c sub32.c $ llvm-objdump -S -no-show-raw-insn sub32.o sub32.o: file format ELF64-BPF Disassembly of section .text: 0000000000000000 sub32: 0: w0 = w1 1: w0 -= w2 2: exit $ clang -O0 -target bpf -Xclang -target-feature -Xclang +alu32 -c sub32.c $ llvm-objdump -S -no-show-raw-insn sub32.o sub32.o: file format ELF64-BPF Disassembly of section .text: 0000000000000000 sub32: 0: w3 = w2 1: w4 = w1 2: *(u32 *)(r10 - 4) = r1 3: *(u32 *)(r10 - 8) = r2 4: r1 = *(u32 *)(r10 - 4) 5: r2 = *(u32 *)(r10 - 8) 6: w1 -= w2 7: w0 = w1 8: *(u32 *)(r10 - 12) = r3 9: *(u32 *)(r10 - 16) = r4 10: exit
- majke 8y agoCompiling to eBPF is hard! The compiler must: - avoid loops (backwards jumps) - unroll everything - inline everything - no function calls Basically, it's impossible to use clang to generate eBPF without -O2. Sorry.
- justinclift 8y ago>Basically, it's impossible to use clang to generate eBPF without -O2. Sorry. That's a bit confusing, as the comment by praseodym shows the output from doing just that. ???
- Dangeranger 8y agoI believe he is saying that eBPF will disallow certain code to run if it doesn’t meet the standards listed by the parent comment. The article also mentions that eBPF will check whether code is “worthy” or running or rearrange it so that it is worthy. The idea is to compile to assembly which both fixes the bug, and is runnable by eBPF.
- majke 8y agoApologies. Indeed, not all generated clang eBPF is loadable eBPF. So we can sum up: - Clang can compile C - eBPF backend can generate the eBPF bytecode for subset of IR (AFAIU not all IR, not all "C" is okay for eBPF) - verifier can accept or reject the generated eBPF. The goal is to write C in such a way that will be generally sensible C but that will consistently yield loadable eBPF programs. This is not easy! Just recently I was debugging code where clang optmizer decided to merge two loads of 2 bytes into one load of 4 bytes. This made the verifier unhappy - the specific loads are rewritten to helper calls by verifier. The verifier knows what to do on 2-byte load at offset X, and 2-byte load at offset X+2. But it has no idea what to do with 4-byte load at offset X! Writing C programs that will compile to valid, loadable eBPF requires a fair amount of voodoo.
- justinclift 8y agoThanks, that makes sense. :) Kind of sounds like having a specific eBPF transformation pass available in LLVM would be useful. eg having it generate known-to-work-with-eBPF output
- lukego 8y agoInterestingly, you are basically describing the way that LuaJIT compiles Lua code :)
- sreque 8y agoI'm new to this area of work, but has anyone stepped back and thought: "this is not the right way to build eBPF programs"? Would it be better to create a new high-level language and toolset? All this voodoo hackery to try to trick a C compiler into making eBPF-compatible code feels unsustainable.
- asgeir 8y agobcc https://github.com/iovisor/bcc https://github.com/iovisor/bcc if you need low-level abstractions or bpftrace https://github.com/iovisor/bpftrace https://github.com/iovisor/bpftrace if you need something simpler (similar to dtrace)
- jkbs 8y agoCool idea. I played around with it. As majke mentioned we need optimizations or the backend compiler will complain: $ clang -O0 -target bpf filter_alu32.c -emit-llvm -S -c -o filter_alu32.ll $ llc -march=bpf -mattr=+alu32 -filetype=obj filter_alu32.ll LLVM ERROR: Unsupported relocation: try to compile with -O2 or above, or check your static variable usage The code has to be non-trivial to hit the error. I think its the use of eBPF maps that triggers it. What if we turned off the optimizations in the frontend and ran just the backend optimizer, though? I had some success with this approach: $ clang -O0 -target bpf filter_alu32.c -emit-llvm -S -c -o filter_alu32.ll $ opt -O2 filter_alu32.ll | llc -march=bpf -mattr=+alu32 -filetype=obj -o filter_alu32.o $ llvm-objdump -S filter_alu32.o | grep -e -= 45: 1c 21 00 00 00 00 00 00 w1 -= w2 53: 1c 21 00 00 00 00 00 00 w1 -= w2 65: 1c 21 00 00 00 00 00 00 w1 -= w2 You end up with more eBPF instructions if you don't use the frontend optimizer but the code doesn't need any hacks. Perhaps it can be tweaked further by applying frontend optimizations selectively? So thanks for the suggestion. It's another working solution. You can find the code for this experiment here: https://github.com/jsitnicki/cloudflare-blog/commit/acf28b69d823d6abab852cd7aa55b0c5bd79029f https://github.com/jsitnicki/cloudflare-blog/commit/acf28b69...