10 ms·
Comparing the C FFI overhead on various languages
- deleted 4y ago[deleted]
- throw827474737 4y agoSo why isn't C the baseline (and zig and rust being pretty close to it quite expected), but both luajit and julia are significantly faster??
- gallexme 4y agohttps://nullprogram.com/blog/2018/05/27/ https://nullprogram.com/blog/2018/05/27/
- qalmakka 4y agoI'm always pretty surprised when I find out most people writing C or C++ have no idea that PLTs exist. They have a small but not negligible cost.
- eatonphil 4y ago> For the C “FFI” he used standard dynamic linking, not dlopen(). This distinction is important, since it really makes a difference in the benchmark. There’s a potential argument about whether or not this is a fair comparison to an actual FFI, but, regardless, it’s still interesting to measure
- arinlen 4y ago> There’s a potential argument about whether or not this is a fair comparison to an actual FFI, but, regardless, it’s still interesting to measure (...) If there's interest in measuring dynamic linking then wouldn't there be an interest in measuring it on all languages that support dynamic linking?
- jcelerier 4y agoWith clang, just compiling with -fno-plt gives me: jit: 1.003483 ns/call plt: 1.254158 ns/call ind: 1.254616 ns/call GCC does not seem to support it though, even if it accepts the flag and gives me: jit: 1.003483 ns/call plt: 1.502089 ns/call ind: 1.254616 ns/call (tried everything I could think of that would have a chance to make the PLT disappear: cc -fno-plt -Bsymbolic -fno-semantic-interposition -flto -std=c99 -Wall -Wextra -O3 -g3 -Wl,-z,relro,-z,now -o benchmark benchmark.c ./empty.so -ldl without any change on GCC)
- miohtama 4y agoIs there anything akin FFI but with static linking for any foreign (non C) language?
- junon 4y agoThe question as it stands makes a few assumptions I don't think one can make, and as such is a bit tricky to answer cleanly, but I'll try. Yes it's just called linking. The language needs to be aware of calling conventions and perhaps side effects and be prepared for no additional intrinsic support for higher level features. It probably also needs to be able to read C headers, because C symbols do not contain type signatures like many C++ compilers add. There's no "library" or some out of the box solution for this, if that's what you're asking. This boils down to how programs are constructed and, moreso, how CPUs work. In most (all?) cases, anything higher level than straight-up linking is headed toward FFI territory.
- tlb 4y agoCalling WebAssembly from Javascript, sort of? In the early Python 2 era there was an option to build an interpreter binary with statically linked C stubs, and it was noticeably faster and let you access Python data structures from C. I used it for robotics code for speed. It was inconvenient because you had to link in all the modules you needed.
- samatman 4y agoLuaJIT can use the FFI against statically linked object code just fine, I'm not sure if that answers your question since in this context it must be embedded in a C program. It's a hard requirement of static linking that you have just one binary so it might, answer your question that is.
- fweimer 4y agoFor OpenJDK, there is JEP 178: https://openjdk.java.net/jeps/178 https://openjdk.java.net/jeps/178 I haven't seen it used in practice. Ocaml's C-implemented functions are linked statically. But like JNI, the C functions have special names and type signatures, so it is slightly different from, say, ctypes in Python. CGO for Go is statically linked, too. Its overhead stems from significant differences between the Go and C world. The example uses dynamic linking, but it would not have to do that.
- deleted 4y ago[deleted]
- bachmeier 4y agoC, C++, Zig, Rust, D, and Haskell are all similar because they're basically doing the same thing. Someone else linked to the blog post, but Lua and Julia aren't doing the same thing, so they get different results. > both luajit and julia are significantly faster I would be interested if anyone has an example where the difference matters in practice. As soon as you move to the more realistic scenario where you're writing a program that does something other than what is measured by these benchmarks, that's not going to be your biggest concern.
- vvanders 4y agoOh it totally matters, any sort of chatty interface over FFI you will pay for it. There's a reason a lot of gamedev uses luajit, I've personally had to refactor many interfaces to avoid JNI calls as much as possible as there was significant overhead(both in the call and from the VM not being able to optimize around it).
- kllrnohj 4y agoThe reason a lot of gamedev uses luajit is the ease at which it can be embedded. And that's not really even true anymore as the majority of gamedev is using Unreal or Unity, neither of which use luajit.
- vvanders 4y agoIt's not just how easy it is to embed, it's also really small in both code + runtime size. I've shipped it on systems with sub 8mb of total system memory(we used a preallocated 400kb block), until quickjs came along there really wasn't anything comparable. It was also much faster than anything else at the time and regularly beat v8 in the benchmarks I ran. Unity+Unreal are the public engines out there but there's plenty of in-house engines and tool chains you don't really hear about. I wouldn't be surprised if it's still deployed in quite a few contexts.
- kllrnohj 4y ago> I would be interested if anyone has an example where the difference matters in practice. Vulkan. Any sort of binding to Vulkan over a non-trivial FFI (so like, not from C++, Rust, etc...) is going to be murdered by this FFI overhead cost. Especially since for bindings from something like Java you're either paying FFI on every field set on a struct, or you're paying non-trivial marshalling costs to convert from a Java class to a C struct to then finally call the corresponding Vulkan function.
- exebook 4y agoI developed a terminal emulator, file manager and text editor Deodar 8 years ago in JavaScript/V8 with native C++ calls, it worked but I was extremely disappointed by speed, it felt so slow like you need to do a passport control each time you call a C++ function.
- ZiiS 4y agoProbably butter smooth after eight years of V8 development and Moore's Law (well whatever passes for it now).
- kllrnohj 4y agoProbably not since single threaded improvement has barely advanced over the last 8 years, and JS/V8 are still in a single threaded world that stopped existing a decade ago.
- exikyut 4y agoOh nice, with a Norton Commander-alike terminal UI. Screenshots at https://sourceforge.net/projects/deodar/ https://sourceforge.net/projects/deodar/ Last-modified 2018 over at https://github.com/exebook/deodar https://github.com/exebook/deodar I'm not sure where the use of Yacc ends and https://github.com/exebook/elfu https://github.com/exebook/elfu begins in the .yy files (which are sprinkled with very algebraic-looking Unicode throughout). The Pascal-like class definitions may be defined in https://github.com/exebook/intervision https://github.com/exebook/intervision. Very interesting project.
- Koromix 4y agoThe official solutions, node-ffi and node-ffi-napi, are extremely slow, with an overhead hundred of times higher than it should be. I don't know what they do to be so slow. I'm making my own FFI module for Node.js, Koffi, as a much faster alternative. You can see some benchmark here, to compare with node-ffi-napi: https://www.npmjs.com/package/koffi#benchmarks https://www.npmjs.com/package/koffi#benchmarks
- 4y ago
- khoobid_shoma 4y agoI guess it is better to measure CPU time instead of wall time (e.g. using clock() ).
- deleted 4y ago[deleted]
- cube2222 4y agoJust a caveat, not sure if it matters in practice, but this benchmark is using very old versions of many languages it's comparing (5 year old ones).
- kllrnohj 4y agoIt probably matters for a few of the slower ones, like Java, Go, or Dart. It's also going to matter on what platform. eg, Java may have better FFI on x86 than on ARM. Or similarly Dart's FFI may better on ARM than on x86, particularly given Flutter is the primary user these days. And then to make it even more complicated it's also going to potentially depend on the GC being used. For example for Java's JNI it's actually the bookkeeping for the GC that takes the most time in that FFI transition (can't pause the thread to mark the stack for a concurrent GC when it's executing random C code, after all). Which is going to potentially depend on what the specific GC being used requires.
- kllrnohj 4y agoAnother major caveat to this benchmark is it doesn't include any significant marshalling costs. For example, passing strings or arrays from Java to C is much, much slower than passing a single integer. Same is going to be true for a lot (all?) of the GC'd languages, and especially true for strings when the language isn't utf8 natively (as in, even though Java can store in utf8 internally, it doesn't expose that publicly so JNI doesn't benefit)
- jimmaswell 4y agoHow is that possible? It's not just passing pointers?
- lelanthran 4y ago> How is that possible? It's not just passing pointers? No. A Java string is a "pointer" to an array of 16-bit integers (each element is a 2-byte character). A C string is a pointer to an array of 8-bit integers. You have to first convert the Java string to UTF8, then allocate an array of 1-byte unsigned integers, then copy the UTF8 into it, and only then can you pass it to a C function that expects a string.
- spullara 4y agoWill be interesting to see how Project Panama does on this kind of benchmark. https://openjdk.java.net/projects/panama/ https://openjdk.java.net/projects/panama/
- vvanders 4y agoLet's not forget that it's modified UTF-8[1] you get back from JNI lest you think that you'll be able to use the buffer as-is. [1] https://docs.oracle.com/javase/10/docs/specs/jni/types.html#modified-utf-8-strings https://docs.oracle.com/javase/10/docs/specs/jni/types.html#...
- jimmaswell 4y agoGuess I was missing the context, I thought this was just within Java.
- sdze 4y agoCan you try PHP?
- alkonaut 4y agoAny idea why mono is used rather than .NET here?
- DoingIsLearning 4y agoFrom the readme: > My environment: > [...] > - Ubuntu 14.04 x64 > [...] Mono can run on nix targets but .NET itself (not .NET core) is still very much windows only.
- juki 4y ago.NET core is .NET nowadays. The old Windows only .NET framework isn't being developed anymore (other than fixes).
- vopi 4y ago.NET core doesn't exist anymore. It's now just .NET. It is the successor to the old windows only .NET. It is very much not windows only.
- alkonaut 4y agoThe “normal” .NET runs on Linux as much as Java or python. The framework/core divide no longer exists. That’s why I’m asking.
- richardwhiuk 4y agoThe benchmark is ancient.
- KSPAtlas 4y agoWhat about Common Lisp?
- medo-bear 4y agothere is a pretty powerful CFFI package (library) to achieve this, however performance will be very implementation dependent. in case someone wants to try this, the defacto-standard, free open source, speedy implementation is SBCL
- tomas789 4y agoThere is no Python benchmark but you can find a PR claiming it has 123,198ms. That would be a worst one by a wide margin. https://github.com/dyu/ffi-overhead/pull/18 https://github.com/dyu/ffi-overhead/pull/18
- sk0g 4y agoC FFI takes 123 seconds?! That's pretty insane, but if you mean 123.2 ms, it's still very bad. Doesn't feel like that would be the case from using NumPy, PyTorch and the likes, but they also typically run 'fat' functions, where it's one function with a lot of data that returns something. Usually don't chain or loop much there. Edit: the number was for 500 million calls. Yeah, don't think I've ever made that many calls. 123 seconds feels fairly short then, except for demanding workflows like game dev maybe.
- seniorsassycat 4y ago500 million calls in 123 seconds
- NoahKAndrews 4y agoI think that's the time to run the whole benchmark suite. Compare to the results for go, for example.
- remram 4y agocffi is probably the canonical way to do this on Python, I wonder what the performance is there. edit: 30% improvement, still 100x slower than e.g. Rust.
- kevin_thibedeau 4y agoIf you need a fast loop in Python then switch to Cython.
- remram 4y agoThat's fine for a tight loop. Performance might still matter in a bigger application. This benchmark is measuring the overhead, which is relevant in all contexts; the fact that it does it with a loop is a statistical detail.
- ryukoposting 4y agoThis ia a cool concept, but the implementation is contrived (as many others describe). e.g. JNI array marshalling/unmarshalling has a lot of overhead. The Nim version is super outdated too (not sure about the other languages).
- SemanticStrengh 4y agoJava has a new API for FFI called the foreign memory interface
- dunefox 4y ago> - julia 0.6.3 That's an ancient version, the current version is v1.7.2.
- Sukera 4y agoI don't think that matters here - the FFI interface hasn't changed and I wouldn't expect it to differ significantly.
- sk0g 4y agoFor a game scripting language, Wren posts a pretty bad result here. Think it has isn't explicitly game focused though. The version tested is quite old however, having released in 2016.
- haberman 4y agoSome of the results look outdated. The Dart results look bad (25x slower than C), but looking at the code (https://github.com/dyu/ffi-overhead/tree/master/dart https://github.com/dyu/ffi-overhead/tree/master/dart) it appears to be five years old. Dart has a new FFI as of Dart 2.5 (2019): https://medium.com/dartlang/announcing-dart-2-5-super-charged-development-328822024970 https://medium.com/dartlang/announcing-dart-2-5-super-charge... I'm curious how the new FFI would fare in these benchmarks.
- meibo 4y agoSame with C#, it only benchmarks an old version of mono and not .NET Core, which has received several big performance boosts in recent releases.
- kcb 4y agoBecause their environment is using an Ubuntu version from 8 years ago. So a better title would be "Comparing the C FFI overhead on various languages in 2014"
- elcritch 4y agoActually looks like most of the languages are seriously outdated. Nim and Julia are both way outdated, Elixir is pretty outdated.
- ta988 4y agoJava has project Panama coming that may improve things a little.
- mhh__ 4y agoNeeds LTO, with that it will have 0 overhead in the compiled languages. D can actually compile the C code in this test now.
- TazeTSchnitzel 4y agoIt seems Rust has basically no overhead versus C, but it could have negative overhead if you use cross-language LTO. Of course, you can do LTO between C files too, so that would be unfair. But I think this sets it apart from languages that, even with a highly optimised FFI, don't have compiler support for LTO with C code.
- Normal_gaussian 4y agoFor those of us not up on our compilation acronyms - this is Link Time Optimisation.
- TazeTSchnitzel 4y agoAh yes indeed. LTO means the compiler can do certain optimisations across compilation units (in C/C++ those are your .c/.cpp files, in Rust the unit is the entire crate), notably inlining.
- DonHopkins 4y ago>negative overhead underhead
- otikik 4y agoOvertail
- moefh 4y agoI'm not sure LTO is the correct term for this. What makes C slower than these other languages is that the external function is in a dynamic library. Calling external functions like this is notoriously slow because it involves a double indirection -- every call jumps to the PLT's function entry, which jumps to the actual function. It's not that hard to do better in C if you load the library manually with `dlopen` and read the function pointer with `dlysym`. In this case, you directly call the function given its address value. But that's still slower than having the function address "hardcoded" in the machine code, which is what you get when statically linking in C. That's also what the languages with negative overhead do when JIT-compiling (they basically statically link at runtime).
- dgan 4y agoI had to run it to believe, I confirm it's 183 seconds(!) for python3 on my laptop Also, OCaml because I was interested (milliseconds): ocaml(int,noalloc,native) = 2022 ocaml(int,alloc,native) = 2344 ocaml(int,untagged,native) = 1912 ocaml(int32,noalloc,native) = 1049 ocaml(int32,alloc,native) = 1556 ocaml(int32,boxed,native) = 7544
- WalterBright 4y agoThe D programming language has literally a zero overhead to interface with C. The same calling conventions are used, the types are the same. D can also access C code by simply importing a .c file: import foo; // call functions from foo.c analogously to how you can `#include "foo.h"` in C++.
- planetis 4y agoThat Nim version has just left kindergarten and is prepping for elementary.
- thot_experiment 4y agon.b. this is using an absolutely ancient version of node, though i'm not sure that would change anything, worth nothing
- ksec 4y agoMissing 2014, or 2018 in the title.
- bfrog 4y agoGo looks horribly slow, I thought segmented stacks have gone away to improve this?
- SolitudeSF 4y agothis benchmark is afwul (which is expected)