8 ms·

Neat Rust Tricks: Passing Closures to C

215 points by rabidferret 7y ago

cheez 7y ago
Is this a neat trick or just standard operating procedure for calling C from <your favorite lang>? As it was billed as a trick, I was expecting some sort of runtime code generation to pass the data pointer and some jump instruction to jump to the right spot and unpack the data pointer. Maybe I just overcomplicate things ;-)
- Mathnerd314 7y ago
  It does seem quite similar to Haskell FFI code: https://github.com/bobfrank/hasqlite/blob/4e38801d969a43e88bba72afd2c8057cc2ac7569/SQLite3.hsc#L497 https://github.com/bobfrank/hasqlite/blob/4e38801d969a43e88b... The "neat" factor comes from how little type wrangling and unsafe code is needed.
  - Ericson2314 7y ago
    I believe this actually JITs a trampoline with libffi, so only one code pointer is needed, not separate code and data pointers. (Also hi, go contribute to Nixpkgs again!)
- DougBTX 7y ago
  I’d say that “standard procedure” would be to do it the same way as it would be done in C: define a struct, allocate one somewhere, then pass a pointer to it as the data pointer. Using the anonymous struct which represents the closure itself seems like skipping a step, the user doesn’t need to spell out which values are stored in the struct.
  - zozbot234 7y ago
    If the language supports closures which capture variables from their surrounding environment, there's no way around using "the closure itself" as your data object. After all, "the user" is not expected to "know" what any given closure is capturing from the environment; part of the point of closures is implementing a sort of information hiding.
- marcan_42 7y ago
  It's standard procedure. I've done the exact same thing when wrapping C APIs into Python using Cython, several times. You pass the Python closure as the void *data and then register a shared generic callback which casts it and calls it. Easy. Getting the memory management right is slightly tricky, but not too bad. Fun fact: you can't safely do this with ctypes. Since it is called as pure Python, it cannot do watertight Python exception handling in a callback context (because even if you have a try/except block, an exception can always happen right before or after it), and ctypes provides no usable internal way of doing it - it just eats exceptions inside callbacks. This is what motivated me to rewrite Ceph's librbd bindings from ctypes to Cython.
  - cheez 7y ago
    I thought as much, thanks for the confirmation :-)
jgtrosh 7y ago
How does this relate to nested functions in C? (And resulting “infectious executable stacks”?) https://nullprogram.com/blog/2019/11/15/ https://nullprogram.com/blog/2019/11/15/
- Diggsey 7y ago
  It doesn't. This is just showing the normal way that callbacks are implemented in vanilla C and how you would make that programming pattern interoperate with Rust closures. Neither one relies on the compiler trickery/runtime code generation described in the earlier article.
- zozbot234 7y ago
  The executable stack trick is only required if you want to implement closures that can be called as if they were plain C functions, with only a function pointer and no extra (void *) argument.
- rabidferret 7y ago
  It doesn't relate to it at all. The issues around linking to problematic object files mentioned in that article will apply to Rust as well, but that's unrelated to the subject of this article, it's a property of the linker you're using and the toolchain used to compile whatever C dependencies you have
- zabzonk 7y ago
  C doesn't have nested functions - they are a GCC extension.
- richardwhiuk 7y ago
  The problems there don't apply I believe because Rust closures don't require an executable stack.
  - mmastrac 7y ago
    That's correct - a Rust closure generally [1] can't be converted to a function pointer as it requires both code and state. [1] https://github.com/rust-lang/rust/issues/39817 https://github.com/rust-lang/rust/issues/39817
kazinator 7y ago
You can pass closures to C as C functions in TXR Lisp, a language I created. Example: http://rosettacode.org/wiki/Window_creation#Win32.2FWin64 http://rosettacode.org/wiki/Window_creation#Win32.2FWin64 In this program, a translation of Microsoft's "Your First Windows Program" from MSDN, defun is used to define a WindowsProc callback. defun generates a lambda under the hood, which carries a lexical scope. The lambda is passed directly to Win32 as a callback, which is nicely called for repainting the window. (Or at least, things appear that way to the programmer.) Setting this up requires a few steps. We need a target function, of course, which can be any callable object. Then there is this incantation: (deffi-cb wndproc-fn LRESULT (HWND UINT LPARAM WPARAM)) The deffi-cb operator takes a name and some type specifications: return type and parameters. The name is defined as a function; so here we get a function called wndproc-fn. This function is a converter. If we pass it a Lisp function, it gives back a FFI closure object. Then in the program, we instantiate this closure object, and stick it into the WNDPROC structure as required by the Windows API. Here we use the above wndproc-fn converter to obtain WindowProc in the shape of a FFI closure: (let* ((hInstance (GetModuleHandle nil)) (wc (new WNDCLASS lpfnWndProc [wndproc-fn WindowProc] ... The lpfnWndProc member of the WNDCLASS FFI structure is defined as having the FFI type closure; that will correspond to a function pointer on the C side. The rest is just Windows: (RegisterClass wc) register the class, and then CreateWindow with that class by name and so on.
- kazinator 7y ago
  Here is another example of callbacks at work from the TXR Lisp test suite: using the C library funtion qsort to sort a Lisp array of strings. http://www.kylheku.com/cgit/txr/tree/tests/017/qsort.tl http://www.kylheku.com/cgit/txr/tree/tests/017/qsort.tl It's done in two ways, as UTF-8 char * strings and as wchar_t * strings. What's used as the callback is the function cmp-str which is in TXR Lisp's standard library. A lambda expression could be used instead. Also tested is the perpetration of a non-local control transfer out of the callback, instead of the normal return. This properly cleans up the temporary memory allocated for the string conversions.
  - cellularmitosis 7y ago
tedunangst 7y ago
Now call qsort with a closure.
- jstarks 7y ago
  I’m sure we could have rustc generate a trampoline function directly on the stack like gcc can. What could go wrong?
  - saagarjha 7y ago
    Nothing, if your Rust implementation is bug-free ;)
- kazinator 7y ago
  This is the TXR Lisp interactive listener of TXR 228. Quit with :quit or Ctrl-D on empty line. Ctrl-X ? for cheatsheet. 1> (with-dyn-lib nil (deffi qsort "qsort" void ((ptr (array wstr)) size-t size-t closure)) (deffi-cb qsort-cb int ((ptr wstr-d) (ptr wstr-d)))) #:lib-0005 2> (let ((vec #("the" "quick" "brown" "fox" "jumped" "over" "the" "lazy" "dogs"))) (prinl vec) (qsort vec (length vec) (sizeof wstr) [qsort-cb (lambda (a b) (cmp-str a b))]) (prinl vec)) #("the" "quick" "brown" "fox" "jumped" "over" "the" "lazy" "dogs") #("brown" "dogs" "fox" "jumped" "lazy" "over" "quick" "the" "the") #("brown" "dogs" "fox" "jumped" "lazy" "over" "quick" "the" "the") The lambda is pointless; we could create the FFI closure directly from cmp-str with [qsort-cb cmp-str]. It shows more clearly that we can use any closure.
- rabidferret 7y ago
  `void ()(void, everything_else)` is in my personal experience a much more common API than that of `qsort` (probably for exactly the point you're pedantically trying to make), so I chose to focus on that API for this article. There's really no reason to pass a rust closure to `qsort` instead of sorting in Rust. That said, if there's demand for real world use cases that require passing Rust closures to C APIs that take only a function pointer and not a data pointer, I'll be happy to write a follow up.
thenewwazoo 7y ago
This is a really well-written article about a common way to trampoline Rust methods into C embedded RTOS tasks, which often work in exactly this way. I've done just this in the service of getting ChibiOS and uC/OS II running Rust code.
jonny383 7y ago
Rust is already doomed. The amount of literature being published about either comparisons or compatibility with C is a strong indicator C is here to stay.
- cellularmitosis 7y ago
  If Rust is intended to replace C, wouldn't you expect lots of this sort of literature? i.e. isn't this actually a sign of its _success_?
  - steveklabnik 7y ago
    Also, being able to add Rust to an existing C or C++ codebase was an important design consideration. Big projects like Firefox aren’t just going to re-write millions of lines of code all at once.
    - The_rationalist 7y ago
      And this is why rust is not going to succeed. I has not a great compatibility with c++.
  - jonny383 7y ago
    No. It shows that people are still struggling with changing the way in which they write software to the "rust" way. The attitude of falling back to C or using unsafe rust just undermines the premise of the argument of why you should use rust. This is just like the node.js craze a few years ago - people will rant on trying to justify why you should use rust and write the "rust" way before realising that what they already had worked as intended. A true replacement for C (when one is finally developed) will remove all of these doubts and back-shadowing behaviour almost instantly (kind of like the react way of ux did) EDIT: typo
    - MaulingMonkey 7y ago
      > It shows that people are still struggling with changing the way in which they write software to the "rust" way. It shows no such thing. I generally work on relatively small ~1MLOC C++ codebases. There are codebases out there measured in the hundreds of MLOC. These are not the comparatively tiny javascript codebases you find React used in - where additional milliseconds of download / parse / evaluation time has a measurable effect on your user retention statistics. There is no "near instant" at 100M+ LOC scales. There is only incremental rewrites, and incremental rewrites means making your new code talk to your old code, and to other people's existing code. This means interop with existing C ABIs. There is no such thing as a C "replacement" that can't talk to an existing C ABI, or expose an existing C ABI. Of course, a C ABI doesn't mean C. It's more frequently C++ in my ecosystem, for example. But it could just as easily be Rust, or any number of other languages capable of exposing a C ABI.
KenanSulayman 7y ago
Interestingly this is very similar to how I implemented passing closures into JavaScriptCore as hooks for JS class invocations ("function calls"). [0] Essentially it's taking advantage of the fact that closures are static methods with "implicit" data pointers. It should be fairly obvious that this is a massive violation of safety and undefined behavior, and most likely to break when debugging symbols etc. are inserted. The safest way to do this until Rust has figured out a stable-enough-ABI for closure passing would be a thread-local trampoline, I guess. Not very nice.. [0] https://github.com/psychonautwiki/rust-ul/blob/master/src/helpers_internal.rs#L67-L118 https://github.com/psychonautwiki/rust-ul/blob/master/src/he...
- Gibbon1 7y ago
  I read an article by a guy talking about stupid C tricks. One of them was to 'mangle' raw pointers into an index before passing them. And then de-magle them to get back a raw pointer. Advantages are you can pass meta data with the 'pointer'. Which also allows you to invalidate a pointer. The pointer can't be dereferenced. The enclosed variables/data isn't accessible and cannot be modified by the target. For callbacks the overhead likely isn't significant.
- doomrobo 7y ago
  Where's the UB? It casts a boxed closure to a raw pointer, and then back to a boxed closure. There's no tricky introspection being done here.
  - KenanSulayman 7y ago
    I'm not entirely sure you read the code I'm referring to. There's no box there.
dmitrygr 7y ago
> If you’re not familiar with C’s syntax, here’s the equivalent signature in Rust Author is hilarious. Who is familiar with that but not c?
- alkonaut 7y ago
  I came to Rust without writing C before. Most of my experience with C comes from problems exactly like this. I doubt I'm alone in this.

215 points by rabidferret 7y ago

Is this a neat trick or just standard operating procedure for calling C from <your favorite lang>? As it was billed as a trick, I was expecting some sort of runtime code generation to pass the data pointer and some jump instruction to jump to the right spot and unpack the data pointer. Maybe I just overcomplicate things ;-)

Mathnerd314 7y ago
It does seem quite similar to Haskell FFI code: https://github.com/bobfrank/hasqlite/blob/4e38801d969a43e88bba72afd2c8057cc2ac7569/SQLite3.hsc#L497 https://github.com/bobfrank/hasqlite/blob/4e38801d969a43e88b... The "neat" factor comes from how little type wrangling and unsafe code is needed.
- Ericson2314 7y ago
  I believe this actually JITs a trampoline with libffi, so only one code pointer is needed, not separate code and data pointers. (Also hi, go contribute to Nixpkgs again!)
DougBTX 7y ago
I’d say that “standard procedure” would be to do it the same way as it would be done in C: define a struct, allocate one somewhere, then pass a pointer to it as the data pointer. Using the anonymous struct which represents the closure itself seems like skipping a step, the user doesn’t need to spell out which values are stored in the struct.
- zozbot234 7y ago
  If the language supports closures which capture variables from their surrounding environment, there's no way around using "the closure itself" as your data object. After all, "the user" is not expected to "know" what any given closure is capturing from the environment; part of the point of closures is implementing a sort of information hiding.
marcan_42 7y ago
It's standard procedure. I've done the exact same thing when wrapping C APIs into Python using Cython, several times. You pass the Python closure as the void *data and then register a shared generic callback which casts it and calls it. Easy. Getting the memory management right is slightly tricky, but not too bad. Fun fact: you can't safely do this with ctypes. Since it is called as pure Python, it cannot do watertight Python exception handling in a callback context (because even if you have a try/except block, an exception can always happen right before or after it), and ctypes provides no usable internal way of doing it - it just eats exceptions inside callbacks. This is what motivated me to rewrite Ceph's librbd bindings from ctypes to Cython.
- cheez 7y ago
  I thought as much, thanks for the confirmation :-)

jgtrosh 7y ago

How does this relate to nested functions in C? (And resulting “infectious executable stacks”?) https://nullprogram.com/blog/2019/11/15/ https://nullprogram.com/blog/2019/11/15/

Diggsey 7y ago
It doesn't. This is just showing the normal way that callbacks are implemented in vanilla C and how you would make that programming pattern interoperate with Rust closures. Neither one relies on the compiler trickery/runtime code generation described in the earlier article.
zozbot234 7y ago
The executable stack trick is only required if you want to implement closures that can be called as if they were plain C functions, with only a function pointer and no extra (void *) argument.
rabidferret 7y ago
It doesn't relate to it at all. The issues around linking to problematic object files mentioned in that article will apply to Rust as well, but that's unrelated to the subject of this article, it's a property of the linker you're using and the toolchain used to compile whatever C dependencies you have
zabzonk 7y ago
C doesn't have nested functions - they are a GCC extension.
richardwhiuk 7y ago
The problems there don't apply I believe because Rust closures don't require an executable stack.
- mmastrac 7y ago
  That's correct - a Rust closure generally [1] can't be converted to a function pointer as it requires both code and state. [1] https://github.com/rust-lang/rust/issues/39817 https://github.com/rust-lang/rust/issues/39817

kazinator 7y ago

You can pass closures to C as C functions in TXR Lisp, a language I created. Example: http://rosettacode.org/wiki/Window_creation#Win32.2FWin64 http://rosettacode.org/wiki/Window_creation#Win32.2FWin64 In this program, a translation of Microsoft's "Your First Windows Program" from MSDN, defun is used to define a WindowsProc callback. defun generates a lambda under the hood, which carries a lexical scope. The lambda is passed directly to Win32 as a callback, which is nicely called for repainting the window. (Or at least, things appear that way to the programmer.) Setting this up requires a few steps. We need a target function, of course, which can be any callable object. Then there is this incantation: (deffi-cb wndproc-fn LRESULT (HWND UINT LPARAM WPARAM)) The deffi-cb operator takes a name and some type specifications: return type and parameters. The name is defined as a function; so here we get a function called wndproc-fn. This function is a converter. If we pass it a Lisp function, it gives back a FFI closure object. Then in the program, we instantiate this closure object, and stick it into the WNDPROC structure as required by the Windows API. Here we use the above wndproc-fn converter to obtain WindowProc in the shape of a FFI closure: (let* ((hInstance (GetModuleHandle nil)) (wc (new WNDCLASS lpfnWndProc [wndproc-fn WindowProc] ... The lpfnWndProc member of the WNDCLASS FFI structure is defined as having the FFI type closure; that will correspond to a function pointer on the C side. The rest is just Windows: (RegisterClass wc) register the class, and then CreateWindow with that class by name and so on.

kazinator 7y ago
Here is another example of callbacks at work from the TXR Lisp test suite: using the C library funtion qsort to sort a Lisp array of strings. http://www.kylheku.com/cgit/txr/tree/tests/017/qsort.tl http://www.kylheku.com/cgit/txr/tree/tests/017/qsort.tl It's done in two ways, as UTF-8 char * strings and as wchar_t * strings. What's used as the callback is the function cmp-str which is in TXR Lisp's standard library. A lambda expression could be used instead. Also tested is the perpetration of a non-local control transfer out of the callback, instead of the normal return. This properly cleans up the temporary memory allocated for the string conversions.
- cellularmitosis 7y ago

tedunangst 7y ago

Now call qsort with a closure.

jstarks 7y ago
I’m sure we could have rustc generate a trampoline function directly on the stack like gcc can. What could go wrong?
- saagarjha 7y ago
  Nothing, if your Rust implementation is bug-free ;)
kazinator 7y ago
This is the TXR Lisp interactive listener of TXR 228. Quit with :quit or Ctrl-D on empty line. Ctrl-X ? for cheatsheet. 1> (with-dyn-lib nil (deffi qsort "qsort" void ((ptr (array wstr)) size-t size-t closure)) (deffi-cb qsort-cb int ((ptr wstr-d) (ptr wstr-d)))) #:lib-0005 2> (let ((vec #("the" "quick" "brown" "fox" "jumped" "over" "the" "lazy" "dogs"))) (prinl vec) (qsort vec (length vec) (sizeof wstr) [qsort-cb (lambda (a b) (cmp-str a b))]) (prinl vec)) #("the" "quick" "brown" "fox" "jumped" "over" "the" "lazy" "dogs") #("brown" "dogs" "fox" "jumped" "lazy" "over" "quick" "the" "the") #("brown" "dogs" "fox" "jumped" "lazy" "over" "quick" "the" "the") The lambda is pointless; we could create the FFI closure directly from cmp-str with [qsort-cb cmp-str]. It shows more clearly that we can use any closure.
rabidferret 7y ago
`void ()(void, everything_else)` is in my personal experience a much more common API than that of `qsort` (probably for exactly the point you're pedantically trying to make), so I chose to focus on that API for this article. There's really no reason to pass a rust closure to `qsort` instead of sorting in Rust. That said, if there's demand for real world use cases that require passing Rust closures to C APIs that take only a function pointer and not a data pointer, I'll be happy to write a follow up.

thenewwazoo 7y ago

This is a really well-written article about a common way to trampoline Rust methods into C embedded RTOS tasks, which often work in exactly this way. I've done just this in the service of getting ChibiOS and uC/OS II running Rust code.

jonny383 7y ago

Rust is already doomed. The amount of literature being published about either comparisons or compatibility with C is a strong indicator C is here to stay.

cellularmitosis 7y ago
If Rust is intended to replace C, wouldn't you expect lots of this sort of literature? i.e. isn't this actually a sign of its _success_?
- steveklabnik 7y ago
  Also, being able to add Rust to an existing C or C++ codebase was an important design consideration. Big projects like Firefox aren’t just going to re-write millions of lines of code all at once.
  - The_rationalist 7y ago
    And this is why rust is not going to succeed. I has not a great compatibility with c++.
- jonny383 7y ago
  No. It shows that people are still struggling with changing the way in which they write software to the "rust" way. The attitude of falling back to C or using unsafe rust just undermines the premise of the argument of why you should use rust. This is just like the node.js craze a few years ago - people will rant on trying to justify why you should use rust and write the "rust" way before realising that what they already had worked as intended. A true replacement for C (when one is finally developed) will remove all of these doubts and back-shadowing behaviour almost instantly (kind of like the react way of ux did) EDIT: typo
  - MaulingMonkey 7y ago
    > It shows that people are still struggling with changing the way in which they write software to the "rust" way. It shows no such thing. I generally work on relatively small ~1MLOC C++ codebases. There are codebases out there measured in the hundreds of MLOC. These are not the comparatively tiny javascript codebases you find React used in - where additional milliseconds of download / parse / evaluation time has a measurable effect on your user retention statistics. There is no "near instant" at 100M+ LOC scales. There is only incremental rewrites, and incremental rewrites means making your new code talk to your old code, and to other people's existing code. This means interop with existing C ABIs. There is no such thing as a C "replacement" that can't talk to an existing C ABI, or expose an existing C ABI. Of course, a C ABI doesn't mean C. It's more frequently C++ in my ecosystem, for example. But it could just as easily be Rust, or any number of other languages capable of exposing a C ABI.

KenanSulayman 7y ago

Interestingly this is very similar to how I implemented passing closures into JavaScriptCore as hooks for JS class invocations ("function calls"). [0] Essentially it's taking advantage of the fact that closures are static methods with "implicit" data pointers. It should be fairly obvious that this is a massive violation of safety and undefined behavior, and most likely to break when debugging symbols etc. are inserted. The safest way to do this until Rust has figured out a stable-enough-ABI for closure passing would be a thread-local trampoline, I guess. Not very nice.. [0] https://github.com/psychonautwiki/rust-ul/blob/master/src/helpers_internal.rs#L67-L118 https://github.com/psychonautwiki/rust-ul/blob/master/src/he...

Gibbon1 7y ago
I read an article by a guy talking about stupid C tricks. One of them was to 'mangle' raw pointers into an index before passing them. And then de-magle them to get back a raw pointer. Advantages are you can pass meta data with the 'pointer'. Which also allows you to invalidate a pointer. The pointer can't be dereferenced. The enclosed variables/data isn't accessible and cannot be modified by the target. For callbacks the overhead likely isn't significant.
doomrobo 7y ago
Where's the UB? It casts a boxed closure to a raw pointer, and then back to a boxed closure. There's no tricky introspection being done here.
- KenanSulayman 7y ago
  I'm not entirely sure you read the code I'm referring to. There's no box there.

dmitrygr 7y ago

> If you’re not familiar with C’s syntax, here’s the equivalent signature in Rust Author is hilarious. Who is familiar with that but not c?

alkonaut 7y ago
I came to Rust without writing C before. Most of my experience with C comes from problems exactly like this. I doubt I'm alone in this.