6 ms·

I only had to read the first few sentences of the first point to see that the author thinks problems are easy because they don't understand the difficulties. P

by roca 5y ago

I only had to read the first few sentences of the first point to see that the author thinks problems are easy because they don't understand the difficulties. Parsing C function prototypes doesn't give you enough information to write safe language bindings. For example if you see "char* make_stuff(const char* param);", you don't know whether the memory pointed to by 'param' can be reused after the function call, or whether the function actually took ownership of it. You don't even know how many bytes of memory 'param' has to point to, because you don't know whether 'param' points to a null-terminated string or something else. Likewise you don't know how many bytes of memory the returned pointer points to, you don't know whether the caller is responsible for freeing it, and if they are you don't know how to free it. You don't know whether it's safe to call this function concurrently from multiple threads. You don't know whether 'param' or the return value are allowed to be NULL or not. And this is nearly the simplest possible example!

TimTheTinker 5y ago
You raise a good point, but the OP's point still stands. All we'd need is comments in header files with the extra type information encoded, and automated FFI would be possible. Not to say that would be easy... building a type system on top of a language that lacks it is anything but easy, but it's possible. TypeScript is an excellent example.
- roca 5y ago
  The OP's point is very clearly "we could just parse C header files and do it all automatically", not "we could parse C header files and a bunch of extra handwritten metadata and then do the rest automatically". The latter point is certainly more defensible but it's not the point they made, and it's much weaker ... I mean of course if you add extra metadata you can generate whatever FFI glue you need. The devil is then in the details.
  - philipswood 5y ago
    I took his point in general to be a rant about the unwillingness to give priority and effort to things that are schleppy, but doable. An unwillingness to address the devil in the details. The main idea being that an unwillingness to solve these problems once, generally and completely is dooming "us" to solve it partly, inefficiently and repeatedly. I would take the detail he gives for the specific cases as incomplete. Specifically this one. I would NOT say that the article as a whole is "clearly "we could just parse C header files and do it all automatically"" All of these should be doable. Each of these obviously has significant schleppy or technical difficulties/constraints. For none of these does he address the actual reasons they haven't been done or suggest plans for overcoming them. I kinda agree with him at a high-level in this case FFI _should_ be solvable. I think for this case he's just saying "there should be _some_ way to automatically generate FFI glue code (and we should be reusing it)"
    - roca 5y ago
      I respect you for generously steelmanning the OP's argument here but it is very clear the OP's intent is "just parse C header files and do it all automatically": > not only could you technically parse .h files and turn them into JNI Well, no, technically you can't do that at all.
      - philipswood 5y ago
        OK. Fair interpretation. The OP didn't have much leg to really stand on for this point and glosses over any of the actual difficult work that would need to be done to make this utopia from the 70/80/90s materialize. It's not a great post, and its ideas are not that clearly worked out. He could seriously work on his tone. It could be parsed minimally as a just a grumpy rant with a bit of "get off the lawn" and a few wrong ideas. Mostly a waste of time. I'm steelmanning it because personally I'd prefer to get as much as possible from what I've read. Even if it wasn't necessarily put it in the first place (maximizing my utility from it, for my purposes). I'm not debating the OP. If you're acting from a position of keeping him honest. Kudos. So I don't really need to engage you on the point then. I'm agreeing that you'd probably be able to take down the OP in an argument. Or at least would force him to up his game considerable to be able to actually make his points clearly. BTW, I really like your use of "steelmanning" here. Actually looking at his other points after this exchange: (2) I think modern VMs have amazing engineering - is he asking for hacks? Or what? (3) Cloud offerings could be more coherent. But there are real market and organisational constraints. (4) Drag and drop UI designers would be nice, and I miss them, but even the good ones used to produce terrible code when used innocently. (5) I think modern compilers are pretty awesome. Not sure what he wants done. (6) Yeah. We could build better more compact systems. His unhappiness at the state of the world. Hmm. Moaning about it like that doesn't really do much to solve it...
- jolux 5y ago
  I think having to manually annotate C headers with extra type information starts to reduce the benefits of automatically generating FFI interfaces. Rust already has bindgen which goes pretty far.
- nixpulvis 5y ago
  Did you just imply that we put the comments of C headers into our types?! Comments are lower than whitespace as far as the compiler should be sure. As I'm sure you are aware https://github.com/rust-lang/rust-bindgen https://github.com/rust-lang/rust-bindgen does basically what you want. I'm actually quite curious where exactly rust-bindgen falls short in the eyes of the author. I'm not as familiar with the other FFI libraries the OP linked.
  - DonHopkins 5y ago
    When is SWIG finally going to support Generalized Whitespace Overloading in C++ 2000? It's been 23 years, 2 months, and 19 days since the technical details were published in AT&T Labs Technical Report no. 42! https://www.stroustrup.com/whitespace98.pdf https://www.stroustrup.com/whitespace98.pdf
- masklinn 5y ago
  > All we'd need is comments in header files with the extra type information encoded, and automated FFI would be possible. That’s SWIG. It’s something like 15 years old, possibly more. > building a type system on top of a language that lacks it is anything but easy, but it's possible. TypeScript is an excellent example. That has nothing to do with the issue at hand. You can compile haskell to c, that does not help FFI-ing to unannotated C functions.
  - DonHopkins 5y ago
    https://en.wikipedia.org/wiki/SWIG https://en.wikipedia.org/wiki/SWIG >Initial release February 1996; 25 years ago http://www.swig.org/history.html http://www.swig.org/history.html >July, 1995. Dave develops SWIG while working in the Theoretical Physics Division at Los Alamos National Laboratory. Originally, it was conceived as an extension building tool for a customized scripting language that had been created for the Connection Machine 5. David Beazley, the author of SWIG, is a brilliant programmer, mad scientist, and excellent presenter. https://en.wikipedia.org/wiki/David_M._Beazley https://en.wikipedia.org/wiki/David_M._Beazley https://www.dabeaz.com/ https://www.dabeaz.com/ Check out his many talks about programming, especially his epic PyCon 2014 talk on his work as an expert on a patent infringement case. https://www.youtube.com/user/dabeazllc/videos https://www.youtube.com/user/dabeazllc/videos https://www.youtube.com/watch?v=RZ4Sn-Y7AP8 https://www.youtube.com/watch?v=RZ4Sn-Y7AP8 >David Beazley: Discovering Python - PyCon 2014 >So, what happens when you lock a Python programmer in a secret vault containing 1.5 TBytes of C++ source code and no internet connection? Find out as I describe how I used Python as a secret weapon of "discovery" in an epic legal battle. >Slides can be found at: https://speakerdeck.com/pycon2014 https://speakerdeck.com/pycon2014 and https://github.com/PyCon/2014-slides https://github.com/PyCon/2014-slides
injidup 5y ago
My thoughts exactly. For somebody complaining that these tools don't exist because people are too lazy or whatever he himself is too lazy to do a quick google search to see that these tools do exist. I was using SWIG (http://www.swig.org/ http://www.swig.org/) 15+20 years ago to bind C/C++ to Perl and Java and it still exists. However your point above stands in that you have to annotate the headers with enough information about memory ownership to make it work. The tone of the article also goes mad bigot when he starts whining about (AI) ( Aliens and immigrants ) taking our jobs for less money.
- strken 5y ago
  As someone who found an old I-94 in his wallet last week and spent a while as an alien, I try to be sympathetic to those sort of rants. It is genuinely frustrating when employers solve problems by finding a cheaper labour force instead of automating an unnecessary process or fixing the underlying technical issue. A company I once worked for that will remain nameless paid an entire team of people in the Philippines to read global weather reports and respond to them instead of integrating with the relevant APIs, solely because it would have been a higher upfront cost to get expensive US-based engineers to do the work, and I found it infuriating: partly because the time of the two groups was valued so differently, but also because it was just so inelegant and pointless.
- psyc 5y ago
  I didn’t take the AI bit to be about “tik er jurbs” but rather about employers’ attitude towards developers. That would be more in keeping with the theme of the overall rant.
- Animats 5y ago
  * I was using SWIG 15+20 years ago to bind C/C++ to Perl and Java and it still exists.* (OK, who broke the * for italics markup?) I've used it. It took the entire thing you want to call and generated one giant file of munged C. That's sort of where that idea takes you. But the author is on to something. There's a C to Rust translator. It sucks, because it works by emulating C pointer arithmetic in unsafe Rust, using its own set of primitives along the lines of "offset this pointer by this much". Now you have ugly, unsafe Rust. What's needed is something that infers the meaning of an ambiguous function call from the code. Something that reasons like this: Function call: int read(char* buf, size_t n) Analysis: Is "buf" an array, or a reference to one item? Examine C code for "read". Is it ever used in an array context, like subscripting or pointer arithmetic? Possible results are "no, definitely not an array", "yes, definitely an array", and "can't decide, code is too confusing". One would like the third case to be rare. If it's an array, how big is it? Possible results are inferred from the highest subscript seen, as an expression. Is it an expression on some variable at the interface? If it can be determined that the length of buf is n, the function call can be changed to a Rust slice for which .len() will work. Pointer arithmetic needs to be analyzed by symbolic execution, treating each pointer as an (array, pointer into array) pair. If the array associated with a pointer never changes during the lifetime of the pointer, tracked all the way down the call tree, then the pointer can be represented as a subscript. Subarrays passed by pointer become slices. This won't work all the time. Sometimes you'll need hints from the user. It would be tempting to guess using something like GPT-3, doing the conversion to safe code, and seeing if it worked. Most of these things are idioms, which is how humans convert them. Translation can be wrong in two ways - subscripts going out of range and being caught, and things using too much memory because a worst case overestimated something. All the heavy work is in figuring out the equivalent, but not identical, data representation. Once that's figured out, converting the executable code is reasonably straightforward. There's already a dumb C to Rust translator for most of that.
ajarmst 5y ago
The whole thing is a paean to Dunning-Kruger.
solresol 5y ago
... and yet, Ada95 managed to include (as part of the standard for the language) the ability to bind to C, C++, Fortran and Cobol. All of which have absurdly different calling conventions and assumptions about memory.
- roca 5y ago
  I don't know anything about Ada95's FFI but it can't magically answer the questions I posed. Either it requires extra information not in the prototypes, or it makes assumptions about the C code that won't always be true (and will have catastrophic consequences when they're not true), or it gives up on safe binding and offloads responsibility for handling those issues to the programmer.
CheezeIt 5y ago
You can perfectly well make a tool that emits incomplete JS API’s with some human interaction to specify details of ownership and object representation. In you could reuse much of it for multiple target languages. In fact, you can also make FFI wrappers automatically, and handle the memory safety with wrappers on the JS side.
raverbashing 5y ago
You're right. Because C conflates pointers with arrays. It conflates giving an array to a function with giving a space for it to write because there's no difference (besides the 'in'/'out' keyword which I'm not even sure is official) C is defective out of the box.
smsm42 5y ago
All those difficulties are real, and yet, FFI exists and is useful. Of course it won't work in 100% cases, but it would work in many cases, because all those things that you describe aren't random - usually libraries have certain conventions on what the pointers mean, who frees them, etc. What the author is asking for is extending the basic service FFI provides to make it more useable and accessible. Still wouldn't solve 100% of causes, and still the questions like "is it thread safe" would never be part of it - so what? There are many use cases which can live without answering this question.
dvfjsdhgfv 5y ago
> I only had to read the first few sentences of the first point to see that the author thinks problems are easy because they don't understand the difficulties. It's because you haven't read it till the end. He explains these things haven't been solved exactly because they are difficult and (he believes) unmonetizeable.

by roca 5y ago

TimTheTinker 5y ago

You raise a good point, but the OP's point still stands. All we'd need is comments in header files with the extra type information encoded, and automated FFI would be possible. Not to say that would be easy... building a type system on top of a language that lacks it is anything but easy, but it's possible. TypeScript is an excellent example.

roca 5y ago
The OP's point is very clearly "we could just parse C header files and do it all automatically", not "we could parse C header files and a bunch of extra handwritten metadata and then do the rest automatically". The latter point is certainly more defensible but it's not the point they made, and it's much weaker ... I mean of course if you add extra metadata you can generate whatever FFI glue you need. The devil is then in the details.
- philipswood 5y ago
  I took his point in general to be a rant about the unwillingness to give priority and effort to things that are schleppy, but doable. An unwillingness to address the devil in the details. The main idea being that an unwillingness to solve these problems once, generally and completely is dooming "us" to solve it partly, inefficiently and repeatedly. I would take the detail he gives for the specific cases as incomplete. Specifically this one. I would NOT say that the article as a whole is "clearly "we could just parse C header files and do it all automatically"" All of these should be doable. Each of these obviously has significant schleppy or technical difficulties/constraints. For none of these does he address the actual reasons they haven't been done or suggest plans for overcoming them. I kinda agree with him at a high-level in this case FFI _should_ be solvable. I think for this case he's just saying "there should be _some_ way to automatically generate FFI glue code (and we should be reusing it)"
  - roca 5y ago
    I respect you for generously steelmanning the OP's argument here but it is very clear the OP's intent is "just parse C header files and do it all automatically": > not only could you technically parse .h files and turn them into JNI Well, no, technically you can't do that at all.
    - philipswood 5y ago
      OK. Fair interpretation. The OP didn't have much leg to really stand on for this point and glosses over any of the actual difficult work that would need to be done to make this utopia from the 70/80/90s materialize. It's not a great post, and its ideas are not that clearly worked out. He could seriously work on his tone. It could be parsed minimally as a just a grumpy rant with a bit of "get off the lawn" and a few wrong ideas. Mostly a waste of time. I'm steelmanning it because personally I'd prefer to get as much as possible from what I've read. Even if it wasn't necessarily put it in the first place (maximizing my utility from it, for my purposes). I'm not debating the OP. If you're acting from a position of keeping him honest. Kudos. So I don't really need to engage you on the point then. I'm agreeing that you'd probably be able to take down the OP in an argument. Or at least would force him to up his game considerable to be able to actually make his points clearly. BTW, I really like your use of "steelmanning" here. Actually looking at his other points after this exchange: (2) I think modern VMs have amazing engineering - is he asking for hacks? Or what? (3) Cloud offerings could be more coherent. But there are real market and organisational constraints. (4) Drag and drop UI designers would be nice, and I miss them, but even the good ones used to produce terrible code when used innocently. (5) I think modern compilers are pretty awesome. Not sure what he wants done. (6) Yeah. We could build better more compact systems. His unhappiness at the state of the world. Hmm. Moaning about it like that doesn't really do much to solve it...
jolux 5y ago
I think having to manually annotate C headers with extra type information starts to reduce the benefits of automatically generating FFI interfaces. Rust already has bindgen which goes pretty far.
nixpulvis 5y ago
Did you just imply that we put the comments of C headers into our types?! Comments are lower than whitespace as far as the compiler should be sure. As I'm sure you are aware https://github.com/rust-lang/rust-bindgen https://github.com/rust-lang/rust-bindgen does basically what you want. I'm actually quite curious where exactly rust-bindgen falls short in the eyes of the author. I'm not as familiar with the other FFI libraries the OP linked.
- DonHopkins 5y ago
  When is SWIG finally going to support Generalized Whitespace Overloading in C++ 2000? It's been 23 years, 2 months, and 19 days since the technical details were published in AT&T Labs Technical Report no. 42! https://www.stroustrup.com/whitespace98.pdf https://www.stroustrup.com/whitespace98.pdf
masklinn 5y ago
> All we'd need is comments in header files with the extra type information encoded, and automated FFI would be possible. That’s SWIG. It’s something like 15 years old, possibly more. > building a type system on top of a language that lacks it is anything but easy, but it's possible. TypeScript is an excellent example. That has nothing to do with the issue at hand. You can compile haskell to c, that does not help FFI-ing to unannotated C functions.
- DonHopkins 5y ago
  https://en.wikipedia.org/wiki/SWIG https://en.wikipedia.org/wiki/SWIG >Initial release February 1996; 25 years ago http://www.swig.org/history.html http://www.swig.org/history.html >July, 1995. Dave develops SWIG while working in the Theoretical Physics Division at Los Alamos National Laboratory. Originally, it was conceived as an extension building tool for a customized scripting language that had been created for the Connection Machine 5. David Beazley, the author of SWIG, is a brilliant programmer, mad scientist, and excellent presenter. https://en.wikipedia.org/wiki/David_M._Beazley https://en.wikipedia.org/wiki/David_M._Beazley https://www.dabeaz.com/ https://www.dabeaz.com/ Check out his many talks about programming, especially his epic PyCon 2014 talk on his work as an expert on a patent infringement case. https://www.youtube.com/user/dabeazllc/videos https://www.youtube.com/user/dabeazllc/videos https://www.youtube.com/watch?v=RZ4Sn-Y7AP8 https://www.youtube.com/watch?v=RZ4Sn-Y7AP8 >David Beazley: Discovering Python - PyCon 2014 >So, what happens when you lock a Python programmer in a secret vault containing 1.5 TBytes of C++ source code and no internet connection? Find out as I describe how I used Python as a secret weapon of "discovery" in an epic legal battle. >Slides can be found at: https://speakerdeck.com/pycon2014 https://speakerdeck.com/pycon2014 and https://github.com/PyCon/2014-slides https://github.com/PyCon/2014-slides

injidup 5y ago

My thoughts exactly. For somebody complaining that these tools don't exist because people are too lazy or whatever he himself is too lazy to do a quick google search to see that these tools do exist. I was using SWIG (http://www.swig.org/ http://www.swig.org/) 15+20 years ago to bind C/C++ to Perl and Java and it still exists. However your point above stands in that you have to annotate the headers with enough information about memory ownership to make it work. The tone of the article also goes mad bigot when he starts whining about (AI) ( Aliens and immigrants ) taking our jobs for less money.

strken 5y ago
As someone who found an old I-94 in his wallet last week and spent a while as an alien, I try to be sympathetic to those sort of rants. It is genuinely frustrating when employers solve problems by finding a cheaper labour force instead of automating an unnecessary process or fixing the underlying technical issue. A company I once worked for that will remain nameless paid an entire team of people in the Philippines to read global weather reports and respond to them instead of integrating with the relevant APIs, solely because it would have been a higher upfront cost to get expensive US-based engineers to do the work, and I found it infuriating: partly because the time of the two groups was valued so differently, but also because it was just so inelegant and pointless.
psyc 5y ago
I didn’t take the AI bit to be about “tik er jurbs” but rather about employers’ attitude towards developers. That would be more in keeping with the theme of the overall rant.
Animats 5y ago
* I was using SWIG 15+20 years ago to bind C/C++ to Perl and Java and it still exists.* (OK, who broke the * for italics markup?) I've used it. It took the entire thing you want to call and generated one giant file of munged C. That's sort of where that idea takes you. But the author is on to something. There's a C to Rust translator. It sucks, because it works by emulating C pointer arithmetic in unsafe Rust, using its own set of primitives along the lines of "offset this pointer by this much". Now you have ugly, unsafe Rust. What's needed is something that infers the meaning of an ambiguous function call from the code. Something that reasons like this: Function call: int read(char* buf, size_t n) Analysis: Is "buf" an array, or a reference to one item? Examine C code for "read". Is it ever used in an array context, like subscripting or pointer arithmetic? Possible results are "no, definitely not an array", "yes, definitely an array", and "can't decide, code is too confusing". One would like the third case to be rare. If it's an array, how big is it? Possible results are inferred from the highest subscript seen, as an expression. Is it an expression on some variable at the interface? If it can be determined that the length of buf is n, the function call can be changed to a Rust slice for which .len() will work. Pointer arithmetic needs to be analyzed by symbolic execution, treating each pointer as an (array, pointer into array) pair. If the array associated with a pointer never changes during the lifetime of the pointer, tracked all the way down the call tree, then the pointer can be represented as a subscript. Subarrays passed by pointer become slices. This won't work all the time. Sometimes you'll need hints from the user. It would be tempting to guess using something like GPT-3, doing the conversion to safe code, and seeing if it worked. Most of these things are idioms, which is how humans convert them. Translation can be wrong in two ways - subscripts going out of range and being caught, and things using too much memory because a worst case overestimated something. All the heavy work is in figuring out the equivalent, but not identical, data representation. Once that's figured out, converting the executable code is reasonably straightforward. There's already a dumb C to Rust translator for most of that.

ajarmst 5y ago

The whole thing is a paean to Dunning-Kruger.

solresol 5y ago

... and yet, Ada95 managed to include (as part of the standard for the language) the ability to bind to C, C++, Fortran and Cobol. All of which have absurdly different calling conventions and assumptions about memory.

roca 5y ago
I don't know anything about Ada95's FFI but it can't magically answer the questions I posed. Either it requires extra information not in the prototypes, or it makes assumptions about the C code that won't always be true (and will have catastrophic consequences when they're not true), or it gives up on safe binding and offloads responsibility for handling those issues to the programmer.

CheezeIt 5y ago

You can perfectly well make a tool that emits incomplete JS API’s with some human interaction to specify details of ownership and object representation. In you could reuse much of it for multiple target languages. In fact, you can also make FFI wrappers automatically, and handle the memory safety with wrappers on the JS side.

raverbashing 5y ago

You're right. Because C conflates pointers with arrays. It conflates giving an array to a function with giving a space for it to write because there's no difference (besides the 'in'/'out' keyword which I'm not even sure is official) C is defective out of the box.

smsm42 5y ago

All those difficulties are real, and yet, FFI exists and is useful. Of course it won't work in 100% cases, but it would work in many cases, because all those things that you describe aren't random - usually libraries have certain conventions on what the pointers mean, who frees them, etc. What the author is asking for is extending the basic service FFI provides to make it more useable and accessible. Still wouldn't solve 100% of causes, and still the questions like "is it thread safe" would never be part of it - so what? There are many use cases which can live without answering this question.

dvfjsdhgfv 5y ago

> I only had to read the first few sentences of the first point to see that the author thinks problems are easy because they don't understand the difficulties. It's because you haven't read it till the end. He explains these things haven't been solved exactly because they are difficult and (he believes) unmonetizeable.