4 ms·

This is a distraction from the post. But the thread may attract minds who are glad to correct. The post shows an example where code calls into shared-object fi

This is a distraction from the post. But the thread may attract minds who are glad to correct. The post shows an example where code calls into shared-object files from picolisp. Something I have been wondering: why are the inspection options so weak for SO files? I expect you should be able to introspect a SO. In response, you would receive function and struct documentation, and clear type information about functions. Relevant stack overflow question, https://unix.stackexchange.com/questions/61940/introspection-of-the-interface-of-a-shared-library https://unix.stackexchange.com/questions/61940/introspection.... "For C-type functions, you'll only get function names, not argument types or return values. For C++ functions, you'll get mangled names - pipe the output of that command through c++filt to get function names and argument types (still no return values though)." You could change Elf to fix it. An alternative, the compiler could do it. Gcc could run an early preprocessor round to pick up metadata and type information. It would freeze this metadata to blobs in the code area. Through this, you could have a standard set of introspection methods built with that compiler. I have been programming on Windows recently. In some situations, in order to build against a DLL you need need an accompanying "lib" file. If you do not have one, you can perform a circus to create one from the DLL. What have I missed?

skissane 8y ago
I like the idea of this tool: https://github.com/rpav/c2ffi https://github.com/rpav/c2ffi (Although despite planning to use it for some personal project I haven't got around to it yet...) It parses C/C++/Objective-C into JSON metadata. It uses clang/LLVM so the parsing/etc should be very accurate. So, to implement your idea, you could just embed this JSON into an ELF section. (Or, if you don't like JSON, convert your JSON to some other format, such as S-expressions or protobuf or whatever.)
emmelaich 8y ago
Some people are trying. See Stephen Kell's[1] work: https://www.thestrangeloop.com/2014/liberating-the-smalltalk-lurking-in-c-and-unix.html https://www.thestrangeloop.com/2014/liberating-the-smalltalk... https://slideslive.com/38902419/adding-runtime-type-information-to-the-gnu-toolchain-and-runtime https://slideslive.com/38902419/adding-runtime-type-informat... [1] http://www.cl.cam.ac.uk/~srk31/ http://www.cl.cam.ac.uk/~srk31/
avodonosov 8y ago
Metadata exists actually, in corresponding .h files.
- Firadeoclus 8y ago
  They're (a) in no way connected to the .so files, (b) not necessarily available, (c) incomplete information since build information (e.g. compiler, command line options) is missing, and (d) much harder to parse than necessary.
  - kazinator 8y ago
    You don't need special compiler options to use a shared object. If you got it as a binary, there is nothing to compile. The incomplete information in header files isn't the compiler command line options, but the nuances of the API semantics. Like if a structure is passed and the structure contains pointers. Who owns the structure and must free it? Who owns the pointers inside it? Oooh, here is a functional argument: what parts of the API can be safely invoked from that callback? That size argument, is that bytes or array elements? And so on.
    - Firadeoclus 8y ago
      > You don't need special compiler options to use a shared object. If you got it as a binary, there is nothing to compile. I was referring to the header file. Unfortunately, with C/C++ you need compile options to fully interpret source code. > The incomplete information in header files isn't the compiler command line options, but the nuances of the API semantics. No doubt there's much more to API semantics than basic type information. But the latter is still valuable.
  - avodonosov 8y ago
    There are shortcomings, but they do exist, and if you have a practical goal, may provide a descent and easy enough solution.
    - Firadeoclus 8y ago
keithnz 8y ago
I guess in the windows world COM was supposed to fix this. There was a special interface, mainly for VB, which would expose introspection so things could be discovered.
AceJohnny2 8y ago
Mostly because on all mainstream OSes you see today, C is the lowest-common-denominator ABI that a lot of the code is written in, and for historical reasons C does not require or expect any extra information. Thus, it's not standard. Which is why some systems have tried to add their own layer of extra metadata, such as Windows COM and GTK's gobject(introspection). I don't know how these systems store their metadata, but ELF at least is flexible enough that you should be able to store your metadata in extra sections of the SO. However, (and I'm just speculating here, hopefully more informed people will correct me) there's also a risk that part of the tool ecosystem doesn't know what to do with the in-ELF metadata and accidentally removes it: "strip" comes to mind, the tool that removes extra symbols and debugging information from binaries and libraries.
- pjmlp 8y ago
  > I don't know how these systems store their metadata, but ELF at least is flexible enough that you should be able to store your metadata in extra sections of the SO. Window executable and dynamic libraries are very flexible and always allowed for resource and metadata sections. COM uses type libraries, which have been replaced by .NET Metadata on UWP (basically COM Runtime reborn, .NET genesis).
- floatboth 8y ago
  gobject-introspection generates metadata from source code, stores it as XML files in $PREFIX/share/gir-1.0 and as binary files in $PREFIX/lib/girepository-1.0 And yeah, seems like strip is one of the reasons why they don't store it in the ELF. But also: cross-platform compatibility (GTK does support Windows and macOS).
kazinator 8y ago
You can't even tell whether a given symbol looked up with dlsym is a function or data! Re:> In some situations, in order to build against a DLL you need need an accompanying "lib" file. If you do not have one .lib files are not required to use a DLL. A DLL is opened with LoadLibrary and symbols can be resolved with GetProcAddress. Those import .lib files are just some quirk of Microsoft Visual C.
- pjmlp 8y ago
  It is not a quirk, it goes back to how Windows did dynamic libraries on Win16 and everyone had to use them, specially when linking at compile time. Also it was how Symbian and Aix did dynamic loading as well (Aix eventually adopted other UNIXes model).
- geocar 8y ago
  > You can't even tell whether a given symbol looked up with dlsym is a function or data! On many modern unixes you can! Functions will usually be in a "text" section (marked execute or read+execute) and data will usually be in a "data" or "bss" section (marked read-only, or read-write). The dl* family of functions don't expose this information, but after getting a symbol you can check the protection bits on the page the pointer is on (using a system-defined mechanism).
  - kazinator 8y ago
    Data could be placed into protected space. I think that a string literal in .text space could be exposed as a dynamic symbol.
- deleted 8y ago
  [deleted]

skissane 8y ago

I like the idea of this tool: https://github.com/rpav/c2ffi https://github.com/rpav/c2ffi (Although despite planning to use it for some personal project I haven't got around to it yet...) It parses C/C++/Objective-C into JSON metadata. It uses clang/LLVM so the parsing/etc should be very accurate. So, to implement your idea, you could just embed this JSON into an ELF section. (Or, if you don't like JSON, convert your JSON to some other format, such as S-expressions or protobuf or whatever.)

emmelaich 8y ago

Some people are trying. See Stephen Kell's[1] work: https://www.thestrangeloop.com/2014/liberating-the-smalltalk-lurking-in-c-and-unix.html https://www.thestrangeloop.com/2014/liberating-the-smalltalk... https://slideslive.com/38902419/adding-runtime-type-information-to-the-gnu-toolchain-and-runtime https://slideslive.com/38902419/adding-runtime-type-informat... [1] http://www.cl.cam.ac.uk/~srk31/ http://www.cl.cam.ac.uk/~srk31/

avodonosov 8y ago

Metadata exists actually, in corresponding .h files.

Firadeoclus 8y ago
They're (a) in no way connected to the .so files, (b) not necessarily available, (c) incomplete information since build information (e.g. compiler, command line options) is missing, and (d) much harder to parse than necessary.
- kazinator 8y ago
  You don't need special compiler options to use a shared object. If you got it as a binary, there is nothing to compile. The incomplete information in header files isn't the compiler command line options, but the nuances of the API semantics. Like if a structure is passed and the structure contains pointers. Who owns the structure and must free it? Who owns the pointers inside it? Oooh, here is a functional argument: what parts of the API can be safely invoked from that callback? That size argument, is that bytes or array elements? And so on.
  - Firadeoclus 8y ago
    > You don't need special compiler options to use a shared object. If you got it as a binary, there is nothing to compile. I was referring to the header file. Unfortunately, with C/C++ you need compile options to fully interpret source code. > The incomplete information in header files isn't the compiler command line options, but the nuances of the API semantics. No doubt there's much more to API semantics than basic type information. But the latter is still valuable.
- avodonosov 8y ago
  There are shortcomings, but they do exist, and if you have a practical goal, may provide a descent and easy enough solution.
  - Firadeoclus 8y ago

keithnz 8y ago

I guess in the windows world COM was supposed to fix this. There was a special interface, mainly for VB, which would expose introspection so things could be discovered.

AceJohnny2 8y ago

Mostly because on all mainstream OSes you see today, C is the lowest-common-denominator ABI that a lot of the code is written in, and for historical reasons C does not require or expect any extra information. Thus, it's not standard. Which is why some systems have tried to add their own layer of extra metadata, such as Windows COM and GTK's gobject(introspection). I don't know how these systems store their metadata, but ELF at least is flexible enough that you should be able to store your metadata in extra sections of the SO. However, (and I'm just speculating here, hopefully more informed people will correct me) there's also a risk that part of the tool ecosystem doesn't know what to do with the in-ELF metadata and accidentally removes it: "strip" comes to mind, the tool that removes extra symbols and debugging information from binaries and libraries.

pjmlp 8y ago
> I don't know how these systems store their metadata, but ELF at least is flexible enough that you should be able to store your metadata in extra sections of the SO. Window executable and dynamic libraries are very flexible and always allowed for resource and metadata sections. COM uses type libraries, which have been replaced by .NET Metadata on UWP (basically COM Runtime reborn, .NET genesis).
floatboth 8y ago
gobject-introspection generates metadata from source code, stores it as XML files in $PREFIX/share/gir-1.0 and as binary files in $PREFIX/lib/girepository-1.0 And yeah, seems like strip is one of the reasons why they don't store it in the ELF. But also: cross-platform compatibility (GTK does support Windows and macOS).

kazinator 8y ago

You can't even tell whether a given symbol looked up with dlsym is a function or data! Re:> In some situations, in order to build against a DLL you need need an accompanying "lib" file. If you do not have one .lib files are not required to use a DLL. A DLL is opened with LoadLibrary and symbols can be resolved with GetProcAddress. Those import .lib files are just some quirk of Microsoft Visual C.

pjmlp 8y ago
It is not a quirk, it goes back to how Windows did dynamic libraries on Win16 and everyone had to use them, specially when linking at compile time. Also it was how Symbian and Aix did dynamic loading as well (Aix eventually adopted other UNIXes model).
geocar 8y ago
> You can't even tell whether a given symbol looked up with dlsym is a function or data! On many modern unixes you can! Functions will usually be in a "text" section (marked execute or read+execute) and data will usually be in a "data" or "bss" section (marked read-only, or read-write). The dl* family of functions don't expose this information, but after getting a symbol you can check the protection bits on the page the pointer is on (using a system-defined mechanism).
- kazinator 8y ago
  Data could be placed into protected space. I think that a string literal in .text space could be exposed as a dynamic symbol.
deleted 8y ago
[deleted]