4 ms·

The added cargo env variable pointing to the path of generated binaries is a nice quality of life fix. I think that was one of the first annoyances I ran into w

The added cargo env variable pointing to the path of generated binaries is a nice quality of life fix. I think that was one of the first annoyances I ran into with cargo and binaries. It was minor but was like a little splinter that stood out from cargo's otherwise nicely integrated build and test capabilities.

jeffdavis 6y ago
I'm curious why there's no similar variable pointing to a built library (e.g. a cdylib).
- steveklabnik 6y ago
  What's the use case?
  - jeffdavis 6y ago
    I feel like I've been pushing my use case plenty already, so please take this as an answer to your question rather than one more push ;-) I'd like to be able to create plugin-style cdylibs that get loaded into a host program. For instance, PostgreSQL extensions are shared libraries (typically written in C) that need to be loaded into a running instance of postgres for integration testing. I am trying to make it reasonable to write such extensions in rust (https://github.com/jeff-davis/postgres-extension.rs https://github.com/jeff-davis/postgres-extension.rs). The environment variable to find the shared library would solve one problem, but there are still a couple more right behind it: * Running "cargo test" tries to build a binary that links to the library to run the unit tests, but this will fail because there are unresolved symbols in the .so file (symbols that call APIs in the host program, in this case PostgreSQL functions like palloc()). In other words, the test binary is useless because it can't load the plugin. * Linux seems fine creating a .so with unresolved symbols, but Mac and Windows need special linker flags. I can document that the linker flags are needed by consumers of my crate, but it would be nice if there was a better/supported way to create plugin-style shared libraries. I tried filing a bug with PR (https://github.com/rust-lang/rust/pull/66204 https://github.com/rust-lang/rust/pull/66204) it was determined that it needed to go through the RFC process. The simplest way to see the problem is to look at this crate: https://github.com/jeff-davis/unresolved https://github.com/jeff-davis/unresolved My problems don't end there, unfortunately. For my crate to really work, I also need to solve the problem of setjmp/longjmp at the edges between C and Rust (https://github.com/rust-lang/rfcs/issues/2625 https://github.com/rust-lang/rfcs/issues/2625).
    - steveklabnik 6y ago
      It's all good; I was asking because I legit didn't know, and couldn't think of anything off the top of my head. I don't work on Cargo, so I'm not as in-tune with its development. So, I am still not 100% sure that I understand, mostly because each package can only have one library, so this would only work if you wanted one plugin. I guess maybe if they were all in a workspace?
      - jeffdavis 6y ago
        Typically one postgres extension would have only one plugin library. So, the idea is that one crate would produce one plugin. I guess it's possible to have more than one plugin per extension, but it never occurred to me and I haven't seen that before. Having the location of the plugin might also allow "cargo install" to work.
weiming 6y ago
I've been using https://docs.rs/assert_cmd/ https://docs.rs/assert_cmd/ for this. (Helps spawn the binary and assert output.)
- epage 6y ago
  Thamks for the call out! I'm excited by the new env var and need to consider its impact on `assert_cmd`.
bsder 6y ago
The real problem though is that cargo doesn't play nice as part of system where cargo doesn't own everything. Cargo needs a lot more ways to be interrogated about the artifacts it produces and control over how they are produced.
- lilyball 6y ago
  Exactly this. It would be really great if for example when building Rust in Nix, Nix could own each of the artifacts, in order to share artifacts between tools that use the same revisions of dependencies, as well as to avoid rebuilding artifacts unnecessarily on every change to a derivation.
  - digikata 6y ago
    How does Nix handle areas with extensive package ecosystems? Python, Node et al? It seems like every distro/super packager has different compromises for this question.
    - lilyball 6y ago
      I haven't investigated Python, but for Node it uses a tool node2nix that fetches the metadata for the entire dependency tree and constructs a Nix expression for each one, such that each expression represents the source for that specific dependency. It then takes the top-level package and makes a call to `nodeEnv.buildNodePackage`, which contains the source for that top-level package and declares the entire recursive dependency tree. buildeNodePackage itself rebuilds the entire node_modules folder from the declared dependencies and reconstructs the lockfile. And then it asks npm to rebuild the package. The end result is this bypasses npm's downloading and managing of node_modules, does it all in a fully declarative fashion, and then hands control over to npm to build the package. Also, each dependency's source is cached independently by Nix, so if multiple packages use the exact same dependency, it doesn't have to be redownloaded (unlike Rust where the entire dependency tree is vendored separately per package).
      - steveklabnik 6y ago

jeffdavis 6y ago

I'm curious why there's no similar variable pointing to a built library (e.g. a cdylib).

steveklabnik 6y ago
What's the use case?
- jeffdavis 6y ago
  I feel like I've been pushing my use case plenty already, so please take this as an answer to your question rather than one more push ;-) I'd like to be able to create plugin-style cdylibs that get loaded into a host program. For instance, PostgreSQL extensions are shared libraries (typically written in C) that need to be loaded into a running instance of postgres for integration testing. I am trying to make it reasonable to write such extensions in rust (https://github.com/jeff-davis/postgres-extension.rs https://github.com/jeff-davis/postgres-extension.rs). The environment variable to find the shared library would solve one problem, but there are still a couple more right behind it: * Running "cargo test" tries to build a binary that links to the library to run the unit tests, but this will fail because there are unresolved symbols in the .so file (symbols that call APIs in the host program, in this case PostgreSQL functions like palloc()). In other words, the test binary is useless because it can't load the plugin. * Linux seems fine creating a .so with unresolved symbols, but Mac and Windows need special linker flags. I can document that the linker flags are needed by consumers of my crate, but it would be nice if there was a better/supported way to create plugin-style shared libraries. I tried filing a bug with PR (https://github.com/rust-lang/rust/pull/66204 https://github.com/rust-lang/rust/pull/66204) it was determined that it needed to go through the RFC process. The simplest way to see the problem is to look at this crate: https://github.com/jeff-davis/unresolved https://github.com/jeff-davis/unresolved My problems don't end there, unfortunately. For my crate to really work, I also need to solve the problem of setjmp/longjmp at the edges between C and Rust (https://github.com/rust-lang/rfcs/issues/2625 https://github.com/rust-lang/rfcs/issues/2625).
  - steveklabnik 6y ago
    It's all good; I was asking because I legit didn't know, and couldn't think of anything off the top of my head. I don't work on Cargo, so I'm not as in-tune with its development. So, I am still not 100% sure that I understand, mostly because each package can only have one library, so this would only work if you wanted one plugin. I guess maybe if they were all in a workspace?
    - jeffdavis 6y ago
      Typically one postgres extension would have only one plugin library. So, the idea is that one crate would produce one plugin. I guess it's possible to have more than one plugin per extension, but it never occurred to me and I haven't seen that before. Having the location of the plugin might also allow "cargo install" to work.

weiming 6y ago

I've been using https://docs.rs/assert_cmd/ https://docs.rs/assert_cmd/ for this. (Helps spawn the binary and assert output.)

epage 6y ago
Thamks for the call out! I'm excited by the new env var and need to consider its impact on `assert_cmd`.

bsder 6y ago

The real problem though is that cargo doesn't play nice as part of system where cargo doesn't own everything. Cargo needs a lot more ways to be interrogated about the artifacts it produces and control over how they are produced.

lilyball 6y ago
Exactly this. It would be really great if for example when building Rust in Nix, Nix could own each of the artifacts, in order to share artifacts between tools that use the same revisions of dependencies, as well as to avoid rebuilding artifacts unnecessarily on every change to a derivation.
- digikata 6y ago
  How does Nix handle areas with extensive package ecosystems? Python, Node et al? It seems like every distro/super packager has different compromises for this question.
  - lilyball 6y ago
    I haven't investigated Python, but for Node it uses a tool node2nix that fetches the metadata for the entire dependency tree and constructs a Nix expression for each one, such that each expression represents the source for that specific dependency. It then takes the top-level package and makes a call to `nodeEnv.buildNodePackage`, which contains the source for that top-level package and declares the entire recursive dependency tree. buildeNodePackage itself rebuilds the entire node_modules folder from the declared dependencies and reconstructs the lockfile. And then it asks npm to rebuild the package. The end result is this bypasses npm's downloading and managing of node_modules, does it all in a fully declarative fashion, and then hands control over to npm to build the package. Also, each dependency's source is cached independently by Nix, so if multiple packages use the exact same dependency, it doesn't have to be redownloaded (unlike Rust where the entire dependency tree is vendored separately per package).
    - steveklabnik 6y ago