4 ms·

I've been beating my head trying to get duckdb to statically link into a Go program (I'm neither an expert with cgo nor ld). If anyone else has been able to do

I've been beating my head trying to get duckdb to statically link into a Go program (I'm neither an expert with cgo nor ld). If anyone else has been able to do this I'd love to see your build steps. https://github.com/marcboeker/go-duckdb https://github.com/marcboeker/go-duckdb produces a non-static binary by default.

knome 4y ago
I'm not familiar with the project. Does it use any net-related code? That won't be static because it will want to load C-libs for using /etc/nsswitch.conf to handle DNS/name stuff. https://stackoverflow.com/questions/33228809/why-is-my-go-application-not-statically-linked https://stackoverflow.com/questions/33228809/why-is-my-go-ap...
- eatonphil 4y ago
  I don't have the source code in a good state to publish yet but here's where I'm at. At some point before this CGO_LDFLAGS does work and the header is found (omit the -ldflags args). But when it goes to statically link it can no longer find the header. CGO_LDFLAGS="-L$(pwd)/duckdb/src/include" CGO_CFLAGS="-I$(pwd)/duckdb/src/include" go build -ldflags '-extldflags " -lstdc++ -lm -lduckdb -static"' # github.com/marcboeker/go-duckdb ../../go/pkg/mod/github.com/marcboeker/go-duckdb@v0.0.0-20220427142532-cd9f33e64d9a/connection.go:4:10: fatal error: duckdb.h: No such file or directory 4 | #include <duckdb.h> | ^~~~~~~~~~ compilation terminated. Edit, nevermind about not being in a good state! Here's my code: https://github.com/multiprocessio/duckdb-tests https://github.com/multiprocessio/duckdb-tests.
  - irq-1 4y ago
    Put the file in quotes. Angle brackets are for built-in files. #include "duckdb.h"
    - eatonphil 4y ago
      That's not my code.
      - eatonphil 4y ago
        But also, just to double check, I modified the vendored code and no difference: CGO_LDFLAGS="-L$(pwd)/duckdb/src/include" CGO_CFLAGS="-I$(pwd)/duckdb/src/include" go build -ldflags '-extldflags " -lstdc++ -lm -lduckdb -static"' # github.com/marcboeker/go-duckdb vendor/github.com/marcboeker/go-duckdb/connection.go:4:10: fatal error: duckdb.h: No such file or directory 4 | #include "duckdb.h" | ^~~~~~~~~~ compilation terminated.
DemocracyFTW2 4y ago
I tried to replace SQLite with DuckDB for a customized install of better-sqlite3[1] and failed. [1] https://github.com/JoshuaWise/better-sqlite3 https://github.com/JoshuaWise/better-sqlite3
- 1egg0myegg0 4y ago
  We have a node client if that would be helpful! https://duckdb.org/docs/api/nodejs https://duckdb.org/docs/api/nodejs
Cwizard 4y ago
I tried the same thing, also failed… I am also not an expert however. But I am very interested in this. Anyone reading this that could point me to some resources that might help?
folago 4y ago
Did you try Zig? https://dev.to/kristoff/zig-makes-go-cross-compilation-just-work-29ho https://dev.to/kristoff/zig-makes-go-cross-compilation-just-...
- ignoramous 4y ago
  Per this post [0] by Andrew Kelley, Zig's lead developer, projects with "large dependency trees" are better off using other tools than rely on Zig's cross-compile magic. DuckDB needs Python3 to build as well, so not sure how easy it might be to get it cross-compile with Zig CC. [0] https://archive.is/7SuAf https://archive.is/7SuAf
  - eatonphil 4y ago
    Also, the issue isn't cross compiling it's just static linking.
    - ignoramous 4y ago
      Gotcha. Targeting musl instead of glibc with Zig CC should get you a statically-linked binary, though, unsure if duckdb and its deps play nice with musl. Personally, a duckdb golang binary interests me. But: I haven't yet mustered enough patience to sit through a time-consuming duckdb build.

knome 4y ago

I'm not familiar with the project. Does it use any net-related code? That won't be static because it will want to load C-libs for using /etc/nsswitch.conf to handle DNS/name stuff. https://stackoverflow.com/questions/33228809/why-is-my-go-application-not-statically-linked https://stackoverflow.com/questions/33228809/why-is-my-go-ap...

eatonphil 4y ago
I don't have the source code in a good state to publish yet but here's where I'm at. At some point before this CGO_LDFLAGS does work and the header is found (omit the -ldflags args). But when it goes to statically link it can no longer find the header. CGO_LDFLAGS="-L$(pwd)/duckdb/src/include" CGO_CFLAGS="-I$(pwd)/duckdb/src/include" go build -ldflags '-extldflags " -lstdc++ -lm -lduckdb -static"' # github.com/marcboeker/go-duckdb ../../go/pkg/mod/github.com/marcboeker/go-duckdb@v0.0.0-20220427142532-cd9f33e64d9a/connection.go:4:10: fatal error: duckdb.h: No such file or directory 4 | #include <duckdb.h> | ^~~~~~~~~~ compilation terminated. Edit, nevermind about not being in a good state! Here's my code: https://github.com/multiprocessio/duckdb-tests https://github.com/multiprocessio/duckdb-tests.
- irq-1 4y ago
  Put the file in quotes. Angle brackets are for built-in files. #include "duckdb.h"
  - eatonphil 4y ago
    That's not my code.
    - eatonphil 4y ago
      But also, just to double check, I modified the vendored code and no difference: CGO_LDFLAGS="-L$(pwd)/duckdb/src/include" CGO_CFLAGS="-I$(pwd)/duckdb/src/include" go build -ldflags '-extldflags " -lstdc++ -lm -lduckdb -static"' # github.com/marcboeker/go-duckdb vendor/github.com/marcboeker/go-duckdb/connection.go:4:10: fatal error: duckdb.h: No such file or directory 4 | #include "duckdb.h" | ^~~~~~~~~~ compilation terminated.

DemocracyFTW2 4y ago

I tried to replace SQLite with DuckDB for a customized install of better-sqlite3[1] and failed. [1] https://github.com/JoshuaWise/better-sqlite3 https://github.com/JoshuaWise/better-sqlite3

1egg0myegg0 4y ago
We have a node client if that would be helpful! https://duckdb.org/docs/api/nodejs https://duckdb.org/docs/api/nodejs

Cwizard 4y ago

I tried the same thing, also failed… I am also not an expert however. But I am very interested in this. Anyone reading this that could point me to some resources that might help?

folago 4y ago

Did you try Zig? https://dev.to/kristoff/zig-makes-go-cross-compilation-just-work-29ho https://dev.to/kristoff/zig-makes-go-cross-compilation-just-...

ignoramous 4y ago
Per this post [0] by Andrew Kelley, Zig's lead developer, projects with "large dependency trees" are better off using other tools than rely on Zig's cross-compile magic. DuckDB needs Python3 to build as well, so not sure how easy it might be to get it cross-compile with Zig CC. [0] https://archive.is/7SuAf https://archive.is/7SuAf
- eatonphil 4y ago
  Also, the issue isn't cross compiling it's just static linking.
  - ignoramous 4y ago
    Gotcha. Targeting musl instead of glibc with Zig CC should get you a statically-linked binary, though, unsure if duckdb and its deps play nice with musl. Personally, a duckdb golang binary interests me. But: I haven't yet mustered enough patience to sit through a time-consuming duckdb build.