5 ms·

Range joins in DuckDB

111 points by hfmuehleisen 4y ago

chewbacha 4y ago
Well I feel silly, based on a slight mis-reading of the title, I totally thought that Range was some company that was acquired by DuckDB.
- deleted 4y ago
  [deleted]
- tracerfett 4y ago
- hawkfish-rmgw 4y ago
  Oh dear I can see that - sorry for the confusion! I'll see if we can come up with something a bit longer. It was a bit nerdy...
- zxv 4y ago
  I also thought that DuckDB had acquired a company named Range. Interesting article regardless!
  - qorrect 4y ago
    I was thinking there was some 10x programmer known only as Range that had joined.
eatonphil 4y ago
I've been beating my head trying to get duckdb to statically link into a Go program (I'm neither an expert with cgo nor ld). If anyone else has been able to do this I'd love to see your build steps. https://github.com/marcboeker/go-duckdb https://github.com/marcboeker/go-duckdb produces a non-static binary by default.
- knome 4y ago
  I'm not familiar with the project. Does it use any net-related code? That won't be static because it will want to load C-libs for using /etc/nsswitch.conf to handle DNS/name stuff. https://stackoverflow.com/questions/33228809/why-is-my-go-application-not-statically-linked https://stackoverflow.com/questions/33228809/why-is-my-go-ap...
  - eatonphil 4y ago
    I don't have the source code in a good state to publish yet but here's where I'm at. At some point before this CGO_LDFLAGS does work and the header is found (omit the -ldflags args). But when it goes to statically link it can no longer find the header. CGO_LDFLAGS="-L$(pwd)/duckdb/src/include" CGO_CFLAGS="-I$(pwd)/duckdb/src/include" go build -ldflags '-extldflags " -lstdc++ -lm -lduckdb -static"' # github.com/marcboeker/go-duckdb ../../go/pkg/mod/github.com/marcboeker/go-duckdb@v0.0.0-20220427142532-cd9f33e64d9a/connection.go:4:10: fatal error: duckdb.h: No such file or directory 4 | #include <duckdb.h> | ^~~~~~~~~~ compilation terminated. Edit, nevermind about not being in a good state! Here's my code: https://github.com/multiprocessio/duckdb-tests https://github.com/multiprocessio/duckdb-tests.
    - irq-1 4y ago
      Put the file in quotes. Angle brackets are for built-in files. #include "duckdb.h"
      - eatonphil 4y ago
        That's not my code.
        eatonphil 4y ago
        But also, just to double check, I modified the vendored code and no difference: CGO_LDFLAGS="-L$(pwd)/duckdb/src/include" CGO_CFLAGS="-I$(pwd)/duckdb/src/include" go build -ldflags '-extldflags " -lstdc++ -lm -lduckdb -static"' # github.com/marcboeker/go-duckdb vendor/github.com/marcboeker/go-duckdb/connection.go:4:10: fatal error: duckdb.h: No such file or directory 4 | #include "duckdb.h" | ^~~~~~~~~~ compilation terminated.
lnsp 4y ago
Great to see new features being implemented. I'm using DuckDB for a thesis project and integrating it into my own Python CLI/web tool has been super easy -- I especially love the direct integration with DataFrames, it makes things really seamless.
crimsoneer 4y ago
I've been using DuckDb a fair bit recently and really enjoy it... When it has slightly better ide support (eg, I can use it in pycharm) and can take in geospatial data, I'll be ecstatic.
orthoxerox 4y ago
DuckDB has become my preferred tool for hardcore data wrangling. Excel is fine for like 80% of data processing tasks, but the remaining twenty percent are a pain, especially when you're CPU bound on a remote desktop. Smuggling the DuckDB JDBC driver onto said remote machine was the most productive infosec violation I've ever committed.
- hawkfish-rmgw 4y ago
  Our lips are sealed.
- minaguib 4y ago
  I don't know why I waited so long to try it. I wrangle a ton of raw and aggregate data locally every day. I've had a 10-year habit of massaging via unix CLI tools and pipes then moving to excel. I guess I didn't wanna write code. Funny thing is I love SQL. But with `duckdb_cli` it's a game-changer. I'm truly truly impressed.
  - ignoramous 4y ago
    > I've had a 10-year habit of massaging via unix CLI tools and pipes then moving to excel. Have you, at any point, considered / used dgsh [0] or a similar tool? If so, how has been your experience with it? [0] https://news.ycombinator.com/item?id=13352659 https://news.ycombinator.com/item?id=13352659
    - minaguib 4y ago
      Interesting. Never tried dgsh before, but yea it looks like my poor-man's CLI workflow for a lot of tasks

111 points by hfmuehleisen 4y ago

Well I feel silly, based on a slight mis-reading of the title, I totally thought that Range was some company that was acquired by DuckDB.

deleted 4y ago
[deleted]
tracerfett 4y ago
hawkfish-rmgw 4y ago
Oh dear I can see that - sorry for the confusion! I'll see if we can come up with something a bit longer. It was a bit nerdy...
zxv 4y ago
I also thought that DuckDB had acquired a company named Range. Interesting article regardless!
- qorrect 4y ago
  I was thinking there was some 10x programmer known only as Range that had joined.

eatonphil 4y ago

I've been beating my head trying to get duckdb to statically link into a Go program (I'm neither an expert with cgo nor ld). If anyone else has been able to do this I'd love to see your build steps. https://github.com/marcboeker/go-duckdb https://github.com/marcboeker/go-duckdb produces a non-static binary by default.

knome 4y ago
I'm not familiar with the project. Does it use any net-related code? That won't be static because it will want to load C-libs for using /etc/nsswitch.conf to handle DNS/name stuff. https://stackoverflow.com/questions/33228809/why-is-my-go-application-not-statically-linked https://stackoverflow.com/questions/33228809/why-is-my-go-ap...
- eatonphil 4y ago
  I don't have the source code in a good state to publish yet but here's where I'm at. At some point before this CGO_LDFLAGS does work and the header is found (omit the -ldflags args). But when it goes to statically link it can no longer find the header. CGO_LDFLAGS="-L$(pwd)/duckdb/src/include" CGO_CFLAGS="-I$(pwd)/duckdb/src/include" go build -ldflags '-extldflags " -lstdc++ -lm -lduckdb -static"' # github.com/marcboeker/go-duckdb ../../go/pkg/mod/github.com/marcboeker/go-duckdb@v0.0.0-20220427142532-cd9f33e64d9a/connection.go:4:10: fatal error: duckdb.h: No such file or directory 4 | #include <duckdb.h> | ^~~~~~~~~~ compilation terminated. Edit, nevermind about not being in a good state! Here's my code: https://github.com/multiprocessio/duckdb-tests https://github.com/multiprocessio/duckdb-tests.
  - irq-1 4y ago
    Put the file in quotes. Angle brackets are for built-in files. #include "duckdb.h"
    - eatonphil 4y ago
      That's not my code.
      - eatonphil 4y ago
        But also, just to double check, I modified the vendored code and no difference: CGO_LDFLAGS="-L$(pwd)/duckdb/src/include" CGO_CFLAGS="-I$(pwd)/duckdb/src/include" go build -ldflags '-extldflags " -lstdc++ -lm -lduckdb -static"' # github.com/marcboeker/go-duckdb vendor/github.com/marcboeker/go-duckdb/connection.go:4:10: fatal error: duckdb.h: No such file or directory 4 | #include "duckdb.h" | ^~~~~~~~~~ compilation terminated.

lnsp 4y ago

Great to see new features being implemented. I'm using DuckDB for a thesis project and integrating it into my own Python CLI/web tool has been super easy -- I especially love the direct integration with DataFrames, it makes things really seamless.

crimsoneer 4y ago

I've been using DuckDb a fair bit recently and really enjoy it... When it has slightly better ide support (eg, I can use it in pycharm) and can take in geospatial data, I'll be ecstatic.

orthoxerox 4y ago

DuckDB has become my preferred tool for hardcore data wrangling. Excel is fine for like 80% of data processing tasks, but the remaining twenty percent are a pain, especially when you're CPU bound on a remote desktop. Smuggling the DuckDB JDBC driver onto said remote machine was the most productive infosec violation I've ever committed.

hawkfish-rmgw 4y ago
Our lips are sealed.
minaguib 4y ago
I don't know why I waited so long to try it. I wrangle a ton of raw and aggregate data locally every day. I've had a 10-year habit of massaging via unix CLI tools and pipes then moving to excel. I guess I didn't wanna write code. Funny thing is I love SQL. But with `duckdb_cli` it's a game-changer. I'm truly truly impressed.
- ignoramous 4y ago
  > I've had a 10-year habit of massaging via unix CLI tools and pipes then moving to excel. Have you, at any point, considered / used dgsh [0] or a similar tool? If so, how has been your experience with it? [0] https://news.ycombinator.com/item?id=13352659 https://news.ycombinator.com/item?id=13352659
  - minaguib 4y ago
    Interesting. Never tried dgsh before, but yea it looks like my poor-man's CLI workflow for a lot of tasks