4 ms·

Moreover, if it's about cross-compilation, why not using zig? https://zig.news/kristoff/building-sqlite-with-cgo-for-every-os-4cic https://zig.news/kristoff/bu

by acqq 3y ago

Moreover, if it's about cross-compilation, why not using zig? https://zig.news/kristoff/building-sqlite-with-cgo-for-every-os-4cic https://zig.news/kristoff/building-sqlite-with-cgo-for-every...

ncruces 3y ago
It's not just about cross compilation. There's also debuggability and other aspects of development. It's an anecdote, but I've had a better experience debugging issues while developing SQLite extensions and VFSes for this driver than I've had in other "cross runtime" / "bindings" work I've done before (with Cgo, JNI, P/Invoke).
- acqq 3y ago
  I would need to read more about that to understand how it could be a problem. You see, I've literally spend decades using C and I can never imagine how adding that what I understand to be huge dependencies like wazero.io and executing huge wasms binaries made from original C sources can be more debuggable than debugging a directly compiled native code -- I'm just disclosing my bias, and I want to be clear about it, I'm certainly not doubting your "better experience." Edit: have you tried that zig workflow for cross-compiling? Edit2: and I'm also aware that zig also uses wasm binary and its own wasm2c, written from scratch, for its own bootstrapping: https://ziglang.org/news/goodbye-cpp/ https://ziglang.org/news/goodbye-cpp/
  - ncruces 3y ago
    I don't mean debugging the C bits (although having a stack trace and an exact line number when I cause SQLite to crash is rather nice). I mean developing a complex wrapper/binding that involves Go calls C, which then calls back to Go, which again calls C, and putting breakpoints on the Go bits multiple layers deep, and being able to inspect the full stack of those Go bits. Last I've tried with Cgo, stuff has a high probability of hanging your entire process (e.g. if the C library acquired some lock or something). There's also the permanent fear that (bugs, API misuse…) leads to the C library corrupting your Go memory, which is much harder to do if the C bits run in a sandbox. SQLite is "imune" to bugs, but not API misuse. Tbh, I haven't had a better experience with JNI, JNA, PInvoke… And yes, I've looked at zig, even used it initially to build the WASM blob. I actually think it'd be a worthy endeavour to build a better Cgo driver. But having Cgo and WASM versions of this in the same repo is just too much effort at this point (I've considered it).
- electroly 3y ago
  I'm curious what issues you had with P/Invoke. I use SQLite's C API from C# using P/Invoke rather than using a high-level .NET library. I've implemented virtual table modules, VFSes, application-defined SQL functions, you name it. I even reach into SQLite and call their internal tokenizer directly. The Visual Studio debugger was key to getting this done productively. With C#'s high level of interoperability with C, I can't imagine a similar "SQLite transpiled to C#" project gaining traction in our ecosystem. I think C# stands out from the crowd on interop with C libraries.
  - ncruces 3y ago
    Issue one: someone else was employed professionally to develope those bindings for you. :) Once that's done correctly and at a high quality, your life is easier. Developing those bindings is another story. But PInvoke is one of the best development stories, I'm not dinging on it.
    - electroly 3y ago
      No, I wrote my own bindings to the C API with P/Invoke. I _don't_ use any other high-level library in .NET. Previously I used C++/CLI, but I migrated it all to vanilla C# with P/Invoke. I have significant experience with the SQLite C API (which is why I don't bother with other people's bindings). With P/Invoke it was a pretty straightforward experience, and specifically the debugger worked great and was crucial to getting it working. I really think that C# with P/Invoke and the Visual Studio debugger is a cut above the rest.
      - ncruces 3y ago
        Oh, great! I assumed you meant the officially supported System.Data.SQLite bindings. Do you have a link or is this internal?
        electroly 3y ago
        The meat of the P/Invoke code is in here: https://github.com/electroly/sqlnotebook/tree/master/src/SqlNotebookScript/Core/SqliteInterop https://github.com/electroly/sqlnotebook/tree/master/src/Sql... The parent directory includes code that uses it. I'm most proud of this SQLite virtual table module that proxies queries to remote ADO.NET connections, allowing you to write joins directly between local SQLite tables and remote SQL Server tables. https://github.com/electroly/sqlnotebook/blob/master/src/SqlNotebookScript/Core/AdoModules/AdoModuleProvider.cs https://github.com/electroly/sqlnotebook/blob/master/src/Sql... This is exposed to the user as a "LINK" option on the "IMPORT DATABASE" statement: https://sqlnotebook.com/import-database-stmt.html https://sqlnotebook.com/import-database-stmt.html I've also got a generic virtual table module that lets me easily write table-valued functions in C#: https://github.com/electroly/sqlnotebook/blob/master/src/SqlNotebookScript/Core/GenericModules/GenericModuleProvider.cs https://github.com/electroly/sqlnotebook/blob/master/src/Sql... I don't have too many table-valued functions yet but one example is the "LIST_XLS_WORKSHEETS" function: https://sqlnotebook.com/list-xls-worksheets-func.html https://sqlnotebook.com/list-xls-worksheets-func.html I have my own implementation of SQLite's grammar so I can embed it inside my larger grammar which adds things like DECLARE, IF, and WHILE: https://github.com/electroly/sqlnotebook/blob/master/src/SqlNotebookScript/Interpreter/SqliteGrammar.cs https://github.com/electroly/sqlnotebook/blob/master/src/Sql... The goal is to provide various "supercharged" features to base SQLite by taking advantage of all the extension points I can. I wish some went further; in particular the virtual table API doesn't "push down" enough of the original query to allow the module to avoid doing N+1 queries in some cases. Always happy to see people using SQLite in unique ways!
- neonsunset 3y ago
  Nothing ever comes close to P/Invoke in ease of use* (while remaining practically zero-overhead if you don't need to marshal strings). JNI is on the opposite side of the spectrum and should never be even compared as it is at least ten times worse developer experiences (and equally higher overhead). *in garbage-collected languages, otherwise it's pretty much like calling C from Rust.
  - ncruces 3y ago
    I'm sorry. The debugging experience of trying to whip up PInvoke wrappers for the COM library for Kinect 2 that worked with Unity Engine's awful version of Mono was charring. Also had “fun” building some Xamarin Android PInvoke-to-JNI bindings. I just died a little bit remembering this.
    - neonsunset 3y ago
      No language has good interop story with Android unless there's no interop and you are using Java/Kotlin. As for the first argument - this really does not represent the average experience in the current day ecosystem. Kinect 2 was a thing 9 years ago and together with Unity it's pretty much an exotic scenario. Unity in general is really not a good example until they get Unity 6 out and move over to vanilla .NET.
acqq 3y ago
To be clearer to those who don't read what's linked but reflexively downvote: zig is not only the language, it's also a build system that compiles C (also in Go CGO use scenarios) better than the default C compilers, allowing order of magnitude easier cross compilation. It's described in the link I've already provided exactly by compiling SqLite for its use in Go!

by acqq 3y ago

Moreover, if it's about cross-compilation, why not using zig? https://zig.news/kristoff/building-sqlite-with-cgo-for-every-os-4cic https://zig.news/kristoff/building-sqlite-with-cgo-for-every...

ncruces 3y ago

It's not just about cross compilation. There's also debuggability and other aspects of development. It's an anecdote, but I've had a better experience debugging issues while developing SQLite extensions and VFSes for this driver than I've had in other "cross runtime" / "bindings" work I've done before (with Cgo, JNI, P/Invoke).

acqq 3y ago
I would need to read more about that to understand how it could be a problem. You see, I've literally spend decades using C and I can never imagine how adding that what I understand to be huge dependencies like wazero.io and executing huge wasms binaries made from original C sources can be more debuggable than debugging a directly compiled native code -- I'm just disclosing my bias, and I want to be clear about it, I'm certainly not doubting your "better experience." Edit: have you tried that zig workflow for cross-compiling? Edit2: and I'm also aware that zig also uses wasm binary and its own wasm2c, written from scratch, for its own bootstrapping: https://ziglang.org/news/goodbye-cpp/ https://ziglang.org/news/goodbye-cpp/
- ncruces 3y ago
  I don't mean debugging the C bits (although having a stack trace and an exact line number when I cause SQLite to crash is rather nice). I mean developing a complex wrapper/binding that involves Go calls C, which then calls back to Go, which again calls C, and putting breakpoints on the Go bits multiple layers deep, and being able to inspect the full stack of those Go bits. Last I've tried with Cgo, stuff has a high probability of hanging your entire process (e.g. if the C library acquired some lock or something). There's also the permanent fear that (bugs, API misuse…) leads to the C library corrupting your Go memory, which is much harder to do if the C bits run in a sandbox. SQLite is "imune" to bugs, but not API misuse. Tbh, I haven't had a better experience with JNI, JNA, PInvoke… And yes, I've looked at zig, even used it initially to build the WASM blob. I actually think it'd be a worthy endeavour to build a better Cgo driver. But having Cgo and WASM versions of this in the same repo is just too much effort at this point (I've considered it).
electroly 3y ago
I'm curious what issues you had with P/Invoke. I use SQLite's C API from C# using P/Invoke rather than using a high-level .NET library. I've implemented virtual table modules, VFSes, application-defined SQL functions, you name it. I even reach into SQLite and call their internal tokenizer directly. The Visual Studio debugger was key to getting this done productively. With C#'s high level of interoperability with C, I can't imagine a similar "SQLite transpiled to C#" project gaining traction in our ecosystem. I think C# stands out from the crowd on interop with C libraries.
- ncruces 3y ago
  Issue one: someone else was employed professionally to develope those bindings for you. :) Once that's done correctly and at a high quality, your life is easier. Developing those bindings is another story. But PInvoke is one of the best development stories, I'm not dinging on it.
  - electroly 3y ago
    No, I wrote my own bindings to the C API with P/Invoke. I _don't_ use any other high-level library in .NET. Previously I used C++/CLI, but I migrated it all to vanilla C# with P/Invoke. I have significant experience with the SQLite C API (which is why I don't bother with other people's bindings). With P/Invoke it was a pretty straightforward experience, and specifically the debugger worked great and was crucial to getting it working. I really think that C# with P/Invoke and the Visual Studio debugger is a cut above the rest.
    - ncruces 3y ago
      Oh, great! I assumed you meant the officially supported System.Data.SQLite bindings. Do you have a link or is this internal?
      - electroly 3y ago
        The meat of the P/Invoke code is in here: https://github.com/electroly/sqlnotebook/tree/master/src/SqlNotebookScript/Core/SqliteInterop https://github.com/electroly/sqlnotebook/tree/master/src/Sql... The parent directory includes code that uses it. I'm most proud of this SQLite virtual table module that proxies queries to remote ADO.NET connections, allowing you to write joins directly between local SQLite tables and remote SQL Server tables. https://github.com/electroly/sqlnotebook/blob/master/src/SqlNotebookScript/Core/AdoModules/AdoModuleProvider.cs https://github.com/electroly/sqlnotebook/blob/master/src/Sql... This is exposed to the user as a "LINK" option on the "IMPORT DATABASE" statement: https://sqlnotebook.com/import-database-stmt.html https://sqlnotebook.com/import-database-stmt.html I've also got a generic virtual table module that lets me easily write table-valued functions in C#: https://github.com/electroly/sqlnotebook/blob/master/src/SqlNotebookScript/Core/GenericModules/GenericModuleProvider.cs https://github.com/electroly/sqlnotebook/blob/master/src/Sql... I don't have too many table-valued functions yet but one example is the "LIST_XLS_WORKSHEETS" function: https://sqlnotebook.com/list-xls-worksheets-func.html https://sqlnotebook.com/list-xls-worksheets-func.html I have my own implementation of SQLite's grammar so I can embed it inside my larger grammar which adds things like DECLARE, IF, and WHILE: https://github.com/electroly/sqlnotebook/blob/master/src/SqlNotebookScript/Interpreter/SqliteGrammar.cs https://github.com/electroly/sqlnotebook/blob/master/src/Sql... The goal is to provide various "supercharged" features to base SQLite by taking advantage of all the extension points I can. I wish some went further; in particular the virtual table API doesn't "push down" enough of the original query to allow the module to avoid doing N+1 queries in some cases. Always happy to see people using SQLite in unique ways!
neonsunset 3y ago
Nothing ever comes close to P/Invoke in ease of use* (while remaining practically zero-overhead if you don't need to marshal strings). JNI is on the opposite side of the spectrum and should never be even compared as it is at least ten times worse developer experiences (and equally higher overhead). *in garbage-collected languages, otherwise it's pretty much like calling C from Rust.
- ncruces 3y ago
  I'm sorry. The debugging experience of trying to whip up PInvoke wrappers for the COM library for Kinect 2 that worked with Unity Engine's awful version of Mono was charring. Also had “fun” building some Xamarin Android PInvoke-to-JNI bindings. I just died a little bit remembering this.
  - neonsunset 3y ago
    No language has good interop story with Android unless there's no interop and you are using Java/Kotlin. As for the first argument - this really does not represent the average experience in the current day ecosystem. Kinect 2 was a thing 9 years ago and together with Unity it's pretty much an exotic scenario. Unity in general is really not a good example until they get Unity 6 out and move over to vanilla .NET.

acqq 3y ago

To be clearer to those who don't read what's linked but reflexively downvote: zig is not only the language, it's also a build system that compiles C (also in Go CGO use scenarios) better than the default C compilers, allowing order of magnitude easier cross compilation. It's described in the link I've already provided exactly by compiling SqLite for its use in Go!