4 ms·

Memory safety is something that needs to be mentioned. I was integrating DuckDB into a project and ended up ripping it out after running into memory corruption

by carom 4y ago

Memory safety is something that needs to be mentioned. I was integrating DuckDB into a project and ended up ripping it out after running into memory corruption issue in practice. Upon investigation they had a massive issue of fuzzer found bugs on their GitHub. While I am glad they are fuzzing and finding issues, I cannot ship that onto customer systems. We have a few very good memory safe programming languages at this point. Please do not start a project in C/C++ unless you are truly exceptional and understand memory management and exploitation inside and out. I switched to SQLite on the project since it is one of the more fuzzed applications out there that fit the need. The next embeddable database I use (bonus if it works on cloud) will need to be in a memory safe language.

tyingq 4y ago
If you're writing an embeddable database, exposing it as a C library maximizes the potential user base. Since most languages have some way to do ffi with C, most platforms have C compilers, etc.
- josephg 4y ago
  C FFI goes both ways. It’s also pretty easy to compile rust or zig code into a static or dynamic C library. And then that library can be used directly from C or from anything that has a C FFI (Python, Ruby, javascript, Java, etc etc). I’ve been working on an iOS app lately with some of the core code in rust (shared with other platforms) and the UI in native swift. Changing the API at the boundary is a bit annoying, but the resulting app works great.
  - plagiarist 4y ago
    What are the ergonomics of compiling that binary library? I think this sounds really neat, if you have a build system that produces something like an .xcframework and corresponding package file it wouldn't even be a pain point.
    - josephg 4y ago
      Yeah; thats exactly what I have. I have a script which compiles the rust code for all the combinations of hardware (arm & x86_64 on ios & macos) and produces an .xcframework. Then I just recompile my xcode project. Its a bit hacky, but it works fine. I'm using the swift_bridge crate to generate a lot of the bindings and generate the .xcframework. https://github.com/josephg/replica/blob/master/build_swift.sh https://github.com/josephg/replica/blob/master/build_swift.s...
vbezhenar 4y ago
There're more than one embedded database in Java. They were old and well tested 10 years ago. Why people keep missing this ecosystem is beyond my comprehension.
- marginalia_nu 4y ago
  I say this as someone who does low-level db-adjacent stuff in Java: Java's in a weird spot. On the one hand it almost definitely gets way more shit than it deserves. On the other hand there's a kernel of truth in a lot of the shit that comes its way especially in this area. Low level programming in Java is unbelievably awkward. From 2Gb limits on bytebuffers to the absence of unsigned types. Everything is just a bit too far removed from the actual hardware and OS to do this stuff really well. It's like coding with oven mitts, and undeniably there's a bit of overhead almost everywhere as a result, beyond what the JVM already introduces (which is not a lot). Another big cause of awkwardness in this space is the lack of generic algorithms. Yeah you have <Generics> but they come at a cost of boxing all your primitives, which is just untenable if you need to go fast. So because you're undoubtedly dealing with some awkward custom workaround to the 2Gb byte buffer limit, you end up having to build and maintain a custom standard library of N different sort functions, N different binary searches, N different merge functions, N different bisect functions, N different B*-tree (or whatever) implementations, etc. to support different N widths of data types (probably at least short, int, long, float, double in a dbms). Although it should be noted a lot of this is getting better with JEPs that are in the pipe and ought to arrive in a few years.
  - jiggawatts 4y ago
    C# fixes most of these issues. It has unsigned types, and interop with native calls is generally much more ergonomic than in Java. There are a range of useful built-in types for handling this type of system code, such as SafeHandle[1] and Span<T>[2] where T can be 'byte', etc... Similarly, it has good support for struct (value) types, and it boxes native types far less often than Java does, for massive performance benefits. Last but not least, NET 8 will come with an ahead-of-time native compiler for faster startup times. It's about to get AVX-512 support! [3] [1] https://learn.microsoft.com/en-us/dotnet/api/system.runtime.interopservices.safehandle https://learn.microsoft.com/en-us/dotnet/api/system.runtime.... [2] https://learn.microsoft.com/en-us/dotnet/api/system.span-1 https://learn.microsoft.com/en-us/dotnet/api/system.span-1 [3] https://github.com/dotnet/runtime/issues/35773 https://github.com/dotnet/runtime/issues/35773

by carom 4y ago

tyingq 4y ago

If you're writing an embeddable database, exposing it as a C library maximizes the potential user base. Since most languages have some way to do ffi with C, most platforms have C compilers, etc.

josephg 4y ago
C FFI goes both ways. It’s also pretty easy to compile rust or zig code into a static or dynamic C library. And then that library can be used directly from C or from anything that has a C FFI (Python, Ruby, javascript, Java, etc etc). I’ve been working on an iOS app lately with some of the core code in rust (shared with other platforms) and the UI in native swift. Changing the API at the boundary is a bit annoying, but the resulting app works great.
- plagiarist 4y ago
  What are the ergonomics of compiling that binary library? I think this sounds really neat, if you have a build system that produces something like an .xcframework and corresponding package file it wouldn't even be a pain point.
  - josephg 4y ago
    Yeah; thats exactly what I have. I have a script which compiles the rust code for all the combinations of hardware (arm & x86_64 on ios & macos) and produces an .xcframework. Then I just recompile my xcode project. Its a bit hacky, but it works fine. I'm using the swift_bridge crate to generate a lot of the bindings and generate the .xcframework. https://github.com/josephg/replica/blob/master/build_swift.sh https://github.com/josephg/replica/blob/master/build_swift.s...

vbezhenar 4y ago

There're more than one embedded database in Java. They were old and well tested 10 years ago. Why people keep missing this ecosystem is beyond my comprehension.

marginalia_nu 4y ago
I say this as someone who does low-level db-adjacent stuff in Java: Java's in a weird spot. On the one hand it almost definitely gets way more shit than it deserves. On the other hand there's a kernel of truth in a lot of the shit that comes its way especially in this area. Low level programming in Java is unbelievably awkward. From 2Gb limits on bytebuffers to the absence of unsigned types. Everything is just a bit too far removed from the actual hardware and OS to do this stuff really well. It's like coding with oven mitts, and undeniably there's a bit of overhead almost everywhere as a result, beyond what the JVM already introduces (which is not a lot). Another big cause of awkwardness in this space is the lack of generic algorithms. Yeah you have <Generics> but they come at a cost of boxing all your primitives, which is just untenable if you need to go fast. So because you're undoubtedly dealing with some awkward custom workaround to the 2Gb byte buffer limit, you end up having to build and maintain a custom standard library of N different sort functions, N different binary searches, N different merge functions, N different bisect functions, N different B*-tree (or whatever) implementations, etc. to support different N widths of data types (probably at least short, int, long, float, double in a dbms). Although it should be noted a lot of this is getting better with JEPs that are in the pipe and ought to arrive in a few years.
- jiggawatts 4y ago
  C# fixes most of these issues. It has unsigned types, and interop with native calls is generally much more ergonomic than in Java. There are a range of useful built-in types for handling this type of system code, such as SafeHandle[1] and Span<T>[2] where T can be 'byte', etc... Similarly, it has good support for struct (value) types, and it boxes native types far less often than Java does, for massive performance benefits. Last but not least, NET 8 will come with an ahead-of-time native compiler for faster startup times. It's about to get AVX-512 support! [3] [1] https://learn.microsoft.com/en-us/dotnet/api/system.runtime.interopservices.safehandle https://learn.microsoft.com/en-us/dotnet/api/system.runtime.... [2] https://learn.microsoft.com/en-us/dotnet/api/system.span-1 https://learn.microsoft.com/en-us/dotnet/api/system.span-1 [3] https://github.com/dotnet/runtime/issues/35773 https://github.com/dotnet/runtime/issues/35773