3 ms·

There're more than one embedded database in Java. They were old and well tested 10 years ago. Why people keep missing this ecosystem is beyond my comprehension.

marginalia_nu 4y ago
I say this as someone who does low-level db-adjacent stuff in Java: Java's in a weird spot. On the one hand it almost definitely gets way more shit than it deserves. On the other hand there's a kernel of truth in a lot of the shit that comes its way especially in this area. Low level programming in Java is unbelievably awkward. From 2Gb limits on bytebuffers to the absence of unsigned types. Everything is just a bit too far removed from the actual hardware and OS to do this stuff really well. It's like coding with oven mitts, and undeniably there's a bit of overhead almost everywhere as a result, beyond what the JVM already introduces (which is not a lot). Another big cause of awkwardness in this space is the lack of generic algorithms. Yeah you have <Generics> but they come at a cost of boxing all your primitives, which is just untenable if you need to go fast. So because you're undoubtedly dealing with some awkward custom workaround to the 2Gb byte buffer limit, you end up having to build and maintain a custom standard library of N different sort functions, N different binary searches, N different merge functions, N different bisect functions, N different B*-tree (or whatever) implementations, etc. to support different N widths of data types (probably at least short, int, long, float, double in a dbms). Although it should be noted a lot of this is getting better with JEPs that are in the pipe and ought to arrive in a few years.
- jiggawatts 4y ago
  C# fixes most of these issues. It has unsigned types, and interop with native calls is generally much more ergonomic than in Java. There are a range of useful built-in types for handling this type of system code, such as SafeHandle[1] and Span<T>[2] where T can be 'byte', etc... Similarly, it has good support for struct (value) types, and it boxes native types far less often than Java does, for massive performance benefits. Last but not least, NET 8 will come with an ahead-of-time native compiler for faster startup times. It's about to get AVX-512 support! [3] [1] https://learn.microsoft.com/en-us/dotnet/api/system.runtime.interopservices.safehandle https://learn.microsoft.com/en-us/dotnet/api/system.runtime.... [2] https://learn.microsoft.com/en-us/dotnet/api/system.span-1 https://learn.microsoft.com/en-us/dotnet/api/system.span-1 [3] https://github.com/dotnet/runtime/issues/35773 https://github.com/dotnet/runtime/issues/35773
  - crabbone 4y ago
    No... if Java isn't used in system programming, which is what storage, and databases sort of is, C# has no hope of doing it. At least due to tradition. Same way how nobody will write Linux utilities in C++, even though it's technically possible. It's just a tradition. Storage people have a very strong allergy to anything that comes from Microsoft. You probably would have a better chance of writing storage applications in Common Lisp than C#. Only Microsoft uses its own storage products in places that matter. Even just having to integrate with Microsoft's products is seen as a huge pain that's not worth the effort. And, truth be told... I want it to stay that way. Companies like Microsoft should be kept as far away as possible from critical infrastructure. We failed on few fronts, and are paying through the nose for it. It's nice that at least on this front, this is not a problem. Also, on a personal level, no other language makes me want to puke as much as C# does. I mean, to the point that if the only programming jobs available in the world were the ones that ask for programming in C#, I'd go farming or cleaning... whatever. Just not touch that steaming pile ever again.
- crabbone 4y ago
  I worked on a distributed filesystem which had its administration tools written in Java. The system itself was written mostly in C, and in some weird Haskell / C++ generated templates that compiled to C source. I've known about a bunch of "sister" storage projects which were written in a similar combo: typically the product itself would be C (or eventually, D, more recently -- Rust), but a language like Java would be inconceivable for a system like that. Few things that these projects had in common: * Home-made concurrency (i.e. they didn't use pthreads, instead they used some green threads designed specifically for this project). * Home-made allocator. Again, similar to the threads, the default allocator options available to system programming aren't good for mission-critical software. * Home-made logging. Due to high volume, speed and storage requirement it's typically a very involved system that is far more sophisticated than any Java built-in or 3rd party logging library. * Home-made collections / containers. The reasons to have this is very similar to the reason to have custom allocators -- improved ability to deal with memory-related errors. * Home-made drivers and IP stack, or at least partial re-implementation of IP and higher-level protocols. Again, because of memory-related concerns. * Virtually no 3rd-party libraries. For close to 2M lines of C code for the core part of the system, we had like... some compression algorithm library as a dependency... and that was it, iirc. I mean, these kinds of products need to rebuild something rather minimal like C almost from the ground up... just the mere thought of using Java in this kind of environment would've been laughed out of the room. I know that databases, like relational databases are often written in Java, but in my experience, those are mostly used by Java applications and rarely leave the boundaries of the Java ecosystem. People coming from storage background rather than pure Java background wouldn't think about Java databases more than as toy projects really.
carom 4y ago
Hm, I don't think anyone is missing the Java ecosystem. It is widely used. In our case there were much better choices for interfacing with eBPF hooks. I am happy to use Java if I am doing graph database work or forced to interact with an Apache project. It is fine language and the developers are cheap. Often a good choice.
dunefox 4y ago
DuckDB is designed and used for data science, ml, etc. and Java isn't. It's as simple as that.
- marginalia_nu 4y ago
  JEPs are in the pipe to fix that though.

marginalia_nu 4y ago

I say this as someone who does low-level db-adjacent stuff in Java: Java's in a weird spot. On the one hand it almost definitely gets way more shit than it deserves. On the other hand there's a kernel of truth in a lot of the shit that comes its way especially in this area. Low level programming in Java is unbelievably awkward. From 2Gb limits on bytebuffers to the absence of unsigned types. Everything is just a bit too far removed from the actual hardware and OS to do this stuff really well. It's like coding with oven mitts, and undeniably there's a bit of overhead almost everywhere as a result, beyond what the JVM already introduces (which is not a lot). Another big cause of awkwardness in this space is the lack of generic algorithms. Yeah you have <Generics> but they come at a cost of boxing all your primitives, which is just untenable if you need to go fast. So because you're undoubtedly dealing with some awkward custom workaround to the 2Gb byte buffer limit, you end up having to build and maintain a custom standard library of N different sort functions, N different binary searches, N different merge functions, N different bisect functions, N different B*-tree (or whatever) implementations, etc. to support different N widths of data types (probably at least short, int, long, float, double in a dbms). Although it should be noted a lot of this is getting better with JEPs that are in the pipe and ought to arrive in a few years.

jiggawatts 4y ago
C# fixes most of these issues. It has unsigned types, and interop with native calls is generally much more ergonomic than in Java. There are a range of useful built-in types for handling this type of system code, such as SafeHandle[1] and Span<T>[2] where T can be 'byte', etc... Similarly, it has good support for struct (value) types, and it boxes native types far less often than Java does, for massive performance benefits. Last but not least, NET 8 will come with an ahead-of-time native compiler for faster startup times. It's about to get AVX-512 support! [3] [1] https://learn.microsoft.com/en-us/dotnet/api/system.runtime.interopservices.safehandle https://learn.microsoft.com/en-us/dotnet/api/system.runtime.... [2] https://learn.microsoft.com/en-us/dotnet/api/system.span-1 https://learn.microsoft.com/en-us/dotnet/api/system.span-1 [3] https://github.com/dotnet/runtime/issues/35773 https://github.com/dotnet/runtime/issues/35773
- crabbone 4y ago
  No... if Java isn't used in system programming, which is what storage, and databases sort of is, C# has no hope of doing it. At least due to tradition. Same way how nobody will write Linux utilities in C++, even though it's technically possible. It's just a tradition. Storage people have a very strong allergy to anything that comes from Microsoft. You probably would have a better chance of writing storage applications in Common Lisp than C#. Only Microsoft uses its own storage products in places that matter. Even just having to integrate with Microsoft's products is seen as a huge pain that's not worth the effort. And, truth be told... I want it to stay that way. Companies like Microsoft should be kept as far away as possible from critical infrastructure. We failed on few fronts, and are paying through the nose for it. It's nice that at least on this front, this is not a problem. Also, on a personal level, no other language makes me want to puke as much as C# does. I mean, to the point that if the only programming jobs available in the world were the ones that ask for programming in C#, I'd go farming or cleaning... whatever. Just not touch that steaming pile ever again.
crabbone 4y ago
I worked on a distributed filesystem which had its administration tools written in Java. The system itself was written mostly in C, and in some weird Haskell / C++ generated templates that compiled to C source. I've known about a bunch of "sister" storage projects which were written in a similar combo: typically the product itself would be C (or eventually, D, more recently -- Rust), but a language like Java would be inconceivable for a system like that. Few things that these projects had in common: * Home-made concurrency (i.e. they didn't use pthreads, instead they used some green threads designed specifically for this project). * Home-made allocator. Again, similar to the threads, the default allocator options available to system programming aren't good for mission-critical software. * Home-made logging. Due to high volume, speed and storage requirement it's typically a very involved system that is far more sophisticated than any Java built-in or 3rd party logging library. * Home-made collections / containers. The reasons to have this is very similar to the reason to have custom allocators -- improved ability to deal with memory-related errors. * Home-made drivers and IP stack, or at least partial re-implementation of IP and higher-level protocols. Again, because of memory-related concerns. * Virtually no 3rd-party libraries. For close to 2M lines of C code for the core part of the system, we had like... some compression algorithm library as a dependency... and that was it, iirc. I mean, these kinds of products need to rebuild something rather minimal like C almost from the ground up... just the mere thought of using Java in this kind of environment would've been laughed out of the room. I know that databases, like relational databases are often written in Java, but in my experience, those are mostly used by Java applications and rarely leave the boundaries of the Java ecosystem. People coming from storage background rather than pure Java background wouldn't think about Java databases more than as toy projects really.

carom 4y ago

Hm, I don't think anyone is missing the Java ecosystem. It is widely used. In our case there were much better choices for interfacing with eBPF hooks. I am happy to use Java if I am doing graph database work or forced to interact with an Apache project. It is fine language and the developers are cheap. Often a good choice.

dunefox 4y ago

DuckDB is designed and used for data science, ml, etc. and Java isn't. It's as simple as that.

marginalia_nu 4y ago
JEPs are in the pipe to fix that though.