4 ms·
How well can you access graphics drivers and cuda cores for doing any kind of local large language model stuff using java? I know that there is a project called
by Onawa 3y ago
How well can you access graphics drivers and cuda cores for doing any kind of local large language model stuff using java? I know that there is a project called MLC for machine learning compilation that is meant to be written in Python and then transcribed using machine learning to a native language for the platform so that you can use mobile processors and laptops and stuff more efficiently for small language models.
- water9 3y agoJava has the Java native interface where you can interface with c/c++ written code
- nradov 3y agoJNI is kind of a hassle. If most of the heavy lifting is being done by native code then using Java becomes pointless. Might as well just go with Python which makes calling native code a bit easier.
- cempaka 3y agoProject Panama is meant to make this a lot nicer, and they often explicitly mention trying to stay competitive in machine learning as a key impetus for it. But yeah, JNI is so awful even the language architects are compelled to trash it and apologize for how bad it is.
- water9 3y agoI cant argue with you there, but Java isn’t that much slower than c++ now a-days. especially on an X86 platform where there’s been lots of time to tune the JVM.
- nradov 3y agoSure, Java performance is good but again if most of the real work is being done in native code then it doesn't really matter.