4 ms·

I feel like for many tasks, there's a certain "good enough" threshold that local small LMs can do as good but private and no cloud LLM is needed. I think the fu

I feel like for many tasks, there's a certain "good enough" threshold that local small LMs can do as good but private and no cloud LLM is needed. I think the future is mostly on-device SLMs and their agentic coordination. In that sense, a local agentic framework (js/ts based) would be soon very relevant.

digdugdirk 2y ago
Any reason why you're calling out a need for a framework to be js/ts based? There's plenty of python frameworks in active development, some of which have js bindings/libraries.
- mentalgear 2y ago
  minimal setup. Anything python requires the host machine to have the correct python version, pm and libs installed (which is far more than normal users can do), or have it compiled within a virtual python executable (big!). Web-native technology requires minimal setup as it can basically run in any browser (or electron) as is.
  - PaulHoule 2y ago
    uv has revolutionized the Python situation, mostly. I recently updated YOShInOn's Python environment to be repeatable, the python packaging part was pretty simple, but getting CUDA running in WSL2 was a little tricky. Turns out my "game ready" NVIDIA drivers in Windows install a certain version of the base CUDA libs in WSL somehow. You have to install another 5 libraries with deb packaging inside the WSL which is not too hard but I realized I had a version mismatch about 1/3 of the way through but decided to barrel ahead. I installed most of the deb's manually but got fatigued and installed the last one automatically. Somehow it all works so I'm not messing with it, but I am a bit intimidated about what to do if I have problems and need to tear it down.
    - mentalgear 2y ago
      uv is still a cli tool, maybe ok for tech enthusiasts. No normal user will ever install a cli tool. And even for knowledgable users like yourself, there are a manifold of different problems as you stated.
      - PaulHoule 2y ago
        It can be called from another application and hidden, probably a mostly Java application could uv up a Python environment for one or more subsystems. My beef is with CUDA being not just one thing but about 5 libs if you wanna use PyTorch. In the old days I figured how to make conda packages for all that NVIDIA stuff so I could single step install any version of Tensorflow and it was 100% correct. Just then they changed Tensorflow so it was 95% correct but worked most of the time for most people and my system didn’t work anymore.
        mentalgear 2y ago
        This again requires Java Virtual Machine to be installed on the system, which most normal user don't have and won't install.

digdugdirk 2y ago

Any reason why you're calling out a need for a framework to be js/ts based? There's plenty of python frameworks in active development, some of which have js bindings/libraries.

mentalgear 2y ago
minimal setup. Anything python requires the host machine to have the correct python version, pm and libs installed (which is far more than normal users can do), or have it compiled within a virtual python executable (big!). Web-native technology requires minimal setup as it can basically run in any browser (or electron) as is.
- PaulHoule 2y ago
  uv has revolutionized the Python situation, mostly. I recently updated YOShInOn's Python environment to be repeatable, the python packaging part was pretty simple, but getting CUDA running in WSL2 was a little tricky. Turns out my "game ready" NVIDIA drivers in Windows install a certain version of the base CUDA libs in WSL somehow. You have to install another 5 libraries with deb packaging inside the WSL which is not too hard but I realized I had a version mismatch about 1/3 of the way through but decided to barrel ahead. I installed most of the deb's manually but got fatigued and installed the last one automatically. Somehow it all works so I'm not messing with it, but I am a bit intimidated about what to do if I have problems and need to tear it down.
  - mentalgear 2y ago
    uv is still a cli tool, maybe ok for tech enthusiasts. No normal user will ever install a cli tool. And even for knowledgable users like yourself, there are a manifold of different problems as you stated.
    - PaulHoule 2y ago
      It can be called from another application and hidden, probably a mostly Java application could uv up a Python environment for one or more subsystems. My beef is with CUDA being not just one thing but about 5 libs if you wanna use PyTorch. In the old days I figured how to make conda packages for all that NVIDIA stuff so I could single step install any version of Tensorflow and it was 100% correct. Just then they changed Tensorflow so it was 95% correct but worked most of the time for most people and my system didn’t work anymore.
      - mentalgear 2y ago
        This again requires Java Virtual Machine to be installed on the system, which most normal user don't have and won't install.