5 ms·

Can someone explain why these types of projects are compelling (running an LLM in the user's browser?) I guess privacy and ease-of-use? So you don't need to d

Can someone explain why these types of projects are compelling (running an LLM in the user's browser?) I guess privacy and ease-of-use? So you don't need to download llama.cpp, and run it in the terminal or something?

Veen 3y ago
Yeah, ease of use. I haven't wanted to bother with running local LLMs until now because I expected it would be complicated and time consuming (and I'm technically competent). I managed to get this up-and-running in about five minutes.
- sourcecodeplz 3y ago
  I thought the same but llama.cpp is literally two downloads in the same folder and then you double click an .exe
  - EGreg 3y ago
    You lost me at .exe
  - woah 3y ago
    Reminds me of the HN comments on the original dropbox announcement: https://news.ycombinator.com/item?id=8863 https://news.ycombinator.com/item?id=8863
    - AmericanChopper 3y ago
      I know this comment is a bit of a meme, but in my experience “well actually you can already do this on Linux” is almost universally a calling card of an impending bad opinion.
      - tpoacher 3y ago
        Why bad? OP from 2007 still has a point. Actually, given dropbox's deterioration over time, FTP+SVN is sounding pretty good to me right now.
        AmericanChopper 3y ago
        Because the “do it yourself on linux” advocate tends to ignore all of the reasons why a person might prefer an easy to use, 3rd party managed services. Which could include not wanting to accept all of the compromises required to use Linux for personal computing, not having the competence to implement a Linux solution, or not having the motivation or time to maintain their own Linux-based services. So even when the solutions they advocate are perfectly decent, it tends to be associated with a lack of insight into how small of a segment people with the same competencies and preferences as them represent.
zhte415 3y ago
As well as privacy, cheaper and more scalable for a web service to use a user's resources than server side jumps out at me.
b_mc2 3y ago
A reason I like it is I have an "older" AMD GPU which is no longer supported by ROCm (sort of AMDs version of Cuda) which means running locally I'm either trying to figure out older ROCm builds to use my GPU and running into dependency issues or using my CPU which isn't that great either. But with WebGPU I'm able to run these models on my GPU which has been much faster than using the .cpp builds. Its also fairly easy to route a Flask server to these models with websockets, so with that I've been able to run python and pass data to the model to run on the GPU and pass the response back to the program. Again, there's probably a better way but its cool to have my own personal API for a LLM.

Veen 3y ago

Yeah, ease of use. I haven't wanted to bother with running local LLMs until now because I expected it would be complicated and time consuming (and I'm technically competent). I managed to get this up-and-running in about five minutes.

sourcecodeplz 3y ago
I thought the same but llama.cpp is literally two downloads in the same folder and then you double click an .exe
- EGreg 3y ago
  You lost me at .exe
- woah 3y ago
  Reminds me of the HN comments on the original dropbox announcement: https://news.ycombinator.com/item?id=8863 https://news.ycombinator.com/item?id=8863
  - AmericanChopper 3y ago
    I know this comment is a bit of a meme, but in my experience “well actually you can already do this on Linux” is almost universally a calling card of an impending bad opinion.
    - tpoacher 3y ago
      Why bad? OP from 2007 still has a point. Actually, given dropbox's deterioration over time, FTP+SVN is sounding pretty good to me right now.
      - AmericanChopper 3y ago
        Because the “do it yourself on linux” advocate tends to ignore all of the reasons why a person might prefer an easy to use, 3rd party managed services. Which could include not wanting to accept all of the compromises required to use Linux for personal computing, not having the competence to implement a Linux solution, or not having the motivation or time to maintain their own Linux-based services. So even when the solutions they advocate are perfectly decent, it tends to be associated with a lack of insight into how small of a segment people with the same competencies and preferences as them represent.

zhte415 3y ago

As well as privacy, cheaper and more scalable for a web service to use a user's resources than server side jumps out at me.

b_mc2 3y ago

A reason I like it is I have an "older" AMD GPU which is no longer supported by ROCm (sort of AMDs version of Cuda) which means running locally I'm either trying to figure out older ROCm builds to use my GPU and running into dependency issues or using my CPU which isn't that great either. But with WebGPU I'm able to run these models on my GPU which has been much faster than using the .cpp builds. Its also fairly easy to route a Flask server to these models with websockets, so with that I've been able to run python and pass data to the model to run on the GPU and pass the response back to the program. Again, there's probably a better way but its cool to have my own personal API for a LLM.