4 ms·

Haven't people been saying that for the last decade? I mean, eventually they will be right, maybe "about" means next year, or maybe a decade later? They just ha

Haven't people been saying that for the last decade? I mean, eventually they will be right, maybe "about" means next year, or maybe a decade later? They just have to stop making huge improvements for a few years and the investment will dry up. I really wasn't interested in computer hardware anymore (they are fast enough!) until I discovered the world of running LLMs and other AI locally. Now I actually care about computer hardware again. It is weird, I wouldn't have even opened this HN thread a year ago.

vlovich123 2y ago
What makes local AI interesting to you vs larger remote models like ChatGPT and Claude?
- adriancr 2y ago
  Not OP but for me a big thing is privacy, I can feed it personal documents and expect those to not leak. It has zero cost, hardware is already there. I'm not captive to some remote company. I can fiddle and integrate with other home sensors / automation as I want.
  - hentrep 2y ago
    Curious as I’m of the same mind - what’s your local AI setup? I’m looking to implement a local system that would ideally accommodate voice chat. I know the answer depends on my use case - mostly searching and analysis of personal documents - but would love to hear how you’ve implemented.
    - dgfitz 2y ago
      llama.ccp and time seems to be the general answer to this question.
    - adriancr 2y ago
      If you are just starting up, you can try out 'open-webui' as inspiration. After that you can just use llama.cpp to build out your own things. Hardware side, I just have a beefy server that acts as a router (mellanox card to provider fiber optic and local fiber network), firewall, wifi access point, zigbee coordinator, host to various services, camera video feed ingestion and processing, and so on...
      - red-iron-pine 2y ago
        define brefy server
- epicureanideal 2y ago
  Lack of ideological capture of the public models.
- seanmcdirmid 2y ago
  Control and freedom. You can use unharmonious models and hacks to existing models, also latency, you can actually use AI for a lot more applications when it is running locally.

vlovich123 2y ago

What makes local AI interesting to you vs larger remote models like ChatGPT and Claude?

adriancr 2y ago
Not OP but for me a big thing is privacy, I can feed it personal documents and expect those to not leak. It has zero cost, hardware is already there. I'm not captive to some remote company. I can fiddle and integrate with other home sensors / automation as I want.
- hentrep 2y ago
  Curious as I’m of the same mind - what’s your local AI setup? I’m looking to implement a local system that would ideally accommodate voice chat. I know the answer depends on my use case - mostly searching and analysis of personal documents - but would love to hear how you’ve implemented.
  - dgfitz 2y ago
    llama.ccp and time seems to be the general answer to this question.
  - adriancr 2y ago
    If you are just starting up, you can try out 'open-webui' as inspiration. After that you can just use llama.cpp to build out your own things. Hardware side, I just have a beefy server that acts as a router (mellanox card to provider fiber optic and local fiber network), firewall, wifi access point, zigbee coordinator, host to various services, camera video feed ingestion and processing, and so on...
    - red-iron-pine 2y ago
      define brefy server
epicureanideal 2y ago
Lack of ideological capture of the public models.
seanmcdirmid 2y ago
Control and freedom. You can use unharmonious models and hacks to existing models, also latency, you can actually use AI for a lot more applications when it is running locally.