11 ms·

Let's do this. But entirely local. Local obsidian, local LLM, and all open source. That's the future I want.

by xorvoid 1y ago

smm11 1y ago
Apple then.
- p_ing 1y ago
  That rules out the open source part.
eadmund 1y ago
Local Org mode, local LLM, all orchestrated with Emacs, all free software. If only I were retired and had infinite time!
gchamonlive 1y ago
Open-weights only are also not enough, we need control of the dataset and training pipeline. The average user like me wouldn't be able to run pipelines without serious infrastructure, but it's very important to understand how the data is used and how the models are trained, so that we own the model and can assess its biases openly.
- tsimionescu 1y ago
  Good luck understanding the biases in a petabyte of text and images and video, or whatever the training set is.
  - gchamonlive 1y ago
    Do you disagree it's important to have access to the data, ease of assessment notwithstanding?
    - scottyah 1y ago
      It is an interesting question. Of course everyone should have equal access to the data in theory, but I also believe nobody should be forced to offer it for free to others and I don't think I want to spend tax money having the government host and distribute that data. I'm not sure how everyone can have access to the data without necessitating another taking on the burden of providing it.
      - gchamonlive 1y ago
        I think torrent is a very good way to redistribute this type of data. You can even selectively sync and redistribute. I'm also not saying anyone should be forced to disclose training data. I'm only staying that a LLM that's only openweight and not open data/pipeline barely fits the opensource model of the stack mentioned by OP.
    - tsimionescu 1y ago
      I view it as more or less irrelevant. LLMs are fundamentally black boxes. Whether you run the black box locally or use it remotely, whether you train it yourself or use a pretrained version, whether you have access to the training set or not, it's completely irrelevant to control. Using an LLM means giving up control and understanding of the process. Whether it's OpenAI or the training data-guided algorithm that controls the process, it's still not you. Now, running local models instead of using them as a SaaS has a clear purpose: the price of your local model won't suddenly increase ten fold once you start depending on it, like the SaaS models might. Any level of control beyond that is illusory with LLMs.
      - gchamonlive 1y ago
        I on the other hand think it's irrelevant if a technology is a blackbox or not. If it's supposed to fit the opensource/FOSS model of the original post having access to precursors is just as important as having access to the weights. It's fine for models to have open-weights and closed data. It's only barely fitting the opensource model IMHO though.
noahbrier 1y ago
Seems like this might be possible with opencode? Haven't played much.
buzzy_hacker 1y ago
LLMs are making open source programs both more viable and more valuable. I have many programs I use that I wish were a little different, but even if they were open source, it would take a while to acquaint myself with the source code organization to make these changes. LLMs, on the other hand, are pretty good at small self-contained changes like tweaks or new minor features. This makes it easier to modify open source programs, but also means that if a program isn't open source, I can't make these changes at all. Before, I wasn't going to make the change anyway, but now that I actually can, the ability to make changes (i.e. the program is open source) becomes much more important.
- blks 1y ago
  So you’re just storing bunch of forks of open source projects with some AI-generated changes applied to them?
nwsm 1y ago
Is local not infeasible for models of useful size (at least on a typical dev machine with <= 64GB RAM and a single GPU)
CountGeek 1y ago
Maybe this is of interest https://laurentcazanove.com/blog/obsidian-rag-api https://laurentcazanove.com/blog/obsidian-rag-api
BirAdam 1y ago
LM Studio + aider

Let's do this. But entirely local. Local obsidian, local LLM, and all open source. That's the future I want.

smm11 1y ago

Apple then.

p_ing 1y ago
That rules out the open source part.

eadmund 1y ago

Local Org mode, local LLM, all orchestrated with Emacs, all free software. If only I were retired and had infinite time!

gchamonlive 1y ago

Open-weights only are also not enough, we need control of the dataset and training pipeline. The average user like me wouldn't be able to run pipelines without serious infrastructure, but it's very important to understand how the data is used and how the models are trained, so that we own the model and can assess its biases openly.

tsimionescu 1y ago
Good luck understanding the biases in a petabyte of text and images and video, or whatever the training set is.
- gchamonlive 1y ago
  Do you disagree it's important to have access to the data, ease of assessment notwithstanding?
  - scottyah 1y ago
    It is an interesting question. Of course everyone should have equal access to the data in theory, but I also believe nobody should be forced to offer it for free to others and I don't think I want to spend tax money having the government host and distribute that data. I'm not sure how everyone can have access to the data without necessitating another taking on the burden of providing it.
    - gchamonlive 1y ago
      I think torrent is a very good way to redistribute this type of data. You can even selectively sync and redistribute. I'm also not saying anyone should be forced to disclose training data. I'm only staying that a LLM that's only openweight and not open data/pipeline barely fits the opensource model of the stack mentioned by OP.
  - tsimionescu 1y ago
    I view it as more or less irrelevant. LLMs are fundamentally black boxes. Whether you run the black box locally or use it remotely, whether you train it yourself or use a pretrained version, whether you have access to the training set or not, it's completely irrelevant to control. Using an LLM means giving up control and understanding of the process. Whether it's OpenAI or the training data-guided algorithm that controls the process, it's still not you. Now, running local models instead of using them as a SaaS has a clear purpose: the price of your local model won't suddenly increase ten fold once you start depending on it, like the SaaS models might. Any level of control beyond that is illusory with LLMs.
    - gchamonlive 1y ago
      I on the other hand think it's irrelevant if a technology is a blackbox or not. If it's supposed to fit the opensource/FOSS model of the original post having access to precursors is just as important as having access to the weights. It's fine for models to have open-weights and closed data. It's only barely fitting the opensource model IMHO though.

noahbrier 1y ago

Seems like this might be possible with opencode? Haven't played much.

buzzy_hacker 1y ago

LLMs are making open source programs both more viable and more valuable. I have many programs I use that I wish were a little different, but even if they were open source, it would take a while to acquaint myself with the source code organization to make these changes. LLMs, on the other hand, are pretty good at small self-contained changes like tweaks or new minor features. This makes it easier to modify open source programs, but also means that if a program isn't open source, I can't make these changes at all. Before, I wasn't going to make the change anyway, but now that I actually can, the ability to make changes (i.e. the program is open source) becomes much more important.

blks 1y ago
So you’re just storing bunch of forks of open source projects with some AI-generated changes applied to them?

nwsm 1y ago

Is local not infeasible for models of useful size (at least on a typical dev machine with <= 64GB RAM and a single GPU)

CountGeek 1y ago

Maybe this is of interest https://laurentcazanove.com/blog/obsidian-rag-api https://laurentcazanove.com/blog/obsidian-rag-api

BirAdam 1y ago

LM Studio + aider