3 ms·

Gimme an actual example instead of downvoting, help me learn. Edit on that too : makes me think of OpenAI Whisper as a service via /e/OS and supposedly anonymo

Gimme an actual example instead of downvoting, help me learn. Edit on that too : makes me think of OpenAI Whisper as a service via /e/OS and supposedly anonymous proxying (by mixing), namely running STT remotely. That would be an actual potential usage... but IMHO that's low end enough to be run locally. So I'm still looking for an application here.

wat10000 1y ago
Are you looking for a general application of LLMs too large to run locally? Because anything you might use remote inference for, you might want to use privately.
- utopiah 1y ago
  Sure that'd do, what - useful thing (according to someone specific requirements, maybe hallucinations are OK, maybe not) that - needs privacy (for example generating code that will be open source probably does not need that) - can't be run locally - can be trusted to actually process as said it does
  - wat10000 1y ago
    I've found LLMs to be extremely useful for writing helper scripts (debugger enhancements, analyzing large disassembly dumps, that sort of thing) and as a next-level source code search. Take a large code base and you get an error in component A involving a type in component B and it's not immediately obvious how the two are connected. I've had great success in giving an LLM the error message and access to the code and asking it how B got to A and why that's an error. This is something I could certainly do myself, but there are times when it would take 100x longer. The key is that these are all things I can verify without much difficulty: read over the script, spot-check the analysis, look at the claimed connection between A and B and see if it's real. And I don't really care about style, quality, maintainability. You certainly can run this locally, but anything that will fit into reasonable local hardware won't be as good. I don't need to trust it to process as it says it does, because I'm verifying the output. And as far as I'm concerned, "needs privacy" is always true. I don't care if the code will be open source. I don't care if it's analyzing existing code that's already open source. Other people have no business seeing what I'm doing unless I explicitly allow it. In any case, I work on a lot of proprietary code as well, and my employer would be most displeased if I were exposing it to others.
fragmede 1y ago
> would I actually run on it if I'm not a spammer? > Gimme an actual example instead of downvoting, help me learn. Basically you asked a bunch of people on a privacy minded forum, why should they be allowed to encrypt their data? What are you (they) hiding!? Are you a spammer??? Apple is beloved for their stance on privacy, and you basically called everyone who thinks that's more than marketing, a spammer. And before you start arguing no you didn't, it doesn't matter that you didn't, what matters is that that's how your comment made people feel. You can say they're the stupid ones because that's not what you wrote, but if you're genuinely asking for feedback about the downvotes, there you are. You seriously can't imagine any reason to want to use an LLM privately other than to use it to write spam bots and to spam people? At the very least expand your scope past spamming to, like, also using it to write ransomware. The proprietary models that can't be run locally are SOTA and local models, even if they can come close, simply aren't what people want.
- utopiah 1y ago
  Seems I formulated my question in a way that wasn't clear. I specifically like the privacy aspect (even though honestly I think most people in this forum claim they do and yet they rely on BigTech with which they their data, so IMHO most people on HN are not as demanding as you describe) the question is precisely about what to run. FWIW I specifically keep a page on self hosting AI (you can check if you want to see if it's real at https://fabien.benetou.fr/Content/SelfHostingArtificialIntelligence https://fabien.benetou.fr/Content/SelfHostingArtificialIntel... ) so again, the privacy aspect is crucial to me. The question is ... what to actually run. What can't be run locally that would be useful. What model for which tasks. For example coding (which I don't think models are good enough for) typically would NOT need this because one wouldn't share any PII over there, hopefully would even instead publish the resulting code as open source. So my provocation about spammer is... because they often ARE actual users of LLMs. They use LLM for their language capabilities, namely craft a message that is always slightly different yet convey (roughly) the same meaning (the scam) to avoid detection. A random person though using LLM might NOT be OK with hallucinations when they use it for their own private journal or chat. So... what for? Edit: I did use Apple for years, recommending by someone at Mozilla, and I moved away from them earlier this year precisely because even though they are better than others, e.g Google, IMHO it's just not good enough for me. No intermediary is better than one with closed source.

wat10000 1y ago

Are you looking for a general application of LLMs too large to run locally? Because anything you might use remote inference for, you might want to use privately.

utopiah 1y ago
Sure that'd do, what - useful thing (according to someone specific requirements, maybe hallucinations are OK, maybe not) that - needs privacy (for example generating code that will be open source probably does not need that) - can't be run locally - can be trusted to actually process as said it does
- wat10000 1y ago
  I've found LLMs to be extremely useful for writing helper scripts (debugger enhancements, analyzing large disassembly dumps, that sort of thing) and as a next-level source code search. Take a large code base and you get an error in component A involving a type in component B and it's not immediately obvious how the two are connected. I've had great success in giving an LLM the error message and access to the code and asking it how B got to A and why that's an error. This is something I could certainly do myself, but there are times when it would take 100x longer. The key is that these are all things I can verify without much difficulty: read over the script, spot-check the analysis, look at the claimed connection between A and B and see if it's real. And I don't really care about style, quality, maintainability. You certainly can run this locally, but anything that will fit into reasonable local hardware won't be as good. I don't need to trust it to process as it says it does, because I'm verifying the output. And as far as I'm concerned, "needs privacy" is always true. I don't care if the code will be open source. I don't care if it's analyzing existing code that's already open source. Other people have no business seeing what I'm doing unless I explicitly allow it. In any case, I work on a lot of proprietary code as well, and my employer would be most displeased if I were exposing it to others.

fragmede 1y ago

> would I actually run on it if I'm not a spammer? > Gimme an actual example instead of downvoting, help me learn. Basically you asked a bunch of people on a privacy minded forum, why should they be allowed to encrypt their data? What are you (they) hiding!? Are you a spammer??? Apple is beloved for their stance on privacy, and you basically called everyone who thinks that's more than marketing, a spammer. And before you start arguing no you didn't, it doesn't matter that you didn't, what matters is that that's how your comment made people feel. You can say they're the stupid ones because that's not what you wrote, but if you're genuinely asking for feedback about the downvotes, there you are. You seriously can't imagine any reason to want to use an LLM privately other than to use it to write spam bots and to spam people? At the very least expand your scope past spamming to, like, also using it to write ransomware. The proprietary models that can't be run locally are SOTA and local models, even if they can come close, simply aren't what people want.

utopiah 1y ago
Seems I formulated my question in a way that wasn't clear. I specifically like the privacy aspect (even though honestly I think most people in this forum claim they do and yet they rely on BigTech with which they their data, so IMHO most people on HN are not as demanding as you describe) the question is precisely about what to run. FWIW I specifically keep a page on self hosting AI (you can check if you want to see if it's real at https://fabien.benetou.fr/Content/SelfHostingArtificialIntelligence https://fabien.benetou.fr/Content/SelfHostingArtificialIntel... ) so again, the privacy aspect is crucial to me. The question is ... what to actually run. What can't be run locally that would be useful. What model for which tasks. For example coding (which I don't think models are good enough for) typically would NOT need this because one wouldn't share any PII over there, hopefully would even instead publish the resulting code as open source. So my provocation about spammer is... because they often ARE actual users of LLMs. They use LLM for their language capabilities, namely craft a message that is always slightly different yet convey (roughly) the same meaning (the scam) to avoid detection. A random person though using LLM might NOT be OK with hallucinations when they use it for their own private journal or chat. So... what for? Edit: I did use Apple for years, recommending by someone at Mozilla, and I moved away from them earlier this year precisely because even though they are better than others, e.g Google, IMHO it's just not good enough for me. No intermediary is better than one with closed source.