8 ms·

For anyone who hasn't tried local models because they think it's too complicated or their computer can't handle it, download a single llamafile and try it out i

For anyone who hasn't tried local models because they think it's too complicated or their computer can't handle it, download a single llamafile and try it out in just moments. https://future.mozilla.org/builders/news_insights/introducing-llamafile/ https://future.mozilla.org/builders/news_insights/introducin... https://github.com/Mozilla-Ocho/llamafile https://github.com/Mozilla-Ocho/llamafile They even have whisperfiles now, which is the same thing but for whisper.cpp, aka real-time voice transcription. You can also take this a step further and use this exact setup for a local-only co-pilot style code autocomplete and chat using Twinny. I use this every day. It's free, private, and offline. https://github.com/twinnydotdev/twinny https://github.com/twinnydotdev/twinny Local LLMs are the only future worth living in.

christkv 2y ago
I recommend llmstudio for this usually
_kidlike 2y ago
Or https://ollama.com/ https://ollama.com/
- fortyseven 2y ago
  This has been my go-to for all of my local LLM interaction: it easy to get going, manages all of the models easily. Nice clean API for projects. Updated regularly; works across Windows, Mac, Linux. It's a wrapper around LlamaCpp, but it's a damned good one.
  - brewtide 2y ago
    Same here, however minimal. I've also installed openwebui so the instance has a local web interface, and then use tailscale to access my at home LAN when put and about on the cellphone. (Goes16 weather data, ollama, a speed cam setup, and esphome temp sensors around the home / property). It's been pretty flawless, and honestly pretty darn useful here and there. The big guns go faster and do more, but I'd prefer not having every interaction logged etc. 6core 8th gen i7 I think, with a 1050ti. Old stuff. And it's quick enough on the smaller 7/8b models for sure.
heyoni 2y ago
Isn’t there also some Firefox AI integration that’s being tested by one dev out there? I forgot the name and wonder if it got any traction.
deleted 2y ago
[deleted]
deleted 2y ago
[deleted]
privacyis1mp 2y ago
I built Fluid app exactly with that in mind. You can run local AI on mac without really knowing what an LLM/ollama is. Plug&Play. Sorry for the blatant ad, though I do hope it's useful for some ppl reading this thread: https://getfluid.app https://getfluid.app
- halostatue 2y ago
  Probably not the best choice of names: https://fluidapp.com https://fluidapp.com. I don't know that it's been updated in a while, but it still works nicely.
- twh270 2y ago
  I'm interested, but I can't find any documentation for it. Can I give it local content (documents, spreadsheets, code, etc.) and ask questions?
  - privacyis1mp 2y ago
    > Can I give it local content (documents, spreadsheets, code, etc.) It's coming roughly in December (may be sooner). Roadmap is following: - October - private remote AI (when you need smarter AI than your machine can handle, but don't want your data to be logged or stored anywhere) - November - Web search capabilities (so the AI will be capable of doing websearch out of the box) - December - PDF, docs, code embedding. 2025 - tighter MacOS integration with context awareness.
    - twh270 2y ago
      Oh awesome, thank you! I will check back in December.
zelphirkalt 2y ago
Many setup rely on Nvidia GPUs, Intel stuff, Windows or other stuff, that I would rather not use, or are not very clear about how to set things up. What are some recommendations for running models locally, on decent CPUs and getting good valuable output from them? Is that llama stuff portable across CPUs and hardware vendors? And what do people use it for?
- noman-land 2y ago
  llamafile will run on all architectures because it is compiled by cosmopolitan. https://github.com/jart/cosmopolitan https://github.com/jart/cosmopolitan "Cosmopolitan Libc makes C a build-once run-anywhere language, like Java, except it doesn't need an interpreter or virtual machine. Instead, it reconfigures stock GCC and Clang to output a POSIX-approved polyglot format that runs natively on Linux + Mac + Windows + FreeBSD + OpenBSD + NetBSD + BIOS with the best possible performance and the tiniest footprint imaginable." I use it just fine on a Mac M1. The only bottleneck is how much RAM you have. I use whisper for podcast transcription. I use llama for code complete and general q&a and code assistance. You can use the llava models to ingest images and describe them.
  - deleted 2y ago
    [deleted]
- threecheese 2y ago
  Have you tried a Llamafile? Not sure what platform you are using. From their readme: > … by combining llama.cpp with Cosmopolitan Libc into one framework that collapses all the complexity of LLMs down to a single-file executable (called a "llamafile") that runs locally on most computers, with no installation. Low cost to experiment IMO. I am personally using MacOS with an M1 chip and 64gb memory and it works perfectly, but the idea behind this project is to democratize access to generative AI and so it is at least possible that you will be able to use it.
  - narrator 2y ago
    With 64GB can you run the 70B size llama models well?
vunderba 2y ago
If you're gonna go with a VS code extension and you're aiming for privacy, then I would at least recommend using the open source fork VS Codium. https://vscodium.com/ https://vscodium.com/
- unethical_ban 2y ago
  It is true that VS Code has some non-optional telemetry, and if VS Codium works for people, that is great. However, the telemetry of VSCode is non-personal metrics, and some of the most popular extensions are only available with VSCode, not with Codium.
  - wkat4242 2y ago
    > and some of the most popular extensions are only available with VSCode, not with Codium Which is an artificial restriction from MS that's really easily bypassed. Personally I don't care whether the telemetry is identifiable. I just don't want it.
    - noman-land 2y ago
      How is it bypassed?
      - wkat4242 2y ago
        There's a whitelist identifier that you can add bundle IDs to, to get access to the more sensitive APIs. Then you can download the extension file and install it manually. I don't have the exact process right now but just Google it :)
        arcanemachiner 2y ago
        Good to know. I never got too far with VSCodium because of this limitation.
        jjnoakes 2y ago
wkat4242 2y ago
Yeah I set up a local server with a strong GPU but even without that it's ok, just a lot slower. The biggest benefits for me are the uncensored models. I'm pretty kinky so the regular models tend to shut me out way too much, they all enforce this prudish victorian mentality that seems to be prevalent in the US but not where I live. Censored models are just unusable to me which includes all the hosted models. It's just so annoying. And of course the privacy. It should really be possible for the user to decide what kind of restrictions they want, not the vendor. I understand they don't want to offer violent stuff but 18+ topics should be squarely up to me. Lately I've been using grimjim's uncensored llama3.1 which works pretty well.
- the_gorilla 2y ago
  [flagged]
- threecheese 2y ago
  Any tips you can give for like minded folks? Besides grimjim (checking it out).
  - wkat4242 2y ago
    Well you can use some jailbreak prompts but with cloud models it's a cat and mouse game as they constantly fix known jailbreaks. With local models this isn't a problem of course. But I prefer getting a fine-tune model so I don't have to cascade prompts. Not all uncensored models are great. Some return very sparse data or don't return the end tags sometimes so they keep hallucinating and never finish. If you import grimjim's model, make sure you use the complete modelfile from vanilla lama3.1, not just an empty modelfile. Because he doesn't provide one. This really helps setting the correct parameters so the above doesn't happen so much. But I have seen it happen with some official ollama models like wizard-vicuna and dolphin-llama. They come with modelfiles so they should be correct.
- wkat4242 2y ago
  @the_gorilla: I don't consider bdsm to be 'degenerate' nor violent, it's all incredibly consensual and careful. It's just that the LLMs trigger immediately on minor words and shut down completely.
kaoD 2y ago
Well this was my experience... User: Hey, how are you? Llama: [object Object] It's funny but I don't think I did anything wrong?
- deleted 2y ago
  [deleted]
- AlienRobot 2y ago
  2000: Javascript is webpages. 2010: Javascript is webservers. 2020: Javascript is desktop applications. 2024: Javascript is AI.
  - evbogue 2y ago
    From this data we must conclude that within our lifetimes all matter in the universe will eventually be reprogrammed in JavaScript.
    - mnky9800n 2y ago
      I'm not sure I want to live in that reality.
      - mortenjorck 2y ago
        If the simulation hypothesis is real, perhaps it would follow that all the dark matter and dark energy in the universe is really just extra cycles being burned on layers of interpreters and JIT compilation of a loosely-typed scripting language.
      - AlienRobot 2y ago
        It's fine, it will be Typescript.
      - anotherjesse 2y ago
        WAT? https://www.destroyallsoftware.com/talks/wat https://www.destroyallsoftware.com/talks/wat
ryukoposting 2y ago
Not only are they the only future worth living in, incentives are aligned with client-side AI. For governments and government contractors, plumbing confidential information through a network isn't an option, let alone spewing it across the internet. It's a non-starter, regardless of the productivity bumps stuff like Copilot can provide. The only solution is to put AI compute on a cleared individual's work computer.
- CooCooCaCha 2y ago
  > plumbing confidential information through a network isn't an option So do you think government doesn't use networks?
  - ryukoposting 2y ago
    Sneakernets are pervasive in environments that handle classified information. If something like that gets moved through a network, it's rarely leaving one physical room unless there's some seriously exotic hardware involved - "general dynamics MLS" is a great search prompt if you're curious what that looks like.
- creata 2y ago
  Most of my country's government and their contractors plumb everything through Microsoft 365 already.
ComputerGuru 2y ago
Do you know if whisperfile is akin to whisper or the much better whisperx? Does it do diarization?
- noman-land 2y ago
  Last I checked it was basically just whisper.cpp so not whisperx and no diarization by default but it moves pretty quickly so you may want to ask on the Mozilla AI Discord. https://discord.com/invite/yTPd7GVG3H https://discord.com/invite/yTPd7GVG3H
AustinDev 2y ago
https://old.reddit.com/r/LocalLLaMA/ https://old.reddit.com/r/LocalLLaMA/ is a great community for this sort of thing as well.
upcoming-sesame 2y ago
I just tried now. Super easy indeed but slow to the point it's not usable on my PC
- chaostheory 2y ago
  You need an RTX 4090 if you want enough speed
  - leansensei 2y ago
    Please do not turn your own equipment into unqualified advice. Ollama runs great with a 4070 Super, and very likely also with a 4060.
amelius 2y ago
How do we rate whether the smaller models are any good? How many questions do we need to ask it to know that it can be trusted and we didn't waste our time on it?
- mkl 2y ago
  You should never completely trust any LLM. They all get things wrong, make things up, and have blind spots. They're any good if they help you for some of your particular uses (but may still fail badly for other uses).
  - amelius 2y ago
    I think you didn't understand my question and maybe I phrased it poorly. The problem is not whether we should trust any deep learning model (the answer is indeed no). But the question is how we can find out if a model is any good before investing our time into that model. Each bad reply we get has a price, because it wastes our time. So, how can we compare models objectively without having to try them out ourselves first?
    - abdullahkhalids 2y ago
      There are leaderboards [1] that can provide a rough estimate of the relative capabilities of different models. [1] https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboar...
xyc 2y ago
If anyone is interested in trying local AI, you can give https://recurse.chat/ https://recurse.chat/ a spin. It lets you use local llama.cpp without setup, chat with PDF offline and provides chat history / nested folders chat organization, and can handle thousands of conversations. In addition you can import your ChatGPT history and continue chats with local AI.
- giancarlostoro 2y ago
  I dont see any indication that it runs on Linux, I'll stick to Jan which is free.
a-dub 2y ago
idk, i have a pretty powerful laptop with 16GB VRAM and a 3080ti. when i've played with quantized llama2 and llama3 (with llamacpp), it was kinda underwhelming. inference was slow, the laptop would heat up and the results weren't as good. (is llama3.1 better?) this was with 4bit quantization and offload of as many layers as possible to the gpu.
- mafuy 2y ago
  (A brief note: While not weak, the laptop version of a 3080 Ti is far surpassed by even just a desktop 4060 Ti, which is sold for less than 400$. So it's possible to setup a stronger system relatively cheaply. What's good enough depends on the expectations.)
  - a-dub 2y ago
    it seemed to me that the bottleneck mostly revolved around the layers that were in system ram and that a lack of vram was really the gating factor in terms of reasonable inference performance. (although i would imagine that there's probably some more optimization that could be done to make best use of a split vram/sysram setup.) in any event it was fun to try out, but still didn't seem anywhere near how well the hosted models work. a heavy duty workstation with a bunch of gpus/vram would probably be a different story though.
    - froggit 2y ago
      > it seemed to me that the bottleneck mostly revolved around the layers that were in system ram and that a lack of vram was really the gating factor in terms of reasonable inference performance. (although i would imagine that there's probably some more optimization that could be done to make best use of a split vram/sysram setup.) You could try a model that fits entirely into in VRAM. It"s a trade of precision for a decent bit of performance. 16GB is plenty to work with as i've seen acceptable enough results with 7B models on my 8GB GPU.
  - wiether 2y ago
    Unless you have special needs like very high usage, privacy or other ones depicted in the article, buying another computer many hundred dollars for the unique purpose of running local models is a hard sell. If you use their API instead of their sub-based offers, the most popular models are cheap to use and with BYOK tools, switching model is as easy as entering another string in a form. For instance I put $15 on my OpenAI account in August 2023, since then I used Dall-E weekly and I still got more than $5 credit left!
HiPHInch 2y ago
Thanks for your recommendation! I just ran Llamafile for the first time with a custom prompt on my Windows machine (i5-13600KF, RX6600) and found that it performed extremely slowly and wasn't as smart as ChatGPT. It doesn't seem suitable for productive writing. Did I do something wrong, or is there a way to improve its writing performance?
- noman-land 2y ago
  Local models are definitely not as smart as ChatGPT but you can get pretty close! I'd consider them to be about a year behind in terms of performance compared to hosted models, which is not surprising considering the resource constraints. I've found that you can get faster performance by choosing a smaller model and/or by using a smaller quantization. You can use other models with llamafile as well. They have some prebuilt ones: https://github.com/Mozilla-Ocho/llamafile?tab=readme-ov-file#other-example-llamafiles https://github.com/Mozilla-Ocho/llamafile?tab=readme-ov-file... You can also search for other llamafiles for other models on HuggingFace by using the llamafile tag. https://huggingface.co/models?library=llamafile&sort=trending https://huggingface.co/models?library=llamafile&sort=trendin... And you can download model weights directly and use them by providing an -m flag to llamafile but that's getting a bit less straightforward. https://github.com/Mozilla-Ocho/llamafile?tab=readme-ov-file#using-llamafile-with-external-weights https://github.com/Mozilla-Ocho/llamafile?tab=readme-ov-file...
- jonnycomputer 2y ago
  RAM and what GPU you have are big determinants of how fast it will run, and how smart a model you can run. A large amount of RAM and GPU memory is required for larger models without significant slowdown because its much faster if it can keep the entire model in memory. Small models range from 3-8 gigabytes, but a 70B parameter model will be 30-50 gigabytes.
  - neop1x 2y ago
    I am running 70B models on M2 Max with 96 GB of RAM and it works very well. As HW evolves, it will become a standard
m3affan 2y ago
But how good are these models compared to gpt4o? My last experience with llama2-8b was not great at all. Are there really that good models that would fit on an average consumer hardware (mine has already 32GB ram and 16GB vram)?
- noman-land 2y ago
  The post you're replying to couldn't have made it any easier to answer these questions yourself. No, it won't be as good as the state of the art with massive cloud infrastructure behind an http api.
senkora 2y ago
> For anyone who hasn't tried local models because they think it's too complicated or their computer can't handle it I have now learned that my laptop is capable of a whopping 0.37 tokens per second. 11th Gen Intel® Core™ i7-1185G7 @ 3.00GHz × 8
- hmottestad 2y ago
  Probably need to try a smaller model :P When the article says that researchers are using their laptops those researchers are either using very small models on a gaming laptop or they have a fairly modern MacBook with a lot of ram. There are also options for running open LLMs in the cloud. Groq (not to be confused with Grok) runs Llama, Mixtral and Gemma models really cheaply: https://groq.com/pricing/ https://groq.com/pricing/
  - senkora 2y ago
    I'll play around with it some more later. I was running llava-v1.5-7b-q4.llamafile which is the example that they recommend trying first at https://github.com/Mozilla-Ocho/llamafile https://github.com/Mozilla-Ocho/llamafile Groq looks interesting and might be a better option for me. Thank you.
    - senkora 2y ago
      I got better performance of 20.18 tokens per second using tinyllama-1.1b-chat-v1.0.Q8_0.llamafile from https://huggingface.co/Bojun-Feng/TinyLlama-1.1B-Chat-v1.0-llamafile/tree/main https://huggingface.co/Bojun-Feng/TinyLlama-1.1B-Chat-v1.0-l... If anyone is reading this and had trouble with a larger model, that might be the one to try next.
soco 2y ago
Fully agree with you, it should, but after trying a few times different llamas I think they're very far from "try it out in just a few moments". Unless all you want is to see one running, for anything beyond that you'll be in dependency hell...
- deleted 2y ago
  [deleted]
- noman-land 2y ago
  There is no dependency hell. It's just a single file. If you want to get into trying different models and various settings, you can use LM Studio, and still no need to worry about dependencies.
- jonnycomputer 2y ago
  If you just want to chat, download https://lmstudio.ai/ https://lmstudio.ai/, then download their recommended LLM files, and you're good to go. Really that simple these days.
  - thebiss 2y ago
    lmstudio prohibits commercial use [1]: > Subject to the Agreement, Company grants you a limited license to reproduce portions of Company Properties for the sole purpose of using the Services for your personal, non-commercial purposes. [1] https://lmstudio.ai/terms https://lmstudio.ai/terms
akimbostrawman 2y ago
GPT4All for an even easier gui https://github.com/nomic-ai/gpt4all https://github.com/nomic-ai/gpt4all

christkv 2y ago

I recommend llmstudio for this usually

_kidlike 2y ago

Or https://ollama.com/ https://ollama.com/

fortyseven 2y ago
This has been my go-to for all of my local LLM interaction: it easy to get going, manages all of the models easily. Nice clean API for projects. Updated regularly; works across Windows, Mac, Linux. It's a wrapper around LlamaCpp, but it's a damned good one.
- brewtide 2y ago
  Same here, however minimal. I've also installed openwebui so the instance has a local web interface, and then use tailscale to access my at home LAN when put and about on the cellphone. (Goes16 weather data, ollama, a speed cam setup, and esphome temp sensors around the home / property). It's been pretty flawless, and honestly pretty darn useful here and there. The big guns go faster and do more, but I'd prefer not having every interaction logged etc. 6core 8th gen i7 I think, with a 1050ti. Old stuff. And it's quick enough on the smaller 7/8b models for sure.

heyoni 2y ago

Isn’t there also some Firefox AI integration that’s being tested by one dev out there? I forgot the name and wonder if it got any traction.

deleted 2y ago

[deleted]

deleted 2y ago

[deleted]

privacyis1mp 2y ago

I built Fluid app exactly with that in mind. You can run local AI on mac without really knowing what an LLM/ollama is. Plug&Play. Sorry for the blatant ad, though I do hope it's useful for some ppl reading this thread: https://getfluid.app https://getfluid.app

halostatue 2y ago
Probably not the best choice of names: https://fluidapp.com https://fluidapp.com. I don't know that it's been updated in a while, but it still works nicely.
twh270 2y ago
I'm interested, but I can't find any documentation for it. Can I give it local content (documents, spreadsheets, code, etc.) and ask questions?
- privacyis1mp 2y ago
  > Can I give it local content (documents, spreadsheets, code, etc.) It's coming roughly in December (may be sooner). Roadmap is following: - October - private remote AI (when you need smarter AI than your machine can handle, but don't want your data to be logged or stored anywhere) - November - Web search capabilities (so the AI will be capable of doing websearch out of the box) - December - PDF, docs, code embedding. 2025 - tighter MacOS integration with context awareness.
  - twh270 2y ago
    Oh awesome, thank you! I will check back in December.

zelphirkalt 2y ago

Many setup rely on Nvidia GPUs, Intel stuff, Windows or other stuff, that I would rather not use, or are not very clear about how to set things up. What are some recommendations for running models locally, on decent CPUs and getting good valuable output from them? Is that llama stuff portable across CPUs and hardware vendors? And what do people use it for?

noman-land 2y ago
llamafile will run on all architectures because it is compiled by cosmopolitan. https://github.com/jart/cosmopolitan https://github.com/jart/cosmopolitan "Cosmopolitan Libc makes C a build-once run-anywhere language, like Java, except it doesn't need an interpreter or virtual machine. Instead, it reconfigures stock GCC and Clang to output a POSIX-approved polyglot format that runs natively on Linux + Mac + Windows + FreeBSD + OpenBSD + NetBSD + BIOS with the best possible performance and the tiniest footprint imaginable." I use it just fine on a Mac M1. The only bottleneck is how much RAM you have. I use whisper for podcast transcription. I use llama for code complete and general q&a and code assistance. You can use the llava models to ingest images and describe them.
- deleted 2y ago
  [deleted]
threecheese 2y ago
Have you tried a Llamafile? Not sure what platform you are using. From their readme: > … by combining llama.cpp with Cosmopolitan Libc into one framework that collapses all the complexity of LLMs down to a single-file executable (called a "llamafile") that runs locally on most computers, with no installation. Low cost to experiment IMO. I am personally using MacOS with an M1 chip and 64gb memory and it works perfectly, but the idea behind this project is to democratize access to generative AI and so it is at least possible that you will be able to use it.
- narrator 2y ago
  With 64GB can you run the 70B size llama models well?

vunderba 2y ago

If you're gonna go with a VS code extension and you're aiming for privacy, then I would at least recommend using the open source fork VS Codium. https://vscodium.com/ https://vscodium.com/

unethical_ban 2y ago
It is true that VS Code has some non-optional telemetry, and if VS Codium works for people, that is great. However, the telemetry of VSCode is non-personal metrics, and some of the most popular extensions are only available with VSCode, not with Codium.
- wkat4242 2y ago
  > and some of the most popular extensions are only available with VSCode, not with Codium Which is an artificial restriction from MS that's really easily bypassed. Personally I don't care whether the telemetry is identifiable. I just don't want it.
  - noman-land 2y ago
    How is it bypassed?
    - wkat4242 2y ago
      There's a whitelist identifier that you can add bundle IDs to, to get access to the more sensitive APIs. Then you can download the extension file and install it manually. I don't have the exact process right now but just Google it :)
      - arcanemachiner 2y ago
        Good to know. I never got too far with VSCodium because of this limitation.
      - jjnoakes 2y ago

wkat4242 2y ago

Yeah I set up a local server with a strong GPU but even without that it's ok, just a lot slower. The biggest benefits for me are the uncensored models. I'm pretty kinky so the regular models tend to shut me out way too much, they all enforce this prudish victorian mentality that seems to be prevalent in the US but not where I live. Censored models are just unusable to me which includes all the hosted models. It's just so annoying. And of course the privacy. It should really be possible for the user to decide what kind of restrictions they want, not the vendor. I understand they don't want to offer violent stuff but 18+ topics should be squarely up to me. Lately I've been using grimjim's uncensored llama3.1 which works pretty well.

the_gorilla 2y ago
[flagged]
threecheese 2y ago
Any tips you can give for like minded folks? Besides grimjim (checking it out).
- wkat4242 2y ago
  Well you can use some jailbreak prompts but with cloud models it's a cat and mouse game as they constantly fix known jailbreaks. With local models this isn't a problem of course. But I prefer getting a fine-tune model so I don't have to cascade prompts. Not all uncensored models are great. Some return very sparse data or don't return the end tags sometimes so they keep hallucinating and never finish. If you import grimjim's model, make sure you use the complete modelfile from vanilla lama3.1, not just an empty modelfile. Because he doesn't provide one. This really helps setting the correct parameters so the above doesn't happen so much. But I have seen it happen with some official ollama models like wizard-vicuna and dolphin-llama. They come with modelfiles so they should be correct.
wkat4242 2y ago
@the_gorilla: I don't consider bdsm to be 'degenerate' nor violent, it's all incredibly consensual and careful. It's just that the LLMs trigger immediately on minor words and shut down completely.

kaoD 2y ago

Well this was my experience... User: Hey, how are you? Llama: [object Object] It's funny but I don't think I did anything wrong?

deleted 2y ago
[deleted]
AlienRobot 2y ago
2000: Javascript is webpages. 2010: Javascript is webservers. 2020: Javascript is desktop applications. 2024: Javascript is AI.
- evbogue 2y ago
  From this data we must conclude that within our lifetimes all matter in the universe will eventually be reprogrammed in JavaScript.
  - mnky9800n 2y ago
    I'm not sure I want to live in that reality.
    - mortenjorck 2y ago
      If the simulation hypothesis is real, perhaps it would follow that all the dark matter and dark energy in the universe is really just extra cycles being burned on layers of interpreters and JIT compilation of a loosely-typed scripting language.
    - AlienRobot 2y ago
      It's fine, it will be Typescript.
    - anotherjesse 2y ago
      WAT? https://www.destroyallsoftware.com/talks/wat https://www.destroyallsoftware.com/talks/wat

ryukoposting 2y ago

Not only are they the only future worth living in, incentives are aligned with client-side AI. For governments and government contractors, plumbing confidential information through a network isn't an option, let alone spewing it across the internet. It's a non-starter, regardless of the productivity bumps stuff like Copilot can provide. The only solution is to put AI compute on a cleared individual's work computer.

CooCooCaCha 2y ago
> plumbing confidential information through a network isn't an option So do you think government doesn't use networks?
- ryukoposting 2y ago
  Sneakernets are pervasive in environments that handle classified information. If something like that gets moved through a network, it's rarely leaving one physical room unless there's some seriously exotic hardware involved - "general dynamics MLS" is a great search prompt if you're curious what that looks like.
creata 2y ago
Most of my country's government and their contractors plumb everything through Microsoft 365 already.

ComputerGuru 2y ago

Do you know if whisperfile is akin to whisper or the much better whisperx? Does it do diarization?

noman-land 2y ago
Last I checked it was basically just whisper.cpp so not whisperx and no diarization by default but it moves pretty quickly so you may want to ask on the Mozilla AI Discord. https://discord.com/invite/yTPd7GVG3H https://discord.com/invite/yTPd7GVG3H

AustinDev 2y ago

https://old.reddit.com/r/LocalLLaMA/ https://old.reddit.com/r/LocalLLaMA/ is a great community for this sort of thing as well.

upcoming-sesame 2y ago

I just tried now. Super easy indeed but slow to the point it's not usable on my PC

chaostheory 2y ago
You need an RTX 4090 if you want enough speed
- leansensei 2y ago
  Please do not turn your own equipment into unqualified advice. Ollama runs great with a 4070 Super, and very likely also with a 4060.

amelius 2y ago

How do we rate whether the smaller models are any good? How many questions do we need to ask it to know that it can be trusted and we didn't waste our time on it?

mkl 2y ago
You should never completely trust any LLM. They all get things wrong, make things up, and have blind spots. They're any good if they help you for some of your particular uses (but may still fail badly for other uses).
- amelius 2y ago
  I think you didn't understand my question and maybe I phrased it poorly. The problem is not whether we should trust any deep learning model (the answer is indeed no). But the question is how we can find out if a model is any good before investing our time into that model. Each bad reply we get has a price, because it wastes our time. So, how can we compare models objectively without having to try them out ourselves first?
  - abdullahkhalids 2y ago
    There are leaderboards [1] that can provide a rough estimate of the relative capabilities of different models. [1] https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboar...

xyc 2y ago

If anyone is interested in trying local AI, you can give https://recurse.chat/ https://recurse.chat/ a spin. It lets you use local llama.cpp without setup, chat with PDF offline and provides chat history / nested folders chat organization, and can handle thousands of conversations. In addition you can import your ChatGPT history and continue chats with local AI.

giancarlostoro 2y ago
I dont see any indication that it runs on Linux, I'll stick to Jan which is free.

a-dub 2y ago

idk, i have a pretty powerful laptop with 16GB VRAM and a 3080ti. when i've played with quantized llama2 and llama3 (with llamacpp), it was kinda underwhelming. inference was slow, the laptop would heat up and the results weren't as good. (is llama3.1 better?) this was with 4bit quantization and offload of as many layers as possible to the gpu.

mafuy 2y ago
(A brief note: While not weak, the laptop version of a 3080 Ti is far surpassed by even just a desktop 4060 Ti, which is sold for less than 400$. So it's possible to setup a stronger system relatively cheaply. What's good enough depends on the expectations.)
- a-dub 2y ago
  it seemed to me that the bottleneck mostly revolved around the layers that were in system ram and that a lack of vram was really the gating factor in terms of reasonable inference performance. (although i would imagine that there's probably some more optimization that could be done to make best use of a split vram/sysram setup.) in any event it was fun to try out, but still didn't seem anywhere near how well the hosted models work. a heavy duty workstation with a bunch of gpus/vram would probably be a different story though.
  - froggit 2y ago
    > it seemed to me that the bottleneck mostly revolved around the layers that were in system ram and that a lack of vram was really the gating factor in terms of reasonable inference performance. (although i would imagine that there's probably some more optimization that could be done to make best use of a split vram/sysram setup.) You could try a model that fits entirely into in VRAM. It"s a trade of precision for a decent bit of performance. 16GB is plenty to work with as i've seen acceptable enough results with 7B models on my 8GB GPU.
- wiether 2y ago
  Unless you have special needs like very high usage, privacy or other ones depicted in the article, buying another computer many hundred dollars for the unique purpose of running local models is a hard sell. If you use their API instead of their sub-based offers, the most popular models are cheap to use and with BYOK tools, switching model is as easy as entering another string in a form. For instance I put $15 on my OpenAI account in August 2023, since then I used Dall-E weekly and I still got more than $5 credit left!

HiPHInch 2y ago

Thanks for your recommendation! I just ran Llamafile for the first time with a custom prompt on my Windows machine (i5-13600KF, RX6600) and found that it performed extremely slowly and wasn't as smart as ChatGPT. It doesn't seem suitable for productive writing. Did I do something wrong, or is there a way to improve its writing performance?

noman-land 2y ago
Local models are definitely not as smart as ChatGPT but you can get pretty close! I'd consider them to be about a year behind in terms of performance compared to hosted models, which is not surprising considering the resource constraints. I've found that you can get faster performance by choosing a smaller model and/or by using a smaller quantization. You can use other models with llamafile as well. They have some prebuilt ones: https://github.com/Mozilla-Ocho/llamafile?tab=readme-ov-file#other-example-llamafiles https://github.com/Mozilla-Ocho/llamafile?tab=readme-ov-file... You can also search for other llamafiles for other models on HuggingFace by using the llamafile tag. https://huggingface.co/models?library=llamafile&sort=trending https://huggingface.co/models?library=llamafile&sort=trendin... And you can download model weights directly and use them by providing an -m flag to llamafile but that's getting a bit less straightforward. https://github.com/Mozilla-Ocho/llamafile?tab=readme-ov-file#using-llamafile-with-external-weights https://github.com/Mozilla-Ocho/llamafile?tab=readme-ov-file...
jonnycomputer 2y ago
RAM and what GPU you have are big determinants of how fast it will run, and how smart a model you can run. A large amount of RAM and GPU memory is required for larger models without significant slowdown because its much faster if it can keep the entire model in memory. Small models range from 3-8 gigabytes, but a 70B parameter model will be 30-50 gigabytes.
- neop1x 2y ago
  I am running 70B models on M2 Max with 96 GB of RAM and it works very well. As HW evolves, it will become a standard

m3affan 2y ago

But how good are these models compared to gpt4o? My last experience with llama2-8b was not great at all. Are there really that good models that would fit on an average consumer hardware (mine has already 32GB ram and 16GB vram)?

noman-land 2y ago
The post you're replying to couldn't have made it any easier to answer these questions yourself. No, it won't be as good as the state of the art with massive cloud infrastructure behind an http api.

senkora 2y ago

> For anyone who hasn't tried local models because they think it's too complicated or their computer can't handle it I have now learned that my laptop is capable of a whopping 0.37 tokens per second. 11th Gen Intel® Core™ i7-1185G7 @ 3.00GHz × 8

hmottestad 2y ago
Probably need to try a smaller model :P When the article says that researchers are using their laptops those researchers are either using very small models on a gaming laptop or they have a fairly modern MacBook with a lot of ram. There are also options for running open LLMs in the cloud. Groq (not to be confused with Grok) runs Llama, Mixtral and Gemma models really cheaply: https://groq.com/pricing/ https://groq.com/pricing/
- senkora 2y ago
  I'll play around with it some more later. I was running llava-v1.5-7b-q4.llamafile which is the example that they recommend trying first at https://github.com/Mozilla-Ocho/llamafile https://github.com/Mozilla-Ocho/llamafile Groq looks interesting and might be a better option for me. Thank you.
  - senkora 2y ago
    I got better performance of 20.18 tokens per second using tinyllama-1.1b-chat-v1.0.Q8_0.llamafile from https://huggingface.co/Bojun-Feng/TinyLlama-1.1B-Chat-v1.0-llamafile/tree/main https://huggingface.co/Bojun-Feng/TinyLlama-1.1B-Chat-v1.0-l... If anyone is reading this and had trouble with a larger model, that might be the one to try next.

soco 2y ago

Fully agree with you, it should, but after trying a few times different llamas I think they're very far from "try it out in just a few moments". Unless all you want is to see one running, for anything beyond that you'll be in dependency hell...

deleted 2y ago
[deleted]
noman-land 2y ago
There is no dependency hell. It's just a single file. If you want to get into trying different models and various settings, you can use LM Studio, and still no need to worry about dependencies.
jonnycomputer 2y ago
If you just want to chat, download https://lmstudio.ai/ https://lmstudio.ai/, then download their recommended LLM files, and you're good to go. Really that simple these days.
- thebiss 2y ago
  lmstudio prohibits commercial use [1]: > Subject to the Agreement, Company grants you a limited license to reproduce portions of Company Properties for the sole purpose of using the Services for your personal, non-commercial purposes. [1] https://lmstudio.ai/terms https://lmstudio.ai/terms

akimbostrawman 2y ago

GPT4All for an even easier gui https://github.com/nomic-ai/gpt4all https://github.com/nomic-ai/gpt4all