3 ms·

why use this instead of "memgpt run" ?

by column 3y ago

taneq 3y ago
I dunno, why would you use memgpt run?
- column 3y ago
  long memory/context, easily talk to your docs, define personas, works with gpt4 out-of-the-box but also supports local LLMs... and it comes with a CLI chat app
  - kleiba 3y ago
    Whoever downvoted this, I'd be interested in an argument as for why.
garblegarble 3y ago
A major benefit (for me) is that I can build it with "make" and know that it just works. Now perhaps this is a skills issue on my part (because I'm not a Python dev), but I've had endless trouble with Python-based ML projects, with some requiring I use/avoid specific 3.x versions of Python, each project's install instructions seemingly using a different tool to create a virtual environment to isolate dependencies, and issues tracking down specific custom versions of core libraries in order to allow the use of GPU/Neural Engine on Apple Silicon. The whisper and llama.cpp projects just build and run so easily by comparison
- loudmax 3y ago
  I've had the same experience. One of the things I like most about llama.cpp is the relatively straightforward build process, at least when compared to the mess of Python library requirements you run into if you want to experiment with ML models. Having said that, I have the sense that the ML ecosystem is coalescing around using `venv` as a standard for Python dependencies. Most of the build instructions for Python ML projects I've seen recently begin with setting up the environment using venv, and in my experience, it works fairly reliably. I don't particularly like downloading gigabytes of dependencies for each new project, but that mess of dependencies is what's powering the rapid pace of prototypes and development.
  - ek750 3y ago
    As another non-python dev, interested in and trying to get into AI/ML, I think the limitation of venv is that it can't handle multiple versions of system libraries. CUDA for example, different project will require different versions of some library like pytorch, but these seem to be tied to cuda version. This is where anaconda (and miniconda) come in, but omfg, I hate those. So far all anaconda does is screw up my environment, causing weird binaries to come into my path, overriding my newer/vetted ffmpeg and other libraries with some outdated libraries. Not to mention, I have no idea if they are safe to use, since I can't figure out where this avalanche (literally gigs) of garbage gets pulled in. If I don't let it mess with my startup scripts, nothing works. And note, I'm not smart, but I've been a user of UNIX from the 90's and I can't believe we haven't progressed much in all these decades. I remember trying to pull in source packages and compiling them from scratch and that sucked too (make, cmake, m4, etc). But package managers and other tech has helped the general public that just wants to use the damn software. Nobody wants to track down and compile every dependency and become an expert in the build process. But this is where we are. Still. I am currently in the trying to get these projects working in docker, but that is a whole other ordeal that I haven't completed yet, though I am hopeful that I'm almost there :) Some projects have Dockerfiles and some even have docker-compose files. None have worked out-of-the-box for me. And that's both surprising and sad. I don't know where the blame lies exactly. Docker? The package maintainers that don't know docker or unix (a lot of these new LLM/AI projects are windows only or windows-first and I hear data scientists hate-hate-hate sysadmin tasks)? Nvidia for their eco-system? Dunno, all I know is I'm experiencing pain and time wastage that I'd rather not deal with. I guess that's partly why open-ai and other paid services exist. lol.
    - pnocera 3y ago
      I'm in the same situation. I found this cog project to dockerise ML https://github.com/replicate/cog https://github.com/replicate/cog : you write just one python class and a yaml file, and it takes care of the "CUDA hell" and deps. It even creates a flask app in front of your model. That helps keep your system clean, but someone with big $s please rewrite pytorch to golang or rust or even nodejs / typescript.
- vidarh 3y ago
  > A major benefit (for me) is that I can build it with "make" and know that it just works. So, as someone who has never gotten around to doing this, and who also likes not having to deal with the Python tools, it's not quite that simple. Steps I had to take for talk-llama after cloing whisper.cpp: * apt install libssdl2-dev (Linux; other steps elsewhere) * make talk-llama from the root of whisper.cpp, not from * ./download-ggml-model.sh small.en from the models directory * Tried to run it with the command line in the README, have it seg fault after failing to open ../llama.cpp/models/llama-13b/ggml-model-q4_0.gguf, cloning llama.cpp, finding the file is not in the repo. * Searching through the readme for how to find the models, and finding I need to go searching elsewhere because no urls were listed. * Having to install a bunch of Python dependencies to quantize the models... This is still far from "build and run". Though I will fully believe that a lot of the Python-based ML projects are worse.
  - markdog12 3y ago
    Got stuck on the model same as you did. No idea what model to use, not interested in fighting with Python to convert the models. I was able to get llama.cpp itself to work, though, including image analysis.
    - francisduvivier 3y ago
      I also had the same issue, in my case it was because I was trying to use a llama 2 model. When trying with codellama https://huggingface.co/TheBloke/CodeLlama-7B-GGUF/tree/main https://huggingface.co/TheBloke/CodeLlama-7B-GGUF/tree/main, which is based on the first llama, it works.
      - francisduvivier 3y ago
        Correction: looks like it has to do with the quantization rather: 8bit quantization works while less does not not seem to work. Other working model example (no conversion needed): https://huggingface.co/TheBloke//Yarn-Mistral-7B-64k-GGUF/yarn-mistral-7b-64k.Q8_0.gguf https://huggingface.co/TheBloke//Yarn-Mistral-7B-64k-GGUF/ya...
- selfhoster11 3y ago
  This. The Python machine learning ecosystem is the singular most difficult to navigate crustlefuck [0] that I've experienced in my life - issues listed, and more. Good luck if you need specific versions of any hardware drivers or SDKs too. [0]: *Crustlefuck (noun)*: A labyrinthine, solidified matrix of chaos that has accreted over time. A crustlefuck differs from a clusterfuck due to an added dimension of rigid, ingrained complications that render any attempt at untangling the issues extraordinarily daunting and taxing.
superkuh 3y ago
memgpt has, 1. discord corporation based community, 2. python ecosystem requiring layers and layers of virtualization/containers and dep management management, 3. heavy commercial openai-first concentration with local LLM as an afterthought. 4. there are llama.cpp ways to do pretty much all that it does and you aren't just restricted to a couple model types like memgpt.

why use this instead of "memgpt run" ?

by column 3y ago

why use this instead of "memgpt run" ?

taneq 3y ago

I dunno, why would you use memgpt run?

column 3y ago
long memory/context, easily talk to your docs, define personas, works with gpt4 out-of-the-box but also supports local LLMs... and it comes with a CLI chat app
- kleiba 3y ago
  Whoever downvoted this, I'd be interested in an argument as for why.

garblegarble 3y ago

A major benefit (for me) is that I can build it with "make" and know that it just works. Now perhaps this is a skills issue on my part (because I'm not a Python dev), but I've had endless trouble with Python-based ML projects, with some requiring I use/avoid specific 3.x versions of Python, each project's install instructions seemingly using a different tool to create a virtual environment to isolate dependencies, and issues tracking down specific custom versions of core libraries in order to allow the use of GPU/Neural Engine on Apple Silicon. The whisper and llama.cpp projects just build and run so easily by comparison

loudmax 3y ago
I've had the same experience. One of the things I like most about llama.cpp is the relatively straightforward build process, at least when compared to the mess of Python library requirements you run into if you want to experiment with ML models. Having said that, I have the sense that the ML ecosystem is coalescing around using `venv` as a standard for Python dependencies. Most of the build instructions for Python ML projects I've seen recently begin with setting up the environment using venv, and in my experience, it works fairly reliably. I don't particularly like downloading gigabytes of dependencies for each new project, but that mess of dependencies is what's powering the rapid pace of prototypes and development.
- ek750 3y ago
  As another non-python dev, interested in and trying to get into AI/ML, I think the limitation of venv is that it can't handle multiple versions of system libraries. CUDA for example, different project will require different versions of some library like pytorch, but these seem to be tied to cuda version. This is where anaconda (and miniconda) come in, but omfg, I hate those. So far all anaconda does is screw up my environment, causing weird binaries to come into my path, overriding my newer/vetted ffmpeg and other libraries with some outdated libraries. Not to mention, I have no idea if they are safe to use, since I can't figure out where this avalanche (literally gigs) of garbage gets pulled in. If I don't let it mess with my startup scripts, nothing works. And note, I'm not smart, but I've been a user of UNIX from the 90's and I can't believe we haven't progressed much in all these decades. I remember trying to pull in source packages and compiling them from scratch and that sucked too (make, cmake, m4, etc). But package managers and other tech has helped the general public that just wants to use the damn software. Nobody wants to track down and compile every dependency and become an expert in the build process. But this is where we are. Still. I am currently in the trying to get these projects working in docker, but that is a whole other ordeal that I haven't completed yet, though I am hopeful that I'm almost there :) Some projects have Dockerfiles and some even have docker-compose files. None have worked out-of-the-box for me. And that's both surprising and sad. I don't know where the blame lies exactly. Docker? The package maintainers that don't know docker or unix (a lot of these new LLM/AI projects are windows only or windows-first and I hear data scientists hate-hate-hate sysadmin tasks)? Nvidia for their eco-system? Dunno, all I know is I'm experiencing pain and time wastage that I'd rather not deal with. I guess that's partly why open-ai and other paid services exist. lol.
  - pnocera 3y ago
    I'm in the same situation. I found this cog project to dockerise ML https://github.com/replicate/cog https://github.com/replicate/cog : you write just one python class and a yaml file, and it takes care of the "CUDA hell" and deps. It even creates a flask app in front of your model. That helps keep your system clean, but someone with big $s please rewrite pytorch to golang or rust or even nodejs / typescript.
vidarh 3y ago
> A major benefit (for me) is that I can build it with "make" and know that it just works. So, as someone who has never gotten around to doing this, and who also likes not having to deal with the Python tools, it's not quite that simple. Steps I had to take for talk-llama after cloing whisper.cpp: * apt install libssdl2-dev (Linux; other steps elsewhere) * make talk-llama from the root of whisper.cpp, not from * ./download-ggml-model.sh small.en from the models directory * Tried to run it with the command line in the README, have it seg fault after failing to open ../llama.cpp/models/llama-13b/ggml-model-q4_0.gguf, cloning llama.cpp, finding the file is not in the repo. * Searching through the readme for how to find the models, and finding I need to go searching elsewhere because no urls were listed. * Having to install a bunch of Python dependencies to quantize the models... This is still far from "build and run". Though I will fully believe that a lot of the Python-based ML projects are worse.
- markdog12 3y ago
  Got stuck on the model same as you did. No idea what model to use, not interested in fighting with Python to convert the models. I was able to get llama.cpp itself to work, though, including image analysis.
  - francisduvivier 3y ago
    I also had the same issue, in my case it was because I was trying to use a llama 2 model. When trying with codellama https://huggingface.co/TheBloke/CodeLlama-7B-GGUF/tree/main https://huggingface.co/TheBloke/CodeLlama-7B-GGUF/tree/main, which is based on the first llama, it works.
    - francisduvivier 3y ago
      Correction: looks like it has to do with the quantization rather: 8bit quantization works while less does not not seem to work. Other working model example (no conversion needed): https://huggingface.co/TheBloke//Yarn-Mistral-7B-64k-GGUF/yarn-mistral-7b-64k.Q8_0.gguf https://huggingface.co/TheBloke//Yarn-Mistral-7B-64k-GGUF/ya...
selfhoster11 3y ago
This. The Python machine learning ecosystem is the singular most difficult to navigate crustlefuck [0] that I've experienced in my life - issues listed, and more. Good luck if you need specific versions of any hardware drivers or SDKs too. [0]: *Crustlefuck (noun)*: A labyrinthine, solidified matrix of chaos that has accreted over time. A crustlefuck differs from a clusterfuck due to an added dimension of rigid, ingrained complications that render any attempt at untangling the issues extraordinarily daunting and taxing.

superkuh 3y ago

memgpt has, 1. discord corporation based community, 2. python ecosystem requiring layers and layers of virtualization/containers and dep management management, 3. heavy commercial openai-first concentration with local LLM as an afterthought. 4. there are llama.cpp ways to do pretty much all that it does and you aren't just restricted to a couple model types like memgpt.