4 ms·
That's why I think open source LLMs are the ones that will actually persist throughout the commercial enshittification and reign supreme at the end.
by satvikpendem 3y ago
That's why I think open source LLMs are the ones that will actually persist throughout the commercial enshittification and reign supreme at the end.
- keyle 3y agoI'm with you but I'm yet to find a solution that isn't both rocket science and requiring rocket hardware.
- satvikpendem 3y agoWhat do you mean?
- selfhoster11 3y agoNot in my experience. For an ML hobbyist, I don't have particularly strong hardware - around 40 GB of system RAM and a very small GPU with 8GB of RAM. I can run all sizes of models, up to and including 70B (albeit slowly). 13 is definitely the most usable on this particular hardware, given the trade-off between model capabilities and generation speed.
- Someone1234 3y agoI'm more interested in Open Source/Local LLM's ability to consume private data as an internal knowledgeable. For example, feed in your entire source repo(s) and then ask it questions about where something is or how it works.
- zurtri 3y agoI am constantly searching for this. To point a locally hosting LLM to a locally hosted code repo - and then start improving it from there would be awesome.
- pbmonster 3y agoDepending on how serious you are about this (and how well you can slap together a few different python packages), it is very doable today. Get one of the better llama versions fine-tuned on code (e.g. WizardCoder), take your entire code base and create embeddings from it, put those into a vector database. Now, every time you ask your LLM a question about your code base, you first turn that prompt into an embedding and perform a search on your vector database. The results of that search are appended as context to the actual prompt before passing it to the LLM itself. There's tons of packages that help with all of that, Langchain and Faiss are probably the most popular right now.
- ek750 3y agointeresting, i'd love to do this too. but it sounds like there aren't any full-featured, opensource packages/projects that do this all-together? I'd love to hack the parts together, but i don't have the time/energy these days to do it. thanks for the helpful keywords though, it helps point me in the right direction.
- nicce 3y agoYou would still need a huge training and data set for model before it can do something useful from your private data.
- selfhoster11 3y agoRetrieval Augmented Generation (RAG) is a good stopgap measure. It avoids the need for a lot of work on customising the LLM by supplying the relevant data via vector DB queries and letting LLM remain as a "calculator for words" that will just rephrase/extract the answer from the DB query result.