7 ms·
The most frustrating thing about the many, many clones of this exact type of idea is that pretty much all of them require OpenAI. Stop doing that. You will ha
by chaxor 3y ago
The most frustrating thing about the many, many clones of this exact type of idea is that pretty much all of them require OpenAI.
Stop doing that.
You will have way more users if you make OpenAI (or anything that requires cloud) the 'technically possible but pretty difficult art of hoops to make it happen' option, instead of the other way around.
The best way to make these apps IMO is to make them work entirely locally, with an easy string that's swappable in a .toml file to any huggingface model. Then if you really want OpenAI crap, you can make it happen with some other docker secret or `pass` chain or something with a key, while changing up the config.
The default should be local first, do as much as possible, and then if the user /really/ wants to, make the collated prompt send a very few set of tokens to openAI.
- zikohh 3y agoHave you seen PrivateGPT. It's quite good and free.
- yoyopa 3y agowhat hardware do you need for that?
- drdaeman 3y agoConsumer-grade, AFAIK it's GPT4All with LLaMA.
- EGreg 3y agoLink?
- hoopsman 3y agoPresumably https://github.com/imartinez/privateGPT https://github.com/imartinez/privateGPT
- raffraffraff 3y agoThat's doesn't really answer the question. Neither does the github "System requirements" section, which I find disappointing. Ideally, it should give minimum memory requirements and rudimentary performance benchmark table for a sample data set, across a handful of setups (eg, Intel CPU, Apple M1, AMD CPU, with/without a bunch of common GPUs). With that information I would know whether or not it's worth my time even trying it out on my laptop. Edit: lol, I went through 2 pages of the issues on the github page and most of them could be avoided by putting this basic information into the system requirements: https://github.com/imartinez/privateGPT/issues/174 https://github.com/imartinez/privateGPT/issues/174 https://github.com/imartinez/privateGPT/issues/179 https://github.com/imartinez/privateGPT/issues/179 https://github.com/imartinez/privateGPT/issues/104 https://github.com/imartinez/privateGPT/issues/104 https://github.com/imartinez/privateGPT/issues/141 https://github.com/imartinez/privateGPT/issues/141 https://github.com/imartinez/privateGPT/issues/282 https://github.com/imartinez/privateGPT/issues/282 https://github.com/imartinez/privateGPT/issues/316 https://github.com/imartinez/privateGPT/issues/316 https://github.com/imartinez/privateGPT/issues/333 https://github.com/imartinez/privateGPT/issues/333 ... And lots more! Some of these people have 128gb memory and 32 cores, and still find it "very slow". Others having memory pool errors. Some of the answers hand-waving at needing "a more better computer" I reckon a lot of these issues could be closed and linked to a single ticket for proper hardware requirements in the readme.
- qingcharles 3y agoIt runs (slowly) on my 6 year old i5 laptop.
- mvkel 3y agoIt's not nearly usable. It's functional in that it spits out a response. Can that response be used for anything useful? No.
- deleted 3y ago[deleted]
- Der_Einzige 3y agoEven GPT-3.5-turbo-16K isn't good enough for most retrieval augmented generation tasks. Locally ran LLMs are far worse. I don't like it either, but for now, if you want good RAG, you have to use GPT-4
- sheeshkebab 3y agoAmen, local first should be the default for anything that sucks all my data. Although until these things can do my laundry none of them deserve any of my compute time either.
- trolan 3y agoThe only OpenAI 'crap' being used here is to generate the embeddings. Right now, OpenAI has some of the best and cheapest embeddings possible, especially for personal projects. Once the vectors are created tho, you're completely off the cloud if you so choose. You can always swap out the embedding generator too, because LangChain abstracts that for your exact gripes. Everything else is already using huggingface here and can be swapped out for any other model besides GPT2 which supports the prompts.
- space_fountain 3y agoDo you have citations on OpenAI embeddings being some of the cheapest and best? The signs I've seen points almost in the opposite direction?
- colobas 3y agoCan you elucidate on what those signs are? Thanks in advance
- muggermuch 3y agoAs per the Massive Text Embedding Benchmark (MTEB) Leaderboard maintained by Huggingface, OpenAI's embedding models are not the best. https://huggingface.co/spaces/mteb/leaderboard https://huggingface.co/spaces/mteb/leaderboard Of course, that's far from saying that they're the worst, or even headed that way. Just not the best (those would be a couple of fully opensource models, including those of the Instructor family, which we use at my workplace).
- ben_w 3y agoThe only embeddings I currently see listed on https://openai.com/pricing https://openai.com/pricing are Ada v2, at $0.1/million tokens. Even if the alternative is free, how much do you value your time, how long will it take to set up an alternative, and how much use will you get out of it? If you're getting less than a million tokens and it takes half an hour longer to set up, you'd better be a student with zero literally income because that cost of time matches the UN abject poverty level. This is also why it's never been the year of linux on the desktop, and why most businesses still don't use Libre Office and GIMP. I can't speak for quality; even if I used that API directly, this whole area is changing too fast to really keep up.
- AussieWog93 3y agoHonestly, for me, getting good quality results matters way more than keeping my searches private. And for that, nothing compares with GPT4.
- jstummbillig 3y agoWhat do you (or anyone else, feel free to chime in) do with other LLMs that makes them useable for anything that is not strictly tinkering? Here is my premise: We are past the wonder stage. I want to actually get stuff done efficiently. From what I have tested so far, the only model that allows me to do that halfway reliably is GPT-4. Am I incompetent or are we really just wishfully thinking in HN spirit that other LLMs are a lot better at being applied to actual tasks that require a certain level of quality, consistency and reliability?
- xmprt 3y agoThis line of thinking only works if it's impossible to imagine a world where OpenAI isn't the leader. In 2 years if the non OpenAI models are better then it will serve us much better to allow these tools to work with other models as well.
- jstummbillig 3y agoSince OpenAI is all just APIs with simple interfaces, I don't think that plugging a different, capable model in whatever tool you are building is going to be an issue.
- mcmoor 3y agoI still wonder what makes GPT-4 so much better than its contemporaries. That's why I saw tons of people trying to explain how GPT-4 works starting from simple neural network distasteful, tons of people already knew and do that but none of them is nearly close to GPT-4.l
- two_in_one 3y agoFrom my understanding. GPT-4 is the biggest, or one of the biggest. It was trained on low quality internet datasets, like the others. What makes it different is post-training on custom data with human supervision. We know they even outsourced that to Africa. Second, they integrated it with external tools. Like Python interpreter, internet browser. But the first is most important. Also most likely they have experimented and found some tricks which make it bit better.
- JimmyRuska 3y agoIt's difficult to compete. A small business might answer 10,000 requests to their chat bot. The options are - Pay openai less than $50mo - Manage cloud gpus, hire ml engineers > $1000/mo - Buy a local 4090 and put it under someone's desk, $no reliability +$1500 fixed Any larger business will need scalability and you still can't compete with openai pricing. Maybe one of you startup inclined people can make an openllama startup that charges by request and allows for finetuning, vector storage
- srcthr 3y agoPeople don't scale. This is personal. Only 3 is a good choice for people in a site with the name hacker something.
- whoiscroberts 3y agoI have a 4080, let’s do a startup. #cancode #hashomelab
- tikkun 3y agoDoing this. We soft launched yesterday with a paid Falcon-40B playground - 3 models for now Falcon 40b instruct, uncensored, and base. Adding API and per token pricing this week. https://api.llm-utils.org/ https://api.llm-utils.org/ And more models coming soon. Vector storage isn’t on the roadmap (what stops using a separate vector store from working well? Could add to roadmap but want to add understand more first), and we could add fine tuning if it’s a common request.
- JimmyRuska 3y agoLots of people using LLMs to make chat bots from their existing datasets: customer service troubleshooting, FAQs, billing, scheduling. Being able to upload their own pdfs, spreadsheets, docx, crawl their home page, lets the chat bot become personalized to their use case. While you could locally query your own vectordb before prompting, people buy paid service so they won't have to manage any of the technical details. If people can drag and drop some files from their nas, you parse them with apache tika or similar https://tika.apache.org/ https://tika.apache.org/ , they can start using personalized branded bots. It also lets you do things like refusing to answer, if the vector database returns nothing and the use case requires a specific answer from the docs only (not the llm to make stuff up).
- akira2501 3y ago> is that pretty much all of them require OpenAI. They're not here to release an actual product. They're here to release part of a CV proving they have "OpenAI" experience. I'm assuming this is the result of OpenAI not actually having any homegrown certification program of their own.
- gumby 3y ago> OpenAI not actually having any homegrown certification program A bit off topic but where are certifications (e.g. Cisco, Microsoft) useful? I am sure they are useful (both to candidates and companies) because people go to the effort to get these certs, and if they were useless everyone would have stopped long ago. I don't assume people do it for ego satisfaction. But I've never worked anywhere where it has come up as an interview criterion (nobody has ever pointed it out when we are looking at a resume, for example). Is it a big business thing? Is it just an HR thing?
- n4te 3y agoI've only ever seen it as a people who don't have a job thing.
- tw04 3y agoMainly when applying for a corporate job where you have 0 referrals. It's a guidepost that you at least have some idea what you're doing and are worth interviewing when people can't find someone who knows you and your previous work.
- codingdave 3y agoIt is a consulting / business partner thing. Different levels in the business partner programs require minimum number of certified employees in your consulting firm. So if you work in that slice of the industry, certifications matter. Outside of that... not so much.
- halfcat 3y agoYears ago, companies could get discounts if they were a “certified gold partner” or whatever. To be a partner, the company would need a certain number of certifications among their employees, so there was tangible value to companies who either used a ton of Microsoft licensing or Cisco/Dell hardware, or resold those to their own clients (better discount equating to higher margin). In some cases, getting the higher level certifications like Cisco CCIE was a virtual guarantee of a good job. I feel like this has become less of a thing in recent years, but I’m not involved in that space anymore.
- hospitalJail 3y agoClosedAI has freaked me out with how much power they have, and how irresponsible they are with it. I'm so horrified that they are going to take away the ability to ask medical questions when the AMA comes knocking at their door.
- tudorw 3y agoI don't liked closedAI either, this seems like the first tech I've played with in a long time that was great on day 1, and seems to get progressively less great over time.
- persedes 3y agoGpt4all does exactly that. You can choose between local model or bring your own openai token.
- d4rkp4ttern 3y agoDoes it provide a uniform interface that includes: retries, caching, streaming?
- dmezzetti 3y agotxtai makes it easy to use Hugging Face embeddings and Faiss, all local and configurable. https://github.com/neuml/txtai https://github.com/neuml/txtai paperai is a sub-project focused on processing medical/scientific papers. https://github.com/neuml/paperai https://github.com/neuml/paperai Disclaimer: I am the author of both
- anton5mith2 3y agohttps://mudler.pm/posts/smart-slackbot-for-teams/ https://mudler.pm/posts/smart-slackbot-for-teams/ with LocalAI?
- behnamoh 3y agoThis is a well-written tutorial and it's exactly what I was looking for! Thanks so much.
- bicx 3y ago> Stop doing that. This commanding attitude on HN seems to be getting worse lately. Not a fan.
- bendtb 3y ago#Chaxor: I fully agree with this. I am not keen on this being a one horse race, and for privacy reasons would like to deploy these models locally. However, it seems for many programmers it is somewhat easy to build something that can query into OPenAI so they can put it on their resume. Do you know of any FAISS / open source / one-click install w<windows app here I can search in my PDFs via vectors? I can see Secondbrain.sh will have the function in the future, but currently it does not. I have around 500 documents I want to be able to search in.
- raghavankl 3y agoWhen I was creating this tool, I made sure to abstract out the reliance on just one LLM or vector db. Instead, I focused on using langchain/huggingface for tokenization, embedding, and conversational modeling. This was done purposely so that it would be simple to replace the OpenAI dependencies with any other LLMs if needed.