5 ms·
How long till there's a self-hosted CoPilot based on llama or one of the other plethora of open source models? That will be a sizable market I predict. I'd jum
by baryphonic 3y ago
How long till there's a self-hosted CoPilot based on llama or one of the other plethora of open source models? That will be a sizable market I predict.
I'd jump into it if I had time/resources
- whimsicalism 3y agoHow long until Hacker News commentators wake up to the fact that the OSS models are terrible on complex tasks compared to GPT-3.5 & GPT-4
- baryphonic 3y agoI'd agree to an extent w/ GPT-4. GPT-3.5, not at all.
- whimsicalism 3y agoWhat OSS model comes close to 3.5?
- baryphonic 3y agoFor complex tasks? Just about any of them. Alpaca, Llama isn't bad (it's inferior but relatively close), other fine-tuned models. To be more precise, they're all about as as bad as GPT-3.5 at complex tasks, which isn't great. I don't live and breathe this stuff, but I do use GPT-3.5 frequently, and many of the open source models I've tried are surprisingly close to GPT-3.5.
- whimsicalism 3y agoI don't think the evidence bears that out [0]. I agree that GPT-4 is way better than GPT-3.5, but I don't think most of the OSS are even close to GPT-3.5. Vicuna is closer for simple tasks/conversation, but it still doesn't match GPT-3.5 elsewhere IMO, even though GPT-3.5 is also not great at complex tasks. [0]: https://github.com/microsoft/guidance/blob/main/notebooks/chatgpt_vs_open_source_on_harder_tasks.ipynb https://github.com/microsoft/guidance/blob/main/notebooks/ch...
- baryphonic 3y agoThis is fair; my only evidence is my personal experience, most of which has consisted of trying models on Huggingface or similar, which isn't persuasive. I will say based on my cursory glance that a lot of the tasks here seem odd for a chat AI, though there are certainly applications that might use them. E.g. asking if someone insulted another person given some transcript of their conversation seems like a somewhat tough sell to me. Nevertheless, ChatGPT performed better. Was that because ChatGPT has a better training set, a better architecture or more examples in it training set related to the questions? Is that even knowable? Anyway, cool notebook.
- nathan_compton 3y agoI've been following this stuff pretty closely and so far absolutely nothing out there in the open model world is even remotely close to GPT4 for coding. Gpt3.5 maybe, but I honestly find GPT3.5 almost useless.
- baryphonic 3y agoI haven't been given access to GPT-4 yet. I've heard good things, but also seen the order-of-magnitude price jump.