6 ms·
How do these small models compare to gpt4 for coding and technical questions? I noticed that gpt3.5 is practically useless to me (either wrong or too generic),
by p1esk 3y ago
How do these small models compare to gpt4 for coding and technical questions?
I noticed that gpt3.5 is practically useless to me (either wrong or too generic), while gpt4 provides a decent answer 80% of the time.
- modeless 3y agoThey are not close to GPT-4. Yet. But the rate of improvement is higher than I expected. I think there will be open source models at GPT-4 level that can run on consumer GPUs within a year or two. Possibly requiring some new techniques that haven't been invented yet. The rate of adoption of new techniques that work is incredibly fast. Of course, GPT-5 is expected soon, so there's a moving target. And I can't see myself using GPT-4 much after GPT-5 is available, if it represents a significant improvement. We are quite far from "good enough".
- vitorgrs 3y agoI believe one of the problems that OSS models need to solve, is... dataset. All of them lack a good and large dataset. And this is most noticiable if you ask anything that is not in English-American-ish.
- mattigames 3y agoMaybe it should be an independent model in charge only of converting your question to American English and back, instead of trying to make a single model speak all languages
- jitl 3y agoThere’s a lot of world knowledge that is just not present in an American English corpus. For example knowledge of world cuisine & culture. There’s precious few good English sources on Sichuan cooking.
- vitorgrs 3y agoI don't think this is a good idea. A good model if we are really aiming at anything that resembles AGI (or even a good LLM like GPT4) is a model that have world knowledge. The world is not just English.
- p1esk 3y agoI’m both excited and scared to think about this “significant improvement” over GPT-4. It can make our jobs a lot easier or it can take our jobs.
- stavros 3y agoIsn't that the same? At some point, your job becomes so easy that anyone can do it.
- Spivak 3y agoIt's weird for programmers to be worried about getting automated out of a job when my job as a programmer is basically to try as hard as I can to automate myself out of a job.
- Der_Einzige 3y agoYou’re supposed to automate yourself out but not tell anyone. Didn’t you see that old simpsons episode from the 90s about the self driving trucks? The drivers rightfully STFU about their innovation and cashed in on great work life balance and Homer ruined it by blabbered about it to everyone, causing the drivers to try to go after him. We are trying to keep SWE salaries up, and lowering the barrier to entry will drop them.
- deleted 3y ago[deleted]
- rmbyrro 3y agoI expect the demand for SWE to grow faster than productivity gains.
- Der_Einzige 3y agoThe idea that demand scales to fill supply doesn’t work when supply becomes effectively infinite. Induction from the past is likely wrong in this case
- OfSanguineFire 3y agoCurious thought: at some point a competitor’s AI might become so advanced, you can just ask it to tell you how to create your own, analogous system. Easier than trying to catch up on your own. Corporations will have to include their own trade secrets among the things that AIs aren’t presently allowed to talk about like medical issues or sex.
- p1esk 3y agoHow to create my own LLM? Step 1: get a billion dollars. That’s your main trade secret.
- deleted 3y ago[deleted]
- chongli 3y agoWhat is inherent about AIs that requires spending a billion dollars? Humans learn a lot of things from very little input. Seems to me there's no reason, in principle, that AIs could not do the same. We just haven't figured out how to build them yet. What we have right now, with LLMs, is a very crude brute-force method. That suggests to me that we really don't understand how cognition works, and much of this brute computation is actually unnecessary.
- nemothekid 3y agoIf we knew how to build humans for cheap, then it wouldn't require spending a billion dollars. Your reasoning is circular. It's precisely because we don't know how to build these LLMs cheaply that one must so spend so much money to build them.
- chongli 3y agoThe point is that it's not inherently necessary to spend a billion dollars. We just haven't figured it out yet, and it's not due to trade secrets. Transistors used to cost a billion times more than they do now [1]. Do you have any reason to suspect AIs to be different? [1] https://spectrum.ieee.org/how-much-did-early-transistors-cost https://spectrum.ieee.org/how-much-did-early-transistors-cos...
- 0xDEF 3y ago>I think there will be open source models at GPT-4 level that can run on consumer GPUs within a year or two. There is indeed already open source models rivaling ChatGPT-3.5 but GPT-4 is an order of magnitude better. The sentiment that GPT-4 is going to be surpassed by open source models soon is something I only notice on HN. Makes me suspect people here haven't really tried the actual GPT-4 but instead the various scammy services like Bing that claim they are using GPT-4 under the hood when they are clearly not.
- rmbyrro 3y agoMakes me suspect you don't follow HN user base very closely.
- refulgentis 3y agoYou're 100% right and I apologize that you're getting downvoted, in solidarity I will eat downvotes with you. HNs funny right now because LLMs are all over the front page constantly, but there's a lot of HN "I am an expert because I read comments sections" type behavior. So many not even wrong comments that start from "I know LLaMa is local and C++ is a programming language and I know LLaMa.cpp is on GitHub and software improves and I've heard of Mistral."
- CSMastermind 3y agoMistral's latest just released model is well below GPT-3 out of the box. I've seen people speculate that with fine-tuning and RLHF you could get GPT-3 like performance out of it but it's still too early to tell. I'm in agreement with you, I've been following this field for a decade now and GPT-4 did seem to cross a magical threshold for me where it was finally good enough to not just be a curiosity but a real tool. I try to test every new model I can get my hands on and it remains the only one to cross that admittedly subjective threshold for me.
- rmbyrro 3y agoStill, for a 7B model, this is quite impressive.
- espadrine 3y ago> Mistral's latest just released model is well below GPT-3 out of the box The early information I see implies it is above. Mind you, that is mostly because GPT-3 was comparatively low: for instance its 5-shot MMLU score was 43.9%, while Llama2 70B 5-shot was 68.9%[0]. Early benchmarks[1] give Mixtral scores above Llama2 70B on MMLU (and other benchmarks), thus transitively, it seems likely to be above GPT-3. Of course, GPT-3.5 has a 5-shot score of 70, and it is unclear yet whether Mixtral is above or below, and clearly it is below GPT-4’s 86.5. The dust needs to settle, and the official inference code needs to be released, before there is certainty on its exact strength. (It is also a base model, not a chat finetune; I see a lot of people saying it is worse, simply because they interact with it as if it was a chatbot.) [0]: https://paperswithcode.com/sota/multi-task-language-understanding-on-mmlu https://paperswithcode.com/sota/multi-task-language-understa... [1]: https://github.com/open-compass/MixtralKit#comparison-with-other-models https://github.com/open-compass/MixtralKit#comparison-with-o...
- brucethemoose2 3y agoHave you played with finetunes, like Cybertron? Augmented in wrappers and retrievers like GPT is? It's not there yet, but its waaaay closer than the plain Mistral chat release.
- idonotknowwhy 3y agoIf you can run yi34b, you can run phind-codellama. It's much better than yi and mistral for code questions. I use it daily. More useful than gpt3 for coding, not as good as gpt4, except that I can copy and paste secrets into it without sending them to openai.
- mark_l_watson 3y agoThanks, I will give codellama a try.
- sharemywin 3y agowhat types of things do you ask ChatGPT to do for you regarding coding?
- yodsanklai 3y agoTypically a few lines snippets that would require me a few minutes of thinking but that ChatGPT will provide immediately. It often works, but there are setbacks. For instance, if I'm lazy and don't very carefully check the code, it can produce bugs and cancel the benefits. It can be useful, but I can see how it'll generate a class of lazy coders who can't think by themselves and just try to get the answer from ChatGPT. An amplified Stack Overflow syndrome.
- valval 3y agoOpen source models will probably catch up at the same rate as open source search engines have caught up to Google search.
- apantel 3y agoOne thing people should keep in mind when reading others’ comments about how good an LLM is at coding, is that the capability of the model will vary depending on the programming language. GPT-4 is phenomenal at Java because it probably ate an absolutely enormous amount of Java in training. Also, Java is a well-managed language with good backwards-compatibility, so patterns in code written at different times are likely to be compatible with each other. Finally, Java has been designed so that it is hard for the programmer to make mistakes. GPT-4 is great for Java because Java is great for GPT-4: it provides what the LLM needs to be great.