5 ms·
They are not close to GPT-4. Yet. But the rate of improvement is higher than I expected. I think there will be open source models at GPT-4 level that can run on
by modeless 3y ago
They are not close to GPT-4. Yet. But the rate of improvement is higher than I expected. I think there will be open source models at GPT-4 level that can run on consumer GPUs within a year or two. Possibly requiring some new techniques that haven't been invented yet. The rate of adoption of new techniques that work is incredibly fast.
Of course, GPT-5 is expected soon, so there's a moving target. And I can't see myself using GPT-4 much after GPT-5 is available, if it represents a significant improvement. We are quite far from "good enough".
- vitorgrs 3y agoI believe one of the problems that OSS models need to solve, is... dataset. All of them lack a good and large dataset. And this is most noticiable if you ask anything that is not in English-American-ish.
- mattigames 3y agoMaybe it should be an independent model in charge only of converting your question to American English and back, instead of trying to make a single model speak all languages
- jitl 3y agoThere’s a lot of world knowledge that is just not present in an American English corpus. For example knowledge of world cuisine & culture. There’s precious few good English sources on Sichuan cooking.
- vitorgrs 3y agoI don't think this is a good idea. A good model if we are really aiming at anything that resembles AGI (or even a good LLM like GPT4) is a model that have world knowledge. The world is not just English.
- p1esk 3y agoI’m both excited and scared to think about this “significant improvement” over GPT-4. It can make our jobs a lot easier or it can take our jobs.
- stavros 3y agoIsn't that the same? At some point, your job becomes so easy that anyone can do it.
- Spivak 3y agoIt's weird for programmers to be worried about getting automated out of a job when my job as a programmer is basically to try as hard as I can to automate myself out of a job.
- Der_Einzige 3y agoYou’re supposed to automate yourself out but not tell anyone. Didn’t you see that old simpsons episode from the 90s about the self driving trucks? The drivers rightfully STFU about their innovation and cashed in on great work life balance and Homer ruined it by blabbered about it to everyone, causing the drivers to try to go after him. We are trying to keep SWE salaries up, and lowering the barrier to entry will drop them.
- deleted 3y ago[deleted]
- rmbyrro 3y agoI expect the demand for SWE to grow faster than productivity gains.
- Der_Einzige 3y agoThe idea that demand scales to fill supply doesn’t work when supply becomes effectively infinite. Induction from the past is likely wrong in this case
- OfSanguineFire 3y agoCurious thought: at some point a competitor’s AI might become so advanced, you can just ask it to tell you how to create your own, analogous system. Easier than trying to catch up on your own. Corporations will have to include their own trade secrets among the things that AIs aren’t presently allowed to talk about like medical issues or sex.
- p1esk 3y agoHow to create my own LLM? Step 1: get a billion dollars. That’s your main trade secret.
- deleted 3y ago[deleted]
- chongli 3y agoWhat is inherent about AIs that requires spending a billion dollars? Humans learn a lot of things from very little input. Seems to me there's no reason, in principle, that AIs could not do the same. We just haven't figured out how to build them yet. What we have right now, with LLMs, is a very crude brute-force method. That suggests to me that we really don't understand how cognition works, and much of this brute computation is actually unnecessary.
- nemothekid 3y agoIf we knew how to build humans for cheap, then it wouldn't require spending a billion dollars. Your reasoning is circular. It's precisely because we don't know how to build these LLMs cheaply that one must so spend so much money to build them.
- chongli 3y agoThe point is that it's not inherently necessary to spend a billion dollars. We just haven't figured it out yet, and it's not due to trade secrets. Transistors used to cost a billion times more than they do now [1]. Do you have any reason to suspect AIs to be different? [1] https://spectrum.ieee.org/how-much-did-early-transistors-cost https://spectrum.ieee.org/how-much-did-early-transistors-cos...
- 0xDEF 3y ago>I think there will be open source models at GPT-4 level that can run on consumer GPUs within a year or two. There is indeed already open source models rivaling ChatGPT-3.5 but GPT-4 is an order of magnitude better. The sentiment that GPT-4 is going to be surpassed by open source models soon is something I only notice on HN. Makes me suspect people here haven't really tried the actual GPT-4 but instead the various scammy services like Bing that claim they are using GPT-4 under the hood when they are clearly not.
- rmbyrro 3y agoMakes me suspect you don't follow HN user base very closely.
- refulgentis 3y agoYou're 100% right and I apologize that you're getting downvoted, in solidarity I will eat downvotes with you. HNs funny right now because LLMs are all over the front page constantly, but there's a lot of HN "I am an expert because I read comments sections" type behavior. So many not even wrong comments that start from "I know LLaMa is local and C++ is a programming language and I know LLaMa.cpp is on GitHub and software improves and I've heard of Mistral."