5 ms·
Microsoft says GPT 3.5 has 20B parameters?
- samsepi0l121 3y agoSo it's possible to run gpt-3.5-turbo on a local machine?
- Flux159 3y agoLink to the paper directly https://arxiv.org/pdf/2310.17680.pdf https://arxiv.org/pdf/2310.17680.pdf The system that they describe, Codefusion is interesting because it's a diffusion model for generating code rather than an autoregressive model like most LLM code generators.
- airstrike 3y agoAlso on HuggingFace: https://huggingface.co/papers/2310.17680 https://huggingface.co/papers/2310.17680 > We evaluate CodeFusion on the task of natural language to code generation for Bash, Python, and Microsoft Excel conditional formatting (CF) rules. Microsoft Excel conditional formatting, really? The future we want is one in which that abomination has died out, not one in which it is thriving.
- mistrial9 3y agoding ding ding ding !! prizes for the spectator who correctly names the Tip of the Iceberg for AI-assisted-communications at every level. You have heard of evolution? creativity ? Free-Will ? now, with AI assisted communication technology, every road leads back to MSFT ! So you wondered about Proust, or that line in Hamlet? surprise - it is only the entry topic to get back to a clever rhyming ad-jingle for your next software subscription update. btw- our agent is authorized to check your bank account balances while you renew. according to that user agreement you acknowledged. /fiction
- wing-_-nuts 3y agoI know you're being tongue in cheek but this is why the evolution of open source models capable of cheaply running on your own hardware (i.e. the r/locallamma folks) is so important.
- superkuh 3y agoIt says gpt3.5-turbo has 20B parameters, which I believe. But it says gpt3.5 (text-davinci-003) has 175B parameters, which I also believe.
- WendyTheWillow 3y agoI just want the 100k context window Anthropic gave Claude, but for GPT. Claude likes to hallucinate when I ask it to build chapter notes for a class I’m taking, and I don’t want to have to break up the text into tiny bits for GPT…
- cjonas 3y agoThe large context is the reason its hallucinating and gpt would likely have the same problem. You can see this effect even on gpt-3.5-turbo-16k. Performance and accuracy are directly correlated with context size
- danielbln 3y agoThat's just GPT3.5 having a pretty spotty attention. This becomes much less of a problem if you use gpt-4-32k.
- Obscurity4340 3y ago*confabulate. Hallucination is StableDiffusion
- behnamoh 3y agoThe lost-in-the-middle phenomenon makes me _not_ want larger context windows because the model cannot effectively use it. It seems like we need another arch to process large ctx size.
- msp26 3y agoPerformance on long contexts just isn't very good right now. Relevant paper: Lost in the Middle: How Language Models Use Long Contexts https://arxiv.org/abs/2307.03172 https://arxiv.org/abs/2307.03172
- _ea1k 3y agoThat sounds incredible given how powerful the model is.
- sidcool 3y agoDidn't it leak earlier that it is 100 billion? And GPT 4 is 1.17 Trillion?
- nacs 3y agoThat 1 trillion number was quickly confirmed to be false. It's believed that GPT4 is a Mixture of Experts model -- it's 8 smaller models (estimated around 200B each) and a router picks the best model to redirect to based on input.
- hencoappel 3y agoSo 800B + a router network, so not far off a trillion?
- bmitc 3y agoJust eating up water and fossil fuels.
- behnamoh 3y agoThis was expected because open source models of the same size already beat GPT-3.5 in many ways. And Mistral 7B makes you think if huge parameters are even needed for something like GPT-3.5 level.
- marisha 3y agoPlease enlighten me. Which open-source models beat GPT3.5?
- dang 3y agoIt's against HN's guidelines to editorialize titles like this. From https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html: "Please use the original title, unless it is misleading or linkbait; don't editorialize." If you want to say what you think is important about an article, that's fine, but do it by adding a comment to the thread. Then your view will be on a level playing field with everyone else's: https://hn.algolia.com/?dateRange=all&page=0&prefix=false&sort=byDate&type=comment&query=%22level%20playing%20field%22%20by:dang https://hn.algolia.com/?dateRange=all&page=0&prefix=false&so...
- matusp 3y agoThis seems like a valid exception from this rule. The interesting tidbit here is what the title suggests, even though it is not the title of the paper. It should be possible to point out to facts like these hidden in papers, reports, etc without having to create a separate article about it. You can of course add a comment as you suggested, but that does not reflect how people use HN.
- nmstoker 3y agoI could be wrong but I believe the original comment that's used as the title here comes from this tweet: https://twitter.com/felix_red_panda/status/1718916631512949248?t=jmCkeVH1Hyyu4vmwY-NhQg&s=19 https://twitter.com/felix_red_panda/status/17189166315129492...
- Alifatisk 3y agoDid openAi ever publicize how many params GPT-4 had?
- leobg 3y agoThat would certainly explain the pricing (gpt-3.5 vs davinci).
- kaspermarstal 3y agoMhm, interesting development, the paper has been withdrawn.