9 ms·
GPT-4-turbo preliminary benchmark results on code-editing
- xeckr 3y agoBack in April it would only generate a handful of tokens per second. The speed improvements for GPT-4 are staggering. I wonder how much of it is because Microsoft is making GPUs rain on OpenAI, and how much of it is due to improvements to the model and its scaffolding.
- andy_xor_andrew 3y agoI think the general (completely speculative and unconfirmed) consensus is that "Turbo" models are somehow quantized, or otherwise modified for much faster and cheaper inference, at the expense of some quality (the definition of "some" is unknown here). well, that's how it was for GPT-3.5 anyway. The "turbo" flavor was faster and cheaper, but seemed to have slightly worse output (again, this is all going by subjective measurement; it could entirely be the imagination of AI bros)
- xeckr 3y agoI agree that this is the case for GPT-3.5, at least subjectively. However, with GPT-4 Turbo, it seems that performance has improved. If they got it to be faster using quantization, then they must have also found a way to offset any resulting performance losses.
- letitgo12345 3y agoThey have tons of usage data by now to figure out which queries to devote model capacity to
- kridsdale3 3y agoPerhaps we owe it to Nvidia. Perhaps a huge batch of H100 arrived and the premium models can run there instead of the A100.
- smodad 3y agoBased on rumors, it seems that because the demand is so high and supply so short, that Nvidia is having to select who gets the cards. I bet they're likely thinking about who can do the most impactful work and OpenAI would definitely be in the running for the short list of companies who are actually shipping. So I bet OpenAI / Microsoft got a lot of the newest cards.
- hooloovoo_zoo 3y agoThey’re definitely thinking about impact… to their overall margins.
- deleted 3y ago[deleted]
- jiggawatts 3y agoH100 + quantisation + algorithmic improvements would be sufficient to explain the speed boost. If you "have enough compute" available -- which OpenAI definitely does -- the best current technique is to use mixed precision with post-quantisation fine tuning to restore performance. That's most probably how all of the "turbo" models work. Take a model that was initially 16 or 32 bits per parameter during training, quantise it down to a mixture of 4, 8, and 16 bits, and then fix it up with an additional training pass that uses the original full-fat model's predictions as the loss function. With access to the raw parameters, it's possible to do this training such that all of the output weights are considered and adjusted during this phase instead of just the top word. Third parties fine-tuning against GPT4 chats can't do this, even with the collected samples, because they only have individual selected tokens/words instead of the full probability distribution.
- kridsdale3 3y agoTo use a graphical term, would it be fair to call that process "dithering"?
- jiggawatts 3y ago
- exo-pla-net 3y agoSo it appears that GPT-4-Turbo is indeed (at least marginally) smarter than the previous GPT-4, just as Altman claimed. Also, it's faster and cheaper, with a massive context window. Exciting!
- doctoboggan 3y agoYes I was really worried this was going to be "better" as in faster and cheaper but not "better" as in smarter. I've been playing with it for the past day and haven't noticed it any smarter per se, but also haven't noticed it dummer either.
- kridsdale3 3y agoIt's a big surprise. I assumed something would have to have been "lobotomized" to make this speed increase. I'd love to know what they did.
- bigyikes 3y agoWell, the knowledge cutoff is much more recent (I think sama said April 2023?), so having more, newer data might be a significant contributing factor.
- Racing0461 3y agoIf this is the case, why use the word turbo in the name which has the baggage of faster but worse reasoning.
- jquery 3y agoPerhaps it may have worse reasoning for some tasks, so having "turbo" lets people know they can still try the old version if turbo doesn't work for them? Kind of like 3.5 vs 3.5 Turbo.
- japhyr 3y agoThe increased context size will have the most significant impact on my work. That's where I run into limitations, when reviewing written work and code. I've been feeding written work into GPT in chunks, and I'm really happy to be able to feed in whole pieces. (I don't have it revise anything for me, I just have a specific prompt for exactly the kind of feedback I want on written work.) I tried Claude because of the larger context size, but I've been disappointed so far. I find Claude much more likely to just compliment my writing, whereas GPT will identify strengths and areas that could be improved.
- ttul 3y agoThe progress here is remarkable. A year ago, we didn’t even have ChatGPT. LLM completions were cool but so hard to use and definitely there was nothing accessible to non-nerds.
- cloudking 3y agoHas anyone been able to access the 128k context window? I'm not seeing that option in the API playground
- throw03172019 3y agoYes, it’s fantastic. It is called gpt-4-1106-preview
- infecto 3y agoYou might want to double check. The latest turbo models for 4 and 3.5 don’t differentiate. There is only one model and it has 128k context.
- kridsdale3 3y agoI wonder if 3.5 / 4 is really just "agents are on" or not.
- thequadehunter 3y agoThe reason he's not seeing it is because it's only available as an API endpoint. It doesn't show up in playground.
- infecto 3y agoNope, they definitely show up in the playground.
- weird-eye-issue 3y agoConfidently incorrect
- cloudking 3y agoI think I need to be in the higher usage tier to unlock the full 128k context length
- johnxie 3y agoYes, it should be under gpt-4-1106-preview.
- Racing0461 3y agoIs this just the api or does it work on chatgpt also?
- dmm 3y agoI haven't seen any changes in the chatgpt plus interface yet.
- replwoacause 3y agoMe either
- adamsmark 3y agoYou can try it out in the playground but not the consumer ChatGPT website. You will be limited to 10K tokens per minute if you are not in any spending tier e.g $150 per month will give you bigger context window.
- Racing0461 3y agoreddit thread on the opposite experience - https://www.reddit.com/r/ChatGPT/comments/17prwlg/gpt4_turbo_is_unusable_for_coding_and_various/ https://www.reddit.com/r/ChatGPT/comments/17prwlg/gpt4_turbo...
- a_wild_dandan 3y agoWe can always count on Reddit to provide the cold, hard anecdotes.
- vunderba 3y agoThat Reddit user mentions something that I have noticed happening with infuriatingly increasing frequency in the last few weeks (at least on the chat GPT interface) - the "fill in the rest here" or "remainder of code" snipped sections even when you EXPLICITLY instruct ChatGPT to include the fully modified code in its responses. It's like the AI equivalent of "the rest is left as an exercise to the reader" you'd find in old textbooks.
- meiraleal 3y agoThis is so annoying. And it is getting worse. And a great way for OpenAI to hinder competition and then copy the products being developed using their APIs. This is openai cheating on their userbase.
- Racing0461 3y agoAgreed, this is single handedly making me hate chatgpt for coding. I wish OpenAI would respect it's paying users enough to say what the changes and limitations are. (but i guess like google, i am not openai's customer. Their investors are. i am the product). gpt4 should really be called DecartesGPT with this bs.
- replwoacause 3y agoSame. It hasn’t blocked me from learning with it but it is a hindrance.
- deleted 3y ago[deleted]
- meiraleal 3y agoThe past days ChatGPT went from a great pair programming helper to a useless antipathetic intern, the quality of generated code dropped visibly. The context seems to be bigger in the chatgpt plus version too but it got dumber.
- jpdus 3y agoFor other (non-code) benchmarks, people are having the opposite experience: "I benchmarked on SAT reading, which is a nice human reference for reasoning ability. Took 3 sections (67 questions) from an official 2008-2009 test (2400 scale) and got the following results, here a SAT-like test: - GPT3.5 - 690 (10 wrong) - GPT4 - 770 (3 wrong) - GPT4-turbo (one section at time) - 740 (5 wrong) - GPT4-turbo (3 sections at once, 9K tokens) - 730 (6 wrong)" Source: https://twitter.com/wangzjeff/status/1721934560919994823?t=PcAm8yVbU_odyqK9e53MAA&s=19 https://twitter.com/wangzjeff/status/1721934560919994823?t=P...
- exo-pla-net 3y agoN=1
- dazzaji 3y agoDoes anybody know if 2008-2009 SAT is in the training set for these models? Assuming so, I’d be especially interested in head-to-head evals on this type of non-code benchmark for problem sets not already in the training data, to see how it performs on fresh situations.
- rafaelero 3y agoProbably not a statistically significant difference there.
- Terretta 3y agoWhat did you mean by "opposite"? You seem to be suggesting it got a bit worse, and the aider article seems to suggest gpt4 got a bit worse, although much faster at being a bit worse, while gpt3.5 got worse, then better, while faster.
- reitzensteinm 3y agoThe Aider article has been updated with the complete results. Previously Turbo was leading slightly. So far any difference is in the noise. However, in my opinion the first attempt score is more important, and Turbo does genuinely seem to lead there. There's still a possibility the updated training data has tainted the results.
- vouaobrasil 3y agoProgrammers here seem excited about the potential of this new version...but I can't help but wonder at how naive this attitude really is. Even if AI never becomes intelligent like us, if it can emulate this intelligence in enough domains, then it has a serious chance of being dangerous. It's already pretty much guaranteed that it will put almost everyone out of a job, turning the vast majority of humans into content-consuming sloths. Does it really make sense to play with this kind of power?
- ketzo 3y agoIt’s way too late to pretend AI won’t have an impact on programming as a profession. Better to be excited and learn the tool as it develops than to stick your head in the sand. A world where AI has put literally everyone “out of a job,” meanwhile, is still so far from our current reality that IMO, it’s not worth making practical day-to-day decisions on unless you are directly involved in the development or regulation of AI.
- vouaobrasil 3y agoThat is exactly the prisoner's dilemma. Besides, I already quit being a programmer this year :)
- ketzo 3y agoHey, fair enough, more power to you then. Have fun touching some grass, friend!
- TerrifiedMouse 3y ago> meanwhile, is still so far from our current reality 2 years ago something like ChatGPT (as limited as it is) was “far from our current reality”. I think it’s worthwhile to think ahead.
- ketzo 3y agoI understand your broader point, which is that progress in this space will be discontinuous and hard to see until it’s already happened. But two years ago, GPT-3 was already astounding. ChatGPT made some pretty huge gains, and GPT4 more still, but the thing that exists today was relatively predictable, compared to a world where human effort is no longer important.
- kristianp 3y agoAider sounds like a cool tool, I'll have to try it out. I'm assuming it makes use of your local files and edits them for you? Are there any other programming assistant packages that use the chatgpt api like this? Regarding rate limits, it might be an idea to have configurable delays built in to the testing code to prevent hitting limits.