11 ms·
GPT-4 Turbo with Vision is a step backwards for coding
- skenderbeu 3y agoWe're missing the elephant in the room. Who's going to maintain the code? You think GPT5 and Llama4 aren't going to be opinionated and change your code going forward.
- anotherpaulg 3y agoOpenAI just released GPT-4 Turbo with Vision and it performs worse on aider’s coding benchmark suites than all the previous GPT-4 models. In particular, it seems much more prone to “lazy coding” than the GPT-4 Turbo preview models.
- jimmywetnips 3y agoI've definitely run into this personally. But even even I explicitly tell it to not skip implementation and to generate fully functional code, it says that it understands but continues right into omitting things again. It was honestly shocking because we're so used to it understanding our commands that a blatant disregard like that made me seriously wonder what kind of laziness layer they added
- _giorgio_ 3y agoThey should offer different models at this point. This laziness occurs over and over, so why bother with omniscience.
- j45 3y agoThe laziness layer seems to be to be an assistant but not a replacement or doing the task.
- lyu07282 3y agoI suspect they might be worried it could reproduce copyrighted code in certain circumstances, so their solution was to condition the model to never produce large continuous chunks of code. It was a very noticeable change across the board.
- s-lambert 3y agoI thought it would be for performance, since it doesn't output all of the code then each reply is shorter/quicker. Although you can still ask it to generate more of the code but that introduces some latency so there's less overall load.
- satvikpendem 3y agoPeople hypothesized that OpenAI added laziness in order to save money on token generation, since they are burning through GPU time.
- joenot443 3y agoThis has been my conclusion too. Given it’s a product I’m paying monthly for it seems super regressive to have to trick it into doing what it used to do just fine.
- satvikpendem 3y agoI assume even the paid version is still a loss leader, 20 dollars per month is nothing in GPU time compared to how much they must be spending to generate the output.
- whywhywhywhy 3y agoI'd probably pay triple to go back to the pre-"Dev Day" product at this point
- jumpCastle 3y agoYou can use older models with the api.
- memothon 3y agoThanks again for running all these benchmarks with model releases. They are really helpful to track progress!
- CGamesPlay 3y agoReally appreciate the thoroughness you apply to evaluating models for use with Aider. Did you adjust the prompt at all for the newer models?
- kingkongjaffa 3y agoIts not just for coding, the base “gpt-4” model seems better than the latest preview model https://platform.openai.com/docs/models/continuous-model-upgrades https://platform.openai.com/docs/models/continuous-model-upg...
- superkuh 3y agoThe -turbo models in the past have been much worse too. gpt-3.5-turbo is way way worse than text-davinci-003 (gpt-3.5). The -turbos are correspondingly priced. gpt-4-turbo is ~1/3 price of gpt-4, 6.6x more expensive than gpt-3.5-turbo-instruct and 20x gpt-3.5-turbo.
- j45 3y agoI wish base gpt-4 was available in the chat product, miss it.
- pps 3y agoI'm not using ChatGPT now, but isn't this the old GPT-4? https://chat.openai.com/g/g-YyyyMT9XH-chatgpt-classic https://chat.openai.com/g/g-YyyyMT9XH-chatgpt-classic
- j45 3y agoThanks!
- paradite 3y agoYou can get the baseline GPT-4 model without new system prompts via ChatGPT Classic: https://chat.openai.com/g/g-YyyyMT9XH-chatgpt-classic https://chat.openai.com/g/g-YyyyMT9XH-chatgpt-classic It is an official GPT provided by ChatGPT with GPT-4 as backend.
- deleted 3y ago[deleted]
- kgeist 3y agoIf it was possible to hook into token selection process (kind of like JSON restricted grammar but using custom scripts), then it would be possible to detect that GPT-4 is about to add "# impement code here" and then we could force it select a different set of tokens which would make GPT4 generate a proper method body.
- mike_hearn 3y agoThat's called guidance and the problem is that it has to be done carefully or else you'll just get rephrasings that work around the block. I think a better approach is multi-pass coding along with fine-tuning or prompting to use a particular form of TODO comment. Aider can already do a form of fake "fill in the middle" by making it emit diffs. If it notices that some code has been filled out lazily, it could go back and ask it to do the next chunk of work. Given that large tasks are normally split up into small tasks by programmers anyway, this seems like a natural approach that is required for scaling up regardless.
- macrolime 3y agoAnother thing I have noticed is that if you use ChatGPT and it at some points uses Bing to look up something, it becomes super lazy afterwards, going from page long responses on average to a single paragraph.
- e6quisitory 3y agoUse ChatGPT Classic.
- Amivit 3y agoYou can/should make a custom GPT that isn't allowed to use Bing. Works much better that way
- sigmoid10 3y agoIt probably has to do with the extended context window. Keeping websites in there is kind of a hassle. But I actually consider that a feature, not a bug. If I have ChatGPT use the internet, I don't want a full page answer - especially not on the relatively slow GPT4. It's also a hassle if you're unsure about the validity of the output. In that case I might as well browse myself. Just give me a short preview so I can either start searching on my own or ask more questions.
- barfbagginus 3y agoIf answer is too lazy, you can tell it to elaborate. However, repairing a lazy context is sometimes slow and unreliable. To avoid that, use backtracking and up the pressure for detailed answers. Then consider taking the least lazy of 2 or 3 samples. A good prompt for detailed answers is Critique Of Though, an enhanced chain of thought technique. You ask for a search and a detailed response with simple sections including analysis, critique and key assumptions. It will expend more tokens, get more ideas out, and achieve higher accuracy. It will also be less lazy and more liable to recover from laziness or mistakes. TLDR; if GPT4 is being lazy, backtrack and request a detailed multi section critical analysis.
- kgabis 3y agoSo the more advanced the AI, the more human-like it becomes. Senior Programmer level AI will spend all computing resources browsing memes.
- dontupvoteme 3y agoGood thing Claude's a massive step forward.
- Zetobal 3y agoIt's worthless until they open up the api for private use.
- rfw300 3y agoI’ve been using the Claude 3 API since the models were announced. I believe it’s generally available (though capacity constrained & rate limited at present).
- 098799 3y agoYou do have to give them the company name though (however inconsequential that is)
- dontupvoteme 3y agoYou can make something up. I don't' have a name yet
- Zetobal 3y agotaxes...
- gorbypark 3y agoI had my Anthropic account banned (presumably) because I was testing out the vision capabilities and took a photo of a Japanese kitchen knife and asked it to "translate the characters on the knife into English". This wasn't a Claude Pro account, but an API account, so it's extra weird because what if I had some product based off the API, and an end user asked/searched for something taboo..does my entire business get taken offline? Good thing this was just a test account with like $10 in credit on it. They haven't responded to my "account suspension appeal" which is just a google form to enter your email address, not even a box to enter any details. Anyways, Claude 3 Opus is pretty great for coding (I think better in most cases than the GPT4-Turbo previews) but I'm a bit weary of Anthropic now.
- tedsanders 3y agoInterestingly, GPT-4 Turbo with Vision is at the top of the LiveCodeBench Leaderboard: https://livecodebench.github.io/leaderboard.html https://livecodebench.github.io/leaderboard.html (GPT-4 Turbo with Vision has a knowledge cutoff of Dec 2023, so filter to Jan 2024+ to minimize the chance of contamination.) In general, my take is that each model has its own personality, which can cause it to do better or worse on different sorts of tasks. From evaluating many LLMs, I've found that it's almost never the case that one model is better than an another at everything. When an eval only has a certain type of problem (e.g., only edits to long codebases, or only short self-contained competition problems), it's not clear how homogeneously its performance rankings will generalize to other coding tasks. Unfortunately, if you're a developer using an LLM API, the best thing to do is to test all of the models from all the providers to see which works best for your use case. (I work at OpenAI, so feel free to discount my opinions as much as you like.)
- anshumankmr 3y agoHi Ted, since I have been using GPT 4 pretty much every day, I have a few questions about the performance, We had been using 1106 preview for several months to generate SQL queries for a project, but one fine day in February, it stopped responding and it used to respond like so "As a language model, I do not have the ability to generate queries etc...". This lasted for a few hours. Anyway, switching to 0125-preview which helped us immediately resolve the problem. We have been using that for whenever we have code generation related tasks unless we are doing FAQ stuff (where GPT 3.5 Turbo was good enough). However, off late, I am noticing some really inconsistent behaviours in 0125-preview where it responds inconsistently for certain problems, ie one time it works with a detailed prompt and other time it doesn't. I know these models are predicting the next most likely token which is not always deterministic. So I was hoping for the ability to fine tune GPT 4 Turbo some time soon. Is that on the roadmap for Open AI?
- hmottestad 3y agoI don’t work for OpenAI but I do remember them saying that a select few customers would be invited to test out fine tuning GPT-4, and that was several months ago now. They said they would prioritise those who had previously fine tuned GPT-3.5 Turbo.
- klohto 3y agoI would be curious to see if the results improve by using DSPy to improve your prompts (and also reevaluate which prompts work better on the newest model).
- golergka 3y agoI posted this before, I'll post this again: GPT getting lazier is not an objectively bad thing. I don't copy code that it generates but ask it about more high level concepts more often, and have to instruct it not to generate imports and other boilerplate code. In most cases, this “lazy” generation saves time and tokens and is exactly what I need.
- Simon321 3y agoYes that's true, but if you ask to give the full code specifically it should do do
- nottorp 3y agoInteresting, I only trust them for boilerplates that I hate writing 100 times...
- Culonavirus 3y agoIt can give you ideas and lead you to new paths when problem solving, you just have to be aware that its knowledge is planet wide but inch deep. I lost the count of how many times it conceptually gets "above the target" rather nicely and then its implementation is like a blind person throwing darts. I also lost the count of how many times it describes the code it wrote and it's like it has two brains, one which writes the descriprion and the other, ahem, a little bit "slower", that writes the code. Classic debate looks like this: - Hey how do you implement X in lang Y using Z? - Certainly! Blablablah this code adds 1 and the return is 3! - Your code returns 5 and it seems to add 2, fix it. - I apologize for the oversight, here's the fixed version (replies with the same, maybe slightly altered, but still broken, code) Well I guess ultimately one can't expect miracles from a statistics based token generation machine. Sometimes I do wonder if the entire gen AI craze of the last few years is just one massive bubble and we're actually nowhere near AGI. All the evidence I see when interacting with these models points towards them "knowing" things, but not "understanding" things, a context aware planet-scale Wikipedia. (Don't get me wrong, I still think LLMs are life changing for language specific tasks like translation etc., but they're just not in any way new forms of intelligent beings, which is what a lot of mainstream population and even some investors seem to think).
- pr337h4m 3y agoHow hard could it be to let ChatGPT Plus users choose model versions? (especially when older versions are accessible through the API)
- antonvs 3y agoThey may not want that because e.g. it would reduce the amount of interaction their latest model gets. In general, being able to force-upgrade your users is a big advantage.
- paradite 3y agoWell you can get the baseline GPT-4 model without new system prompts via ChatGPT Classic: https://chat.openai.com/g/g-YyyyMT9XH-chatgpt-classic https://chat.openai.com/g/g-YyyyMT9XH-chatgpt-classic It is an official GPT provided by ChatGPT with GPT-4 as backend.
- jeswin 3y agoA big limitation with GPT4 Turbo (and Claude 3) for coding is the output token size. The only way to overcome the 4k limitation is by generating a file (if it fits), and feeding it back to generate the second and so on. For this reason, GPT4-32k is my preferred model for codegen. I wish there were cheaper options.
- lynguist 3y agoCan you use 32k with Chat?
- nomel 3y agoChat is a fairly terrible interface for real work, since you can’t modify anything, meaning the context can get easily poisoned. I much prefer the API playgrounds, and third party interfaces, that slow editing both my input and the responses.
- mentos 3y agoWhat 3rd party tools do you recommend?
- jeswin 3y agoShameless plug. I have a VS Code extension that's very nearly ready. Codespin CLI tools (ready to use): https://github.com/codespin-ai/codespin https://github.com/codespin-ai/codespin VS Code extension for the CLI tool (soon): https://www.youtube.com/watch?v=2TJqosFmkao https://www.youtube.com/watch?v=2TJqosFmkao I'll do a Show HN in a week or two.
- paradite 3y agoI've been using 16x Prompt to construct prompts with source code context and pasting the final prompt into ChatGPT web UI: https://prompt.16x.engineer/ https://prompt.16x.engineer/ Disclosure: I built it.
- nomel 3y ago
- Zetobal 3y agoThe only thing I learned in the last year that you can't really benchmark llms at all. Above a certain level it's just edge case against edge case or script kiddies and multi billion corps optimizing their fine tune against the test.
- squarefoot 3y agoI swear I spent 3 full seconds wondering what in the world could GPT-4 have in common with Turbo Vision.
- rlawson 3y agoSame!
- wiseowise 3y ago> GPT is a step backwards for coding There, fixed the title for you.
- DrSiemer 3y agoGPT-3.5 performance for basic programming tasks used to be just fine, but over the past few weeks the output quality has dropped dramatically. All of this tweaking definitely has it's downsides.
- rrr_oh_man 3y agoThey’re probably retraining something right now
- nicce 3y agoProbably just reducing resources with the cost of quality. GPT-4 suddenly has started be much faster.
- pps 3y agoIf you prefer using GPT-3.5 due to its lower price or speed, wouldn't it be better to switch to Haiku? People were even able to match the performance of Opus when they added a couple of examples to the prompt.
- DrSiemer 3y ago"Unfortunately, Claude.ai is only available in certain regions right now." GPT 3.5 used to be good enough, so I never bothered getting a paid account. I also heard some reports about 3.5 actually being better for the type of coding tasks I usually offload.
- debian3 3y agoYou can use haiku for free here if you dont need the api https://labs.perplexity.ai https://labs.perplexity.ai
- novaRom 3y agoWhy someone still would use GPT-3.5 in 2024? There are tens of fully open models available which beat GPT-3.5 in every possible skill and you can run them locally.
- varispeed 3y agoMaybe I am bit dim, but how one can choose GPT-4 Turbo? Is this available from https://chat.openai.com/ https://chat.openai.com/ ?
- sunaookami 3y agoIt will get rolled out on chat.openai.com in the future. You must use the API or the OpenAI Playground currently. The GPT-4 you see on chat.openai.com is an older GPT-4-Turbo version.
- varispeed 3y agoIs there something I can run locally that will mimic the stock Web interface, but using my API keys?
- sunaookami 2y agoI find big-AGI pretty good: https://github.com/enricoros/big-AGI https://github.com/enricoros/big-AGI
- ChrisRR 3y agoI feel like degrading the discipline of programming/development to "coding" is a bigger step backwards. Coding is used in programming, but if you're just churning out code then you're not developing well-architected, maintainable and safe software It's like saying that accountant is just adding. I think come then of tax year you'd be avoiding an accountant who says they've got experience in adding.
- ryan-duve 3y agoI lean in the opposite direction. When someone random like a new neighbor initially asks me what I do, I say "I'm a coder at an insurance company". I teach fifth graders Python in an "Advanced Coding Club". When people ask me what I do for hobbies, one of them is "learning new languages to code with". I will only go into more detail if they are also technical and want more details about what it is I code. I don't think of it as degrading the thing that I do. I think of it as boiling it down to the simplest description, and I find it more refreshing than "software developer" or "computer programmer" or f"{word} engineer".
- searchableguy 3y agoI use what impact and benefit my work has to answer what I do. In your insurance case, I would say something like "I build tools to shield businesses from unexpected disasters like earthquakes or floods" or "I help people worry less about expenses during an emergency" If someone asks me more, then I might add on that I work on software to automate claim process or similar.
- floor_ 3y ago[flagged]
- tokai 3y agoBeside your tone, the post is not even about semantics at all.
- SkyBelow 3y ago
- naiv 3y agoI am a bit lost looking at the models Can the following be assumed: - The gpt-4-preview models are history now - gpt-4-turbo-2024-* are the now released models - There will be no more 'preview' models released in the '4' branch ?
- deleted 3y ago[deleted]
- submeta 3y ago> In particular, it seems much more prone to “lazy coding” than the existing GPT-4 Turbo “preview” models. The previous model (without vision) was already „lazy“. It will omit large portions of code and wants you to merge your changes into previous answers. Then try hard to force him to give the full code, no omissions. That‘s why I reach for Claude 3 more and more. Its clntext window is larger, and it gives me full detailed answers, no omissions. But it is hallucinating more, my impression. Mentioning packages / functions that are not available. But all in all a superb choice in addition to ChatGPT4.