8 ms·
ChatGPT Forgot My Startup
- throwawayadvsec 3y agoDid anyone try to look at the parameters if they're available in the browser requests to see if that may be related?(temperature?) or maybe are they doing quantization or some other kind of optimization?
- jerpint 3y agoThis is a very important detail. Temperature > 0 means a different result every time
- breckenedge 3y agoThe first response could easily have been a lucky hallucination.
- gs17 3y agoVery likely. I asked it about a programming language I work on (which it doesn't know the first thing about) and it hallucinates features based on its name, sometimes accurately. "CollegeAI" has some obvious potential use cases.
- BulgarianIdiot 3y agoVery unlikely. That's a pretty accurate description of the startup. "CollegeAI" doesn't suggest at all it's an admission matching platform. The hot take would've been it's a college driven by AI, or college specializing in degrees related to AI or god knows what.
- gs17 3y agoIf it hallucinates that it's to help people get into college (which isn't impossible to guess) everything else flows naturally from that.
- BulgarianIdiot 3y agoEverything is possible. We're discussing what's likely and not likely. I don't buy the hallucination theory. BTW, technically it "hallucinates" everything, because it has no connection to the outside world, only memories to base its answer on. But if the hallucinations are correct, then being less correct is still a problem. If you can show it's more correct in other ways, be my guest.
- johnwheeler 3y agoEveryone says GPT 4 is getting much worse. OpenAI says nothing changed. OpenAI, seems we have a problem.
- imtringued 3y agoIt is probably catastrophic forgetting.
- atherton33 3y agoWhat's likely changed is your expectations.
- deleted 3y ago[deleted]
- introvertme 3y agoany link to openai refuting the claim?
- shapefrog 3y agoOpenAI Employee: GPT-4 has been static since March - https://news.ycombinator.com/item?id=36155267 https://news.ycombinator.com/item?id=36155267
- JimtheCoder 3y agoMaybe the AI really is learning from it's human inputs, thus becoming less intelligent...
- SkyPuncher 3y agoPeople are familiar enough with LLMs to understand the nuance in their limitations. Many of these only become obvious when you transition from using it as a toy to using it as a tool. We a novel tool like this comes out people take wild guesses about how to use a tool, jumping well past its capabilities. When they see limits, they jump back to zero (or close to zero) and try again. Once your familiar with a tool, you know how to use it near its limits. Naturally, you’re going to feel more pain when you cross the limit. It won’t feel like you’re far away from a known good. I suspect this is essentially all that’s going on. P
- DangitBobby 3y agoGPT 3.5 seems to have forgotten that Go has generics now. They definitely need to add recent training data.
- heyoni 3y agoDid it forget? I actually checked this out recently and generics were introduced in 1.18 while chatgpt is only trained on golang up to 1.17. I realize it could also be trained on alpha release for 1.18 but it doesn’t seem to be and that’s a plausible explanation.
- refulgentis 3y agoAnswers aren't deterministic.* Especially in ChatGPT, it's impossible to compare. Quick pass: - questions are different. - answer = function of several variables, most importantly, entire chat. then temperature, top-p... - models haven't changed. - entirely possible chatgpt's temperature, top-p etc. changed but model didn't see images of gpt3.5 at 0 temp and 0.3 on this tweet, 0.3 makes up an answer like february, 0 gets "sorry, as an AI..." https://twitter.com/jpohhhh/status/1665007972114743297?s=20 https://twitter.com/jpohhhh/status/1665007972114743297?s=20 * unless temp = 0 and all previous messages in the chat are the same. On ChatGPT, you can't even see all the messages in the chat System messages aren't visible
- svaha1728 3y agoEinstein’s quote about God not playing dice with the universe seems appropriate here. ChatGPT’s universe is governed by probability as it applies to a training set.
- furyofantares 3y agoThis is 3.5 and it seems pretty likely the original is just made up from the name of the thing, not actually something it has in its training.
- colinplamondon 3y agoOpenAI seems to be prioritizing the mobile experience over web. Their mobile app is extremely fast with GPT-4, faster than web. I’d imagine once mobile is well-established, they’ll equalize it out. Totally makes sense, if I’m right. They should just state it publicly.
- photoGrant 3y agoI have had a significant decrease in usable GPT-4 via Chat. Anyone to say otherwise hasn't been using it daily as a plus user since first access. It's absolutely hindered in comparison. But also twice as fast.