3 ms·
Think about what the model is doing: It’s sampling from a statistical distribution of likely tokens, one of which is the correctly spelled variable name.
by catlifeonmars 2y ago
Think about what the model is doing:
It’s sampling from a statistical distribution of likely tokens, one of which is the correctly spelled variable name.
- sk11001 2y agoMy point is that in previous versions it was sampling more correctly more often than it has been recently.
- rany_ 2y agoFrom what I've heard, the reason it's doing that is to add an AI fingerprint. It has to pick less likely tokens to encode this information to the detriment of the output quality. Unfortunately it's only hearsay but it made sense to me so I thought I'd share.
- elliotto 2y agoPeople seem to have noticed that chatgpt has good days and bad days. we don't really know what openai is doing in the background, so it could be that some days you get assigned a really bad B in an A/B model deployment test, or maybe they're throttling the performance because they're at peak, or likely some other factor. But it's an observed phenomenon that it performs well and then doesn't.
- TeMPOraL 2y agoYes. The presence of the correctly spelled name earlier in the context should've dominated the distribution so much that it should've been extremely unlikely to select an incorrectly spelled variable name in the place where the correct name should go.
- catlifeonmars 2y agoWhat if the model got more training data? There’s no guarantee that the training data causes the model to converge on correctness. It’s not like this stuff is curated for specific problems.