4 ms·
Author here — thanks for explaining this. My expectation is that there will be incremental updates to the model, so while I'm providing the model `gpt-4` for c
by imdsm 4y ago
Author here — thanks for explaining this.
My expectation is that there will be incremental updates to the model, so while I'm providing the model `gpt-4` for completions, I'm recording the actual model, `gpt-4-0314` in this case, along with the result.
I don't want to monitor (and potentially miss) model updates, which is likely as this is very much a fire and forget project that I'll review over time. One per day seems more sensible to me than a large batch per month, as the daily generations are more likely to track model changes if they become frequent.
Will be exciting to see where we are in a few months!
- simonw 4y agoBut if each image is random, what does an image being "better" one day actually tell you about the underlying model? That's why I suggested generating 16 models each time instead of just 1 a- if all 16 are noticeably better than the previous day you've learned something a lot more interesting than if just one appears to be better than the previous one.
- imdsm 4y agoI don't think any individual sample will tell us much. I think we'll need to review this in 3 months, 6 months, 12 months etc, and look for patterns of changes. Ultimately, this is just a fun side project, something to leave running and check back on once every month or two.
- easygenes 4y agoI suggest the author generate a few different examples per day, each with a different temperature setting. 0.0 temperature ought to be deterministic on the same model, but would be interesting to look for trends at higher temperatures over time.
- imdsm 4y agoThat's a great idea, thanks easygenes!