5 ms·
I believe the logic is fine. You seem to think you need multiple data points from the same version of the model (i.e. multiple samples per day at least) would b
by brabel 3y ago
I believe the logic is fine.
You seem to think you need multiple data points from the same version of the model (i.e. multiple samples per day at least) would be necessary to judge the actual performance on each particular day.
That's worse logic. How would you visualize the very large sample you would get? Even with the current 118 samples (one per day) it's already difficult to find a pattern.
Would you "average" the samples?? That would not help IMO, you would need to average the score of each image, which requires either manually doing it or finding a reliable algorithm to do it automatically, but good luck with that.
So, a sample per day which allows clearly visualizing any change in the results over months and years is a valuable thing to do and I find it hard to improve on the methodology. You just need to keep in mind that one single picture from the sample is not enough, no one is going to disagree with that... but that doesn't make it "bad logic" and it's pretty thoughtless to say so.
- bhouston 3y agoI agree that trying to determine the distribution of these drawings is hard because it isn't a simple floating point number in its current form. But maybe you could covert it to a linear monotonic measure? You could pass it to an image recognition model and see record the degree to which it thinks it is an: animal horse unicorn Basically if it fails to be a unicorn, see if it is a horse and if it fails to be a horse check if it is an animal. This gives you some type of linear measure if you place these three measures along the same axis adjacently. Then you can transform each image to a floating point number and then characterize the distribution.
- itsoktocry 3y ago>You could pass it to an image recognition model and see record the degree to which it thinks it is... We don't care what another algorithm "thinks". We want to see if what it draws is humanly interpretable as a unicorn.
- bhouston 3y agoThen you could use mechanical turk to have them rate each image to figure out how close it is to a Unicorn...
- butler14 3y agoMechanical turk 'workers' use ChatGPT
- imdsm 3y agoWe could but this is also a fun project, which is why when I checked it today I was surprised that what I saw was not a turd with eyes (2023-05-18) nor a strange sea creature (2023-07-08) but something which, for the first time I think, actually resembled a unicorn. I appreciate all the comments around determinism, sampling, scientific method, but as I said when I posted this just after building, it really is just for fun and to see, over time, if the general mish mash of outputs become more refined without any changes to the prompt (which doesn't aid it through CoT/ToT or improving on previous attempts etc.)
- civilitty 3y agoYou don’t have to justify yourself to the HN peanut gallery :-)
- ActivePattern 3y agoThen simply train the model to predict whether a human can interpret it as a unicorn.