3 ms·

I don't buy this explanation, yes we are getting "constant data", but most of that is completely useless like, traffic noise, looking at the wall etc, and most

• Traubenfuchs 2y ago
  > the same shit my girlfriend says every day. Have you tried increasing the temperature?
• mattnewton 2y ago
  On the richness of the data- When you were a child every time you looked at any wall, how many wikipedia articles would it have taken to describe that entire experience? The model needs to discover what is “noise” by rereading something as sparse as Wikipedia over and over. Think of it as understanding films purely from the script. Or learning what sand is and how it behaves physically from Wikipedia rather than picking it up and having it run through your fingers. The experience of breathing and staring at a wall has an incredible amount of data that has to be learned before it is noise and is boring, and I’d argue it would take many Wikipedia articles to describe it fully. And on the novelty function- SGD does update things based on a loss and how unexpected things were to the model. But that’s very crude compared to what humans are doing, where everything gets the same learning rate multiple, and we feed nearly random data to the model. There is a lot of evidence that if you pick your training set carefully you can train a lot faster, and I think humans are picking their own training set.