3 ms·
* advancing the state of the art It says right at the top: "we get 91.8% accuracy versus the previous best of 90.2%" on a standard sentiment corpus. In additio
by kleiba 10y ago
* advancing the state of the art
It says right at the top: "we get 91.8% accuracy versus the previous best of 90.2%" on a standard sentiment corpus. In addition, their method needs less training data than previous approaches.
* Were there any insights learned that give a deeper understanding of these phenomena?
The main appeal lies in the fact that a model trained on a (1) different and (2) very general task basically "in passing" also learned to predict sentiment (i.e., a specialized task that more or less arose from the domain the general model was trained on), and pretty much through a single neuron (out of the 4096 used). The authors speculate that this might be a general effect that could also be transferred to other prediction tasks.
- emcq 10y agoIf the main contribution here is the quality of the model and its interesting and powerful representation of text, I hope OpenAI does something distruptively different and releases the weights and trained model. The accidental sentiment neuron is a function of the model, distribution of the input dataset, and the optimizer finding nice saddle points. Insight into these foundational components would make these results amazing. It sounded like training on other datasets doesn't have the same sentiment properties, which provides a lever to explore these concepts more. At the moment it feels like the Google cat neuron. It attracted a lot of intrigue but the individual contribution from that in terms of research was more on the infrastructure side, and few people seem to refer back to that publication at this point. That said OpenAIs mission in itself doesn't necessarily require novel research. For example, the gym is fostering a competitive atmosphere for the community to work on RL which hopefully leads to more progress in the field. Training a model for a month is difficult and if it has captured interesting phenomena it seems in the interest of the community to release the weights and model. It would be hard for the community to reproduce this without a month of compute and 83M Amazon reviews.
- mappingbabeljc 10y agoHi there, the weights and model are here: https://github.com/openai/generating-reviews-discovering-sentiment https://github.com/openai/generating-reviews-discovering-sen...
- emcq 10y agoThis is awesome, thanks! My apologies I must have missed it somewhere.