Thanks! It's really not that crazy. Normally you're given an image and predict a label. For example ImageNet has 1000 labels. Here the idea is exactly the same,

Thanks! It's really not that crazy. Normally you're given an image and predict a label. For example ImageNet has 1000 labels. Here the idea is exactly the same, but we're predicting one of ~20,000 words one by one in sequence until a special terminating word is predicted (think of it as predicting the dot at the end of a sentence), at which point the RNN stops generating. There's a little more to it but that's the gist. These works all present a different model, but it's not clear what works best because they all popped up so quickly and I think we didn't really have a chance to settle and agree on common evaluation criteria. Since the task is so new, I chose the simplest/cleanest way to extend a Recurrent Network to condition on image data (through additive interactions). The others chose a bit more fancier models but don't have experiments showing that they necessarily perform better than the simpler models or by how much. I don't yet know what Google did, but so far it looks like a model very similar to mine. Also, all of the other works work on the level of entire images (as a single giant blob of 256x256x3 pixels), while my paper is the only one that also breaks down images into parts and objects, and there are parts of my paper dealing with that.