The LSTM idea is good, but the ordering problem stems from the LSTM assumption that the input is in some way ordered (i.e. words in a text, or frames in a video

The LSTM idea is good, but the ordering problem stems from the LSTM assumption that the input is in some way ordered (i.e. words in a text, or frames in a video). I would instead try separate inputs for each image, followed by an aggregation layer. I think a maximum-pooling approach makes most sense for most of the labels: there seem to be some photos for each restaurant providing all the information regarding certain labels. When there are five interior images, and one showing a patio, you want the latter to determine the label "outdoor dining". For other characteristics, such as "elegant" (I'm making up these labels because I read the post yesterday and don't remember the specifics) average pooling might be a better fit. In fact, a simple concatenation might work well, allowing the model to learn these things on its own. With differing numbers of images per restaurant that wouldn't work, though. Instead, one might try both MAX and AVG pooling, followed by concatenation. Or, if available, play around with other types of averaging, such as trimming outliers. On another point: I was slightly put off by the criticism of the data structure. That structure follows directly from the problem being solved and isn't some oversight making life more complicated than necessary. To work with such data, I tend to throw it into a rails application and export whatever format I need (although I'd use a python solution if I were more familiar with them).