I totally understand your decision, but I bet a 1D deep convolutional network would do really well at this given a larger dataset. You can also do a lot of data

I totally understand your decision, but I bet a 1D deep convolutional network would do really well at this given a larger dataset. You can also do a lot of data augmentation by speed changes, filters, adding reverb, etc.