3 ms·

I’m looking for actual research. I’m wrapping up some work on a stripped-down version of the lenet architecture and extemely low resolution non-representational

I’m looking for actual research. I’m wrapping up some work on a stripped-down version of the lenet architecture and extemely low resolution non-representational data (3x3 greyscale pixels). The results appear to be better than chance and I’m quite frankly stunned - I expected complete garbage. Maybe I’ve missed something, but if the results are simply due to chance, I had an utterly extraordinary case of bad luck. (edit:clarity)

enchiridion 6y ago
So are you making LeNet shallower somehow?
- thedragonline 6y ago
  The pooling layers are gone along with the first RELU. I've also reduced the number of outputs in each convolution layer.
  - enchiridion 6y ago
    That's quite a bit removed. I'd be surprised if it could do much of anything. I guess by using low res data you are also shrinking the problem.
    - thedragonline 6y ago
      I personally suspect that there are surprises in store for me by rooting around in this space. I have a .caffemodel that appears to correctly label better than chance, so my next step is to create a test dataset that's independent of original and see what happens. Personally, I find it unsettling that GPT 3 now contains 175 billion parameters. Why not go in the opposite direction and test the lower bound of what is possible?
verdverm 6y ago
You probably can avoid convolution for a 3x3 grey scale. Have you run your test multiple times with varied training sets? Did it perform well on validation data or test data? Honestly, it sounds like a data set that is easily memorized
- thedragonline 6y ago
  >Have you run your test multiple times with varied training sets? Some background - I've been working on this for several months now, experimenting with various CNNs (settling on a modified LeNet), hyperparameters and parameters. The bulk of the experiments have been failures - a typical scenario is the loss function decreasing in the training phase, but winding up unable to correctly predict labels in the test phase. There has been a progression however - from predicting either label A or B (but not both), to predicting both (but no better than chance), to doing a little bit better than chance. Maybe I'm fooling myself - I don't know. That's why I've been scouring the Internet for similar kinds of work (and not finding anything truly useful) and now reaching out to HN. If there's work out there that definitively rules out working in this space, I'm all ears. Otherwise I'm going to keep on experimenting. (edit:formatting)

enchiridion 6y ago

So are you making LeNet shallower somehow?

thedragonline 6y ago
The pooling layers are gone along with the first RELU. I've also reduced the number of outputs in each convolution layer.
- enchiridion 6y ago
  That's quite a bit removed. I'd be surprised if it could do much of anything. I guess by using low res data you are also shrinking the problem.
  - thedragonline 6y ago
    I personally suspect that there are surprises in store for me by rooting around in this space. I have a .caffemodel that appears to correctly label better than chance, so my next step is to create a test dataset that's independent of original and see what happens. Personally, I find it unsettling that GPT 3 now contains 175 billion parameters. Why not go in the opposite direction and test the lower bound of what is possible?

verdverm 6y ago

You probably can avoid convolution for a 3x3 grey scale. Have you run your test multiple times with varied training sets? Did it perform well on validation data or test data? Honestly, it sounds like a data set that is easily memorized

thedragonline 6y ago
>Have you run your test multiple times with varied training sets? Some background - I've been working on this for several months now, experimenting with various CNNs (settling on a modified LeNet), hyperparameters and parameters. The bulk of the experiments have been failures - a typical scenario is the loss function decreasing in the training phase, but winding up unable to correctly predict labels in the test phase. There has been a progression however - from predicting either label A or B (but not both), to predicting both (but no better than chance), to doing a little bit better than chance. Maybe I'm fooling myself - I don't know. That's why I've been scouring the Internet for similar kinds of work (and not finding anything truly useful) and now reaching out to HN. If there's work out there that definitively rules out working in this space, I'm all ears. Otherwise I'm going to keep on experimenting. (edit:formatting)