5 ms·

A 5 layer CNN is absurdly shallow, so this isn't particularly surprising. I routinely work with 150+ layer CNNs - that's fairly standard practice if you want hi

by highd 9y ago

A 5 layer CNN is absurdly shallow, so this isn't particularly surprising. I routinely work with 150+ layer CNNs - that's fairly standard practice if you want high-quality results.

iandanforth 9y ago
They only generated 2000 images to work with so I'm not sure any cnn can be expected to do terribly well.
- highd 9y ago
  True - it would be interesting to use, say, the coding layer of an autoencoder trained on one of the many face datasets and then training only on the last few layers to fit their data.
halflings 9y ago
Since when are 150+ layer CNNs fairly standard? Inception-v3 has about 50 layers, and that's considered a lot (requires considerable processing power to train).
- highd 9y ago
  I would say since ResNet beat state of the art on ImageNet in 2015. It's not standard, but that's what state-of-art can take.
forgotpw1123 9y ago
This comment is absurd. "Your CNN sucks, I know this because I work with better ones all the time. (insert metric that doesn't mean much)" Oh, BTW I do ML consulting lol
- zo7 9y ago
  VGG has 16-19 layers, Inception has ~50 layers, and ResNet has 150 layers. All of which were the state of the art at one point in time over the last ~2-3 years. A more faithful comparison with CNNs would've used one of these models pre-trained on a much larger face dataset. Oh, BTW I do ML consulting lol
  - forgotpw1123 9y ago
    I don't think it's anywhere near conclusive yet that more layers = better. It's pretty telling that the current state-of-the-art is combining a bunch of layers together in a pseudo random fashion. Nobody understands how these things work to the point that we can make a formula or equation to produce better CNN's, or even predict which models will be more effective to any accuracy. You think more layers is better, because the best models we have happen to have the most layers? Some deep understanding of concepts there.
    - zo7 9y ago
      I don't think I or the parent comment are necessarily suggesting that more layers are better, but are pointing out that the fact that they're only using 5 layers suggests that they're not using a state of the art architecture. You can't faithfully say "oh a CNN cannot model this relationship" when it wasn't a thorough evaluation. (especially given that they don't mention modern face recognition systems like DeepFace or FaceNet, which I'd be interested to see if there's any correlation between the embeddings they produce if a simple PCA model works so well) Also don't be so dismissive, we have a strong enough empirical and intuitive understanding of CNNs that we're able to make thoughtful improvements over time. In fact the insight behind the ResNet paper was noticing that adding layers doesn't improve performance and that training error actually degrades as layers are added – the solution to this was to construct the network so that it learns residual mappings that only modify the input rather than completely transform it. The whole point of that paper was solving this degradation problem so they could use some ridiculously deep architecture like a 150-layer network to get better results.
    - nl 9y ago
      the current state-of-the-art is combining a bunch of layers together in a pseudo random fashion Please don't say things like this. Neural network is nothing like a random process - people understand very well when to use what kind of layer, and when to add more layers. It's generally pretty well accepted that more layers = better because it gives you better ability to deal with multi-dimensional relationships between features. There are two reasons not to add layers: 1) If you are overfitting. In visual tasks this is fairly rare, and there are other better ways to combat this. 2) It's harder to train. Until ResNet came out, the idea of training 100+ layers was considered unapproachable. Even now, more layers make a much harder training task.
- TTPrograms 9y ago
  It's just not a fair comparison. No reason to get in a tiff.
- highd 9y ago
  Please don't be a jerk on HN. 5 layer CNNs are not best practice anymore, and it's not a fair comparison. Your flippant attitude is unwelcome.
nl 9y ago
5 layers is AlexNet, which is a decent starting point for visual tasks.

by highd 9y ago

A 5 layer CNN is absurdly shallow, so this isn't particularly surprising. I routinely work with 150+ layer CNNs - that's fairly standard practice if you want high-quality results.

iandanforth 9y ago

They only generated 2000 images to work with so I'm not sure any cnn can be expected to do terribly well.

highd 9y ago
True - it would be interesting to use, say, the coding layer of an autoencoder trained on one of the many face datasets and then training only on the last few layers to fit their data.

halflings 9y ago

Since when are 150+ layer CNNs fairly standard? Inception-v3 has about 50 layers, and that's considered a lot (requires considerable processing power to train).

highd 9y ago
I would say since ResNet beat state of the art on ImageNet in 2015. It's not standard, but that's what state-of-art can take.

forgotpw1123 9y ago

This comment is absurd. "Your CNN sucks, I know this because I work with better ones all the time. (insert metric that doesn't mean much)" Oh, BTW I do ML consulting lol

zo7 9y ago
VGG has 16-19 layers, Inception has ~50 layers, and ResNet has 150 layers. All of which were the state of the art at one point in time over the last ~2-3 years. A more faithful comparison with CNNs would've used one of these models pre-trained on a much larger face dataset. Oh, BTW I do ML consulting lol
- forgotpw1123 9y ago
  I don't think it's anywhere near conclusive yet that more layers = better. It's pretty telling that the current state-of-the-art is combining a bunch of layers together in a pseudo random fashion. Nobody understands how these things work to the point that we can make a formula or equation to produce better CNN's, or even predict which models will be more effective to any accuracy. You think more layers is better, because the best models we have happen to have the most layers? Some deep understanding of concepts there.
  - zo7 9y ago
    I don't think I or the parent comment are necessarily suggesting that more layers are better, but are pointing out that the fact that they're only using 5 layers suggests that they're not using a state of the art architecture. You can't faithfully say "oh a CNN cannot model this relationship" when it wasn't a thorough evaluation. (especially given that they don't mention modern face recognition systems like DeepFace or FaceNet, which I'd be interested to see if there's any correlation between the embeddings they produce if a simple PCA model works so well) Also don't be so dismissive, we have a strong enough empirical and intuitive understanding of CNNs that we're able to make thoughtful improvements over time. In fact the insight behind the ResNet paper was noticing that adding layers doesn't improve performance and that training error actually degrades as layers are added – the solution to this was to construct the network so that it learns residual mappings that only modify the input rather than completely transform it. The whole point of that paper was solving this degradation problem so they could use some ridiculously deep architecture like a 150-layer network to get better results.
  - nl 9y ago
    the current state-of-the-art is combining a bunch of layers together in a pseudo random fashion Please don't say things like this. Neural network is nothing like a random process - people understand very well when to use what kind of layer, and when to add more layers. It's generally pretty well accepted that more layers = better because it gives you better ability to deal with multi-dimensional relationships between features. There are two reasons not to add layers: 1) If you are overfitting. In visual tasks this is fairly rare, and there are other better ways to combat this. 2) It's harder to train. Until ResNet came out, the idea of training 100+ layers was considered unapproachable. Even now, more layers make a much harder training task.
TTPrograms 9y ago
It's just not a fair comparison. No reason to get in a tiff.
highd 9y ago
Please don't be a jerk on HN. 5 layer CNNs are not best practice anymore, and it's not a fair comparison. Your flippant attitude is unwelcome.

nl 9y ago

5 layers is AlexNet, which is a decent starting point for visual tasks.