3 ms·

What is the advantage of using this instead of OverFeat for natural images? I have been operating on the assumption that the strange license of OverFeat is not

What is the advantage of using this instead of OverFeat for natural images? I have been operating on the assumption that the strange license of OverFeat is not a problem, since you are using someone's software to feed in input, and generate a bunch of floating point values. Are the floating point outputs of a program then "tainted" by the license of the code as well? If not, why not use the state of the art, which handles both localization AND classification? I can see using ccv for custom datasets, though I am also assuming the 6GB TITAN requirement for GPU use is only for ImageNet, and not for a custom dataset? TITAN GPUs are pretty expensive... I am very impressed by the result, as I have not been able to even approach OverFeat with my own work (pylearn2 + theano) thus far. Will definitely be experimenting with this in the future. Thanks!

liuliu 13y ago
From my understanding, you cannot deploy OverFeat in production because its license is only for research and evaluation purposes. You still need TITAN for any reasonable custom datasets (with non-trivial data). The current CPU implementation doesn't support Dropout, therefore, you can only play it with CIFAR-10 (./bin/cifar-10.c) dataset if you don't have a GPU. This is an preliminary implementation, I do plan to finish up CPU training part to be on-par with GPU in subsequent releases.
- kastnerkyle 13y ago
  So you are storing the entire dataset on the GPU then, to speed up processing? Or is there support for a "minibatch" mode that sends chunks at a time to be processed? Unless this more an issue with the model size of the "Krizhevsky net" being on the order of 6GB? Also, does this mean you managed to get a 2D convolutional kernel optimized for Kepler architectures? If so, that is awesome! Alex's code is still only optimized for Fermi architectures if I recall correctly.
  - liuliu 13y ago
    Not really the whole dataset. For ImageNet, I have mini_batch size of 256, and I need to allocate the whole network on GPU for this mini_batch (which is 256 * the neurons in the network), and plus parameters (parameters is about 200MiB * 3 for updates and momentum). Also to speed up certain operations, the data need to be reshaped, and there are 500MiB scratch space just for that purpose. In total, I am using close to 6GiB GPU memory. You probably can get to only use 4GiB memory if the batch size is 128. The code is never optimized to the extreme. I optimized the code to the point of being able to finish at reasonable time (9 days for 100 epochs). The convolutional kernel is parametrized (with template and some macro tricks, forward and backward propagation convolutional kernels are parametrized into hundreds of small functions) and the best parameters were chose with a mini benchmark in the beginning of training process.
    - kastnerkyle 13y ago
      So... duh. You would need > 1.2 TB to fit all of ImageNet on the card :). Thanks for clarifying, and pardon my brain lapse! Also, thanks for putting this out there - if I get some time I may send some pull requests your way. Awesome stuff.
      - liuliu 13y ago
        You need 1.2TB SSD for sure to train the complete ImageNet dataset. The data is loaded into GPU memory only one batch (256 images) at a time. But the loading part will be the bottleneck if you use a rotational disk.

liuliu 13y ago

From my understanding, you cannot deploy OverFeat in production because its license is only for research and evaluation purposes. You still need TITAN for any reasonable custom datasets (with non-trivial data). The current CPU implementation doesn't support Dropout, therefore, you can only play it with CIFAR-10 (./bin/cifar-10.c) dataset if you don't have a GPU. This is an preliminary implementation, I do plan to finish up CPU training part to be on-par with GPU in subsequent releases.

kastnerkyle 13y ago
So you are storing the entire dataset on the GPU then, to speed up processing? Or is there support for a "minibatch" mode that sends chunks at a time to be processed? Unless this more an issue with the model size of the "Krizhevsky net" being on the order of 6GB? Also, does this mean you managed to get a 2D convolutional kernel optimized for Kepler architectures? If so, that is awesome! Alex's code is still only optimized for Fermi architectures if I recall correctly.
- liuliu 13y ago
  Not really the whole dataset. For ImageNet, I have mini_batch size of 256, and I need to allocate the whole network on GPU for this mini_batch (which is 256 * the neurons in the network), and plus parameters (parameters is about 200MiB * 3 for updates and momentum). Also to speed up certain operations, the data need to be reshaped, and there are 500MiB scratch space just for that purpose. In total, I am using close to 6GiB GPU memory. You probably can get to only use 4GiB memory if the batch size is 128. The code is never optimized to the extreme. I optimized the code to the point of being able to finish at reasonable time (9 days for 100 epochs). The convolutional kernel is parametrized (with template and some macro tricks, forward and backward propagation convolutional kernels are parametrized into hundreds of small functions) and the best parameters were chose with a mini benchmark in the beginning of training process.
  - kastnerkyle 13y ago
    So... duh. You would need > 1.2 TB to fit all of ImageNet on the card :). Thanks for clarifying, and pardon my brain lapse! Also, thanks for putting this out there - if I get some time I may send some pull requests your way. Awesome stuff.
    - liuliu 13y ago
      You need 1.2TB SSD for sure to train the complete ImageNet dataset. The data is loaded into GPU memory only one batch (256 images) at a time. But the loading part will be the bottleneck if you use a rotational disk.