3 ms·
82% accuracy on CIFAR-10? Is this a joke? State of the art on CIFAR-10 is 98.5% accuracy. That chart showing them near the top of a top-1 accuracy graph is bord
by modeless 8y ago
82% accuracy on CIFAR-10? Is this a joke? State of the art on CIFAR-10 is 98.5% accuracy. That chart showing them near the top of a top-1 accuracy graph is borderline fraudulent.
Why not compare apples to apples? Take a CNN architecture that gets >95% accuracy, shrink it down until it only gets 82% accuracy, then run it on commercially available non-spiking NN hardware like a Movidius Myriad or Apple's Neural Engine and measure the speed and power consumption.
- stochastic_monk 8y agoThe chart is only covering so-called VPUs. This architecture claims to be best in class, but it's definitely unfair to omit how well state of the art models perform on standard hardware (IE, [GCT]PUs).
- modeless 8y agoTX2 is on the chart and it is a GPU. But reading between the lines, the TX2 numbers on the chart are for ImageNet, not CIFAR-10. The TX2 is classifying 224x224 images into 1000 classes. Brainchip is classifying 32x32 images into 10 classes. But accuracy and power consumption are directly compared on the chart almost as if the tasks are similar. That's why I say it's borderline fraudulent.
- stochastic_monk 8y agoThank you for clarifying. I only looked through several of the chips before assuming they were all the same. I agree, and I don't see what case would lead me to want to be stuck with one particular model for one kind of data, especially of the error rate is dozens of times higher.
- chadmeister 8y agoOh wow that is a very important detail I had overlooked as well. Looking at them now the comparisons seem rather blatantly misleading.
- alfalfasprout 8y agoI agree there's something very fishy here. Some of the points on that chart list the architectures, others don't. For instance, Movidius Myriad VPU 2 can run a variety of different architectures. Also... LeNet isn't THAT bad on CIFAR-10. It can easily reach 75% after 200 epochs. I think these specialized chips can be useful in mobile or low power applications, but for servers the flexibility of a GPU or reprogrammable FPGA is worthwhile. Advancements in ML are frequent and nobody wants to be stuck with hardware that can't run a newer architecture.