3 ms·

How much longer do they have to be trained (the article talked about this but didn't give numbers)? Do they need more data than neural nets with FP ops? Any est

by ktta 9y ago

How much longer do they have to be trained (the article talked about this but didn't give numbers)? Do they need more data than neural nets with FP ops? Any estimates? One thing that looks attractive to me is that FPGAs can be used very effectively for this purpose because these will take up much less LUTs and nets for specific purpose can be manually programmed in HDL.

regularfry 9y ago
You don't even need FPGA's for this sort of network, necessarily. Take a look at how WISARD worked: https://scholar.google.co.uk/scholar?q=igor%20Aleksander%20WISARD https://scholar.google.co.uk/scholar?q=igor%20Aleksander%20W... It's "just" a bunch of RAM.
- ktta 9y ago
  Correct me if I'm wrong, but that looks like it needs specialized hardware. I'm thinking about how FPGAs with now being available on AWS can be used for neural networks. They are also much better than GPUs for in-house deployment because they consume a lot less power which in turn reduces cooling power. They also don't get 'refreshed' every year which can be a good or bad thing because you don't have to upgrade so frequently and possibly lag being on performance compared to rest of the industry. FPGAs have often been acknowledged as being a low power solution for the same precision but they're tremendously difficult to program. Also when you want FP operations on FPGA you're using up a LOT of chip space, because they were meant for GPUs. So this kind of light weight neural nets will be much more suited for FPGAs. I've tried some basic nets but they are better for GPUs since you can fit larger nets without problem and RAM is another problem with FPGAs. I'm thinking that these nets would be perfect since they take up much less space on the FPGA and consume a lot less RAM. (The lower RAM consumption is talked about here[1]) Also, the simple construction of the net with basic operations would help the programming difficulty. [1]: https://youtu.be/Q17HwA5oY4w?t=1m35s https://youtu.be/Q17HwA5oY4w?t=1m35s
  - regularfry 9y ago
    The original implementation used custom hardware, but the original implementation was in 1984, so I think that's forgivable. I don't think the custom hardware is necessary, though. Fundamentally the concept is that you replace a thumping great matrix operation with a set of lookup tables in memory. FPGA's are fun, I'm just not sure you need them for this. RAM is plenty fast.
deepnotderp 9y ago
It's not just being trained longer, that in itself isn't too much of a worry, but rather the issue is that they aren't very good even after retraining, object detection accuracy is essentially the cost of extreme quantization. For example, see Deephi's presentation at HotChips.

by ktta 9y ago

regularfry 9y ago

You don't even need FPGA's for this sort of network, necessarily. Take a look at how WISARD worked: https://scholar.google.co.uk/scholar?q=igor%20Aleksander%20WISARD https://scholar.google.co.uk/scholar?q=igor%20Aleksander%20W... It's "just" a bunch of RAM.

ktta 9y ago
Correct me if I'm wrong, but that looks like it needs specialized hardware. I'm thinking about how FPGAs with now being available on AWS can be used for neural networks. They are also much better than GPUs for in-house deployment because they consume a lot less power which in turn reduces cooling power. They also don't get 'refreshed' every year which can be a good or bad thing because you don't have to upgrade so frequently and possibly lag being on performance compared to rest of the industry. FPGAs have often been acknowledged as being a low power solution for the same precision but they're tremendously difficult to program. Also when you want FP operations on FPGA you're using up a LOT of chip space, because they were meant for GPUs. So this kind of light weight neural nets will be much more suited for FPGAs. I've tried some basic nets but they are better for GPUs since you can fit larger nets without problem and RAM is another problem with FPGAs. I'm thinking that these nets would be perfect since they take up much less space on the FPGA and consume a lot less RAM. (The lower RAM consumption is talked about here[1]) Also, the simple construction of the net with basic operations would help the programming difficulty. [1]: https://youtu.be/Q17HwA5oY4w?t=1m35s https://youtu.be/Q17HwA5oY4w?t=1m35s
- regularfry 9y ago
  The original implementation used custom hardware, but the original implementation was in 1984, so I think that's forgivable. I don't think the custom hardware is necessary, though. Fundamentally the concept is that you replace a thumping great matrix operation with a set of lookup tables in memory. FPGA's are fun, I'm just not sure you need them for this. RAM is plenty fast.

deepnotderp 9y ago

It's not just being trained longer, that in itself isn't too much of a worry, but rather the issue is that they aren't very good even after retraining, object detection accuracy is essentially the cost of extreme quantization. For example, see Deephi's presentation at HotChips.