3 ms·

Are people really using models so big and complex that the parameter space couldn't fit into an on-die cache? A fairly simple 8MB cache can give you 1,000,000 d

Are people really using models so big and complex that the parameter space couldn't fit into an on-die cache? A fairly simple 8MB cache can give you 1,000,000 doubles for your parameter space, and it would allow you to get rid of an entire DRAM interface. It's a serious question, as I've never done any real deep learning...but coming from a world where I once scoffed at a random forest model with 80 parameters, it just seems absurd.

mattnewton 10y ago
Yes. Each layer can have millions of parameters if your data set is large enough. Convolutional networks easily get up there, especially if you add a third dimension that the network can travel across (either space in 3D covnets for medical scans, or time for videos in some experimental archetecture). Say you want to look at a heart in a 3D covnet, that could easily be 512x512x512 for the input alone. In fully connected models, for training efficiency, many features are implemented as one-hot encoded parameters, which turns a single caragory like "state" into 50 parameters. I think there is some active research into sparse representations of this with the same efficiency but I've never seen those techniques, just people piling on more parameters.
sounds 10y ago
I think something that "goes without saying" is that their first-rev design has some essential simplicity to it: - No control flow instructions (though apparently some operations can have a repeat count) - Fundamentally simple architecture This allows them to get through validation and tapeout very quickly.
deepnotderp 10y ago
hahhahahahhaahah The SOTA networks are around 300MB+...
- saosebastiao 10y ago
  Not sure if you meant to laugh at a serious question. I am fully aware of my ignorance of the space. Since it appears you're in the deep learning hardware business, what would be the impediment to using eDRAM or similar? eDRAM is too costly at those sizes for general purpose processors, but I imagine the reduced latency and increased bandwidth would be a huge win for a ridiculously parallel deep learning processor, and would definitely be a tradeoff worth making.
  - deepnotderp 10y ago
    Sorry, that was more of a laugh at the state of deep learning model sizes than anything. Okay, so about eDRAM. There are two types of eDRAM: On-die and on-package. On-die eDRAM refers to manufacturing of DRAM cells on the logic die, which would be a big boon in terms of density since eDRAM cells can be almost 3x as dense as SRAM. The problem however, is that on-chip eDRAM has been impossible to scale beyond 40nm, which mitigates any advantages you would receive from using eDRAM. On-package eDRAM is more interesting but the primary cost in memory access is the physical transportation of the data, which is a physical limit and can't be circumvented. You can call it all sorts of fancy names such as "eDRAM", but the fact of the matter is that you're still moving data. For reference, the projected cost of movement of a 64-bit word on 10nm (ON CHIP) according to Lawrence Livermore national laboratories is ~1pJ, whereas the cost of a 64-bit FLOP is estimated to be 1pJ also. As you can see, the cost of data movement dwarfs the cost of computation. Of course, you gain a lot compared to DRAM, but HBM can offer the same efficiency gains of course. Didn't meant to be rude with the first response. Let me know if you have any other questions, I'd be happy to answer them :)
vincentchu 10y ago
The latest deep learning models are indeed quite large. For comparison, Inception clocks in at "only" 5M parameters, itself a 12x reduction over AlexNet (60M) and VGGNet (180M)! (source: https://arxiv.org/abs/1512.00567 https://arxiv.org/abs/1512.00567) A further point is that even if the model has relatively few parameters, there are advantages to having more memory--- namely, you can do inference on larger batch sizes in one go.
dnautics 10y ago
if you'd like a good answer to that question drop a line.

mattnewton 10y ago

Yes. Each layer can have millions of parameters if your data set is large enough. Convolutional networks easily get up there, especially if you add a third dimension that the network can travel across (either space in 3D covnets for medical scans, or time for videos in some experimental archetecture). Say you want to look at a heart in a 3D covnet, that could easily be 512x512x512 for the input alone. In fully connected models, for training efficiency, many features are implemented as one-hot encoded parameters, which turns a single caragory like "state" into 50 parameters. I think there is some active research into sparse representations of this with the same efficiency but I've never seen those techniques, just people piling on more parameters.

sounds 10y ago

I think something that "goes without saying" is that their first-rev design has some essential simplicity to it: - No control flow instructions (though apparently some operations can have a repeat count) - Fundamentally simple architecture This allows them to get through validation and tapeout very quickly.

deepnotderp 10y ago

hahhahahahhaahah The SOTA networks are around 300MB+...

saosebastiao 10y ago
Not sure if you meant to laugh at a serious question. I am fully aware of my ignorance of the space. Since it appears you're in the deep learning hardware business, what would be the impediment to using eDRAM or similar? eDRAM is too costly at those sizes for general purpose processors, but I imagine the reduced latency and increased bandwidth would be a huge win for a ridiculously parallel deep learning processor, and would definitely be a tradeoff worth making.
- deepnotderp 10y ago
  Sorry, that was more of a laugh at the state of deep learning model sizes than anything. Okay, so about eDRAM. There are two types of eDRAM: On-die and on-package. On-die eDRAM refers to manufacturing of DRAM cells on the logic die, which would be a big boon in terms of density since eDRAM cells can be almost 3x as dense as SRAM. The problem however, is that on-chip eDRAM has been impossible to scale beyond 40nm, which mitigates any advantages you would receive from using eDRAM. On-package eDRAM is more interesting but the primary cost in memory access is the physical transportation of the data, which is a physical limit and can't be circumvented. You can call it all sorts of fancy names such as "eDRAM", but the fact of the matter is that you're still moving data. For reference, the projected cost of movement of a 64-bit word on 10nm (ON CHIP) according to Lawrence Livermore national laboratories is ~1pJ, whereas the cost of a 64-bit FLOP is estimated to be 1pJ also. As you can see, the cost of data movement dwarfs the cost of computation. Of course, you gain a lot compared to DRAM, but HBM can offer the same efficiency gains of course. Didn't meant to be rude with the first response. Let me know if you have any other questions, I'd be happy to answer them :)

vincentchu 10y ago

The latest deep learning models are indeed quite large. For comparison, Inception clocks in at "only" 5M parameters, itself a 12x reduction over AlexNet (60M) and VGGNet (180M)! (source: https://arxiv.org/abs/1512.00567 https://arxiv.org/abs/1512.00567) A further point is that even if the model has relatively few parameters, there are advantages to having more memory--- namely, you can do inference on larger batch sizes in one go.

dnautics 10y ago

if you'd like a good answer to that question drop a line.