6 ms·

Not exactly a fair comparison - the design is limited to run models that can fit entirely in on chip RAM for the FPGA. This greatly reduces power consumption be

by bunnie 2y ago

Not exactly a fair comparison - the design is limited to run models that can fit entirely in on chip RAM for the FPGA. This greatly reduces power consumption because the FPGA does not have to pay the overhead of DRAM PHY's, termination, DRAM chips, etc which is a relatively fixed cost irrespective of capacity. This means the energy cost in terms of energy per bit transfered is much higher for the GPU than the FPGA. Thus the GPU is storing a 110M model in gigabytes of external RAM, and paying the power penalty associated with the excess capacity, while the chosen model 110M fits neatly within the FPGA's on chip RAM, and the design can trim all that overhead accordingly. A more fair comparison would either run a larger model that had both systems hitting external RAM, or they would compare power/performance against some sort of inference ASIC that had all the RAM on chip (maybe a cerebrus, but scaled according to the portion of the wafer actually used for the model). That being said, it's neat that they open sourced their work, and it's worth looking down this path more.

dhruvdh 2y ago
I don't know what you are trying to say here. If one system doesn't need to move as much data because it is more flexible, that is a good thing. What do we gain by making it "fair"?
- alpacalaca 2y ago
  If you're limiting the size of the model to 110 million parameters (105MiB assuming int8) because that's what will fit onto your FPGA then of course it's going to be more energy efficient than a Broadwell era Xeon with a 24GB RTX 3090. It's like concluding that a rickshaw is more efficient than a train, something that will absolutely be true in a technical sense if you're only transporting a single passenger, but makes no sense if you're transporting hundreds if not thousands of passengers. A more apt comparison would have been with a phone made in the past 5 years, even without an AI accelerator chip I'm sure you could manage 20-30+ t/s from a 110m model but this depends entirely on the memory bandwidth of the phone.

by bunnie 2y ago

dhruvdh 2y ago

I don't know what you are trying to say here. If one system doesn't need to move as much data because it is more flexible, that is a good thing. What do we gain by making it "fair"?

alpacalaca 2y ago
If you're limiting the size of the model to 110 million parameters (105MiB assuming int8) because that's what will fit onto your FPGA then of course it's going to be more energy efficient than a Broadwell era Xeon with a 24GB RTX 3090. It's like concluding that a rickshaw is more efficient than a train, something that will absolutely be true in a technical sense if you're only transporting a single passenger, but makes no sense if you're transporting hundreds if not thousands of passengers. A more apt comparison would have been with a phone made in the past 5 years, even without an AI accelerator chip I'm sure you could manage 20-30+ t/s from a 110m model but this depends entirely on the memory bandwidth of the phone.