4 ms·

FPGA developer here. In a former project (with a custom board and FPGA, no processor), we encountered random bugs. We put many checks and finally came to the c

FPGA developer here. In a former project (with a custom board and FPGA, no processor), we encountered random bugs. We put many checks and finally came to the conclusion that our DDR modules were flipping bits randomly, but only on normal load. All test benches were running fine. Putting the modules in a PC did not show any problem. How can you tell your boss "all DDR modules are faulty but run fine in a PC" and not seem crazy? It was when I read about rowhammer attacks that I made the link. Changing the addressing schema completely solved the issue. All these hardware related failures/attacks may not be a threat (yet), but for me, the underlying sand castle we are building things on is very worrying.

CamperBob2 8y ago
Sure you didn't just have a timing problem, such as the all-too-common "Hmm, that wasn't a false path after all" variety?
- leghifla 8y ago
  When I read about rowhammer, I did analyse carefully the memory request pattern and found many similarities to a rowhammer attack on PC. It was several processes issuing small requests on alternating rows, with slowly moving indexes. After I implemented the change and had a stable system, I went as far as implementing a switch of addressing pattern so that, all other things being equal, I could trigger the bug or not. And it worked as expected. So yes, i am very confident in the analysis.
SlowRobotAhead 8y ago
>Changing the addressing schema completely solved the issue. Can you elaborate at all? Do you mean you changing the FPGA to not write to exactly the same spots over and over? And just set up something up similar to wear leveling?
- leghifla 8y ago
  It was very simple, I kept all the "logical" addresses the same, but swapped some low address bits with high address bits at the memory controller interface. With this, the slowly moving indexes became big jumps. This avoids reading the same row over and over again (mixed with other requests).
  - avianes 8y ago
    > This avoids reading the same row over and over again Didn't you notice a huge drop in performance producing row-buffer conflicts over and over ?
    - leghifla 8y ago
      Good question. Jumping around the memory is clearly not good for throughput. Before the change, the mixing of requests between processes was already detrimental for performance anyway. All I can say is that we needed a fixed bandwidth and it was enough, before or after the change.

CamperBob2 8y ago

Sure you didn't just have a timing problem, such as the all-too-common "Hmm, that wasn't a false path after all" variety?

leghifla 8y ago
When I read about rowhammer, I did analyse carefully the memory request pattern and found many similarities to a rowhammer attack on PC. It was several processes issuing small requests on alternating rows, with slowly moving indexes. After I implemented the change and had a stable system, I went as far as implementing a switch of addressing pattern so that, all other things being equal, I could trigger the bug or not. And it worked as expected. So yes, i am very confident in the analysis.

SlowRobotAhead 8y ago

>Changing the addressing schema completely solved the issue. Can you elaborate at all? Do you mean you changing the FPGA to not write to exactly the same spots over and over? And just set up something up similar to wear leveling?

leghifla 8y ago
It was very simple, I kept all the "logical" addresses the same, but swapped some low address bits with high address bits at the memory controller interface. With this, the slowly moving indexes became big jumps. This avoids reading the same row over and over again (mixed with other requests).
- avianes 8y ago
  > This avoids reading the same row over and over again Didn't you notice a huge drop in performance producing row-buffer conflicts over and over ?
  - leghifla 8y ago
    Good question. Jumping around the memory is clearly not good for throughput. Before the change, the mixing of requests between processes was already detrimental for performance anyway. All I can say is that we needed a fixed bandwidth and it was enough, before or after the change.