If you are doing few compute steps per memory access then you are memory bandwidth limited and >100MB is too much RAM requirement per core to not be bottlenecke

If you are doing few compute steps per memory access then you are memory bandwidth limited and >100MB is too much RAM requirement per core to not be bottlenecked by RAM access. Use a GPU or FPGA with HBM.