The OpenAI approach is more amenable to an obvious HW implementation with the block sparsity because the blocks are are GEMM operations are implemented in the f

The OpenAI approach is more amenable to an obvious HW implementation with the block sparsity because the blocks are are GEMM operations are implemented in the first place. There are obviously more available sparse solutions if the block sparsity constraint is relaxed therefore I wouldn't be surprised if the best results come from such a network.