Could one benefit from it by replacing ReLU with sine activation functions in all DNN (conv.) architectures? In this paper they only show that it works better i

Could one benefit from it by replacing ReLU with sine activation functions in all DNN (conv.) architectures? In this paper they only show that it works better in MLPs based (non convolutional) deep neural architectures, if I understood it correctly? Traditional ReLU-CNNs do perform better than Sine-MLPs in one of the tasks(Supp. Table 4).