Another thing you can do is try to optimize hyper parameters with techniques that have fewer or easier hyper parameters. I found several papers where people wer

Another thing you can do is try to optimize hyper parameters with techniques that have fewer or easier hyper parameters. I found several papers where people were using either simulated annealing or differential evolution to optimize the NN’s themselves or the hyperparameters. Some claimed to get good results but with higher, computational cost for that component. I think even a simple NN with few layers could probably pull it off if you already had categorized the types of data you were training the main model with.