You can do greedy feature selection: make a stable train and development and test (and maybe another truer test set, for use later) sets and define an appropria

You can do greedy feature selection: make a stable train and development and test (and maybe another truer test set, for use later) sets and define an appropriate quality measure. Then implement the simplest thing you can think of, train on the training data, calibrate the results on the development set (tuning hyperparameters, etc) and look at the performance on test data. That's your baseline. Now think up of a nice new feature you'd like to add, implement it, and see if you can get the error on the development set to go down. If it goes down a lot (you can decide an appropriate threshold, maybe depending on the computational and human cost of using those features), keep it. Otherwise throw it away. Now repeat this process for every set of features you can think of, documenting the combinations you've tried as you go to see where is the best effort/performance tradeoff. Then test your best model on the test set to see if the performance has really improved. If your model is linear, you will probably see diminishing returns as you implement more, different features. It also helps to look at model errors to see which features are pulling things the wrong way, and then you can add other features to compensate. (feature engineering + linear models feels a lot like writing an old-school AI heuristic program, except you have a computer program give a numeric weight the rules you're writing, so you're free to write a lot (millions) of rules and still get a manageable model)