That's called specification-, model-, or property-based test generation. Google those terms. If older security or verification papers, always try adding "citese

That's called specification-, model-, or property-based test generation. Google those terms. If older security or verification papers, always try adding "citeseerx" and "pdf" to the search to find research publications. The papers will describe the strengths and weaknesses of those tools.