Ask HN: Dumb Machine Learning question – how to get insights, not predictions?

I've been trying to dig into ML and figure out the answer to this question, but so far I'm repeatedly and enthusiastically hitting brick walls. So I thought I'd ask the assembled brains here. Say I have a large corpus of data on, say, esport teams' makeup and performance, along with whether they won or not. (Or alternatively, visitors to a website and whether or not they converted, or whatever.) I've got about 10-20 different fields describing each team - what heroes they picked, what they did in the game, etc - and then a binary win/loss. I know how to train an ML model to predict whether a team will win based on that data. But how do I use an ML model to extract insights about what makes a team more likely to win? Stuff like "if 3 people picked these 3 heroes and then 1 person went to this location on the map, there's a 75% chance that the team wins"? I can find data visualisers allowing humans to pick through data and look for trends, but I can't find a way to get a machine to do it. And I'm sure I'm missing something really obvious here. So - is this doable with ML techniques, and if so, how would you do it?