Based on my experience either Julia or Wolfram Mathematica (the command line version with fewer libraries should be free, as in beer) Or if you want to go all

Based on my experience either Julia or Wolfram Mathematica (the command line version with fewer libraries should be free, as in beer) Or if you want to go all in you can try a proof assistant like Lean which is being developed with a focus toward application in mathematics rather than CS/logic.