One other point not mentioned in other comments: some work was presented at GTC regarding using tensor cores for a low precision solution followed by iterative

One other point not mentioned in other comments: some work was presented at GTC regarding using tensor cores for a low precision solution followed by iterative refinement to a fp64-equivalent solution. IIRC, 2-4x speed up for fp64 dense system solvers.