You don't need matrices to conceptualize these things, you need them to represent them for purposes of calculation. If you want to do calculations on linear map

You don't need matrices to conceptualize these things, you need them to represent them for purposes of calculation. If you want to do calculations on linear maps V -> W, you usually do it with matrices. And if you want to do calculations on bilinear forms on VxW, you also usually do it with matrices.