When not burning sideways, you have the issue of gravity drag. Specifically, all the force spent firing downwards is wasted on fighting gravity rather than gain

When not burning sideways, you have the issue of gravity drag. Specifically, all the force spent firing downwards is wasted on fighting gravity rather than gaining speed. The reason rockets go up is to get above the atmosphere. Without an atmosphere, you'd launch rockets sideways.