It is almost, but not quite, entirely about horizontal velocity. You also need sufficient altitude to be outside the atmosphere, as orbits with altitudes of und

It is almost, but not quite, entirely about horizontal velocity. You also need sufficient altitude to be outside the atmosphere, as orbits with altitudes of under 100km are very short lived. Additionally, it's generally not great to try to spend a lot of time accelerating through the atmosphere. For this reason rockets tend to fly upwards and then once they are out of the bulk of the atmosphere they fly mostly sideways.