Horses definitely got much bigger but they've been big enough to ride since they were domesticated. The small Mongolian pony is still 5-600 pounds. Plenty big

Horses definitely got much bigger but they've been big enough to ride since they were domesticated. The small Mongolian pony is still 5-600 pounds. Plenty big enough to carry ~200 pounds of people + arms.