Presumably the blasting itself could have been done by specialist free men. But the blasting itself is only a small part of the process. First you need to dril

Presumably the blasting itself could have been done by specialist free men. But the blasting itself is only a small part of the process. First you need to drill the holes by hand, that means beating a rock drill with a hammer for hours, just for a single hole, and you need dozens of holes for one blast, countless thousands if you're trying to go through a mountain. There were steam powered machines for drilling rock at this time, but they were cumbersome and temperamental, most of the drilling was still done by hand. Hence the folklore of John Henry racing a power drill to see who could work faster. I believe most of the drilling in the west was done by hand, by Chinese immigrants. Then there is the matter of the rubble. Explosives only break the rock, who hauls that broken rock? That's more manual labor.