Specifically for "out of an abundance of caution", in Google Books there begin to be usages around 1911, based on a very quick search. As noted by someone else

Specifically for "out of an abundance of caution", in Google Books there begin to be usages around 1911, based on a very quick search. As noted by someone else it's a pretty common phrase among American lawyers.