As usual it's tricky to conclude things from Google Books ngrams due to polysemy. If you look at the actual results for "office" in e.g. 1800-1820, the majority

As usual it's tricky to conclude things from Google Books ngrams due to polysemy. If you look at the actual results for "office" in e.g. 1800-1820, the majority of them are for other words that happen to be spelled identically, not for "office" in the sense of "I work in office 802b". From a rough spot-check I'd guess only about 10% of the uses refer to offices in the sense of a place with a desk in which clerical work is done (or related senses, like "office building"). Examples of other, more common uses: Sermons on the Person and Office of the Redeemer Lectures on the Nature and End of the Sacred Office If a coroner be remiss in coming to do his office... The Office of Holy Week in Latin and English And, though it be the office of Bishops and Presbyters to instruct... ...shall not be obliged to serve the office of sheriff.