I'd say in this context distinction makes difference. You also would not say "I am going back to work" when you were sales person visiting customer, you would

I'd say in this context distinction makes difference. You also would not say "I am going back to work" when you were sales person visiting customer, you would say "I am going back to office". You would say "I am going back to work" when you would be on a lunch break as much as you would say "I am going back to office". In work-from-home situation I see only "I am going back to office" as correct statement. With all implications of "going back to work" pointing out you are not "working" at moment you are talking.