I am still having trouble understanding the naive recomputation problem - aren't you forced to recompute due to the nature of convolutional filters? i.e. Thinki

I am still having trouble understanding the naive recomputation problem - aren't you forced to recompute due to the nature of convolutional filters? i.e. Thinking primarily in the first layer - an NxN frame is shifted right by one pixel column. Doesn't this require recomputing the filter response? Or at least getting clever and using something akin to subtracting the "outgoing" pixel response and adding the new column's response for every filter? Or is this only applicable in the larger context when you are basically convolving a whole network inside a larger image, after training?