You aren't prohibited from doing the calculations per se. You are prohibited from doing them for a particular purpose, specifically video playback, in this case

You aren't prohibited from doing the calculations per se. You are prohibited from doing them for a particular purpose, specifically video playback, in this case. If you just want to sit down with a pencil and paper and calculate H264 by yourself, there's nothing illegal about that. It only becomes illegal when you embed it into a video player.