Would it be possible to map the first detected pose to a 3D model, scale and deform it to match the pose, and then use each next pose to manipulate the 3D model

Would it be possible to map the first detected pose to a 3D model, scale and deform it to match the pose, and then use each next pose to manipulate the 3D model (vs generating all the vertices again)? This should result in smooth animation, without artifacts, and joint limits might even help with position estimation.