Pixel averages and Porter-Duff alpha blending will always give fuzzy results. You need sub-pixel geometric info to do a more accurate depth blend. The classic

Pixel averages and Porter-Duff alpha blending will always give fuzzy results. You need sub-pixel geometric info to do a more accurate depth blend. The classic reference is the A-Buffer by Loren Carpenter (SIGGRAPH 1984). One interesting possibility is to go from a binary matte or averaged anti-aliased layer to subpixel geometry by generating a contour (e.g. Marching Squares). Then the coarse layers (e.g. green screen) can be incorporated into a high-resolution A-Buffer rendering pipeline with 3D CGI elements.