The fundamental problem here is that the Nyquist frequency of displays is well below the human visual limit. Furthermore, the display doesn't even attempt to r

The fundamental problem here is that the Nyquist frequency of displays is well below the human visual limit. Furthermore, the display doesn't even attempt to reconstruct a band-limited signal like an audio DAC would: it simply maps samples to pixels. This direct mapping lets you render images with much higher frequency components than you'd otherwise be able to, but as soon as you do limit the bandwidth of the signal (like the DCT is doing here) you get ringing. Of course, this ringing is totally natural, and if the display operated "properly" from a signal processing perspective (like an audio DAC), you'd see it everywhere: a single-pixel line would be below the band limit! To really solve the problem, you have to stop treating pixels as samples, since they're not displayed that way. Instead, you should upsample the image to a higher resolution using a kernel which matches how a display renders pixels, perform your signal processing on this higher-resolution image, then downsample using the same kernel.