3 ms·

Sure, you can maintain state in a texture and iteratively re-apply the shader to it. The problem with HAM is that it potentially requires many iterations. Consi

Sure, you can maintain state in a texture and iteratively re-apply the shader to it. The problem with HAM is that it potentially requires many iterations. Consider the worst case scenario of an input resulting in a single set color operation followed by width x height-1 modify operations. Not too bad for a simple stateful loop on the CPU, always keeping the last color in a register, but with a shader re-applied once per frame to 320x200 pixels you'd be waiting for ~20 minutes for the complete result at 60 fps. Then there's the problem of selecting a useful base palette, if you want consistently decent results regardless of the input. That's maybe the kind of optimization problem that OpenCL can be applied to. So I don't mean to say unsuitable as in impossible, just in the sense that it's exactly the kind of problem that the shader model wasn't designed to solve.

dahart 9y ago
I don't disagree with the idea that ShaderToy wasn't designed for this, or with the idea that it'd be harder to do in a fragment shader than a CPU program. But fragment shaders weren't designed for most of the things you see on ShaderToy. ;) The question is whether it was possible in real-time, and I still think it is. I'm not very interested in the worst case, I think a HAM image with width * height modify ops is very unlikely. I might even be surprised if such a HAM image has even ever existed, outside of perhaps a contrived programmer's example. The average case for a direct HAM implementation would probably be more like hundreds of frames, or a couple of seconds, not tens of thousands. If a modified HAM is on the table, the worst and average cases could be improved considerably. I modified Floyd-Steinberg (which is worst case) so that it runs in more like 10-20 frames, by making it radially symmetric and limiting the propagation distance, which results in negligible differences but vastly improved run times. The palette optimization is also possible in a fragment shader, not quite as easily as OpenCL, but possible. If the question were about transcoding a video stream in real time, that will be harder to do real time. If the optimized palette is fixed, it will be easier. Either way, I wouldn't rule it out. But I would agree it's "hacking" in the sense of abusing things they weren't designed for.
- boomlinde 9y ago
  > I'm not very interested in the worst case, I think a HAM image with width * height modify ops is very unlikely. Sure, but maybe not as far off in some real world cases. Say, black-but-not-perfectly black letter box bars where the nuances could result in long sequences of small alterations to the base black color. This becomes less likely with an optimized palette, and of course you could set a hard limit to length of modify sequences in conversion, though that would have to compromise the quality of conversion. > The average case for a direct HAM implementation would probably be more like hundreds of frames, or a couple of seconds, not tens of thousands. Then the problem, with respect to the original question, is the definition of real-time. For the purpose of a game like No Man's Sky for example I'd think hundreds or even tens of frames would be unacceptable. Don't get me wrong, I love technology applied to the "wrong problem" and see that as an end in itself sometimes :). I'm just considering the feasibility of accurate HAM in realtime using shaders, and am really enjoying the discussion on what compromises have to be made for it to perform adequately in a shader. But with respect to the original question I can't quite bring myself to say "yes"
  - dahart 9y ago
    > Then the problem, with respect to the original question, is the definition of real-time. Yeah, true. But there are two factors here that still meet a reasonable definition of real-time, which is >= 30fps in my book. First, a predictable number of frames of latency still allows 30fps throughput. Second, this encoding can be done in multi-pass batches in a single frame. The dependency of one sweep of modify ops doesn't need to wait for a full frame to do the next sweep. This is very common on ShaderToy where people build dependent buffer chains. It's perfectly reasonable (outside ShaderToy) to do 10 sweeps in a one frame cycle. I don't know what the limit would be on a standard GPU these days, it might be much higher. Consider that shaders on ShaderToy are routinely doing several hundred tap texture samples per pixel, and/or several hundred Ray marching iterations per pixel. The amount of work the GPU can do now in real time is incredible. You can combine the inter-frame sweeps with cross-frame passes, and you only lose latency but not throughput. There's a likely chance you can also unroll once or twice for a single pass. If you unrolled once, and had 10 passes inter-frame, and had a 10 frame latency, you could in theory get a modify radius of 200 pixels, while maintaining 30fps throughput with 10 frames of latency, assuming I haven't taxed out the GPU at this point. So, you're right, it would be silly & hard to do this, but I still think HAM encoding could probably be done in real time, with perhaps some compromises or quality limitations to keep the encoding predictable, or at least have a known reasonable upper bound.

dahart 9y ago

I don't disagree with the idea that ShaderToy wasn't designed for this, or with the idea that it'd be harder to do in a fragment shader than a CPU program. But fragment shaders weren't designed for most of the things you see on ShaderToy. ;) The question is whether it was possible in real-time, and I still think it is. I'm not very interested in the worst case, I think a HAM image with width * height modify ops is very unlikely. I might even be surprised if such a HAM image has even ever existed, outside of perhaps a contrived programmer's example. The average case for a direct HAM implementation would probably be more like hundreds of frames, or a couple of seconds, not tens of thousands. If a modified HAM is on the table, the worst and average cases could be improved considerably. I modified Floyd-Steinberg (which is worst case) so that it runs in more like 10-20 frames, by making it radially symmetric and limiting the propagation distance, which results in negligible differences but vastly improved run times. The palette optimization is also possible in a fragment shader, not quite as easily as OpenCL, but possible. If the question were about transcoding a video stream in real time, that will be harder to do real time. If the optimized palette is fixed, it will be easier. Either way, I wouldn't rule it out. But I would agree it's "hacking" in the sense of abusing things they weren't designed for.

boomlinde 9y ago
> I'm not very interested in the worst case, I think a HAM image with width * height modify ops is very unlikely. Sure, but maybe not as far off in some real world cases. Say, black-but-not-perfectly black letter box bars where the nuances could result in long sequences of small alterations to the base black color. This becomes less likely with an optimized palette, and of course you could set a hard limit to length of modify sequences in conversion, though that would have to compromise the quality of conversion. > The average case for a direct HAM implementation would probably be more like hundreds of frames, or a couple of seconds, not tens of thousands. Then the problem, with respect to the original question, is the definition of real-time. For the purpose of a game like No Man's Sky for example I'd think hundreds or even tens of frames would be unacceptable. Don't get me wrong, I love technology applied to the "wrong problem" and see that as an end in itself sometimes :). I'm just considering the feasibility of accurate HAM in realtime using shaders, and am really enjoying the discussion on what compromises have to be made for it to perform adequately in a shader. But with respect to the original question I can't quite bring myself to say "yes"
- dahart 9y ago
  > Then the problem, with respect to the original question, is the definition of real-time. Yeah, true. But there are two factors here that still meet a reasonable definition of real-time, which is >= 30fps in my book. First, a predictable number of frames of latency still allows 30fps throughput. Second, this encoding can be done in multi-pass batches in a single frame. The dependency of one sweep of modify ops doesn't need to wait for a full frame to do the next sweep. This is very common on ShaderToy where people build dependent buffer chains. It's perfectly reasonable (outside ShaderToy) to do 10 sweeps in a one frame cycle. I don't know what the limit would be on a standard GPU these days, it might be much higher. Consider that shaders on ShaderToy are routinely doing several hundred tap texture samples per pixel, and/or several hundred Ray marching iterations per pixel. The amount of work the GPU can do now in real time is incredible. You can combine the inter-frame sweeps with cross-frame passes, and you only lose latency but not throughput. There's a likely chance you can also unroll once or twice for a single pass. If you unrolled once, and had 10 passes inter-frame, and had a 10 frame latency, you could in theory get a modify radius of 200 pixels, while maintaining 30fps throughput with 10 frames of latency, assuming I haven't taxed out the GPU at this point. So, you're right, it would be silly & hard to do this, but I still think HAM encoding could probably be done in real time, with perhaps some compromises or quality limitations to keep the encoding predictable, or at least have a known reasonable upper bound.