You could also just modulate the lambda used in the trellis quantization so that it is less aggressive in smooth blocks, and more aggressive in textured blocks.

You could also just modulate the lambda used in the trellis quantization so that it is less aggressive in smooth blocks, and more aggressive in textured blocks. It's not as good as being able to change the quantizer, but you can get somewhere around half the benefits of real activity masking by changing lambda alone.