The obvious thing to do would be to store the image in memory as just an array of 8-bit of native unsigned integers (RGBARGBARGBARGBA...) and see where that got

The obvious thing to do would be to store the image in memory as just an array of 8-bit of native unsigned integers (RGBARGBARGBARGBA...) and see where that got you. I assume C# has the equivalent of a Java byte[], except that fortunately it is unsigned in C# instead of signed in Java. I expect that you would get quite a performance boost with just that.