Have you tried to implement one? :-) I worked with a team who did a full implementation from scratch (commercially), and I optimized some parts for certain arch

Have you tried to implement one? :-) I worked with a team who did a full implementation from scratch (commercially), and I optimized some parts for certain architectures. The problem is not in getting the output. The reference decoder decodes just fine. The difficult part is getting the output on time, e.g. getting you decoder to run fast. The reference decoder has not been designed for speed, and getting things to run fast requires architecting everything very differently: it's not just a matter of applying micro-optimizations to reference code. Also, the "intermediate results" might be different for each decoder architecture, so you can't really rely on the reference code for those. So yes, while testing is indeed easy, writing a fast decoder is definitely not a "step by step optimization" process. I'd bet every performant decoder out there was rewritten almost completely at least twice.