Hardware (driver- or OS-level) debug sucks. Commodity hardware debug sucks more. When working at this level, access to hardware probes and external monitoring

Hardware (driver- or OS-level) debug sucks. Commodity hardware debug sucks more. When working at this level, access to hardware probes and external monitoring and manufacturing taps can be invaluable. Boundary and edge conditions and timing races and part steppings and errata rule. For those cases where the errata was written down, or where the vendor deigned to describe what changed between the steppings. There are a number of device-level drivers and operating systems in use where only a very few folks really know the code sufficient to debug this level, too.