I've found myself in all three situations: where performance didn't matter much and something like JSON or XML was fine, where performance mattered and a binary

I've found myself in all three situations: where performance didn't matter much and something like JSON or XML was fine, where performance mattered and a binary format was an obvious win, and one situation where serialized data needed to be as compact as possible and I could only achieve that by hand coding. To me, formats like JSON or XML have one clear advantage: debugging is easier. I can fire up a decoder for a binary format, but that slows down the investigation. That said, most binary formats have a schema file due to necessity of not serializing field names. That schema file makes it easy to have tools that spit out language bindings. And that reduces the tedium of actually writing code that uses (and validates) the data, so I generally prefer those formats for that reason.