Honestly I think I'd use latex (but I'm not recommending from self-publishing experience) - as arcane & annoying as it can be... I think trying to do the typese

Honestly I think I'd use latex (but I'm not recommending from self-publishing experience) - as arcane & annoying as it can be... I think trying to do the typesetting/formatting side of it with markdown or similar would be even more annoying? You can do everything you want (processor-dependent, but e.g. mermaid for diagrams is often supported) in markdown but when you want to make it look pretty as a book you're going to need something like CSS, some way of controlling fonts and image/diagram positioning, etc.