While I don't think readability is a problem if you are judicious with your use of reader macros, the real problem is that they don't play well with editing too

While I don't think readability is a problem if you are judicious with your use of reader macros, the real problem is that they don't play well with editing tools e.g. all those nice Emacs shortcuts like forward-sexp, backward-sexp, etc. won't work correctly with any JSON that you write in your Lisp files. Forget JSON, the problems are evident even when using something like CL-INTERPOL, which makes minimal changes to the read-table.