5 ms·

I'm not too surprised by this, but I'm annoyed that no amount of configuration made those bytes visible again in my editor. Only using hexdump revealed them.

by remram 2y ago

bittercynic 2y ago
My mind went the same place. Anyone know a more convenient way to search larger blocks of text for this?
jrootabega 2y ago
Here's a POC that works in emacs. Doesn't cover all of the relevant characters, but: (setq ;;some other invisible or interesting characters unicode-zero-width-space ?\u200b unicode-zero-width-non-joiner ?\u200c unicode-zero-width-joiner ?\u200d unicode-zero-width-nbsp ?\ufeff unicode-narrow-nbsp ?\u202f unicode-word-joiner ?\u2060 unicode-grapheme-joiner ?\u034f unicode-no-break-space ?\u00a0 unicode-combining-long-stroke ?\u0336 ;;variation selector examples unicode-vs-fe00 ?\ufe00 unicode-vs-fe0f ?\ufe0f unicode-vs-e0100 ?\xe0100) (defun show-glyphless-as-hex (char) (let ((original (elt glyphless-char-display char))) (aset glyphless-char-display char 'hex-code) original)) ;;so you can see what you just replaced (progn (show-glyphless-as-hex unicode-zero-width-space) (show-glyphless-as-hex unicode-zero-width-non-joiner) (show-glyphless-as-hex unicode-zero-width-joiner) (show-glyphless-as-hex unicode-zero-width-nbsp) (show-glyphless-as-hex unicode-word-joiner) (show-glyphless-as-hex unicode-grapheme-joiner) (show-glyphless-as-hex unicode-narrow-nbsp) (show-glyphless-as-hex unicode-no-break-space) ;;these may already be visible if the current conditions don't support them ;;but we'll force them (show-glyphless-as-hex unicode-vs-fe00) (show-glyphless-as-hex unicode-vs-fe0f) (show-glyphless-as-hex unicode-vs-e0100))
- remram 2y ago
  I use vim. It seems like `:set binary enc=latin1` works, though I don't understand why the latin1 part is required.
- TacticalCoder 2y ago
  [dead]
- jrootabega 2y ago
  Here is the bare minimum this is built on, which you can type in yourself if you're paranoid or want to start from the bottom up. Swap in the hexadecimal codepoint of the invisible character after the ?\x (aset glyphless-char-display ?\xfe00 'hex-code)
- jrootabega 2y ago
  And as a higher-level configuration you can set most, maybe even all, of the relevant invisible characters (still not sure how 0x34f grapheme joiner fits in) at once with something like: (custom-set-variables '(glyphless-char-display-control '((format-control . hex-code) (variation-selectors . hex-code)))) This will modify values in glyphless-char-display, but it's OK to modify those directly if you need to.
mdouglass 2y ago
vscode's "Unicode Highlight: Non-basic ASCII" causes the character to get highlighted. Sadly, the more appropriate "Unicode Highlight: Invisible Characters" setting does not reveal them.

I'm not too surprised by this, but I'm annoyed that no amount of configuration made those bytes visible again in my editor. Only using hexdump revealed them.

by remram 2y ago

I'm not too surprised by this, but I'm annoyed that no amount of configuration made those bytes visible again in my editor. Only using hexdump revealed them.

bittercynic 2y ago

My mind went the same place. Anyone know a more convenient way to search larger blocks of text for this?

jrootabega 2y ago

Here's a POC that works in emacs. Doesn't cover all of the relevant characters, but: (setq ;;some other invisible or interesting characters unicode-zero-width-space ?\u200b unicode-zero-width-non-joiner ?\u200c unicode-zero-width-joiner ?\u200d unicode-zero-width-nbsp ?\ufeff unicode-narrow-nbsp ?\u202f unicode-word-joiner ?\u2060 unicode-grapheme-joiner ?\u034f unicode-no-break-space ?\u00a0 unicode-combining-long-stroke ?\u0336 ;;variation selector examples unicode-vs-fe00 ?\ufe00 unicode-vs-fe0f ?\ufe0f unicode-vs-e0100 ?\xe0100) (defun show-glyphless-as-hex (char) (let ((original (elt glyphless-char-display char))) (aset glyphless-char-display char 'hex-code) original)) ;;so you can see what you just replaced (progn (show-glyphless-as-hex unicode-zero-width-space) (show-glyphless-as-hex unicode-zero-width-non-joiner) (show-glyphless-as-hex unicode-zero-width-joiner) (show-glyphless-as-hex unicode-zero-width-nbsp) (show-glyphless-as-hex unicode-word-joiner) (show-glyphless-as-hex unicode-grapheme-joiner) (show-glyphless-as-hex unicode-narrow-nbsp) (show-glyphless-as-hex unicode-no-break-space) ;;these may already be visible if the current conditions don't support them ;;but we'll force them (show-glyphless-as-hex unicode-vs-fe00) (show-glyphless-as-hex unicode-vs-fe0f) (show-glyphless-as-hex unicode-vs-e0100))

remram 2y ago
I use vim. It seems like `:set binary enc=latin1` works, though I don't understand why the latin1 part is required.
TacticalCoder 2y ago
[dead]
jrootabega 2y ago
Here is the bare minimum this is built on, which you can type in yourself if you're paranoid or want to start from the bottom up. Swap in the hexadecimal codepoint of the invisible character after the ?\x (aset glyphless-char-display ?\xfe00 'hex-code)
jrootabega 2y ago
And as a higher-level configuration you can set most, maybe even all, of the relevant invisible characters (still not sure how 0x34f grapheme joiner fits in) at once with something like: (custom-set-variables '(glyphless-char-display-control '((format-control . hex-code) (variation-selectors . hex-code)))) This will modify values in glyphless-char-display, but it's OK to modify those directly if you need to.

mdouglass 2y ago

vscode's "Unicode Highlight: Non-basic ASCII" causes the character to get highlighted. Sadly, the more appropriate "Unicode Highlight: Invisible Characters" setting does not reveal them.