3 ms·

- Submits issue to Chromium requesting it just run webpages through this before displaying them (I've stumbled on old webpages with legitimately broken encoding

- Submits issue to Chromium requesting it just run webpages through this before displaying them (I've stumbled on old webpages with legitimately broken encoding within the past 4-5 months) - Creates Rube Goldberg machine to fix UTF-8 text copy-pasted through TigerVNC (which I was surprised to discover setting LC_ALL doesn't fix) -- Fun trivia w/ VNC, because it's cute: 1. Example from site: #╨┐╤Ç╨╨╨╕╨╗╤î╨╜╨╛╨╡╨┐╨╕╤é╨╨╜╨╕╨╡ 2. Fixed: #правильноепитание 3. What I get when I copy #2 through VNC: #Ð¿ÑÐ°Ð²Ð¸Ð»ÑÐ½Ð¾ÐµÐ¿Ð¸ÑÐ°Ð½Ð¸Ðµ 4. What happens when ftfy sees #3: #правильноепитание 5. What happens when I copy #1 through VNC: #â¨ââ¤Ãâ¨ââ¨ââ¨ââ¨ââ¤Ã®â¨ââ¨ââ¨â¡â¨ââ¨ââ¤Ã©â¨ââ¨ââ¨ââ¨â¡ 6. What happens when I paste #5 to ftfy: #правильноепитание This is absolutely awesome. VNC details: Arch Linux server, Slackware client, both running TigerVNC 1.8.0. Bonus: what happens when I paste the above into Google Translate: "# Proper nutrition" (nice!) Edit: Wow, Arc didn't choke on this. It has good Unicode support. Nice.

rspeer 9y ago
But if Chromium ran ftfy on all text, then you wouldn't be able to read the examples on the page, or in your post :( (In general, I will claim that the only false positives you're likely to encounter using ftfy are test cases for ftfy!)
- exikyut 9y ago
  Very good point. This is an awesome library. My real question is... how on earth does it figure out the text is "okay", that the demangling process is done?
  - rspeer 9y ago
    Each de-mangling step decreases a "cost" metric on the text, based on its length plus the number of unusual combinations of characters. It never really decides that text is "okay", but when there's no step it can take that decreases the cost, it's done. This is an imperfect greedy strategy, incidentally. If it takes multiple steps to fix some text, it's possible that the first step it needs to take is not the one that decreases the cost as much as possible, that it has to go through some awful-looking intermediate state so that everything falls into place for the next step. This is rare, though. I don't think I could come up with an example.
    - exikyut 9y ago
      oooooh. Very interesting. And I think I understand what you mean.

rspeer 9y ago

But if Chromium ran ftfy on all text, then you wouldn't be able to read the examples on the page, or in your post :( (In general, I will claim that the only false positives you're likely to encounter using ftfy are test cases for ftfy!)

exikyut 9y ago
Very good point. This is an awesome library. My real question is... how on earth does it figure out the text is "okay", that the demangling process is done?
- rspeer 9y ago
  Each de-mangling step decreases a "cost" metric on the text, based on its length plus the number of unusual combinations of characters. It never really decides that text is "okay", but when there's no step it can take that decreases the cost, it's done. This is an imperfect greedy strategy, incidentally. If it takes multiple steps to fix some text, it's possible that the first step it needs to take is not the one that decreases the cost as much as possible, that it has to go through some awful-looking intermediate state so that everything falls into place for the next step. This is rare, though. I don't think I could come up with an example.
  - exikyut 9y ago
    oooooh. Very interesting. And I think I understand what you mean.