3 ms·

This only makes sense if your metric is number of code points. You don't actually get better density with this. Base64, encoded as either ASCII or UTF-8, stor

This only makes sense if your metric is number of code points. You don't actually get better density with this. Base64, encoded as either ASCII or UTF-8, stores 6 bits per byte; making it 75% efficient. Base65536 encoded as UTF-8 (by far the most common) will split its characters about evenly between 3-byte and 4-byte encodings (it uses code points up to 165376 roughly evenly distributed [1]). That is 16 bits per 3.5 bytes, or 57% efficient. Much worse. UTF-16 (what Javascript uses) is slightly better; you get a distribution about even between 2 and 4 bytes. That is 16 bits per 3 bytes, or 67% efficient. Better, but not as good as Base64 in ASCII or UTF-8. (However, it is better than Base64 in UTF-16, which is only 37.5% efficient.) Stick with Base64 unless you're strictly counting code points (like tweets do). [1] https://github.com/ferno/base65536/blob/master/get-block-start.json https://github.com/ferno/base65536/blob/master/get-block-sta...

asdfaoeu 11y ago
You could however used the code points in a weighted matter and get better efficiency.
- therein 11y ago
  Only if you are forced to define your "code points" as UTF-8. Maybe due to some dependencies. But it is usually the opposite, dependencies tend to limit you to ASCII if they ever.
arkadiyt 11y ago
"This only makes sense if your metric is number of code points." That is precisely the author's metric. In the "why?" section they write: "I wanted people to be able to share HATETRIS replays via Twitter. Twitter supports tweets of up to 140 characters. "
- colanderman 11y ago
  Yes, I read the article and mentioned its use case of tweets in the last sentence of my post.
deathanatos 11y ago
> This only makes sense if your metric is number of code points. Not quite! :) I wrote a similar tool I called baseunicode[1]. (Glad I'm not the only one who's discovered this!) The author here is using it for tweets (which oddly don't count bytes, they count code points); I wrote my version for copy/pasting data between terminals[2]: copying more than a screen's worth is a hassle, so "screen area" is the metric I'm trying to optimize, which ends up being subtly different from the article's, which is code points for a tweet. (for screen area in a terminal, some characters — CJK in particular — end up taking multiple columns; and combining code points (like acute accent) are basically free!) In my case, it ended up being easier to use base64, because it's installed by default. [2]: SCP is just a pain sometimes. If you need to read a file only privileged by root, AFAIK, it's out of luck. I'm usually going server-to-server, so I also need to remember to -3, which I inevitably forget. You can get around the root issue w/ `ssh source 'sudo cat /foo/bar' | ssh destination 'cat >foo', (or 'sudo bash -c "cat >foo" if you need root on the write…) but more work; often I find I'm already there in both terminals and I just want these bytes from this split in this other split, so I started doing `tar -cz <stuff> | base64`, copy, `base64 -d | tar -xz`, paste. And then screen size mattered, thus baseunicode. [1]: https://github.com/thanatos/baseunicode https://github.com/thanatos/baseunicode
- amptorn 11y ago
  If screen area is your metric, and combining code points are free, I bet you could use [1] to add an arbitrary amount of data to a single character. [1] http://www.eeemo.net/ http://www.eeemo.net/
- adimitrov 11y ago
  > which oddly don't count bytes, they count code points Not so odd, but very sensible! I'm always saddened by this oversight in SMS messages — they're 160 7bit characters, so 140 octets. But those 7bit characters are the GSM charset. If you want to write in other languages than Western European ones, your message will be encoded using UTF-16 — now you only have 70 characters. It might be OK for Chinese, but it's absolutely unusable for Cyrillic. Of course, the limiting factor in SMS is architectural, whereas in Twitter, it's philosophical, so there's your difference.

asdfaoeu 11y ago

You could however used the code points in a weighted matter and get better efficiency.

therein 11y ago
Only if you are forced to define your "code points" as UTF-8. Maybe due to some dependencies. But it is usually the opposite, dependencies tend to limit you to ASCII if they ever.

arkadiyt 11y ago

"This only makes sense if your metric is number of code points." That is precisely the author's metric. In the "why?" section they write: "I wanted people to be able to share HATETRIS replays via Twitter. Twitter supports tweets of up to 140 characters. "

colanderman 11y ago
Yes, I read the article and mentioned its use case of tweets in the last sentence of my post.

deathanatos 11y ago

> This only makes sense if your metric is number of code points. Not quite! :) I wrote a similar tool I called baseunicode[1]. (Glad I'm not the only one who's discovered this!) The author here is using it for tweets (which oddly don't count bytes, they count code points); I wrote my version for copy/pasting data between terminals[2]: copying more than a screen's worth is a hassle, so "screen area" is the metric I'm trying to optimize, which ends up being subtly different from the article's, which is code points for a tweet. (for screen area in a terminal, some characters — CJK in particular — end up taking multiple columns; and combining code points (like acute accent) are basically free!) In my case, it ended up being easier to use base64, because it's installed by default. [2]: SCP is just a pain sometimes. If you need to read a file only privileged by root, AFAIK, it's out of luck. I'm usually going server-to-server, so I also need to remember to -3, which I inevitably forget. You can get around the root issue w/ `ssh source 'sudo cat /foo/bar' | ssh destination 'cat >foo', (or 'sudo bash -c "cat >foo" if you need root on the write…) but more work; often I find I'm already there in both terminals and I just want these bytes from this split in this other split, so I started doing `tar -cz <stuff> | base64`, copy, `base64 -d | tar -xz`, paste. And then screen size mattered, thus baseunicode. [1]: https://github.com/thanatos/baseunicode https://github.com/thanatos/baseunicode

amptorn 11y ago
If screen area is your metric, and combining code points are free, I bet you could use [1] to add an arbitrary amount of data to a single character. [1] http://www.eeemo.net/ http://www.eeemo.net/
adimitrov 11y ago
> which oddly don't count bytes, they count code points Not so odd, but very sensible! I'm always saddened by this oversight in SMS messages — they're 160 7bit characters, so 140 octets. But those 7bit characters are the GSM charset. If you want to write in other languages than Western European ones, your message will be encoded using UTF-16 — now you only have 70 characters. It might be OK for Chinese, but it's absolutely unusable for Cyrillic. Of course, the limiting factor in SMS is architectural, whereas in Twitter, it's philosophical, so there's your difference.