3 ms·

> but in ANSI newline delimiter is translated as "j" ?

ralferoo 4mo ago
\n is ^J (ctrl-J) so there's some logic there, but I wonder if something has been lost in the message. I'd guess it either displays ^J or an inverse-colour J, rather than just a plain lowercase j. Also possible that the j is a red herring and just some random character that's always there. Pasting a URL containing a newline into most browsers just truncates it at the newline, regardless of how much text is after. I only know this from occasionally copying links from a terminal window where the copy somehow added newlines every 80 characters (even though copying this way normally works fine). I'd have to copy the URL with newlines into a text editor, remove the new lines and copy again to be able to paste it.
- thaumasiotes 4mo ago
  > \n is ^J (ctrl-J) so there's some logic there Specifically, J is the 10th letter of the alphabet and therefore ctrl-J is code for ascii 10. Same reason ctrl-D sends EOF and ctrl-I sends tab.
  - mike_hock 4mo ago
    Yes, but piping output containing newlines into wl-copy does not result in j's in the clipboard.
- randallsquared 4mo ago
  Some tool or library is interpreting the newline as two characters (as you note), and then a subsequent step is removing unprintable characters. Things like this used to frequently happen in shells, Perl, PHP, and so on.
- rgoulter 4mo ago
  > \n is ^J (ctrl-J) so there's some logic there The same 'j' as vi uses for 'hjkl'. https://vi.stackexchange.com/questions/42426/why-did-vi-use-hjkl-instead-of-jikl/42428#42428 https://vi.stackexchange.com/questions/42426/why-did-vi-use-...
  - microgpt 4mo ago
    also the same 'j' found in words like 'jujuism', 'jejunities', and 'bejeezus', also by a magical coincidence the same one in most Latin fonts, and even some random text strings such as 'pj$4' But I suppose you're saying ASCII 10 was chosen as newly because it aligns with the down arrow on keyboards of the time. Maybe.
    - rgoulter 4mo ago
      > I suppose you're saying ASCII 10 was chosen as newly because it aligns with the down arrow on keyboards of the time. Maybe. The linked StackExchange has it as: > What character was used for what control code was mostly a matter of bitwise arithmetics. LF is ^J because J happens to be at the corresponding location in the corresponding column of the table (+ 64 in decimal)
      - microgpt 4mo ago
        and did they choose that code because the J key had a down arrow printed on it?
        ahazred8ta 4mo ago
        No, the Baudot / Western Union mechanical teletype code assignments were worked out in the late 1800s and early 1900s. This was later developed into ASCII. Computer keyboards came much later. https://en.wikipedia.org/wiki/Baudot_code https://en.wikipedia.org/wiki/Baudot_code
    - kps 4mo ago
      Other way around. ASCII classified control characters into blocks and newline ended up a bit-flip away from J, so the ADM-3A printed ‘↓’ on the ‘J’ key.
      - Joker_vD 4mo ago
        Yes, and ADM-3A also had "↑" on the "K" key, because \013 is vertical tabulation (VT), and "→" on the "L" because \014 is form feed (FF). It all makes perfect sense. If terminfo database is to be believed, quite a lot of other terminals reused this control sequences too, for some reason.
gucci-on-fleek 4mo ago
This comes up fairly often in TeX, where you can use ^^J to insert a newline character [0]. For example, the following code: \message{before ^^J after} prints the following message: before after This is common in other old software too [1] [2], but TeX is where I see it the most often these days. [0]: https://tex.stackexchange.com/a/64848/270600 https://tex.stackexchange.com/a/64848/270600 [1]: https://superuser.com/q/212874 https://superuser.com/q/212874 [2]: https://en.wikipedia.org/wiki/Caret_notation https://en.wikipedia.org/wiki/Caret_notation
raldi 4mo ago
Yeah, I don't understand this. What broken tool is turning newlines into j?
fsckboy 4mo ago
the ANSI escape sequences for screen erasing start with ESC and end in J. When you display a string on a screen including a newline at the end of it, you need to erase to the end of the line on the screen so you don't leave predecessor garbage which would be confusing. That's how the J got inserted, then whatever is processing this is swallowing the rest of the escape sequence but leaving the J. The fix is there, take the J with the rest of the escape sequence at whatever layer is doing this. ESC[J erase in display (same as ESC[0J) ESC[0J erase from cursor until end of screen ESC[1J erase from cursor to beginning of screen ESC[2J erase entire screen ESC[3J erase saved lines ASCII and then ANSI were invented as in-band communications and terminal control protocols, not "file formats". You thread filters together like beads on a string, where ASCII is somebody's job, and ANSI is somebody's job, and then the rendered text is presented to the user. unix is ingenious how it has accomodated changes and conflicting ideas, incorporating the new squoze next to the old, like UTF-8 because the same people did that. It's not perfect, but it proved "worse is better" because it for a long time it outcompeted its alternatives and never became bloated and overburdened by bureaucratic committee ideas. however today's developers did not "come up" in the same culture of apprenticeship learning so a lot of the clever subtlety has been lost and now worse is starting to be worse, and guess what, there is no better.
- mike_hock 3mo ago
  Interesting hypothesis. What makes it somewhat implausible is the fact that he got lower-case j's. Also, echo '{"foo":"bar"}' | jq -r '.foo' | xxd -p gives me 6261720a (i.e. bar\n), so I don't quite see how he could have gotten j's (or J's for that matter) injected into that pipeline. Maybe with some weird/broken locale settings?
zahlman 4mo ago
I thought it was a strange way to express the problem, too. OP didn't really explain where or how the clipboard text was getting pasted in order to reproduce the issue.

\n is ^J (ctrl-J) so there's some logic there, but I wonder if something has been lost in the message. I'd guess it either displays ^J or an inverse-colour J, rather than just a plain lowercase j. Also possible that the j is a red herring and just some random character that's always there. Pasting a URL containing a newline into most browsers just truncates it at the newline, regardless of how much text is after. I only know this from occasionally copying links from a terminal window where the copy somehow added newlines every 80 characters (even though copying this way normally works fine). I'd have to copy the URL with newlines into a text editor, remove the new lines and copy again to be able to paste it.

thaumasiotes 4mo ago
> \n is ^J (ctrl-J) so there's some logic there Specifically, J is the 10th letter of the alphabet and therefore ctrl-J is code for ascii 10. Same reason ctrl-D sends EOF and ctrl-I sends tab.
- mike_hock 4mo ago
  Yes, but piping output containing newlines into wl-copy does not result in j's in the clipboard.
randallsquared 4mo ago
Some tool or library is interpreting the newline as two characters (as you note), and then a subsequent step is removing unprintable characters. Things like this used to frequently happen in shells, Perl, PHP, and so on.
rgoulter 4mo ago
> \n is ^J (ctrl-J) so there's some logic there The same 'j' as vi uses for 'hjkl'. https://vi.stackexchange.com/questions/42426/why-did-vi-use-hjkl-instead-of-jikl/42428#42428 https://vi.stackexchange.com/questions/42426/why-did-vi-use-...
- microgpt 4mo ago
  also the same 'j' found in words like 'jujuism', 'jejunities', and 'bejeezus', also by a magical coincidence the same one in most Latin fonts, and even some random text strings such as 'pj$4' But I suppose you're saying ASCII 10 was chosen as newly because it aligns with the down arrow on keyboards of the time. Maybe.
  - rgoulter 4mo ago
    > I suppose you're saying ASCII 10 was chosen as newly because it aligns with the down arrow on keyboards of the time. Maybe. The linked StackExchange has it as: > What character was used for what control code was mostly a matter of bitwise arithmetics. LF is ^J because J happens to be at the corresponding location in the corresponding column of the table (+ 64 in decimal)
    - microgpt 4mo ago
      and did they choose that code because the J key had a down arrow printed on it?
      - ahazred8ta 4mo ago
        No, the Baudot / Western Union mechanical teletype code assignments were worked out in the late 1800s and early 1900s. This was later developed into ASCII. Computer keyboards came much later. https://en.wikipedia.org/wiki/Baudot_code https://en.wikipedia.org/wiki/Baudot_code
  - kps 4mo ago
    Other way around. ASCII classified control characters into blocks and newline ended up a bit-flip away from J, so the ADM-3A printed ‘↓’ on the ‘J’ key.
    - Joker_vD 4mo ago
      Yes, and ADM-3A also had "↑" on the "K" key, because \013 is vertical tabulation (VT), and "→" on the "L" because \014 is form feed (FF). It all makes perfect sense. If terminfo database is to be believed, quite a lot of other terminals reused this control sequences too, for some reason.

gucci-on-fleek 4mo ago

This comes up fairly often in TeX, where you can use ^^J to insert a newline character [0]. For example, the following code: \message{before ^^J after} prints the following message: before after This is common in other old software too [1] [2], but TeX is where I see it the most often these days. [0]: https://tex.stackexchange.com/a/64848/270600 https://tex.stackexchange.com/a/64848/270600 [1]: https://superuser.com/q/212874 https://superuser.com/q/212874 [2]: https://en.wikipedia.org/wiki/Caret_notation https://en.wikipedia.org/wiki/Caret_notation

raldi 4mo ago

Yeah, I don't understand this. What broken tool is turning newlines into j?

fsckboy 4mo ago

the ANSI escape sequences for screen erasing start with ESC and end in J. When you display a string on a screen including a newline at the end of it, you need to erase to the end of the line on the screen so you don't leave predecessor garbage which would be confusing. That's how the J got inserted, then whatever is processing this is swallowing the rest of the escape sequence but leaving the J. The fix is there, take the J with the rest of the escape sequence at whatever layer is doing this. ESC[J erase in display (same as ESC[0J) ESC[0J erase from cursor until end of screen ESC[1J erase from cursor to beginning of screen ESC[2J erase entire screen ESC[3J erase saved lines ASCII and then ANSI were invented as in-band communications and terminal control protocols, not "file formats". You thread filters together like beads on a string, where ASCII is somebody's job, and ANSI is somebody's job, and then the rendered text is presented to the user. unix is ingenious how it has accomodated changes and conflicting ideas, incorporating the new squoze next to the old, like UTF-8 because the same people did that. It's not perfect, but it proved "worse is better" because it for a long time it outcompeted its alternatives and never became bloated and overburdened by bureaucratic committee ideas. however today's developers did not "come up" in the same culture of apprenticeship learning so a lot of the clever subtlety has been lost and now worse is starting to be worse, and guess what, there is no better.

mike_hock 3mo ago
Interesting hypothesis. What makes it somewhat implausible is the fact that he got lower-case j's. Also, echo '{"foo":"bar"}' | jq -r '.foo' | xxd -p gives me 6261720a (i.e. bar\n), so I don't quite see how he could have gotten j's (or J's for that matter) injected into that pipeline. Maybe with some weird/broken locale settings?

zahlman 4mo ago

I thought it was a strange way to express the problem, too. OP didn't really explain where or how the clipboard text was getting pasted in order to reproduce the issue.