3 ms·

Agreed. The user of "cat" is responsible if they've created a stream that is invalid input to some other program. Unix philosophy is for programs to do (ideal

Agreed. The user of "cat" is responsible if they've created a stream that is invalid input to some other program. Unix philosophy is for programs to do (ideally) one simple thing well. If the contents of any one file cannot be used directly, then the idea would be to run some other filter program first (e.g. one that strips bytes from the beginning) prior to using "cat" and piping the results somewhere else. In practice though some programs and protocols have text/binary distinction (e.g. FTP). It's not unreasonable to have a mode that hints when bytes are to be used as text. This is frequently done to handle different new-line styles for instance.

ajross 14y ago
...and you think this is an argument in favor of including a BOM?
- makecheck 14y ago
  I prefer to solve the problem in the right place. Raw concatenation of bytes without encoding-awareness introduces the possibility that the bytes will combine in unexpected ways. The presence of something as obvious as a BOM makes it harder to make this mistake, at least during the transition phase to "UTF-8 everywhere". What you really want in this situation is something that forces you to see the potential bug and introduce the correct translation and/or text-concatenation tools to fix it. And yes, at some point in the future enough tools will be truly aware of UTF-8 that the BOM will not have a reason to exist. But right now it has some value.
  - tripzilch 14y ago
    So, first you argue that UTF-8 without BOM might split glyphs at incorrect places, and then it's pointed out that UTF-8 with BOM concatenates files incorrectly, and somehow this is also an argument for using BOM? I'm sorry but it's obvious both approaches have rather symmetrical opportunities to produce bugs. So in that case the best choice seems to be the simplest one. Which is the one without BOM.

ajross 14y ago

...and you think this is an argument in favor of including a BOM?

makecheck 14y ago
I prefer to solve the problem in the right place. Raw concatenation of bytes without encoding-awareness introduces the possibility that the bytes will combine in unexpected ways. The presence of something as obvious as a BOM makes it harder to make this mistake, at least during the transition phase to "UTF-8 everywhere". What you really want in this situation is something that forces you to see the potential bug and introduce the correct translation and/or text-concatenation tools to fix it. And yes, at some point in the future enough tools will be truly aware of UTF-8 that the BOM will not have a reason to exist. But right now it has some value.
- tripzilch 14y ago
  So, first you argue that UTF-8 without BOM might split glyphs at incorrect places, and then it's pointed out that UTF-8 with BOM concatenates files incorrectly, and somehow this is also an argument for using BOM? I'm sorry but it's obvious both approaches have rather symmetrical opportunities to produce bugs. So in that case the best choice seems to be the simplest one. Which is the one without BOM.