4 ms·

The mark isn't useless; it clearly identifies files as UTF-8 so they can be processed as such immediately. Otherwise a program has to "sniff" several bytes to

• 4ad 14y ago
  cat a b c There you go, a BOM in the middle of the file.
  • adobriyan 14y ago
    Unicode-compatible cat(1) would strip BOM from all but the first file. Applying non-Unicode compatible utility to Unicode files of course doesn't work.
    • 4ad 14y ago
      Cat doesn't know and doesn't need to know what kind of input it gets, it's just for concatenatig files. Files themselves can be binary for all that matters (indeed, it was used very often to concatenate tape devices to be used by tar further down in the pipe).
      • makecheck 14y ago
        Agreed. The user of "cat" is responsible if they've created a stream that is invalid input to some other program. Unix philosophy is for programs to do (ideally) one simple thing well. If the contents of any one file cannot be used directly, then the idea would be to run some other filter program first (e.g. one that strips bytes from the beginning) prior to using "cat" and piping the results somewhere else. In practice though some programs and protocols have text/binary distinction (e.g. FTP). It's not unreasonable to have a mode that hints when bytes are to be used as text. This is frequently done to handle different new-line styles for instance.
        • ajross 14y ago
          ...and you think this is an argument in favor of including a BOM?
          • makecheck 14y ago
            I prefer to solve the problem in the right place. Raw concatenation of bytes without encoding-awareness introduces the possibility that the bytes will combine in unexpected ways. The presence of something as obvious as a BOM makes it harder to make this mistake, at least during the transition phase to "UTF-8 everywhere". What you really want in this situation is something that forces you to see the potential bug and introduce the correct translation and/or text-concatenation tools to fix it. And yes, at some point in the future enough tools will be truly aware of UTF-8 that the BOM will not have a reason to exist. But right now it has some value.
            • tripzilch 14y ago
              So, first you argue that UTF-8 without BOM might split glyphs at incorrect places, and then it's pointed out that UTF-8 with BOM concatenates files incorrectly, and somehow this is also an argument for using BOM? I'm sorry but it's obvious both approaches have rather symmetrical opportunities to produce bugs. So in that case the best choice seems to be the simplest one. Which is the one without BOM.
    • kragen 14y ago
      Applying non-Unicode-compatible utilities to UTF-8 files works fine; that's why UTF-8 was invented. Adding a byte-order mark to a UTF-8 file is what causes non-Unicode-compatible utilities to stop working.
  • riffraff 14y ago
    if the BOM appears in the middle of the file than it should be considered a zero width non breaking space (so basically ignored) so why is this an issue?
    • deleted 14y ago
      [deleted]
    • eridius 14y ago
      Only if it's treated as human-readable text. But a great many text files are not, in fact, purely meant as human-readable text.
    • luriel 14y ago
      Here is an issue (that has bitten me and took lots of blood and sweat to find out): cat a b c | grep '^foo'
• bobbydavid 14y ago
  > Q: What do you think about BOMs? > A: Another reason not to use UTF-16. UTF-8 has a BOM too, even though byte order is not an issue in this encoding. This is to manifest that this is a UTF-8 stream. If UTF-8 remains the only popular encoding (as it already is in the internet world), the BOM becomes redundant. In practice, many UTF-8 text files omit BOMs today. That's one of the central points of the article --- "sniffing" is completely unnecessary if you can assume a UTF-8 encoding type. You're right that the mark isn't useless, we want it to be useless.
• unconed 14y ago
  This argument is silly. Why not prefix every UTF-8 string with a BOM then? It's wasteful and unnecessary, because UTF-8's clean structure already makes it trivial to detect, and false positives are all but impossible for real-world text. There's a paper out there that proves this. The UTF-8 BOM was a Microsoft invention. Nobody else uses it, and it breaks tons of things. Two examples off the top of my head: Unix hashbang scripts (i.e. #!/bin/bash), and PHP scripts (the BOM will trigger HTTP header finalization before any code is run).
  • makecheck 14y ago
    You wouldn't prefix every string with it because presumably your API or program's state has already determined the string's encoding. I am not suggesting that every fragment of text has to be explicit (I agree that would be ridiculous). I am only stating facts: there is nothing incorrect about having the mark, a conformant reader must be able to handle the mark, and the mark has some value as a short-cut for avoiding elaborate decoding tricks.
    • unconed 14y ago
      The thing is, the BOM is metadata, it doesn't belong in content. It violates the contract of .txt files, which is: the entire file is a single string of content. Recognizing it at the edges of your program and stripping it out is not the end of the world, but it's annoying and no other (8-bit) encoding works that way. In fact, I find it hard to believe UTF-8 BOMs in MS programs were anything more than a programmer error. Once such files were out in the wild, everyone else had to deal with them.
      • makecheck 14y ago
        There are already plenty of cases that valid UTF-8 readers have to deal with (unused ranges of code points, invalid byte combinations, etc.). Ignoring a BOM is trivial by comparison. A UTF-8 reader honestly doesn't care about the "stringness" of a .txt file because of all the other crap that can be in a byte stream. Older programs do care, but as I've said elsewhere in the thread an ASCII file can remain ASCII (no BOM). There's no reason to BOM-ify an old ASCII file if it really is ASCII and only ASCII-expecting programs will ever use it. Over time these old programs will either be upgraded or go away and it will finally be safe to say that inputs must be UTF-8. At that time, the BOM has no reason to exist.
• jbert 14y ago
  The BOM is in-band signalling. It breaks the ASCII-backwards-compatability of UTF8. BOMs are only necessary where the provenance of data is not known. Normally, there is a context provided which will determine the encoding. Basically, yes - text should be tagged (unless 'utf8 everywhere' wins) but imho the tagging should be external to the content.
  • makecheck 14y ago
    I agree with the use of context, yes; if you already know your input is UTF-8 (e.g. C strings in a program API or a protocol or whatever) there's no point in adding an extra specifier. If a program requires ASCII compatibility in order to work then by all means make the input files ASCII (no BOM), just make sure the files have no true UTF-8 dependencies in them. Once a program supports UTF-8 "properly" however, the BOM is useful as a signal that the input is somewhat complicated. At some point in the future when UTF-8 really is everywhere and programs may no longer even try to sniff encodings, etc. then yes, the BOM has no real reason to exist.
    • jbert 14y ago
      But you have the same problem with text or binary distinctions. On some platforms, no distinction between them needs to be made (quite usefully, adding to tool simplicity and conceptual simplicity). If you use BOM, a tool which can operate on text or binary must be told which it is operating on. This would have to be done via some external context (e.g. cmdline switch). And that would never go away, even in a utf8 everywhere world. Basically, in BOM-world: - You either need to tag each fragment of text or you still need to use external context (e.g. which encoding do I get text columns from my database) - You perpetually need to differentiate between binary and text data for all tools which do nothing more complicated than read and write and in non-BOM-world you: - add to the contextual clues you need anyway something like "any files which you are going to interpret as text on the system should be interpreted as utf8" - when moving data on or off the local the local system, use a network protocol which supports tagging the text payload (e.g. email, http). The problems arise mostly with file shares (or their equivalent, version control systems) where text files are exchanged without an accompanying protocol. That is where "BOM world" or "UTF8 world" will ultimately have to settle their differences. BOM-world would like all systems, everywhere, to make a text/binary distinction for ever. UTF8-world would like to say that textual data lacking a context should be interpreted as UTF8. But feel free to use UTF16/UTF32 for specific purposes or systems.
• ajross 14y ago
  What on earth does "processed as such" mean? UTF-8 can be "processed" anywhere ASCII can, that's the whole point. The only point to the BOM is to distinguish it from UTF-16 (or UCS2, which is usually what UTF-16 degenerates into). And UTF-16 is broken garbage and shouldn't be used. Given that, your last sentence is basically an ode to complexity. Taken to the logical conclusion you'd support any rule, no matter how ridiculous, as long as it is agreed upon as "correct" by ... someone. But I don't agree: the BOM hurts, it doesn't help.
  • makecheck 14y ago
    No, UTF-8 cannot be processed in exactly the same way as ASCII! It is highly compatible with ASCII-assuming environments because it can accept the same data (e.g. it remains a stream of bytes that doesn't have to be converted to some "fat" integer) and any UTF-8 text that just happens to contain only ASCII characters will work just fine with an old ASCII program. But complex multi-byte UTF-8 inputs will not work without special treatment. Historically code pages mapped character values directly to individual bytes: typically redefining the upper half (129+) while leaving the lower numbers the same as ASCII. Programs could display a wide variety of text encodings correctly as long as they knew what the encoding was, and all they had to do was read bytes individually. These encodings could all be handled in a way largely similar to ASCII. UTF-8 however is a multi-byte encoding, which means you could have (say) 3 bytes that combine to form a symbol. Not only that, but in certain forms multiple symbols could imply display as a single glyph (e.g. an accent followed by a letter). A program that does everything the old ASCII way would choke on multi-byte UTF-8, despite being otherwise-compatible. Consider a program with a fixed-size read buffer; if the last byte is only partway through a multi-byte character, that character will be mishandled unless the program knows how to preserve those bytes and "complete" the character when more bytes arrive.
    • ajross 14y ago
      You're thinking from the point of view of a display program that needs to split strings into glyphs. That's a fine application, but very rare. And yes, it's inherently encoding dependent and tends to like wide characters instead of multibyte ones. But introducing a BOM for the sake of that application is a disaster, because it hurts everything else. You can (literally) feed UTF-8 to parsers written 30 years ago and apply all your existing intuition about string handling in C without worry. Unless you deliberately break it by including a binary, non encoding garbage furball at the front of your "file" (and good luck figuring out what a "file" should mean in a OS metaphor designed around streams).
• gwillen 14y ago
  The BOM is not legal UTF-8. Do not put a BOM in your UTF-8 documents. (Accept a BOM in other people's UTF-8 documents, because Postel's Law, and because other people are dumb. But don't perpetuate it...)
• lambda 14y ago
  A lot of software predates UTF-8. For example, scripts on Unix-like (Unix, Linux, BSD) systems. In order to determine that a file should be executed using an interpreter, they look for the ASCII bytes "#!" at the beginning of the file. If those bytes are not present (such as if the file is UTF-8 with a BOM), then it won't be executed with an appropriate interpreter. Now, once the interpreter is found, it can interpret the rest of the text as it sees fit (for instance, interpreting it as proper UTF-8). But the first two characters of the file must be "#!". Furthermore, the BOM is simply a bad idea. It was poorly conceived from the beginning. It means something different at the beginning of a file than it does in the middle (at the beginning, it is the BOM; in the middle, it is a zero-width non-breaking space. This means that you can't simply append files or strings that begin with the BOM, and have the result still be valid. The BOM should not be used for sniffing files, except as a fallback. Sniffing is a terrible way to tell what encoding a file is in. The format should be recorded somewhere (in file metadata, by default, by the locale, or something of the sort). Sniffing should only be used as a last resort.
  • fleitz 14y ago
    "Sniffing is a terrible way to tell what encoding a file is in." In theory yes, in practice it's an awesome way to find out because the sniffer is more accurate then the webserver/mail client,human etc. Just about every piece of software I've written that has to deal with multiple encodings uses a sniffer, even if the format specifies it it's a good idea to check with a sniffer.
• yaix 14y ago
  Just assume that it is UTF-8, no sniffing needed. If it's not, then convert it to UTF-8.
• ori_b 14y ago
  What would you do differently if the mark was there versus if it wasn't? ASCII is a perfect subset of UTF-8, so any operations you would do on UTF-8 are also operations you would do on ASCII. The behavior of your program wouldn't change. The BOM is a no-op when it comes to how your program handles things. However, it has the potential of confusing older programs. It makes things that should be simple (like 'cat') need to be encoding aware, and modal. It means that streaming multiple files one after the other breaks. So, in summary, the BOM doesn't buy you anything, or give you any important information about a file. It does make things harder. It should die.