BOM in UTF-8: good, bad, or ugly?

The usefulness or otherwise of U+FEFF (ZERO WIDTH NON-BREAKING SPACE and BYTE ORDER MARK) in UTF-8 has been subject to reinterpretation over the years. It wasn’t mentioned in the original XML 1.0 Recommendation but was added later, rather like how its use was added to the Unicode Standard.

In the Unicode Standard 2.0, there was no mention of U+FEFF with UTF-8, either in the section on the BOM or in the appendix defining UTF-8.

In the Unicode Standard 3.0, section 13.6, “Specials”, includes:

Although there are never any questions of byte-order with UTF-8 text, this sequence can serve as signature for UTF-8 encoded text where the character set is unmarked.

In the Unicode Standard 5.0, section 3.10, “Unicode Encoding Schemes”, includes:

While there is obviously no need for a byte order signature when using UTF-8, there are occasions when processes convert UTF-16 or UTF-32 data containing a byte order mark into UTF-8. When represented in UTF-8, the byte order mark turns into the byte sequence <EF BB BF>. Its usage at the beginning of a UTF-8 data stream is neither required nor recommended by the Unicode Standard, but its presence does not affect conformance to the UTF-8 encoding scheme. Identification of the <EF BB BF> byte sequence at the beginning of a data stream can, however, be taken as a near-certain indication that the data stream is using the UTF-8 encoding scheme.

So in the Unicode Standard it’s gone from irrelevant to useful to “Oh, if you must”.

(BTW, in other reinterpretations, “Unicode Encoding Scheme” results from splitting the meaning of “UTF”, and the use of U+FEFF to indicate non-breaking is deprecated these days.)

The Unicode FAQ both lists its use as a signature and says to avoid its use where “byte oriented protocols expect ASCII characters at the beginning of a file“. However, I don’t think that XML necessarily counts as one such byte oriented protocol.