1
Fork 0
rust/compiler/rustc_parse/src
Dylan DPC be3d1eb301
Rollup merge of #81856 - Smittyvb:utf16-warn, r=matthewjasper
Suggest character encoding is incorrect when encountering random null bytes

This adds a note whenever null bytes are seen at the start of a token unexpectedly, since those tend to come from UTF-16 encoded files without a [BOM](https://en.wikipedia.org/wiki/Byte_order_mark) (if a UTF-16 BOM appears it won't be valid UTF-8, but if there is no BOM it be both valid UTF-16 and valid but garbled UTF-8). This approach was suggested in https://github.com/rust-lang/rust/issues/73979#issuecomment-653976451.

Closes #73979.
2021-02-27 21:56:15 +01:00
..
lexer Clarify error message wording 2021-02-07 11:02:53 -05:00
parser Combine HasAttrs and HasTokens into AstLike 2021-02-27 00:14:13 -05:00
lib.rs Combine HasAttrs and HasTokens into AstLike 2021-02-27 00:14:13 -05:00
validate_attr.rs ast: Remove some indirection layers from values in key-value attributes 2021-01-09 21:50:39 +03:00