[tgross35]

During discussion about using the new unstable char::is_xid_start and
char::is_xid_continue in the compiler, a point was made that doing so
would mean that the Unicode version used to validate identifiers would
become tied to whatever std supports. However, we are already using
std's version of Unicode in many other places, e.g. via
char::is_whitespace, so this actually seems reasonable to codify.

Thus, add lanugage indicating that the exact Unicode version is
delegated to the standard library. This effectively means that T-lang
does not need to FCP future Unicode updates either.

As also noted on the XID thread, Java provides us some prior art here
https://docs.oracle.com/javase/specs/jls/se23/html/jls-3.html#jls-3.1.

XID discussion: rust-lang/rust#164841

[fmease]

However, we are already using
std's version of Unicode in many other places, e.g. via
char::is_whitespace

That statement seems a bit too vague & misleading.

We're in fact not using char::is_whitespace in the compiler when lexing (esp. in the sense of deciding what's lexicially valid Rust & what isn't) as we're intentionally using the closed set Pattern_White_Space instead of the open set Whitespace. Only ever in error paths are we using char::is_whitespace & in most cases only due to laziness / ignorance but plenty of places throughout the compiler do seem to use rustc_lexer::is_whitespace which is more correct generally speaking.

(I did just find out that we seem to be using char::is_whitespace in select places when parsing format strings but that's T-libs territory & could be considered a bug, too).

[tgross35]

Hm, I thought that at least https://github.com/rust-lang/rust/blob/4c46afa63688fb473618f9b3553d70b0c15a88eb/compiler/rustc_span/src/source_map.rs#L666 was load-bearing in some ways but perhaps not. I think this is still worth it regardless, rather than managing two versions separately or pinning a version, but I'll update the motivation.

[fmease]

SourceMap::indentation_before is only ever used for diagnostics (& perhaps lints at maximum), not for correctness. It's mainly used to properly construct multi-line code suggestions. It should be correct to swap out the char::is_whitespace with rustc_lexer::is_whitespace in there because the lexer would've already rejected the program if it contained code points in White_Space \ Pattern_White_Space (outside of comments & string literals).