bjoernager/rust
bjoernager/rust
1
Fork 0
Code Activity

Remove NtIdent and NtLifetime.

Browse source 
The extra span is now recorded in the new `TokenKind::NtIdent` and
`TokenKind::NtLifetime`. These both consist of a single token, and so
there's no operator precedence problems with inserting them directly
into the token stream.

The other way to do this would be to wrap the ident/lifetime in invisible
delimiters, but there's a lot of code that assumes an interpolated
ident/lifetime fits in a single token, and changing all that code to work with
invisible delimiters would have been a pain. (Maybe it could be done in a
follow-up.)

This change might not seem like much of a win, but it's a first step toward the
much bigger and long-desired removal of `Nonterminal` and
`TokenKind::Interpolated`. That change is big and complex enough that it's
worth doing this piece separately. (Indeed, this commit is based on part of a
late commit in #114647, a prior attempt at that big and complex change.)
This commit is contained in:
Nicholas Nethercote 2024-04-22 19:46:51 +10:00
parent 9a63a42cb7
commit 95e519ecbf
11 changed files with 131 additions and 104 deletions
Download patch file Download diff file Expand all files Collapse all files

23
compiler/rustc_expand/src/proc_macro_server.rs
View file

@ -220,6 +220,12 @@ impl FromInternal<(TokenStream, &mut Rustc<'_, '_>)> for Vec<TokenTree<TokenStre
Ident(sym, is_raw) => {
trees.push(TokenTree::Ident(Ident { sym, is_raw: is_raw.into(), span }))
}
NtIdent(ident, is_raw) => trees.push(TokenTree::Ident(Ident {
sym: ident.name,
is_raw: is_raw.into(),
span: ident.span,
})),
Lifetime(name) => {
let ident = symbol::Ident::new(name, span).without_first_quote();
trees.extend([
@ -227,6 +233,15 @@ impl FromInternal<(TokenStream, &mut Rustc<'_, '_>)> for Vec<TokenTree<TokenStre
TokenTree::Ident(Ident { sym: ident.name, is_raw: false, span }),
]);
}
NtLifetime(ident) => {
let stream = TokenStream::token_alone(token::Lifetime(ident.name), ident.span);
trees.push(TokenTree::Group(Group {
delimiter: pm::Delimiter::None,
stream: Some(stream),
span: DelimSpan::from_single(span),
}))
}
Literal(token::Lit { kind, symbol, suffix }) => {
trees.push(TokenTree::Literal(self::Literal {
kind: FromInternal::from_internal(kind),
@ -259,14 +274,6 @@ impl FromInternal<(TokenStream, &mut Rustc<'_, '_>)> for Vec<TokenTree<TokenStre
}));
}
Interpolated(ref nt) if let NtIdent(ident, is_raw) = &**nt => {
trees.push(TokenTree::Ident(Ident {
sym: ident.name,
is_raw: matches!(is_raw, IdentIsRaw::Yes),
span: ident.span,
}))
}
Interpolated(nt) => {
let stream = TokenStream::from_nonterminal_ast(&nt);
// A hack used to pass AST fragments to attribute and derive