bjoernager/rust
bjoernager/rust
1
Fork 0
Code Activity

Remove NtIdent and NtLifetime.

Browse source 
The extra span is now recorded in the new `TokenKind::NtIdent` and
`TokenKind::NtLifetime`. These both consist of a single token, and so
there's no operator precedence problems with inserting them directly
into the token stream.

The other way to do this would be to wrap the ident/lifetime in invisible
delimiters, but there's a lot of code that assumes an interpolated
ident/lifetime fits in a single token, and changing all that code to work with
invisible delimiters would have been a pain. (Maybe it could be done in a
follow-up.)

This change might not seem like much of a win, but it's a first step toward the
much bigger and long-desired removal of `Nonterminal` and
`TokenKind::Interpolated`. That change is big and complex enough that it's
worth doing this piece separately. (Indeed, this commit is based on part of a
late commit in #114647, a prior attempt at that big and complex change.)
This commit is contained in:
Nicholas Nethercote 2024-04-22 19:46:51 +10:00
parent 9a63a42cb7
commit 95e519ecbf
11 changed files with 131 additions and 104 deletions
Download patch file Download diff file Expand all files Collapse all files

6
compiler/rustc_parse/src/parser/mod.rs
View file

@ -407,6 +407,8 @@ pub(super) fn token_descr(token: &Token) -> String {
(Some(TokenDescription::Keyword), _) => Some("keyword"),
(Some(TokenDescription::ReservedKeyword), _) => Some("reserved keyword"),
(Some(TokenDescription::DocComment), _) => Some("doc comment"),
(None, TokenKind::NtIdent(..)) => Some("identifier"),
(None, TokenKind::NtLifetime(..)) => Some("lifetime"),
(None, TokenKind::Interpolated(node)) => Some(node.descr()),
(None, _) => None,
};
@ -1633,5 +1635,9 @@ pub enum FlatToken {
#[derive(Clone, Debug)]
pub enum ParseNtResult {
Tt(TokenTree),
Ident(Ident, IdentIsRaw),
Lifetime(Ident),
/// This case will eventually be removed, along with `Token::Interpolate`.
Nt(Lrc<Nonterminal>),
}