By replacing them with `{Open,Close}{Param,Brace,Bracket,Invisible}`.
PR #137902 made `ast::TokenKind` more like `lexer::TokenKind` by
replacing the compound `BinOp{,Eq}(BinOpToken)` variants with fieldless
variants `Plus`, `Minus`, `Star`, etc. This commit does a similar thing
with delimiters. It also makes `ast::TokenKind` more similar to
`parser::TokenType`.
This requires a few new methods:
- `TokenKind::is_{,open_,close_}delim()` replace various kinds of
pattern matches.
- `Delimiter::as_{open,close}_token_kind` are used to convert
`Delimiter` values to `TokenKind`.
Despite these additions, it's a net reduction in lines of code. This is
because e.g. `token::OpenParen` is so much shorter than
`token::OpenDelim(Delimiter::Parenthesis)` that many multi-line forms
reduce to single line forms. And many places where the number of lines
doesn't change are still easier to read, just because the names are
shorter, e.g.:
```
- } else if self.token != token::CloseDelim(Delimiter::Brace) {
+ } else if self.token != token::CloseBrace {
```
Rollup of 9 pull requests
Successful merges:
- #135340 (Add `explicit_extern_abis` Feature and Enforce Explicit ABIs)
- #139440 (rustc_target: RISC-V: feature addition batch 2)
- #139667 (cfi: Remove #[no_sanitize(cfi)] for extern weak functions)
- #139828 (Don't require rigid alias's trait to hold)
- #139854 (Improve parse errors for stray lifetimes in type position)
- #139889 (Clean UI tests 3 of n)
- #139894 (Fix `opt-dist` CLI flag and make it work without LLD)
- #139900 (stepping into impls for normalization is unproductive)
- #139915 (replace some #[rustc_intrinsic] usage with use of the libcore declarations)
r? `@ghost`
`@rustbot` modify labels: rollup
Improve parse errors for stray lifetimes in type position
While technically & syntactically speaking lifetimes do begin[^1] types in type contexts (this essentially excludes generic argument lists) and require a following `+` to form a complete type (`'a +` denotes a bare trait object type), the likelihood that a user meant to write a lifetime-prefixed bare trait object type in *modern* editions (Rust ≥2021) when placing a lifetime into a type context is incredibly low (they would need to add at least three tokens to turn it into a *semantically* well-formed TOT: `'a` → `dyn 'a + Trait`).
Therefore let's *lie* in modern editions (just like in PR https://github.com/rust-lang/rust/pull/131239, a precedent if you will) by stating "*expected type, found lifetime*" in such cases which is a lot more a approachable, digestible and friendly compared to "*lifetime in trait object type must be followed by `+`*" (as added in PR https://github.com/rust-lang/rust/pull/69760).
I've also added recovery for "ampersand-less" reference types (e.g., `'a ()`, `'a mut Ty`) in modern editions because it was trivial to do and I think it's not unlikely to occur in practice.
Fixes#133413.
[^1]: For example, in the context of decl macros, this implies that a lone `'a` always matches syntax fragment `ty` ("even if" there's a later macro matcher expecting syntax fragment `lifetime`). Rephrased, lifetimes (in type contexts) *commit* to the type parser.
Namely, use a more sensical primary span.
Don't pretty-print AST nodes for the diagnostic message. Why:
* It's lossy (e.g., it doesn't replicate trailing `+`s in trait objects.
* It's prone to leak error nodes (printed as `(/*ERROR*/)`) since
the LHS can easily represent recovered code (e.g., `fn(i32?) + T`).
Detect and provide suggestion for `&raw EXPR`
When emitting an error in the parser, and we detect that the previous token was `raw` and we *could* have consumed `const`/`mut`, suggest that this may have been a mistyped raw ref expr. To do this, we add `const`/`mut` to the expected token set when parsing `&raw` as an expression (which does not affect the "good path" of parsing, for the record).
This is kind of a rudimentary error improvement, since it doesn't actually attempt to recover anything, leading to some other knock-on errors b/c we still treat `&raw` as the expression that was parsed... but at least we add the suggestion! I don't think the parser grammar means we can faithfully recover `&raw EXPR` early, i.e. during `parse_expr_borrow`.
Fixes#133231
StructLiteralNeedingParens is no longer reachable always giving
precedence to StructLiteralNotAllowedHere.
As an aside: The former error struct shouldn't've existed in the
first place. We should've just used the latter in this branch.
For consistency with `rustc_lexer::TokenKind::Bang`, and because other
`ast::TokenKind` variants generally have syntactic names instead of
semantic names (e.g. `Star` and `DotDot` instead of `Mul` and `Range`).
`BinOpToken` is badly named, because it only covers the assignable
binary ops and excludes comparisons and `&&`/`||`. Its use in
`ast::TokenKind` does allow a small amount of code sharing, but it's a
clumsy factoring.
This commit removes `ast::TokenKind::BinOp{,Eq}`, replacing each one
with 10 individual variants. This makes `ast::TokenKind` more similar to
`rustc_lexer::TokenKind`, which has individual variants for all
operators.
Although the number of lines of code increases, the number of chars
decreases due to the frequent use of shorter names like `token::Plus`
instead of `token::BinOp(BinOpToken::Plus)`.
The one notable test change is `tests/ui/macros/trace_faulty_macros.rs`.
This commit removes the complicated `Interpolated` handling in
`expected_expression_found` that results in a longer error message. But
I think the new, shorter message is actually an improvement.
The original complaint was in #71039, when the error message started
with "error: expected expression, found `1 + 1`". That was confusing
because `1 + 1` is an expression. Other than that, the reporter said
"the whole error message is not too bad if you ignore the first line".
Subsequently, extra complexity and wording was added to the error
message. But I don't think the extra wording actually helps all that
much. In particular, it still says of the `1+1` that "this is expected
to be expression". This repeats the problem from the original complaint!
This commit removes the extra complexity, reverting to a simpler error
message. This is primarily because the traversal is a pain without
`Interpolated` tokens. Nonetheless, I think the error message is
*improved*. It now starts with "expected expression, found `pat`
metavariable", which is much clearer and the real problem. It also
doesn't say anything specific about `1+1`, which is good, because the
`1+1` isn't really relevant to the error -- it's the `$e:pat` that's
important.
It mirrors `ExprKind::Binary`, and contains a `BinOpKind`. This makes
`AssocOp` more like `ExprKind`. Note that the variants removed from
`AssocOp` are all named differently to `BinOpToken`, e.g. `Multiply`
instead of `Mul`, so that's an inconsistency removed.
The commit adds `precedence` and `fixity` methods to `BinOpKind`, and
calls them from the corresponding methods in `AssocOp`. This avoids the
need to create an `AssocOp` from a `BinOpKind` in a bunch of places, and
`AssocOp::from_ast_binop` is removed.
`AssocOp::to_ast_binop` is also no longer needed.
Overall things are shorter and nicer.
When encountering a doc comment without an identifier after, we'd unconditionally state "this doc comment doesn't document anything", swallowing the *actual* error which is that the thing *after* the doc comment wasn't expected. Added a check that the found token is something that "conceptually" closes the previous item before emitting that error, otherwise just complain about the missing identifier.
In both of the following cases, the syntax error follows a doc comment:
```
error: expected identifier, found keyword `Self`
--> $DIR/doc-before-bad-variant.rs:4:5
|
LL | enum TestEnum {
| -------- while parsing this enum
...
LL | Self,
| ^^^^ expected identifier, found keyword
|
= help: enum variants can be `Variant`, `Variant = <integer>`, `Variant(Type, ..., TypeN)` or `Variant { fields: Types }`
```
```
error: expected identifier, found `<`
--> $DIR/doc-before-syntax-error.rs:2:1
|
LL | <>
| ^ expected identifier
```
Fix#71982.
Instead use dcx.abort_if_error() or guar.raise_fatal() instead. These
guarantee that an error actually happened previously and thus we don't
silently abort.
The parser pushes a `TokenType` to `Parser::expected_token_types` on
every call to the various `check`/`eat` methods, and clears it on every
call to `bump`. Some of those `TokenType` values are full tokens that
require cloning and dropping. This is a *lot* of work for something
that is only used in error messages and it accounts for a significant
fraction of parsing execution time.
This commit overhauls `TokenType` so that `Parser::expected_token_types`
can be implemented as a bitset. This requires changing `TokenType` to a
C-style parameterless enum, and adding `TokenTypeSet` which uses a
`u128` for the bits. (The new `TokenType` has 105 variants.)
The new types `ExpTokenPair` and `ExpKeywordPair` are now arguments to
the `check`/`eat` methods. This is for maximum speed. The elements in
the pairs are always statically known; e.g. a
`token::BinOp(token::Star)` is always paired with a `TokenType::Star`.
So we now compute `TokenType`s in advance and pass them in to
`check`/`eat` rather than the current approach of constructing them on
insertion into `expected_token_types`.
Values of these pair types can be produced by the new `exp!` macro,
which is used at every `check`/`eat` call site. The macro is for
convenience, allowing any pair to be generated from a single identifier.
The ident/keyword filtering in `expected_one_of_not_found` is no longer
necessary. It was there to account for some sloppiness in
`TokenKind`/`TokenType` comparisons.
The existing `TokenType` is moved to a new file `token_type.rs`, and all
its new infrastructure is added to that file. There is more boilerplate
code than I would like, but I can't see how to make it shorter.
This is a naming convention used in a handful of spots in the parser for
delimiters. It confused me when I first saw it a long time ago, and I've
never liked it. A web search says "Bra-ket notation" exists in linear
algebra but the terminology has zero prior use in a programming context,
as far as I can tell.
This commit changes it to `open`/`close`, which is consistent with the
rest of the compiler.
`rustc_symbol` is the source of truth for keywords.
rustdoc has its own implicit definition of keywords, via the
`is_doc_keyword`. It (presumably) intends to include all keywords, but
it omits `yeet`.
rustfmt has its own explicit list of Rust keywords. It also (presumably)
intends to include all keywords, but it omits `await`, `builtin`, `gen`,
`macro_rules`, `raw`, `reuse`, `safe`, and `yeet`. Also, it does linear
searches through this list, which is inefficient.
This commit fixes all of the above problems by introducing a new
predicate `is_any_keyword` in rustc and using it in rustdoc and rustfmt.
It documents that it's not the right predicate in most cases.
`rustc_span::symbol` defines some things that are re-exported from
`rustc_span`, such as `Symbol` and `sym`. But it doesn't re-export some
closely related things such as `Ident` and `kw`. So you can do `use
rustc_span::{Symbol, sym}` but you have to do `use
rustc_span::symbol::{Ident, kw}`, which is inconsistent for no good
reason.
This commit re-exports `Ident`, `kw`, and `MacroRulesNormalizedIdent`,
and changes many `rustc_span::symbol::` qualifiers in `compiler/` to
`rustc_span::`. This is a 200+ net line of code reduction, mostly
because many files with two `use rustc_span` items can be reduced to
one.
`parse_expr_bottom` stores `this.token.span` in `lo`, but then fails to
use it in many places where it could. This commit fixes that, and
likewise (to a smaller extent) in `parse_ty_common`.
This PR detects misspelled keywords using two heuristics:
1. Lowercasing the unexpected identifier.
2. Using edit distance to find a keyword similar to the unexpected identifier.
However, it does not detect each and every misspelled keyword to
minimize false positives and ambiguities. More details about the
implementation can be found in the comments.
This commit does the following.
- Renames `collect_tokens_trailing_token` as `collect_tokens`, because
(a) it's annoying long, and (b) the `_trailing_token` bit is less
accurate now that its types have changed.
- In `collect_tokens`, adds a `Option<CollectPos>` argument and a
`UsePreAttrPos` in the return type of `f`. These are used in
`parse_expr_force_collect` (for vanilla expressions) and in
`parse_stmt_without_recovery` (for two different cases of expression
statements). Together these ensure are enough to fix all the problems
with token collection and assoc expressions. The changes to the
`stringify.rs` test demonstrate some of these.
- Adds a new test. The code in this test was causing an assertion
failure prior to this commit, due to an invalid `NodeRange`.
The extra complexity is annoying, but necessary to fix the existing
problems.
Fix ICE in suggestion caused by `⩵` being recovered as `==`
The second suggestion shown here would previously incorrectly assume that the span corresponding to `⩵` was 2 bytes wide composed by 2 1 byte wide chars, so a span pointing at `==` could point only at one of the `=` to remove it. Instead, we now replace the whole thing (as we should have the whole time):
```
error: unknown start of token: \u{2a75}
--> $DIR/unicode-double-equals-recovery.rs:1:16
|
LL | const A: usize ⩵ 2;
| ^
|
help: Unicode character '⩵' (Two Consecutive Equals Signs) looks like '==' (Double Equals Sign), but it is not
|
LL | const A: usize == 2;
| ~~
error: unexpected `==`
--> $DIR/unicode-double-equals-recovery.rs:1:16
|
LL | const A: usize ⩵ 2;
| ^
|
help: try using `=` instead
|
LL | const A: usize = 2;
| ~
```
Fix#127823.
The second suggestion shown here would previously incorrectly assume that the span corresponding to `⩵` was 2 bytes wide composed by 2 1 byte wide chars, so a span pointing at `==` could point only at one of the `=` to remove it. Instead, we now replace the whole thing (as we should have the whole time):
```
error: unknown start of token: \u{2a75}
--> $DIR/unicode-double-equals-recovery.rs:1:16
|
LL | const A: usize ⩵ 2;
| ^
|
help: Unicode character '⩵' (Two Consecutive Equals Signs) looks like '==' (Double Equals Sign), but it is not
|
LL | const A: usize == 2;
| ~~
error: unexpected `==`
--> $DIR/unicode-double-equals-recovery.rs:1:16
|
LL | const A: usize ⩵ 2;
| ^
|
help: try using `=` instead
|
LL | const A: usize = 2;
| ~
```
Use smaller span for suggesting adding `_:` ahead of a type:
```
error: expected one of `(`, `...`, `..=`, `..`, `::`, `:`, `{`, or `|`, found `)`
--> $DIR/anon-params-denied-2018.rs:12:47
|
LL | fn foo_with_qualified_path(<Bar as T>::Baz);
| ^ expected one of 8 possible tokens
|
= note: anonymous parameters are removed in the 2018 edition (see RFC 1685)
help: explicitly ignore the parameter name
|
LL | fn foo_with_qualified_path(_: <Bar as T>::Baz);
| ++
```
