Adds support for .await under the existing async_await feature gate.
Moves macro-like await! syntax to the await_macro feature gate.
Removes support for `await` as a non-keyword under the `async_await`
feature.
It's present within `Token::Interpolated` as an optimization, so that if
a nonterminal is converted to a `TokenStream` multiple times, the
first-computed value is saved and reused.
But in practice it's not needed. `interpolated_to_tokenstream()` is a
cold function: it's only called a few dozen times while compiling rustc
itself, and a few hundred times across the entire `rustc-perf` suite.
Furthermore, when it is called, it is almost always the first
conversion, so no benefit is gained from it.
So this commit removes `LazyTokenStream`, along with the now-unnecessary
`Token::interpolated()`.
As well as a significant simplification, the removal speeds things up
slightly, mostly due to not having to `drop` the `LazyTokenStream`
instances.
It is currently a method of `Token`, but it only is valid to call if
`self` is a `Token::Interpolated`. This commit eliminates the
possibility of misuse by changing it to an associated function that
takes a `Nonterminal`, which also simplifies the call sites.
This requires splitting out a new function, `nonterminal_to_string`.
Deduplicate mismatched delimiter errors
Delay unmatched delimiter errors until after the parser has run to deduplicate them when parsing and attempt recovering intelligently.
Second attempt at #54029, follow up to #53949. Fix#31528.
This commit changes `syntax::fold::Folder` from a functional style
(where most methods take a `T` and produce a new `T`) to a more
imperative style (where most methods take and modify a `&mut T`), and
renames it `syntax::mut_visit::MutVisitor`.
The first benefit is speed. The functional style does not require any
reallocations, due to the use of `P::map` and
`MoveMap::move_{,flat_}map`. However, every field in the AST must be
overwritten; even those fields that are unchanged are overwritten with
the same value. This causes a lot of unnecessary memory writes. The
imperative style reduces instruction counts by 1--3% across a wide range
of workloads, particularly incremental workloads.
The second benefit is conciseness; the imperative style is usually more
concise. E.g. compare the old functional style:
```
fn fold_abc(&mut self, abc: ABC) {
ABC {
a: fold_a(abc.a),
b: fold_b(abc.b),
c: abc.c,
}
}
```
with the imperative style:
```
fn visit_abc(&mut self, ABC { a, b, c: _ }: &mut ABC) {
visit_a(a);
visit_b(b);
}
```
(The reductions get larger in more complex examples.)
Overall, the patch removes over 200 lines of code -- even though the new
code has more comments -- and a lot of the remaining lines have fewer
characters.
Some notes:
- The old style used methods called `fold_*`. The new style mostly uses
methods called `visit_*`, but there are a few methods that map a `T`
to something other than a `T`, which are called `flat_map_*` (`T` maps
to multiple `T`s) or `filter_map_*` (`T` maps to 0 or 1 `T`s).
- `move_map.rs`/`MoveMap`/`move_map`/`move_flat_map` are renamed
`map_in_place.rs`/`MapInPlace`/`map_in_place`/`flat_map_in_place` to
reflect their slightly changed signatures.
- Although this commit renames the `fold` module as `mut_visit`, it
keeps it in the `fold.rs` file, so as not to confuse git. The next
commit will rename the file.
Use a `newtype_index!` within `Symbol`.
This shrinks `Option<Symbol>` from 8 bytes to 4 bytes, which shrinks
`Token` from 24 bytes to 16 bytes. This reduces instruction counts by up
to 1% across a range of benchmarks.
r? @oli-obk
This shrinks `Option<Symbol>` from 8 bytes to 4 bytes, which shrinks
`Token` from 24 bytes to 16 bytes. This reduces instruction counts by up
to 1% across a range of benchmarks.
Because it's an extra type layer that doesn't really help; in a couple
of places it actively gets in the way, and overall removing it makes the
code nicer. It does, however, move `tokenstream::TokenTree` further away
from the `TokenTree` in `quote.rs`.
More importantly, this change reduces the size of `TokenStream` from 48
bytes to 40 bytes on x86-64, which is enough to slightly reduce
instruction counts on numerous benchmarks, the best by 1.5%.
Note that `open_tt` and `close_tt` have gone from being methods on
`Delimited` to associated methods of `TokenTree`.
This commit updates the tokenization of items which are subsequently passed to
`proc_macro` to ensure that span information is preserved on attributes as much
as possible. Previously this area of the code suffered from #43081 where we
haven't actually implemented converting an attribute to to a token tree yet, but
a local fix was possible here.
Closes#47941
rustc: Fix procedural macros generating lifetime tokens
This commit fixes an accidental regression from #50473 where lifetime tokens
produced by procedural macros ended up getting lost in translation in the
compiler and not actually producing parseable code. The issue lies in the fact
that a lifetime's `Ident` is prefixed with `'`. The `glue` implementation for
gluing joint tokens together forgot to take this into account so the lifetime
inside of `Ident` was missing the leading tick!
The `glue` implementation here is updated to create a new `Symbol` in these
situations to manufacture a new `Ident` with a leading tick to ensure it parses
correctly.
Closes#50942
