Simplify `TokenStream` some more
These commits simplify `TokenStream`, remove `ThinTokenStream`, and avoid some clones. The end result is simpler code and a slight perf win on some benchmarks.
r? @petrochenkov
Add non-panicking `maybe_new_parser_from_file` variant
Add (seemingly?) missing `maybe_new_parser_from_file` constructor variant.
Disclaimer: I'm not certain this is the correct approach - just found out we don't have this when working on a Rustfmt PR to catch/prevent more Rust parser panics: https://github.com/rust-lang/rustfmt/pull/3240 and tried to make it work somehow.
Because it's an extra type layer that doesn't really help; in a couple
of places it actively gets in the way, and overall removing it makes the
code nicer. It does, however, move `tokenstream::TokenTree` further away
from the `TokenTree` in `quote.rs`.
More importantly, this change reduces the size of `TokenStream` from 48
bytes to 40 bytes on x86-64, which is enough to slightly reduce
instruction counts on numerous benchmarks, the best by 1.5%.
Note that `open_tt` and `close_tt` have gone from being methods on
`Delimited` to associated methods of `TokenTree`.
53956 panic on include bytes of own file
fix#53956
When using `include_bytes!` on a source file in the project, compiler would panic on subsequent compilations because `expand_include_bytes` would overwrite files in the source_map with no source. This PR changes `expand_include_bytes` to check source_map and use the already existing src, if any.
rustdoc: Replaces fn main search and extern crate search with proper parsing during doctests.
Fixes#21299.
Fixes#33731.
Let me know if there's any additional changes you'd like made!
This commit avoids an allocation when parsing any float and integer
literals that don't involved underscores.
This reduces the number of allocations done for the `tuple-stress`
benchmark by 10%, reducing its instruction count by just under 1%.
syntax: Optimize some literal parsing
Currently in the `wasm-bindgen` project we have a very very large crate that's
procedurally generated, `web-sys`. To generate this crate we parse all of a
browser's WebIDL and we then generate bindings for all of the APIs contained
within.
The resulting Rust file is 18MB large (wow!) and currently takes a very long
time to compile in debug mode. On the nightly compiler a *debug* build takes 90s
for the crate to finish. I was curious what was taking so long and upon
investigating a *massive* portion of the time was spent in the `lit_token`
method of the compiler, primarily formatting strings via `format!`.
Upon some more investigation it looks like the `byte_str_lit` was allocating an
error message once per byte, causing a very large number of allocations to
happen for large literals, of which wasm-bindgen generates quite a few (some are
MB large).
This commit fixes the issue by lazily allocating the error message, only doing
so if the error message is actually needed (which should be never). As a result,
the debug mode compilation time for our `web-sys` crate decreased from 90s to
20s, a very nice improvement! (although we've still got some work to do).
Currently in the `wasm-bindgen` project we have a very very large crate that's
procedurally generated, `web-sys`. To generate this crate we parse all of a
browser's WebIDL and we then generate bindings for all of the APIs contained
within.
The resulting Rust file is 18MB large (wow!) and currently takes a very long
time to compile in debug mode. On the nightly compiler a *debug* build takes 90s
for the crate to finish. I was curious what was taking so long and upon
investigating a *massive* portion of the time was spent in the `lit_token`
method of the compiler, primarily formatting strings via `format!`.
Upon some more investigation it looks like the `byte_str_lit` was allocating an
error message once per byte, causing a very large number of allocations to
happen for large literals, of which wasm-bindgen generates quite a few (some are
MB large).
This commit fixes the issue by lazily allocating the error message, only doing
so if the error message is actually needed (which should be never). As a result,
the debug mode compilation time for our `web-sys` crate decreased from 90s to
20s, a very nice improvement! (although we've still got some work to do).
async/await
This PR implements `async`/`await` syntax for `async fn` in Rust 2015 and `async` closures and `async` blocks in Rust 2018 (tracking issue: https://github.com/rust-lang/rust/issues/50547). Limitations: non-`move` async closures with arguments are currently not supported, nor are `async fn` with multiple different input lifetimes. These limitations are not fundamental and will be removed in the future, however I'd like to go ahead and get this PR merged so we can start experimenting with this in combination with futures 0.3.
Based on https://github.com/rust-lang/rust/pull/51414.
cc @petrochenkov for parsing changes.
r? @eddyb
This is gated on edition 2018 & the `async_await` feature gate.
The parser will accept `async fn` and `async unsafe fn` as fn
items. Along the same lines as `const fn`, only `async unsafe fn`
is permitted, not `unsafe async fn`.The parser will not accept
`async` functions as trait methods.
To do a little code clean up, four fields of the function type
struct have been merged into the new `FnHeader` struct: constness,
asyncness, unsafety, and ABI.
Also, a small bug in HIR printing is fixed: it previously printed
`const unsafe fn` as `unsafe const fn`, which is grammatically
incorrect.
