Implement token-based handling of attributes during expansion
This PR modifies the macro expansion infrastructure to handle attributes in a fully token-based manner. As a result: * Derives macros no longer lose spans when their input is modified by eager cfg-expansion. This is accomplished by performing eager cfg-expansion on the token stream that we pass to the derive proc-macro * Inner attributes now preserve spans in all cases, including when we have multiple inner attributes in a row. This is accomplished through the following changes: * New structs `AttrAnnotatedTokenStream` and `AttrAnnotatedTokenTree` are introduced. These are very similar to a normal `TokenTree`, but they also track the position of attributes and attribute targets within the stream. They are built when we collect tokens during parsing. An `AttrAnnotatedTokenStream` is converted to a regular `TokenStream` when we invoke a macro. * Token capturing and `LazyTokenStream` are modified to work with `AttrAnnotatedTokenStream`. A new `ReplaceRange` type is introduced, which is created during the parsing of a nested AST node to make the 'outer' AST node aware of the attributes and attribute target stored deeper in the token stream. * When we need to perform eager cfg-expansion (either due to `#[derive]` or `#[cfg_eval]`), we tokenize and reparse our target, capturing additional information about the locations of `#[cfg]` and `#[cfg_attr]` attributes at any depth within the target. This is a performance optimization, allowing us to perform less work in the typical case where captured tokens never have eager cfg-expansion run.
This commit is contained in:
parent
25ea6be13e
commit
a93c4f05de
33 changed files with 2046 additions and 1192 deletions
|
@ -2581,19 +2581,17 @@ impl<'a> Parser<'a> {
|
|||
attrs: AttrWrapper,
|
||||
f: impl FnOnce(&mut Self, Vec<ast::Attribute>) -> PResult<'a, P<Expr>>,
|
||||
) -> PResult<'a, P<Expr>> {
|
||||
// FIXME - come up with a nice way to properly forward `ForceCollect`from
|
||||
// the nonterminal parsing code. TThis approach iscorrect, but will cause
|
||||
// us to unnecessarily capture tokens for exprs that have only builtin
|
||||
// attributes. Revisit this before #![feature(stmt_expr_attributes)] is stabilized
|
||||
let force_collect = if attrs.is_empty() { ForceCollect::No } else { ForceCollect::Yes };
|
||||
self.collect_tokens_trailing_token(attrs, force_collect, |this, attrs| {
|
||||
self.collect_tokens_trailing_token(attrs, ForceCollect::No, |this, attrs| {
|
||||
let res = f(this, attrs)?;
|
||||
let trailing = if this.restrictions.contains(Restrictions::STMT_EXPR)
|
||||
&& this.token.kind == token::Semi
|
||||
{
|
||||
TrailingToken::Semi
|
||||
} else {
|
||||
TrailingToken::None
|
||||
// FIXME - pass this through from the place where we know
|
||||
// we need a comma, rather than assuming that `#[attr] expr,`
|
||||
// always captures a trailing comma
|
||||
TrailingToken::MaybeComma
|
||||
};
|
||||
Ok((res, trailing))
|
||||
})
|
||||
|
|
Loading…
Add table
Add a link
Reference in a new issue