rust/compiler/rustc_expand/src/proc_macro_server.rs

Ignoring revisions in .git-blame-ignore-revs. Click here to bypass and see the normal blame view.

847 lines
33 KiB
Rust
Raw Normal View History

use std::ops::{Bound, Range};
2024-02-13 23:28:27 +00:00
use ast::token::IdentIsRaw;
Remove `TokenStreamBuilder`. `TokenStreamBuilder` exists to concatenate multiple `TokenStream`s together. This commit removes it, and moves the concatenation functionality directly into `TokenStream`, via two new methods `push_tree` and `push_stream`. This makes things both simpler and faster. `push_tree` is particularly important. `TokenStreamBuilder` only had a single `push` method, which pushed a stream. But in practice most of the time we push a single token tree rather than a stream, and `push_tree` avoids the need to build a token stream with a single entry (which requires two allocations, one for the `Lrc` and one for the `Vec`). The main `push_tree` use arises from a change to one of the `ToInternal` impls in `proc_macro_server.rs`. It now returns a `SmallVec` instead of a `TokenStream`. This return value is then iterated over by `concat_trees`, which does `push_tree` on each element. Furthermore, the use of `SmallVec` avoids more allocations, because there is always only one or two token trees. Note: the removed `TokenStreamBuilder::push` method had some code to deal with a quadratic blowup case from #57735. This commit removes the code. I tried and failed to reproduce the blowup from that PR, before and after this change. Various other changes have happened to `TokenStreamBuilder` in the meantime, so I suspect the original problem is no longer relevant, though I don't have proof of this. Generally speaking, repeatedly extending a `Vec` without pre-determining its capacity is *not* quadratic. It's also incredibly common, within rustc and many other Rust programs, so if there were performance problems there you'd think it would show up in other places, too.
2022-10-05 10:38:15 +11:00
use pm::bridge::{
server, DelimSpan, Diagnostic, ExpnGlobals, Group, Ident, LitKind, Literal, Punct, TokenTree,
};
use pm::{Delimiter, Level};
2020-04-27 23:26:11 +05:30
use rustc_ast as ast;
use rustc_ast::token;
use rustc_ast::tokenstream::{self, DelimSpacing, Spacing, TokenStream};
use rustc_ast::util::literal::escape_byte_str_symbol;
use rustc_ast_pretty::pprust;
Implement span quoting for proc-macros This PR implements span quoting, allowing proc-macros to produce spans pointing *into their own crate*. This is used by the unstable `proc_macro::quote!` macro, allowing us to get error messages like this: ``` error[E0412]: cannot find type `MissingType` in this scope --> $DIR/auxiliary/span-from-proc-macro.rs:37:20 | LL | pub fn error_from_attribute(_args: TokenStream, _input: TokenStream) -> TokenStream { | ----------------------------------------------------------------------------------- in this expansion of procedural macro `#[error_from_attribute]` ... LL | field: MissingType | ^^^^^^^^^^^ not found in this scope | ::: $DIR/span-from-proc-macro.rs:8:1 | LL | #[error_from_attribute] | ----------------------- in this macro invocation ``` Here, `MissingType` occurs inside the implementation of the proc-macro `#[error_from_attribute]`. Previosuly, this would always result in a span pointing at `#[error_from_attribute]` This will make many proc-macro-related error message much more useful - when a proc-macro generates code containing an error, users will get an error message pointing directly at that code (within the macro definition), instead of always getting a span pointing at the macro invocation site. This is implemented as follows: * When a proc-macro crate is being *compiled*, it causes the `quote!` macro to get run. This saves all of the sapns in the input to `quote!` into the metadata of *the proc-macro-crate* (which we are currently compiling). The `quote!` macro then expands to a call to `proc_macro::Span::recover_proc_macro_span(id)`, where `id` is an opaque identifier for the span in the crate metadata. * When the same proc-macro crate is *run* (e.g. it is loaded from disk and invoked by some consumer crate), the call to `proc_macro::Span::recover_proc_macro_span` causes us to load the span from the proc-macro crate's metadata. The proc-macro then produces a `TokenStream` containing a `Span` pointing into the proc-macro crate itself. The recursive nature of 'quote!' can be difficult to understand at first. The file `src/test/ui/proc-macro/quote-debug.stdout` shows the output of the `quote!` macro, which should make this eaier to understand. This PR also supports custom quoting spans in custom quote macros (e.g. the `quote` crate). All span quoting goes through the `proc_macro::quote_span` method, which can be called by a custom quote macro to perform span quoting. An example of this usage is provided in `src/test/ui/proc-macro/auxiliary/custom-quote.rs` Custom quoting currently has a few limitations: In order to quote a span, we need to generate a call to `proc_macro::Span::recover_proc_macro_span`. However, proc-macros support renaming the `proc_macro` crate, so we can't simply hardcode this path. Previously, the `quote_span` method used the path `crate::Span` - however, this only works when it is called by the builtin `quote!` macro in the same crate. To support being called from arbitrary crates, we need access to the name of the `proc_macro` crate to generate a path. This PR adds an additional argument to `quote_span` to specify the name of the `proc_macro` crate. Howver, this feels kind of hacky, and we may want to change this before stabilizing anything quote-related. Additionally, using `quote_span` currently requires enabling the `proc_macro_internals` feature. The builtin `quote!` macro has an `#[allow_internal_unstable]` attribute, but this won't work for custom quote implementations. This will likely require some additional tricks to apply `allow_internal_unstable` to the span of `proc_macro::Span::recover_proc_macro_span`.
2020-08-02 19:52:16 -04:00
use rustc_data_structures::fx::FxHashMap;
use rustc_data_structures::sync::Lrc;
use rustc_errors::{Diag, ErrorGuaranteed, MultiSpan, PResult};
use rustc_parse::lexer::nfc_normalize;
use rustc_parse::parser::Parser;
use rustc_parse::{new_parser_from_source_str, source_str_to_stream, unwrap_or_emit_fatal};
use rustc_session::parse::ParseSess;
Implement span quoting for proc-macros This PR implements span quoting, allowing proc-macros to produce spans pointing *into their own crate*. This is used by the unstable `proc_macro::quote!` macro, allowing us to get error messages like this: ``` error[E0412]: cannot find type `MissingType` in this scope --> $DIR/auxiliary/span-from-proc-macro.rs:37:20 | LL | pub fn error_from_attribute(_args: TokenStream, _input: TokenStream) -> TokenStream { | ----------------------------------------------------------------------------------- in this expansion of procedural macro `#[error_from_attribute]` ... LL | field: MissingType | ^^^^^^^^^^^ not found in this scope | ::: $DIR/span-from-proc-macro.rs:8:1 | LL | #[error_from_attribute] | ----------------------- in this macro invocation ``` Here, `MissingType` occurs inside the implementation of the proc-macro `#[error_from_attribute]`. Previosuly, this would always result in a span pointing at `#[error_from_attribute]` This will make many proc-macro-related error message much more useful - when a proc-macro generates code containing an error, users will get an error message pointing directly at that code (within the macro definition), instead of always getting a span pointing at the macro invocation site. This is implemented as follows: * When a proc-macro crate is being *compiled*, it causes the `quote!` macro to get run. This saves all of the sapns in the input to `quote!` into the metadata of *the proc-macro-crate* (which we are currently compiling). The `quote!` macro then expands to a call to `proc_macro::Span::recover_proc_macro_span(id)`, where `id` is an opaque identifier for the span in the crate metadata. * When the same proc-macro crate is *run* (e.g. it is loaded from disk and invoked by some consumer crate), the call to `proc_macro::Span::recover_proc_macro_span` causes us to load the span from the proc-macro crate's metadata. The proc-macro then produces a `TokenStream` containing a `Span` pointing into the proc-macro crate itself. The recursive nature of 'quote!' can be difficult to understand at first. The file `src/test/ui/proc-macro/quote-debug.stdout` shows the output of the `quote!` macro, which should make this eaier to understand. This PR also supports custom quoting spans in custom quote macros (e.g. the `quote` crate). All span quoting goes through the `proc_macro::quote_span` method, which can be called by a custom quote macro to perform span quoting. An example of this usage is provided in `src/test/ui/proc-macro/auxiliary/custom-quote.rs` Custom quoting currently has a few limitations: In order to quote a span, we need to generate a call to `proc_macro::Span::recover_proc_macro_span`. However, proc-macros support renaming the `proc_macro` crate, so we can't simply hardcode this path. Previously, the `quote_span` method used the path `crate::Span` - however, this only works when it is called by the builtin `quote!` macro in the same crate. To support being called from arbitrary crates, we need access to the name of the `proc_macro` crate to generate a path. This PR adds an additional argument to `quote_span` to specify the name of the `proc_macro` crate. Howver, this feels kind of hacky, and we may want to change this before stabilizing anything quote-related. Additionally, using `quote_span` currently requires enabling the `proc_macro_internals` feature. The builtin `quote!` macro has an `#[allow_internal_unstable]` attribute, but this won't work for custom quote implementations. This will likely require some additional tricks to apply `allow_internal_unstable` to the span of `proc_macro::Span::recover_proc_macro_span`.
2020-08-02 19:52:16 -04:00
use rustc_span::def_id::CrateNum;
use rustc_span::symbol::{self, sym, Symbol};
use rustc_span::{BytePos, FileName, Pos, SourceFile, Span};
Remove `TokenStreamBuilder`. `TokenStreamBuilder` exists to concatenate multiple `TokenStream`s together. This commit removes it, and moves the concatenation functionality directly into `TokenStream`, via two new methods `push_tree` and `push_stream`. This makes things both simpler and faster. `push_tree` is particularly important. `TokenStreamBuilder` only had a single `push` method, which pushed a stream. But in practice most of the time we push a single token tree rather than a stream, and `push_tree` avoids the need to build a token stream with a single entry (which requires two allocations, one for the `Lrc` and one for the `Vec`). The main `push_tree` use arises from a change to one of the `ToInternal` impls in `proc_macro_server.rs`. It now returns a `SmallVec` instead of a `TokenStream`. This return value is then iterated over by `concat_trees`, which does `push_tree` on each element. Furthermore, the use of `SmallVec` avoids more allocations, because there is always only one or two token trees. Note: the removed `TokenStreamBuilder::push` method had some code to deal with a quadratic blowup case from #57735. This commit removes the code. I tried and failed to reproduce the blowup from that PR, before and after this change. Various other changes have happened to `TokenStreamBuilder` in the meantime, so I suspect the original problem is no longer relevant, though I don't have proof of this. Generally speaking, repeatedly extending a `Vec` without pre-determining its capacity is *not* quadratic. It's also incredibly common, within rustc and many other Rust programs, so if there were performance problems there you'd think it would show up in other places, too.
2022-10-05 10:38:15 +11:00
use smallvec::{smallvec, SmallVec};
use crate::base::ExtCtxt;
trait FromInternal<T> {
fn from_internal(x: T) -> Self;
}
trait ToInternal<T> {
fn to_internal(self) -> T;
}
impl FromInternal<token::Delimiter> for Delimiter {
fn from_internal(delim: token::Delimiter) -> Delimiter {
match delim {
token::Delimiter::Parenthesis => Delimiter::Parenthesis,
token::Delimiter::Brace => Delimiter::Brace,
token::Delimiter::Bracket => Delimiter::Bracket,
token::Delimiter::Invisible => Delimiter::None,
}
}
}
impl ToInternal<token::Delimiter> for Delimiter {
fn to_internal(self) -> token::Delimiter {
match self {
Delimiter::Parenthesis => token::Delimiter::Parenthesis,
Delimiter::Brace => token::Delimiter::Brace,
Delimiter::Bracket => token::Delimiter::Bracket,
Delimiter::None => token::Delimiter::Invisible,
}
}
}
impl FromInternal<token::LitKind> for LitKind {
fn from_internal(kind: token::LitKind) -> Self {
match kind {
token::Byte => LitKind::Byte,
token::Char => LitKind::Char,
token::Integer => LitKind::Integer,
token::Float => LitKind::Float,
token::Str => LitKind::Str,
token::StrRaw(n) => LitKind::StrRaw(n),
token::ByteStr => LitKind::ByteStr,
token::ByteStrRaw(n) => LitKind::ByteStrRaw(n),
2023-03-06 07:42:04 +00:00
token::CStr => LitKind::CStr,
token::CStrRaw(n) => LitKind::CStrRaw(n),
token::Err(_guar) => {
// This is the only place a `pm::bridge::LitKind::ErrWithGuar`
// is constructed. Note that an `ErrorGuaranteed` is available,
// as required. See the comment in `to_internal`.
LitKind::ErrWithGuar
}
token::Bool => unreachable!(),
}
}
}
impl ToInternal<token::LitKind> for LitKind {
fn to_internal(self) -> token::LitKind {
match self {
LitKind::Byte => token::Byte,
LitKind::Char => token::Char,
LitKind::Integer => token::Integer,
LitKind::Float => token::Float,
LitKind::Str => token::Str,
LitKind::StrRaw(n) => token::StrRaw(n),
LitKind::ByteStr => token::ByteStr,
LitKind::ByteStrRaw(n) => token::ByteStrRaw(n),
2023-03-06 07:42:04 +00:00
LitKind::CStr => token::CStr,
LitKind::CStrRaw(n) => token::CStrRaw(n),
LitKind::ErrWithGuar => {
// This is annoying but valid. `LitKind::ErrWithGuar` would
// have an `ErrorGuaranteed` except that type isn't available
// in that crate. So we have to fake one. And we don't want to
// use a delayed bug because there might be lots of these,
// which would be expensive.
#[allow(deprecated)]
let guar = ErrorGuaranteed::unchecked_error_guaranteed();
token::Err(guar)
}
}
}
}
impl FromInternal<(TokenStream, &mut Rustc<'_, '_>)> for Vec<TokenTree<TokenStream, Span, Symbol>> {
fn from_internal((stream, rustc): (TokenStream, &mut Rustc<'_, '_>)) -> Self {
use rustc_ast::token::*;
// Estimate the capacity as `stream.len()` rounded up to the next power
// of two to limit the number of required reallocations.
let mut trees = Vec::with_capacity(stream.len().next_power_of_two());
let mut cursor = stream.trees();
while let Some(tree) = cursor.next() {
let (Token { kind, span }, joint) = match tree.clone() {
tokenstream::TokenTree::Delimited(span, _, delim, tts) => {
let delimiter = pm::Delimiter::from_internal(delim);
trees.push(TokenTree::Group(Group {
delimiter,
stream: Some(tts),
span: DelimSpan {
open: span.open,
close: span.close,
entire: span.entire(),
},
}));
continue;
}
Improve `print_tts` by changing `tokenstream::Spacing`. `tokenstream::Spacing` appears on all `TokenTree::Token` instances, both punct and non-punct. Its current usage: - `Joint` means "can join with the next token *and* that token is a punct". - `Alone` means "cannot join with the next token *or* can join with the next token but that token is not a punct". The fact that `Alone` is used for two different cases is awkward. This commit augments `tokenstream::Spacing` with a new variant `JointHidden`, resulting in: - `Joint` means "can join with the next token *and* that token is a punct". - `JointHidden` means "can join with the next token *and* that token is a not a punct". - `Alone` means "cannot join with the next token". This *drastically* improves the output of `print_tts`. For example, this: ``` stringify!(let a: Vec<u32> = vec![];) ``` currently produces this string: ``` let a : Vec < u32 > = vec! [] ; ``` With this PR, it now produces this string: ``` let a: Vec<u32> = vec![] ; ``` (The space after the `]` is because `TokenTree::Delimited` currently doesn't have spacing information. The subsequent commit fixes this.) The new `print_tts` doesn't replicate original code perfectly. E.g. multiple space characters will be condensed into a single space character. But it's much improved. `print_tts` still produces the old, uglier output for code produced by proc macros. Because we have to translate the generated code from `proc_macro::Spacing` to the more expressive `token::Spacing`, which results in too much `proc_macro::Along` usage and no `proc_macro::JointHidden` usage. So `space_between` still exists and is used by `print_tts` in conjunction with the `Spacing` field. This change will also help with the removal of `Token::Interpolated`. Currently interpolated tokens are pretty-printed nicely via AST pretty printing. `Token::Interpolated` removal will mean they get printed with `print_tts`. Without this change, that would result in much uglier output for code produced by decl macro expansions. With this change, AST pretty printing and `print_tts` produce similar results. The commit also tweaks the comments on `proc_macro::Spacing`. In particular, it refers to "compound tokens" rather than "multi-char operators" because lifetimes aren't operators.
2023-08-08 11:43:44 +10:00
tokenstream::TokenTree::Token(token, spacing) => {
// Do not be tempted to check here that the `spacing`
// values are "correct" w.r.t. the token stream (e.g. that
// `Spacing::Joint` is actually followed by a `Punct` token
// tree). Because the problem in #76399 was introduced that
// way.
//
// This is where the `Hidden` in `JointHidden` applies,
// because the jointness is effectively hidden from proc
// macros.
let joint = match spacing {
Spacing::Alone | Spacing::JointHidden => false,
Spacing::Joint => true,
};
(token, joint)
}
};
// Split the operator into one or more `Punct`s, one per character.
// The final one inherits the jointness of the original token. Any
// before that get `joint = true`.
let mut op = |s: &str| {
assert!(s.is_ascii());
trees.extend(s.bytes().enumerate().map(|(i, ch)| {
let is_final = i == s.len() - 1;
// Split the token span into single chars. Unless the span
// is an unusual one, e.g. due to proc macro expansion. We
// determine this by assuming any span with a length that
// matches the operator length is a normal one, and any
// span with a different length is an unusual one.
let span = if (span.hi() - span.lo()).to_usize() == s.len() {
let lo = span.lo() + BytePos::from_usize(i);
let hi = lo + BytePos::from_usize(1);
span.with_lo(lo).with_hi(hi)
} else {
span
};
Improve `print_tts` by changing `tokenstream::Spacing`. `tokenstream::Spacing` appears on all `TokenTree::Token` instances, both punct and non-punct. Its current usage: - `Joint` means "can join with the next token *and* that token is a punct". - `Alone` means "cannot join with the next token *or* can join with the next token but that token is not a punct". The fact that `Alone` is used for two different cases is awkward. This commit augments `tokenstream::Spacing` with a new variant `JointHidden`, resulting in: - `Joint` means "can join with the next token *and* that token is a punct". - `JointHidden` means "can join with the next token *and* that token is a not a punct". - `Alone` means "cannot join with the next token". This *drastically* improves the output of `print_tts`. For example, this: ``` stringify!(let a: Vec<u32> = vec![];) ``` currently produces this string: ``` let a : Vec < u32 > = vec! [] ; ``` With this PR, it now produces this string: ``` let a: Vec<u32> = vec![] ; ``` (The space after the `]` is because `TokenTree::Delimited` currently doesn't have spacing information. The subsequent commit fixes this.) The new `print_tts` doesn't replicate original code perfectly. E.g. multiple space characters will be condensed into a single space character. But it's much improved. `print_tts` still produces the old, uglier output for code produced by proc macros. Because we have to translate the generated code from `proc_macro::Spacing` to the more expressive `token::Spacing`, which results in too much `proc_macro::Along` usage and no `proc_macro::JointHidden` usage. So `space_between` still exists and is used by `print_tts` in conjunction with the `Spacing` field. This change will also help with the removal of `Token::Interpolated`. Currently interpolated tokens are pretty-printed nicely via AST pretty printing. `Token::Interpolated` removal will mean they get printed with `print_tts`. Without this change, that would result in much uglier output for code produced by decl macro expansions. With this change, AST pretty printing and `print_tts` produce similar results. The commit also tweaks the comments on `proc_macro::Spacing`. In particular, it refers to "compound tokens" rather than "multi-char operators" because lifetimes aren't operators.
2023-08-08 11:43:44 +10:00
let joint = if is_final { joint } else { true };
TokenTree::Punct(Punct { ch, joint, span })
}));
};
match kind {
Eq => op("="),
Lt => op("<"),
Le => op("<="),
EqEq => op("=="),
Ne => op("!="),
Ge => op(">="),
Gt => op(">"),
AndAnd => op("&&"),
OrOr => op("||"),
Not => op("!"),
Tilde => op("~"),
BinOp(Plus) => op("+"),
BinOp(Minus) => op("-"),
BinOp(Star) => op("*"),
BinOp(Slash) => op("/"),
BinOp(Percent) => op("%"),
BinOp(Caret) => op("^"),
BinOp(And) => op("&"),
BinOp(Or) => op("|"),
BinOp(Shl) => op("<<"),
BinOp(Shr) => op(">>"),
BinOpEq(Plus) => op("+="),
BinOpEq(Minus) => op("-="),
BinOpEq(Star) => op("*="),
BinOpEq(Slash) => op("/="),
BinOpEq(Percent) => op("%="),
BinOpEq(Caret) => op("^="),
BinOpEq(And) => op("&="),
BinOpEq(Or) => op("|="),
BinOpEq(Shl) => op("<<="),
BinOpEq(Shr) => op(">>="),
At => op("@"),
Dot => op("."),
DotDot => op(".."),
DotDotDot => op("..."),
DotDotEq => op("..="),
Comma => op(","),
Semi => op(";"),
Colon => op(":"),
2024-04-04 19:03:32 +02:00
PathSep => op("::"),
RArrow => op("->"),
LArrow => op("<-"),
FatArrow => op("=>"),
Pound => op("#"),
Dollar => op("$"),
Question => op("?"),
SingleQuote => op("'"),
2024-02-13 23:28:27 +00:00
Ident(sym, is_raw) => {
trees.push(TokenTree::Ident(Ident { sym, is_raw: is_raw.into(), span }))
}
NtIdent(ident, is_raw) => trees.push(TokenTree::Ident(Ident {
sym: ident.name,
is_raw: is_raw.into(),
span: ident.span,
})),
2024-09-05 05:43:55 -04:00
Lifetime(name, is_raw) => {
let ident = symbol::Ident::new(name, span).without_first_quote();
trees.extend([
TokenTree::Punct(Punct { ch: b'\'', joint: true, span }),
2024-09-05 05:43:55 -04:00
TokenTree::Ident(Ident { sym: ident.name, is_raw: is_raw.into(), span }),
]);
}
2024-09-05 05:43:55 -04:00
NtLifetime(ident, is_raw) => {
let stream =
TokenStream::token_alone(token::Lifetime(ident.name, is_raw), ident.span);
trees.push(TokenTree::Group(Group {
delimiter: pm::Delimiter::None,
stream: Some(stream),
span: DelimSpan::from_single(span),
}))
}
Literal(token::Lit { kind, symbol, suffix }) => {
trees.push(TokenTree::Literal(self::Literal {
kind: FromInternal::from_internal(kind),
symbol,
suffix,
span,
}));
}
DocComment(_, attr_style, data) => {
let mut escaped = String::new();
for ch in data.as_str().chars() {
escaped.extend(ch.escape_debug());
}
let stream = [
2024-02-13 23:28:27 +00:00
Ident(sym::doc, IdentIsRaw::No),
Eq,
TokenKind::lit(token::Str, Symbol::intern(&escaped), None),
]
.into_iter()
.map(|kind| tokenstream::TokenTree::token_alone(kind, span))
.collect();
trees.push(TokenTree::Punct(Punct { ch: b'#', joint: false, span }));
if attr_style == ast::AttrStyle::Inner {
trees.push(TokenTree::Punct(Punct { ch: b'!', joint: false, span }));
}
trees.push(TokenTree::Group(Group {
delimiter: pm::Delimiter::Bracket,
stream: Some(stream),
span: DelimSpan::from_single(span),
}));
}
Interpolated(nt) => {
let stream = TokenStream::from_nonterminal_ast(&nt);
// We used to have an alternative behaviour for crates that
// needed it: a hack used to pass AST fragments to
// attribute and derive macros as a single nonterminal
// token instead of a token stream. Such token needs to be
// "unwrapped" and not represented as a delimited group. We
// had a lint for a long time, but now we just emit a hard
// error. Eventually we might remove the special case hard
// error check altogether. See #73345.
crate::base::nt_pretty_printing_compatibility_hack(&nt, rustc.ecx.sess);
trees.push(TokenTree::Group(Group {
delimiter: pm::Delimiter::None,
stream: Some(stream),
span: DelimSpan::from_single(span),
}))
}
OpenDelim(..) | CloseDelim(..) => unreachable!(),
Eof => unreachable!(),
}
}
trees
}
}
Remove `TokenStreamBuilder`. `TokenStreamBuilder` exists to concatenate multiple `TokenStream`s together. This commit removes it, and moves the concatenation functionality directly into `TokenStream`, via two new methods `push_tree` and `push_stream`. This makes things both simpler and faster. `push_tree` is particularly important. `TokenStreamBuilder` only had a single `push` method, which pushed a stream. But in practice most of the time we push a single token tree rather than a stream, and `push_tree` avoids the need to build a token stream with a single entry (which requires two allocations, one for the `Lrc` and one for the `Vec`). The main `push_tree` use arises from a change to one of the `ToInternal` impls in `proc_macro_server.rs`. It now returns a `SmallVec` instead of a `TokenStream`. This return value is then iterated over by `concat_trees`, which does `push_tree` on each element. Furthermore, the use of `SmallVec` avoids more allocations, because there is always only one or two token trees. Note: the removed `TokenStreamBuilder::push` method had some code to deal with a quadratic blowup case from #57735. This commit removes the code. I tried and failed to reproduce the blowup from that PR, before and after this change. Various other changes have happened to `TokenStreamBuilder` in the meantime, so I suspect the original problem is no longer relevant, though I don't have proof of this. Generally speaking, repeatedly extending a `Vec` without pre-determining its capacity is *not* quadratic. It's also incredibly common, within rustc and many other Rust programs, so if there were performance problems there you'd think it would show up in other places, too.
2022-10-05 10:38:15 +11:00
// We use a `SmallVec` because the output size is always one or two `TokenTree`s.
impl ToInternal<SmallVec<[tokenstream::TokenTree; 2]>>
for (TokenTree<TokenStream, Span, Symbol>, &mut Rustc<'_, '_>)
{
fn to_internal(self) -> SmallVec<[tokenstream::TokenTree; 2]> {
use rustc_ast::token::*;
// The code below is conservative, using `token_alone`/`Spacing::Alone`
2024-05-17 09:54:53 +10:00
// in most places. It's hard in general to do better when working at
// the token level. When the resulting code is pretty-printed by
// `print_tts` the `space_between` function helps avoid a lot of
// unnecessary whitespace, so the results aren't too bad.
let (tree, rustc) = self;
match tree {
TokenTree::Punct(Punct { ch, joint, span }) => {
let kind = match ch {
b'=' => Eq,
b'<' => Lt,
b'>' => Gt,
b'!' => Not,
b'~' => Tilde,
b'+' => BinOp(Plus),
b'-' => BinOp(Minus),
b'*' => BinOp(Star),
b'/' => BinOp(Slash),
b'%' => BinOp(Percent),
b'^' => BinOp(Caret),
b'&' => BinOp(And),
b'|' => BinOp(Or),
b'@' => At,
b'.' => Dot,
b',' => Comma,
b';' => Semi,
b':' => Colon,
b'#' => Pound,
b'$' => Dollar,
b'?' => Question,
b'\'' => SingleQuote,
_ => unreachable!(),
};
Improve `print_tts` by changing `tokenstream::Spacing`. `tokenstream::Spacing` appears on all `TokenTree::Token` instances, both punct and non-punct. Its current usage: - `Joint` means "can join with the next token *and* that token is a punct". - `Alone` means "cannot join with the next token *or* can join with the next token but that token is not a punct". The fact that `Alone` is used for two different cases is awkward. This commit augments `tokenstream::Spacing` with a new variant `JointHidden`, resulting in: - `Joint` means "can join with the next token *and* that token is a punct". - `JointHidden` means "can join with the next token *and* that token is a not a punct". - `Alone` means "cannot join with the next token". This *drastically* improves the output of `print_tts`. For example, this: ``` stringify!(let a: Vec<u32> = vec![];) ``` currently produces this string: ``` let a : Vec < u32 > = vec! [] ; ``` With this PR, it now produces this string: ``` let a: Vec<u32> = vec![] ; ``` (The space after the `]` is because `TokenTree::Delimited` currently doesn't have spacing information. The subsequent commit fixes this.) The new `print_tts` doesn't replicate original code perfectly. E.g. multiple space characters will be condensed into a single space character. But it's much improved. `print_tts` still produces the old, uglier output for code produced by proc macros. Because we have to translate the generated code from `proc_macro::Spacing` to the more expressive `token::Spacing`, which results in too much `proc_macro::Along` usage and no `proc_macro::JointHidden` usage. So `space_between` still exists and is used by `print_tts` in conjunction with the `Spacing` field. This change will also help with the removal of `Token::Interpolated`. Currently interpolated tokens are pretty-printed nicely via AST pretty printing. `Token::Interpolated` removal will mean they get printed with `print_tts`. Without this change, that would result in much uglier output for code produced by decl macro expansions. With this change, AST pretty printing and `print_tts` produce similar results. The commit also tweaks the comments on `proc_macro::Spacing`. In particular, it refers to "compound tokens" rather than "multi-char operators" because lifetimes aren't operators.
2023-08-08 11:43:44 +10:00
// We never produce `token::Spacing::JointHidden` here, which
// means the pretty-printing of code produced by proc macros is
// ugly, with lots of whitespace between tokens. This is
// unavoidable because `proc_macro::Spacing` only applies to
// `Punct` token trees.
Remove `TokenStreamBuilder`. `TokenStreamBuilder` exists to concatenate multiple `TokenStream`s together. This commit removes it, and moves the concatenation functionality directly into `TokenStream`, via two new methods `push_tree` and `push_stream`. This makes things both simpler and faster. `push_tree` is particularly important. `TokenStreamBuilder` only had a single `push` method, which pushed a stream. But in practice most of the time we push a single token tree rather than a stream, and `push_tree` avoids the need to build a token stream with a single entry (which requires two allocations, one for the `Lrc` and one for the `Vec`). The main `push_tree` use arises from a change to one of the `ToInternal` impls in `proc_macro_server.rs`. It now returns a `SmallVec` instead of a `TokenStream`. This return value is then iterated over by `concat_trees`, which does `push_tree` on each element. Furthermore, the use of `SmallVec` avoids more allocations, because there is always only one or two token trees. Note: the removed `TokenStreamBuilder::push` method had some code to deal with a quadratic blowup case from #57735. This commit removes the code. I tried and failed to reproduce the blowup from that PR, before and after this change. Various other changes have happened to `TokenStreamBuilder` in the meantime, so I suspect the original problem is no longer relevant, though I don't have proof of this. Generally speaking, repeatedly extending a `Vec` without pre-determining its capacity is *not* quadratic. It's also incredibly common, within rustc and many other Rust programs, so if there were performance problems there you'd think it would show up in other places, too.
2022-10-05 10:38:15 +11:00
smallvec![if joint {
tokenstream::TokenTree::token_joint(kind, span)
} else {
Remove `TokenStreamBuilder`. `TokenStreamBuilder` exists to concatenate multiple `TokenStream`s together. This commit removes it, and moves the concatenation functionality directly into `TokenStream`, via two new methods `push_tree` and `push_stream`. This makes things both simpler and faster. `push_tree` is particularly important. `TokenStreamBuilder` only had a single `push` method, which pushed a stream. But in practice most of the time we push a single token tree rather than a stream, and `push_tree` avoids the need to build a token stream with a single entry (which requires two allocations, one for the `Lrc` and one for the `Vec`). The main `push_tree` use arises from a change to one of the `ToInternal` impls in `proc_macro_server.rs`. It now returns a `SmallVec` instead of a `TokenStream`. This return value is then iterated over by `concat_trees`, which does `push_tree` on each element. Furthermore, the use of `SmallVec` avoids more allocations, because there is always only one or two token trees. Note: the removed `TokenStreamBuilder::push` method had some code to deal with a quadratic blowup case from #57735. This commit removes the code. I tried and failed to reproduce the blowup from that PR, before and after this change. Various other changes have happened to `TokenStreamBuilder` in the meantime, so I suspect the original problem is no longer relevant, though I don't have proof of this. Generally speaking, repeatedly extending a `Vec` without pre-determining its capacity is *not* quadratic. It's also incredibly common, within rustc and many other Rust programs, so if there were performance problems there you'd think it would show up in other places, too.
2022-10-05 10:38:15 +11:00
tokenstream::TokenTree::token_alone(kind, span)
}]
}
TokenTree::Group(Group { delimiter, stream, span: DelimSpan { open, close, .. } }) => {
Remove `TokenStreamBuilder`. `TokenStreamBuilder` exists to concatenate multiple `TokenStream`s together. This commit removes it, and moves the concatenation functionality directly into `TokenStream`, via two new methods `push_tree` and `push_stream`. This makes things both simpler and faster. `push_tree` is particularly important. `TokenStreamBuilder` only had a single `push` method, which pushed a stream. But in practice most of the time we push a single token tree rather than a stream, and `push_tree` avoids the need to build a token stream with a single entry (which requires two allocations, one for the `Lrc` and one for the `Vec`). The main `push_tree` use arises from a change to one of the `ToInternal` impls in `proc_macro_server.rs`. It now returns a `SmallVec` instead of a `TokenStream`. This return value is then iterated over by `concat_trees`, which does `push_tree` on each element. Furthermore, the use of `SmallVec` avoids more allocations, because there is always only one or two token trees. Note: the removed `TokenStreamBuilder::push` method had some code to deal with a quadratic blowup case from #57735. This commit removes the code. I tried and failed to reproduce the blowup from that PR, before and after this change. Various other changes have happened to `TokenStreamBuilder` in the meantime, so I suspect the original problem is no longer relevant, though I don't have proof of this. Generally speaking, repeatedly extending a `Vec` without pre-determining its capacity is *not* quadratic. It's also incredibly common, within rustc and many other Rust programs, so if there were performance problems there you'd think it would show up in other places, too.
2022-10-05 10:38:15 +11:00
smallvec![tokenstream::TokenTree::Delimited(
tokenstream::DelimSpan { open, close },
DelimSpacing::new(Spacing::Alone, Spacing::Alone),
delimiter.to_internal(),
stream.unwrap_or_default(),
Remove `TokenStreamBuilder`. `TokenStreamBuilder` exists to concatenate multiple `TokenStream`s together. This commit removes it, and moves the concatenation functionality directly into `TokenStream`, via two new methods `push_tree` and `push_stream`. This makes things both simpler and faster. `push_tree` is particularly important. `TokenStreamBuilder` only had a single `push` method, which pushed a stream. But in practice most of the time we push a single token tree rather than a stream, and `push_tree` avoids the need to build a token stream with a single entry (which requires two allocations, one for the `Lrc` and one for the `Vec`). The main `push_tree` use arises from a change to one of the `ToInternal` impls in `proc_macro_server.rs`. It now returns a `SmallVec` instead of a `TokenStream`. This return value is then iterated over by `concat_trees`, which does `push_tree` on each element. Furthermore, the use of `SmallVec` avoids more allocations, because there is always only one or two token trees. Note: the removed `TokenStreamBuilder::push` method had some code to deal with a quadratic blowup case from #57735. This commit removes the code. I tried and failed to reproduce the blowup from that PR, before and after this change. Various other changes have happened to `TokenStreamBuilder` in the meantime, so I suspect the original problem is no longer relevant, though I don't have proof of this. Generally speaking, repeatedly extending a `Vec` without pre-determining its capacity is *not* quadratic. It's also incredibly common, within rustc and many other Rust programs, so if there were performance problems there you'd think it would show up in other places, too.
2022-10-05 10:38:15 +11:00
)]
}
TokenTree::Ident(self::Ident { sym, is_raw, span }) => {
rustc.psess().symbol_gallery.insert(sym, span);
2024-02-13 23:28:27 +00:00
smallvec![tokenstream::TokenTree::token_alone(Ident(sym, is_raw.into()), span)]
}
TokenTree::Literal(self::Literal {
kind: self::LitKind::Integer,
symbol,
suffix,
span,
}) if symbol.as_str().starts_with('-') => {
let minus = BinOp(BinOpToken::Minus);
let symbol = Symbol::intern(&symbol.as_str()[1..]);
let integer = TokenKind::lit(token::Integer, symbol, suffix);
Improve `print_tts` by changing `tokenstream::Spacing`. `tokenstream::Spacing` appears on all `TokenTree::Token` instances, both punct and non-punct. Its current usage: - `Joint` means "can join with the next token *and* that token is a punct". - `Alone` means "cannot join with the next token *or* can join with the next token but that token is not a punct". The fact that `Alone` is used for two different cases is awkward. This commit augments `tokenstream::Spacing` with a new variant `JointHidden`, resulting in: - `Joint` means "can join with the next token *and* that token is a punct". - `JointHidden` means "can join with the next token *and* that token is a not a punct". - `Alone` means "cannot join with the next token". This *drastically* improves the output of `print_tts`. For example, this: ``` stringify!(let a: Vec<u32> = vec![];) ``` currently produces this string: ``` let a : Vec < u32 > = vec! [] ; ``` With this PR, it now produces this string: ``` let a: Vec<u32> = vec![] ; ``` (The space after the `]` is because `TokenTree::Delimited` currently doesn't have spacing information. The subsequent commit fixes this.) The new `print_tts` doesn't replicate original code perfectly. E.g. multiple space characters will be condensed into a single space character. But it's much improved. `print_tts` still produces the old, uglier output for code produced by proc macros. Because we have to translate the generated code from `proc_macro::Spacing` to the more expressive `token::Spacing`, which results in too much `proc_macro::Along` usage and no `proc_macro::JointHidden` usage. So `space_between` still exists and is used by `print_tts` in conjunction with the `Spacing` field. This change will also help with the removal of `Token::Interpolated`. Currently interpolated tokens are pretty-printed nicely via AST pretty printing. `Token::Interpolated` removal will mean they get printed with `print_tts`. Without this change, that would result in much uglier output for code produced by decl macro expansions. With this change, AST pretty printing and `print_tts` produce similar results. The commit also tweaks the comments on `proc_macro::Spacing`. In particular, it refers to "compound tokens" rather than "multi-char operators" because lifetimes aren't operators.
2023-08-08 11:43:44 +10:00
let a = tokenstream::TokenTree::token_joint_hidden(minus, span);
let b = tokenstream::TokenTree::token_alone(integer, span);
Remove `TokenStreamBuilder`. `TokenStreamBuilder` exists to concatenate multiple `TokenStream`s together. This commit removes it, and moves the concatenation functionality directly into `TokenStream`, via two new methods `push_tree` and `push_stream`. This makes things both simpler and faster. `push_tree` is particularly important. `TokenStreamBuilder` only had a single `push` method, which pushed a stream. But in practice most of the time we push a single token tree rather than a stream, and `push_tree` avoids the need to build a token stream with a single entry (which requires two allocations, one for the `Lrc` and one for the `Vec`). The main `push_tree` use arises from a change to one of the `ToInternal` impls in `proc_macro_server.rs`. It now returns a `SmallVec` instead of a `TokenStream`. This return value is then iterated over by `concat_trees`, which does `push_tree` on each element. Furthermore, the use of `SmallVec` avoids more allocations, because there is always only one or two token trees. Note: the removed `TokenStreamBuilder::push` method had some code to deal with a quadratic blowup case from #57735. This commit removes the code. I tried and failed to reproduce the blowup from that PR, before and after this change. Various other changes have happened to `TokenStreamBuilder` in the meantime, so I suspect the original problem is no longer relevant, though I don't have proof of this. Generally speaking, repeatedly extending a `Vec` without pre-determining its capacity is *not* quadratic. It's also incredibly common, within rustc and many other Rust programs, so if there were performance problems there you'd think it would show up in other places, too.
2022-10-05 10:38:15 +11:00
smallvec![a, b]
}
TokenTree::Literal(self::Literal {
kind: self::LitKind::Float,
symbol,
suffix,
span,
}) if symbol.as_str().starts_with('-') => {
let minus = BinOp(BinOpToken::Minus);
let symbol = Symbol::intern(&symbol.as_str()[1..]);
let float = TokenKind::lit(token::Float, symbol, suffix);
Improve `print_tts` by changing `tokenstream::Spacing`. `tokenstream::Spacing` appears on all `TokenTree::Token` instances, both punct and non-punct. Its current usage: - `Joint` means "can join with the next token *and* that token is a punct". - `Alone` means "cannot join with the next token *or* can join with the next token but that token is not a punct". The fact that `Alone` is used for two different cases is awkward. This commit augments `tokenstream::Spacing` with a new variant `JointHidden`, resulting in: - `Joint` means "can join with the next token *and* that token is a punct". - `JointHidden` means "can join with the next token *and* that token is a not a punct". - `Alone` means "cannot join with the next token". This *drastically* improves the output of `print_tts`. For example, this: ``` stringify!(let a: Vec<u32> = vec![];) ``` currently produces this string: ``` let a : Vec < u32 > = vec! [] ; ``` With this PR, it now produces this string: ``` let a: Vec<u32> = vec![] ; ``` (The space after the `]` is because `TokenTree::Delimited` currently doesn't have spacing information. The subsequent commit fixes this.) The new `print_tts` doesn't replicate original code perfectly. E.g. multiple space characters will be condensed into a single space character. But it's much improved. `print_tts` still produces the old, uglier output for code produced by proc macros. Because we have to translate the generated code from `proc_macro::Spacing` to the more expressive `token::Spacing`, which results in too much `proc_macro::Along` usage and no `proc_macro::JointHidden` usage. So `space_between` still exists and is used by `print_tts` in conjunction with the `Spacing` field. This change will also help with the removal of `Token::Interpolated`. Currently interpolated tokens are pretty-printed nicely via AST pretty printing. `Token::Interpolated` removal will mean they get printed with `print_tts`. Without this change, that would result in much uglier output for code produced by decl macro expansions. With this change, AST pretty printing and `print_tts` produce similar results. The commit also tweaks the comments on `proc_macro::Spacing`. In particular, it refers to "compound tokens" rather than "multi-char operators" because lifetimes aren't operators.
2023-08-08 11:43:44 +10:00
let a = tokenstream::TokenTree::token_joint_hidden(minus, span);
let b = tokenstream::TokenTree::token_alone(float, span);
Remove `TokenStreamBuilder`. `TokenStreamBuilder` exists to concatenate multiple `TokenStream`s together. This commit removes it, and moves the concatenation functionality directly into `TokenStream`, via two new methods `push_tree` and `push_stream`. This makes things both simpler and faster. `push_tree` is particularly important. `TokenStreamBuilder` only had a single `push` method, which pushed a stream. But in practice most of the time we push a single token tree rather than a stream, and `push_tree` avoids the need to build a token stream with a single entry (which requires two allocations, one for the `Lrc` and one for the `Vec`). The main `push_tree` use arises from a change to one of the `ToInternal` impls in `proc_macro_server.rs`. It now returns a `SmallVec` instead of a `TokenStream`. This return value is then iterated over by `concat_trees`, which does `push_tree` on each element. Furthermore, the use of `SmallVec` avoids more allocations, because there is always only one or two token trees. Note: the removed `TokenStreamBuilder::push` method had some code to deal with a quadratic blowup case from #57735. This commit removes the code. I tried and failed to reproduce the blowup from that PR, before and after this change. Various other changes have happened to `TokenStreamBuilder` in the meantime, so I suspect the original problem is no longer relevant, though I don't have proof of this. Generally speaking, repeatedly extending a `Vec` without pre-determining its capacity is *not* quadratic. It's also incredibly common, within rustc and many other Rust programs, so if there were performance problems there you'd think it would show up in other places, too.
2022-10-05 10:38:15 +11:00
smallvec![a, b]
}
TokenTree::Literal(self::Literal { kind, symbol, suffix, span }) => {
Remove `TokenStreamBuilder`. `TokenStreamBuilder` exists to concatenate multiple `TokenStream`s together. This commit removes it, and moves the concatenation functionality directly into `TokenStream`, via two new methods `push_tree` and `push_stream`. This makes things both simpler and faster. `push_tree` is particularly important. `TokenStreamBuilder` only had a single `push` method, which pushed a stream. But in practice most of the time we push a single token tree rather than a stream, and `push_tree` avoids the need to build a token stream with a single entry (which requires two allocations, one for the `Lrc` and one for the `Vec`). The main `push_tree` use arises from a change to one of the `ToInternal` impls in `proc_macro_server.rs`. It now returns a `SmallVec` instead of a `TokenStream`. This return value is then iterated over by `concat_trees`, which does `push_tree` on each element. Furthermore, the use of `SmallVec` avoids more allocations, because there is always only one or two token trees. Note: the removed `TokenStreamBuilder::push` method had some code to deal with a quadratic blowup case from #57735. This commit removes the code. I tried and failed to reproduce the blowup from that PR, before and after this change. Various other changes have happened to `TokenStreamBuilder` in the meantime, so I suspect the original problem is no longer relevant, though I don't have proof of this. Generally speaking, repeatedly extending a `Vec` without pre-determining its capacity is *not* quadratic. It's also incredibly common, within rustc and many other Rust programs, so if there were performance problems there you'd think it would show up in other places, too.
2022-10-05 10:38:15 +11:00
smallvec![tokenstream::TokenTree::token_alone(
TokenKind::lit(kind.to_internal(), symbol, suffix),
span,
Remove `TokenStreamBuilder`. `TokenStreamBuilder` exists to concatenate multiple `TokenStream`s together. This commit removes it, and moves the concatenation functionality directly into `TokenStream`, via two new methods `push_tree` and `push_stream`. This makes things both simpler and faster. `push_tree` is particularly important. `TokenStreamBuilder` only had a single `push` method, which pushed a stream. But in practice most of the time we push a single token tree rather than a stream, and `push_tree` avoids the need to build a token stream with a single entry (which requires two allocations, one for the `Lrc` and one for the `Vec`). The main `push_tree` use arises from a change to one of the `ToInternal` impls in `proc_macro_server.rs`. It now returns a `SmallVec` instead of a `TokenStream`. This return value is then iterated over by `concat_trees`, which does `push_tree` on each element. Furthermore, the use of `SmallVec` avoids more allocations, because there is always only one or two token trees. Note: the removed `TokenStreamBuilder::push` method had some code to deal with a quadratic blowup case from #57735. This commit removes the code. I tried and failed to reproduce the blowup from that PR, before and after this change. Various other changes have happened to `TokenStreamBuilder` in the meantime, so I suspect the original problem is no longer relevant, though I don't have proof of this. Generally speaking, repeatedly extending a `Vec` without pre-determining its capacity is *not* quadratic. It's also incredibly common, within rustc and many other Rust programs, so if there were performance problems there you'd think it would show up in other places, too.
2022-10-05 10:38:15 +11:00
)]
}
}
}
}
impl ToInternal<rustc_errors::Level> for Level {
fn to_internal(self) -> rustc_errors::Level {
match self {
Level::Error => rustc_errors::Level::Error,
Level::Warning => rustc_errors::Level::Warning,
Level::Note => rustc_errors::Level::Note,
Level::Help => rustc_errors::Level::Help,
_ => unreachable!("unknown proc_macro::Level variant: {:?}", self),
}
}
}
pub(crate) struct FreeFunctions;
pub(crate) struct Rustc<'a, 'b> {
ecx: &'a mut ExtCtxt<'b>,
def_site: Span,
call_site: Span,
mixed_site: Span,
Implement span quoting for proc-macros This PR implements span quoting, allowing proc-macros to produce spans pointing *into their own crate*. This is used by the unstable `proc_macro::quote!` macro, allowing us to get error messages like this: ``` error[E0412]: cannot find type `MissingType` in this scope --> $DIR/auxiliary/span-from-proc-macro.rs:37:20 | LL | pub fn error_from_attribute(_args: TokenStream, _input: TokenStream) -> TokenStream { | ----------------------------------------------------------------------------------- in this expansion of procedural macro `#[error_from_attribute]` ... LL | field: MissingType | ^^^^^^^^^^^ not found in this scope | ::: $DIR/span-from-proc-macro.rs:8:1 | LL | #[error_from_attribute] | ----------------------- in this macro invocation ``` Here, `MissingType` occurs inside the implementation of the proc-macro `#[error_from_attribute]`. Previosuly, this would always result in a span pointing at `#[error_from_attribute]` This will make many proc-macro-related error message much more useful - when a proc-macro generates code containing an error, users will get an error message pointing directly at that code (within the macro definition), instead of always getting a span pointing at the macro invocation site. This is implemented as follows: * When a proc-macro crate is being *compiled*, it causes the `quote!` macro to get run. This saves all of the sapns in the input to `quote!` into the metadata of *the proc-macro-crate* (which we are currently compiling). The `quote!` macro then expands to a call to `proc_macro::Span::recover_proc_macro_span(id)`, where `id` is an opaque identifier for the span in the crate metadata. * When the same proc-macro crate is *run* (e.g. it is loaded from disk and invoked by some consumer crate), the call to `proc_macro::Span::recover_proc_macro_span` causes us to load the span from the proc-macro crate's metadata. The proc-macro then produces a `TokenStream` containing a `Span` pointing into the proc-macro crate itself. The recursive nature of 'quote!' can be difficult to understand at first. The file `src/test/ui/proc-macro/quote-debug.stdout` shows the output of the `quote!` macro, which should make this eaier to understand. This PR also supports custom quoting spans in custom quote macros (e.g. the `quote` crate). All span quoting goes through the `proc_macro::quote_span` method, which can be called by a custom quote macro to perform span quoting. An example of this usage is provided in `src/test/ui/proc-macro/auxiliary/custom-quote.rs` Custom quoting currently has a few limitations: In order to quote a span, we need to generate a call to `proc_macro::Span::recover_proc_macro_span`. However, proc-macros support renaming the `proc_macro` crate, so we can't simply hardcode this path. Previously, the `quote_span` method used the path `crate::Span` - however, this only works when it is called by the builtin `quote!` macro in the same crate. To support being called from arbitrary crates, we need access to the name of the `proc_macro` crate to generate a path. This PR adds an additional argument to `quote_span` to specify the name of the `proc_macro` crate. Howver, this feels kind of hacky, and we may want to change this before stabilizing anything quote-related. Additionally, using `quote_span` currently requires enabling the `proc_macro_internals` feature. The builtin `quote!` macro has an `#[allow_internal_unstable]` attribute, but this won't work for custom quote implementations. This will likely require some additional tricks to apply `allow_internal_unstable` to the span of `proc_macro::Span::recover_proc_macro_span`.
2020-08-02 19:52:16 -04:00
krate: CrateNum,
rebased_spans: FxHashMap<usize, Span>,
}
impl<'a, 'b> Rustc<'a, 'b> {
pub(crate) fn new(ecx: &'a mut ExtCtxt<'b>) -> Self {
let expn_data = ecx.current_expansion.id.expn_data();
Rustc {
def_site: ecx.with_def_site_ctxt(expn_data.def_site),
call_site: ecx.with_call_site_ctxt(expn_data.call_site),
mixed_site: ecx.with_mixed_site_ctxt(expn_data.call_site),
krate: expn_data.macro_def_id.unwrap().krate,
Implement span quoting for proc-macros This PR implements span quoting, allowing proc-macros to produce spans pointing *into their own crate*. This is used by the unstable `proc_macro::quote!` macro, allowing us to get error messages like this: ``` error[E0412]: cannot find type `MissingType` in this scope --> $DIR/auxiliary/span-from-proc-macro.rs:37:20 | LL | pub fn error_from_attribute(_args: TokenStream, _input: TokenStream) -> TokenStream { | ----------------------------------------------------------------------------------- in this expansion of procedural macro `#[error_from_attribute]` ... LL | field: MissingType | ^^^^^^^^^^^ not found in this scope | ::: $DIR/span-from-proc-macro.rs:8:1 | LL | #[error_from_attribute] | ----------------------- in this macro invocation ``` Here, `MissingType` occurs inside the implementation of the proc-macro `#[error_from_attribute]`. Previosuly, this would always result in a span pointing at `#[error_from_attribute]` This will make many proc-macro-related error message much more useful - when a proc-macro generates code containing an error, users will get an error message pointing directly at that code (within the macro definition), instead of always getting a span pointing at the macro invocation site. This is implemented as follows: * When a proc-macro crate is being *compiled*, it causes the `quote!` macro to get run. This saves all of the sapns in the input to `quote!` into the metadata of *the proc-macro-crate* (which we are currently compiling). The `quote!` macro then expands to a call to `proc_macro::Span::recover_proc_macro_span(id)`, where `id` is an opaque identifier for the span in the crate metadata. * When the same proc-macro crate is *run* (e.g. it is loaded from disk and invoked by some consumer crate), the call to `proc_macro::Span::recover_proc_macro_span` causes us to load the span from the proc-macro crate's metadata. The proc-macro then produces a `TokenStream` containing a `Span` pointing into the proc-macro crate itself. The recursive nature of 'quote!' can be difficult to understand at first. The file `src/test/ui/proc-macro/quote-debug.stdout` shows the output of the `quote!` macro, which should make this eaier to understand. This PR also supports custom quoting spans in custom quote macros (e.g. the `quote` crate). All span quoting goes through the `proc_macro::quote_span` method, which can be called by a custom quote macro to perform span quoting. An example of this usage is provided in `src/test/ui/proc-macro/auxiliary/custom-quote.rs` Custom quoting currently has a few limitations: In order to quote a span, we need to generate a call to `proc_macro::Span::recover_proc_macro_span`. However, proc-macros support renaming the `proc_macro` crate, so we can't simply hardcode this path. Previously, the `quote_span` method used the path `crate::Span` - however, this only works when it is called by the builtin `quote!` macro in the same crate. To support being called from arbitrary crates, we need access to the name of the `proc_macro` crate to generate a path. This PR adds an additional argument to `quote_span` to specify the name of the `proc_macro` crate. Howver, this feels kind of hacky, and we may want to change this before stabilizing anything quote-related. Additionally, using `quote_span` currently requires enabling the `proc_macro_internals` feature. The builtin `quote!` macro has an `#[allow_internal_unstable]` attribute, but this won't work for custom quote implementations. This will likely require some additional tricks to apply `allow_internal_unstable` to the span of `proc_macro::Span::recover_proc_macro_span`.
2020-08-02 19:52:16 -04:00
rebased_spans: FxHashMap::default(),
ecx,
}
}
fn psess(&self) -> &ParseSess {
self.ecx.psess()
}
}
impl server::Types for Rustc<'_, '_> {
type FreeFunctions = FreeFunctions;
type TokenStream = TokenStream;
type SourceFile = Lrc<SourceFile>;
type Span = Span;
type Symbol = Symbol;
}
impl server::FreeFunctions for Rustc<'_, '_> {
fn injected_env_var(&mut self, var: &str) -> Option<String> {
self.ecx.sess.opts.logical_env.get(var).cloned()
}
fn track_env_var(&mut self, var: &str, value: Option<&str>) {
self.psess()
.env_depinfo
.borrow_mut()
.insert((Symbol::intern(var), value.map(Symbol::intern)));
}
fn track_path(&mut self, path: &str) {
self.psess().file_depinfo.borrow_mut().insert(Symbol::intern(path));
}
fn literal_from_str(&mut self, s: &str) -> Result<Literal<Self::Span, Self::Symbol>, ()> {
let name = FileName::proc_macro_source_code(s);
let mut parser =
unwrap_or_emit_fatal(new_parser_from_source_str(self.psess(), name, s.to_owned()));
let first_span = parser.token.span.data();
let minus_present = parser.eat(&token::BinOp(token::Minus));
let lit_span = parser.token.span.data();
let token::Literal(mut lit) = parser.token.kind else {
return Err(());
};
// Check no comment or whitespace surrounding the (possibly negative)
// literal, or more tokens after it.
if (lit_span.hi.0 - first_span.lo.0) as usize != s.len() {
return Err(());
}
if minus_present {
// If minus is present, check no comment or whitespace in between it
// and the literal token.
if first_span.hi.0 != lit_span.lo.0 {
return Err(());
}
// Check literal is a kind we allow to be negated in a proc macro token.
match lit.kind {
token::LitKind::Bool
| token::LitKind::Byte
| token::LitKind::Char
| token::LitKind::Str
| token::LitKind::StrRaw(_)
| token::LitKind::ByteStr
| token::LitKind::ByteStrRaw(_)
| token::LitKind::CStr
| token::LitKind::CStrRaw(_)
| token::LitKind::Err(_) => return Err(()),
token::LitKind::Integer | token::LitKind::Float => {}
}
// Synthesize a new symbol that includes the minus sign.
let symbol = Symbol::intern(&s[..1 + lit.symbol.as_str().len()]);
lit = token::Lit::new(lit.kind, symbol, lit.suffix);
}
let token::Lit { kind, symbol, suffix } = lit;
Ok(Literal {
kind: FromInternal::from_internal(kind),
symbol,
suffix,
span: self.call_site,
})
}
fn emit_diagnostic(&mut self, diagnostic: Diagnostic<Self::Span>) {
let message = rustc_errors::DiagMessage::from(diagnostic.message);
let mut diag: Diag<'_, ()> =
Diag::new(self.psess().dcx(), diagnostic.level.to_internal(), message);
diag.span(MultiSpan::from_spans(diagnostic.spans));
for child in diagnostic.children {
// This message comes from another diagnostic, and we are just reconstructing the
// diagnostic, so there's no need for translation.
#[allow(rustc::untranslatable_diagnostic)]
diag.sub(child.level.to_internal(), child.message, MultiSpan::from_spans(child.spans));
}
Reduce capabilities of `Diagnostic`. Currently many diagnostic modifier methods are available on both `Diagnostic` and `DiagnosticBuilder`. This commit removes most of them from `Diagnostic`. To minimize the diff size, it keeps them within `diagnostic.rs` but changes the surrounding `impl Diagnostic` block to `impl DiagnosticBuilder`. (I intend to move things around later, to give a more sensible code layout.) `Diagnostic` keeps a few methods that it still needs, like `sub`, `arg`, and `replace_args`. The `forward!` macro, which defined two additional methods per call (e.g. `note` and `with_note`), is replaced by the `with_fn!` macro, which defines one additional method per call (e.g. `with_note`). It's now also only used when necessary -- not all modifier methods currently need a `with_*` form. (New ones can be easily added as necessary.) All this also requires changing `trait AddToDiagnostic` so its methods take `DiagnosticBuilder` instead of `Diagnostic`, which leads to many mechanical changes. `SubdiagnosticMessageOp` gains a type parameter `G`. There are three subdiagnostics -- `DelayedAtWithoutNewline`, `DelayedAtWithNewline`, and `InvalidFlushedDelayedDiagnosticLevel` -- that are created within the diagnostics machinery and appended to external diagnostics. These are handled at the `Diagnostic` level, which means it's now hard to construct them via `derive(Diagnostic)`, so instead we construct them by hand. This has no effect on what they look like when printed. There are lots of new `allow` markers for `untranslatable_diagnostics` and `diagnostics_outside_of_impl`. This is because `#[rustc_lint_diagnostics]` annotations were present on the `Diagnostic` modifier methods, but missing from the `DiagnosticBuilder` modifier methods. They're now present.
2024-02-06 16:44:30 +11:00
diag.emit();
}
}
impl server::TokenStream for Rustc<'_, '_> {
fn is_empty(&mut self, stream: &Self::TokenStream) -> bool {
stream.is_empty()
}
fn from_str(&mut self, src: &str) -> Self::TokenStream {
unwrap_or_emit_fatal(source_str_to_stream(
self.psess(),
2019-06-08 19:06:58 +09:00
FileName::proc_macro_source_code(src),
src.to_string(),
Some(self.call_site),
))
}
fn to_string(&mut self, stream: &Self::TokenStream) -> String {
pprust::tts_to_string(stream)
}
fn expand_expr(&mut self, stream: &Self::TokenStream) -> Result<Self::TokenStream, ()> {
// Parse the expression from our tokenstream.
let expr: PResult<'_, _> = try {
let mut p = Parser::new(self.psess(), stream.clone(), Some("proc_macro expand expr"));
let expr = p.parse_expr()?;
if p.token != token::Eof {
p.unexpected()?;
}
expr
};
Make `DiagnosticBuilder::emit` consuming. This works for most of its call sites. This is nice, because `emit` very much makes sense as a consuming operation -- indeed, `DiagnosticBuilderState` exists to ensure no diagnostic is emitted twice, but it uses runtime checks. For the small number of call sites where a consuming emit doesn't work, the commit adds `DiagnosticBuilder::emit_without_consuming`. (This will be removed in subsequent commits.) Likewise, `emit_unless` becomes consuming. And `delay_as_bug` becomes consuming, while `delay_as_bug_without_consuming` is added (which will also be removed in subsequent commits.) All this requires significant changes to `DiagnosticBuilder`'s chaining methods. Currently `DiagnosticBuilder` method chaining uses a non-consuming `&mut self -> &mut Self` style, which allows chaining to be used when the chain ends in `emit()`, like so: ``` struct_err(msg).span(span).emit(); ``` But it doesn't work when producing a `DiagnosticBuilder` value, requiring this: ``` let mut err = self.struct_err(msg); err.span(span); err ``` This style of chaining won't work with consuming `emit` though. For that, we need to use to a `self -> Self` style. That also would allow `DiagnosticBuilder` production to be chained, e.g.: ``` self.struct_err(msg).span(span) ``` However, removing the `&mut self -> &mut Self` style would require that individual modifications of a `DiagnosticBuilder` go from this: ``` err.span(span); ``` to this: ``` err = err.span(span); ``` There are *many* such places. I have a high tolerance for tedious refactorings, but even I gave up after a long time trying to convert them all. Instead, this commit has it both ways: the existing `&mut self -> Self` chaining methods are kept, and new `self -> Self` chaining methods are added, all of which have a `_mv` suffix (short for "move"). Changes to the existing `forward!` macro lets this happen with very little additional boilerplate code. I chose to add the suffix to the new chaining methods rather than the existing ones, because the number of changes required is much smaller that way. This doubled chainging is a bit clumsy, but I think it is worthwhile because it allows a *lot* of good things to subsequently happen. In this commit, there are many `mut` qualifiers removed in places where diagnostics are emitted without being modified. In subsequent commits: - chaining can be used more, making the code more concise; - more use of chaining also permits the removal of redundant diagnostic APIs like `struct_err_with_code`, which can be replaced easily with `struct_err` + `code_mv`; - `emit_without_diagnostic` can be removed, which simplifies a lot of machinery, removing the need for `DiagnosticBuilderState`.
2024-01-03 12:17:35 +11:00
let expr = expr.map_err(|err| {
err.emit();
})?;
// Perform eager expansion on the expression.
let expr = self
.ecx
.expander()
.fully_expand_fragment(crate::expand::AstFragment::Expr(expr))
.make_expr();
// NOTE: For now, limit `expand_expr` to exclusively expand to literals.
// This may be relaxed in the future.
// We don't use `TokenStream::from_ast` as the tokenstream currently cannot
// be recovered in the general case.
match &expr.kind {
ast::ExprKind::Lit(token_lit) if token_lit.kind == token::Bool => {
Ok(tokenstream::TokenStream::token_alone(
2024-02-13 23:28:27 +00:00
token::Ident(token_lit.symbol, IdentIsRaw::No),
expr.span,
))
}
ast::ExprKind::Lit(token_lit) => {
Ok(tokenstream::TokenStream::token_alone(token::Literal(*token_lit), expr.span))
}
2022-10-31 18:30:09 +00:00
ast::ExprKind::IncludedBytes(bytes) => {
let lit = token::Lit::new(token::ByteStr, escape_byte_str_symbol(bytes), None);
Ok(tokenstream::TokenStream::token_alone(token::TokenKind::Literal(lit), expr.span))
2022-10-31 18:30:09 +00:00
}
ast::ExprKind::Unary(ast::UnOp::Neg, e) => match &e.kind {
ast::ExprKind::Lit(token_lit) => match token_lit {
token::Lit { kind: token::Integer | token::Float, .. } => {
Ok(Self::TokenStream::from_iter([
// FIXME: The span of the `-` token is lost when
// parsing, so we cannot faithfully recover it here.
Improve `print_tts` by changing `tokenstream::Spacing`. `tokenstream::Spacing` appears on all `TokenTree::Token` instances, both punct and non-punct. Its current usage: - `Joint` means "can join with the next token *and* that token is a punct". - `Alone` means "cannot join with the next token *or* can join with the next token but that token is not a punct". The fact that `Alone` is used for two different cases is awkward. This commit augments `tokenstream::Spacing` with a new variant `JointHidden`, resulting in: - `Joint` means "can join with the next token *and* that token is a punct". - `JointHidden` means "can join with the next token *and* that token is a not a punct". - `Alone` means "cannot join with the next token". This *drastically* improves the output of `print_tts`. For example, this: ``` stringify!(let a: Vec<u32> = vec![];) ``` currently produces this string: ``` let a : Vec < u32 > = vec! [] ; ``` With this PR, it now produces this string: ``` let a: Vec<u32> = vec![] ; ``` (The space after the `]` is because `TokenTree::Delimited` currently doesn't have spacing information. The subsequent commit fixes this.) The new `print_tts` doesn't replicate original code perfectly. E.g. multiple space characters will be condensed into a single space character. But it's much improved. `print_tts` still produces the old, uglier output for code produced by proc macros. Because we have to translate the generated code from `proc_macro::Spacing` to the more expressive `token::Spacing`, which results in too much `proc_macro::Along` usage and no `proc_macro::JointHidden` usage. So `space_between` still exists and is used by `print_tts` in conjunction with the `Spacing` field. This change will also help with the removal of `Token::Interpolated`. Currently interpolated tokens are pretty-printed nicely via AST pretty printing. `Token::Interpolated` removal will mean they get printed with `print_tts`. Without this change, that would result in much uglier output for code produced by decl macro expansions. With this change, AST pretty printing and `print_tts` produce similar results. The commit also tweaks the comments on `proc_macro::Spacing`. In particular, it refers to "compound tokens" rather than "multi-char operators" because lifetimes aren't operators.
2023-08-08 11:43:44 +10:00
tokenstream::TokenTree::token_joint_hidden(
token::BinOp(token::Minus),
e.span,
),
tokenstream::TokenTree::token_alone(token::Literal(*token_lit), e.span),
]))
}
_ => Err(()),
},
_ => Err(()),
},
_ => Err(()),
}
}
fn from_token_tree(
&mut self,
tree: TokenTree<Self::TokenStream, Self::Span, Self::Symbol>,
) -> Self::TokenStream {
Remove `TokenStreamBuilder`. `TokenStreamBuilder` exists to concatenate multiple `TokenStream`s together. This commit removes it, and moves the concatenation functionality directly into `TokenStream`, via two new methods `push_tree` and `push_stream`. This makes things both simpler and faster. `push_tree` is particularly important. `TokenStreamBuilder` only had a single `push` method, which pushed a stream. But in practice most of the time we push a single token tree rather than a stream, and `push_tree` avoids the need to build a token stream with a single entry (which requires two allocations, one for the `Lrc` and one for the `Vec`). The main `push_tree` use arises from a change to one of the `ToInternal` impls in `proc_macro_server.rs`. It now returns a `SmallVec` instead of a `TokenStream`. This return value is then iterated over by `concat_trees`, which does `push_tree` on each element. Furthermore, the use of `SmallVec` avoids more allocations, because there is always only one or two token trees. Note: the removed `TokenStreamBuilder::push` method had some code to deal with a quadratic blowup case from #57735. This commit removes the code. I tried and failed to reproduce the blowup from that PR, before and after this change. Various other changes have happened to `TokenStreamBuilder` in the meantime, so I suspect the original problem is no longer relevant, though I don't have proof of this. Generally speaking, repeatedly extending a `Vec` without pre-determining its capacity is *not* quadratic. It's also incredibly common, within rustc and many other Rust programs, so if there were performance problems there you'd think it would show up in other places, too.
2022-10-05 10:38:15 +11:00
Self::TokenStream::new((tree, &mut *self).to_internal().into_iter().collect::<Vec<_>>())
}
fn concat_trees(
&mut self,
base: Option<Self::TokenStream>,
trees: Vec<TokenTree<Self::TokenStream, Self::Span, Self::Symbol>>,
) -> Self::TokenStream {
Remove `TokenStreamBuilder`. `TokenStreamBuilder` exists to concatenate multiple `TokenStream`s together. This commit removes it, and moves the concatenation functionality directly into `TokenStream`, via two new methods `push_tree` and `push_stream`. This makes things both simpler and faster. `push_tree` is particularly important. `TokenStreamBuilder` only had a single `push` method, which pushed a stream. But in practice most of the time we push a single token tree rather than a stream, and `push_tree` avoids the need to build a token stream with a single entry (which requires two allocations, one for the `Lrc` and one for the `Vec`). The main `push_tree` use arises from a change to one of the `ToInternal` impls in `proc_macro_server.rs`. It now returns a `SmallVec` instead of a `TokenStream`. This return value is then iterated over by `concat_trees`, which does `push_tree` on each element. Furthermore, the use of `SmallVec` avoids more allocations, because there is always only one or two token trees. Note: the removed `TokenStreamBuilder::push` method had some code to deal with a quadratic blowup case from #57735. This commit removes the code. I tried and failed to reproduce the blowup from that PR, before and after this change. Various other changes have happened to `TokenStreamBuilder` in the meantime, so I suspect the original problem is no longer relevant, though I don't have proof of this. Generally speaking, repeatedly extending a `Vec` without pre-determining its capacity is *not* quadratic. It's also incredibly common, within rustc and many other Rust programs, so if there were performance problems there you'd think it would show up in other places, too.
2022-10-05 10:38:15 +11:00
let mut stream =
if let Some(base) = base { base } else { tokenstream::TokenStream::default() };
for tree in trees {
Remove `TokenStreamBuilder`. `TokenStreamBuilder` exists to concatenate multiple `TokenStream`s together. This commit removes it, and moves the concatenation functionality directly into `TokenStream`, via two new methods `push_tree` and `push_stream`. This makes things both simpler and faster. `push_tree` is particularly important. `TokenStreamBuilder` only had a single `push` method, which pushed a stream. But in practice most of the time we push a single token tree rather than a stream, and `push_tree` avoids the need to build a token stream with a single entry (which requires two allocations, one for the `Lrc` and one for the `Vec`). The main `push_tree` use arises from a change to one of the `ToInternal` impls in `proc_macro_server.rs`. It now returns a `SmallVec` instead of a `TokenStream`. This return value is then iterated over by `concat_trees`, which does `push_tree` on each element. Furthermore, the use of `SmallVec` avoids more allocations, because there is always only one or two token trees. Note: the removed `TokenStreamBuilder::push` method had some code to deal with a quadratic blowup case from #57735. This commit removes the code. I tried and failed to reproduce the blowup from that PR, before and after this change. Various other changes have happened to `TokenStreamBuilder` in the meantime, so I suspect the original problem is no longer relevant, though I don't have proof of this. Generally speaking, repeatedly extending a `Vec` without pre-determining its capacity is *not* quadratic. It's also incredibly common, within rustc and many other Rust programs, so if there were performance problems there you'd think it would show up in other places, too.
2022-10-05 10:38:15 +11:00
for tt in (tree, &mut *self).to_internal() {
stream.push_tree(tt);
}
}
Remove `TokenStreamBuilder`. `TokenStreamBuilder` exists to concatenate multiple `TokenStream`s together. This commit removes it, and moves the concatenation functionality directly into `TokenStream`, via two new methods `push_tree` and `push_stream`. This makes things both simpler and faster. `push_tree` is particularly important. `TokenStreamBuilder` only had a single `push` method, which pushed a stream. But in practice most of the time we push a single token tree rather than a stream, and `push_tree` avoids the need to build a token stream with a single entry (which requires two allocations, one for the `Lrc` and one for the `Vec`). The main `push_tree` use arises from a change to one of the `ToInternal` impls in `proc_macro_server.rs`. It now returns a `SmallVec` instead of a `TokenStream`. This return value is then iterated over by `concat_trees`, which does `push_tree` on each element. Furthermore, the use of `SmallVec` avoids more allocations, because there is always only one or two token trees. Note: the removed `TokenStreamBuilder::push` method had some code to deal with a quadratic blowup case from #57735. This commit removes the code. I tried and failed to reproduce the blowup from that PR, before and after this change. Various other changes have happened to `TokenStreamBuilder` in the meantime, so I suspect the original problem is no longer relevant, though I don't have proof of this. Generally speaking, repeatedly extending a `Vec` without pre-determining its capacity is *not* quadratic. It's also incredibly common, within rustc and many other Rust programs, so if there were performance problems there you'd think it would show up in other places, too.
2022-10-05 10:38:15 +11:00
stream
}
fn concat_streams(
&mut self,
base: Option<Self::TokenStream>,
streams: Vec<Self::TokenStream>,
) -> Self::TokenStream {
Remove `TokenStreamBuilder`. `TokenStreamBuilder` exists to concatenate multiple `TokenStream`s together. This commit removes it, and moves the concatenation functionality directly into `TokenStream`, via two new methods `push_tree` and `push_stream`. This makes things both simpler and faster. `push_tree` is particularly important. `TokenStreamBuilder` only had a single `push` method, which pushed a stream. But in practice most of the time we push a single token tree rather than a stream, and `push_tree` avoids the need to build a token stream with a single entry (which requires two allocations, one for the `Lrc` and one for the `Vec`). The main `push_tree` use arises from a change to one of the `ToInternal` impls in `proc_macro_server.rs`. It now returns a `SmallVec` instead of a `TokenStream`. This return value is then iterated over by `concat_trees`, which does `push_tree` on each element. Furthermore, the use of `SmallVec` avoids more allocations, because there is always only one or two token trees. Note: the removed `TokenStreamBuilder::push` method had some code to deal with a quadratic blowup case from #57735. This commit removes the code. I tried and failed to reproduce the blowup from that PR, before and after this change. Various other changes have happened to `TokenStreamBuilder` in the meantime, so I suspect the original problem is no longer relevant, though I don't have proof of this. Generally speaking, repeatedly extending a `Vec` without pre-determining its capacity is *not* quadratic. It's also incredibly common, within rustc and many other Rust programs, so if there were performance problems there you'd think it would show up in other places, too.
2022-10-05 10:38:15 +11:00
let mut stream =
if let Some(base) = base { base } else { tokenstream::TokenStream::default() };
for s in streams {
stream.push_stream(s);
}
Remove `TokenStreamBuilder`. `TokenStreamBuilder` exists to concatenate multiple `TokenStream`s together. This commit removes it, and moves the concatenation functionality directly into `TokenStream`, via two new methods `push_tree` and `push_stream`. This makes things both simpler and faster. `push_tree` is particularly important. `TokenStreamBuilder` only had a single `push` method, which pushed a stream. But in practice most of the time we push a single token tree rather than a stream, and `push_tree` avoids the need to build a token stream with a single entry (which requires two allocations, one for the `Lrc` and one for the `Vec`). The main `push_tree` use arises from a change to one of the `ToInternal` impls in `proc_macro_server.rs`. It now returns a `SmallVec` instead of a `TokenStream`. This return value is then iterated over by `concat_trees`, which does `push_tree` on each element. Furthermore, the use of `SmallVec` avoids more allocations, because there is always only one or two token trees. Note: the removed `TokenStreamBuilder::push` method had some code to deal with a quadratic blowup case from #57735. This commit removes the code. I tried and failed to reproduce the blowup from that PR, before and after this change. Various other changes have happened to `TokenStreamBuilder` in the meantime, so I suspect the original problem is no longer relevant, though I don't have proof of this. Generally speaking, repeatedly extending a `Vec` without pre-determining its capacity is *not* quadratic. It's also incredibly common, within rustc and many other Rust programs, so if there were performance problems there you'd think it would show up in other places, too.
2022-10-05 10:38:15 +11:00
stream
}
2022-06-17 22:10:07 -04:00
fn into_trees(
&mut self,
stream: Self::TokenStream,
) -> Vec<TokenTree<Self::TokenStream, Self::Span, Self::Symbol>> {
FromInternal::from_internal((stream, self))
}
}
impl server::SourceFile for Rustc<'_, '_> {
fn eq(&mut self, file1: &Self::SourceFile, file2: &Self::SourceFile) -> bool {
Lrc::ptr_eq(file1, file2)
}
fn path(&mut self, file: &Self::SourceFile) -> String {
2022-12-23 17:34:23 +00:00
match &file.name {
FileName::Real(name) => name
.local_path()
.expect("attempting to get a file path in an imported file in `proc_macro::SourceFile::path`")
.to_str()
.expect("non-UTF8 file path in `proc_macro::SourceFile::path`")
.to_string(),
_ => file.name.prefer_local().to_string(),
}
}
fn is_real(&mut self, file: &Self::SourceFile) -> bool {
file.is_real_file()
}
}
impl server::Span for Rustc<'_, '_> {
fn debug(&mut self, span: Self::Span) -> String {
if self.ecx.ecfg.span_debug {
format!("{span:?}")
} else {
format!("{:?} bytes({}..{})", span.ctxt(), span.lo().0, span.hi().0)
}
}
fn source_file(&mut self, span: Self::Span) -> Self::SourceFile {
self.psess().source_map().lookup_char_pos(span.lo()).file
}
fn parent(&mut self, span: Self::Span) -> Option<Self::Span> {
2021-04-18 14:27:04 +02:00
span.parent_callsite()
}
fn source(&mut self, span: Self::Span) -> Self::Span {
span.source_callsite()
}
fn byte_range(&mut self, span: Self::Span) -> Range<usize> {
let source_map = self.psess().source_map();
let relative_start_pos = source_map.lookup_byte_offset(span.lo()).pos;
let relative_end_pos = source_map.lookup_byte_offset(span.hi()).pos;
2023-04-19 13:04:01 +02:00
Range { start: relative_start_pos.0 as usize, end: relative_end_pos.0 as usize }
}
fn start(&mut self, span: Self::Span) -> Self::Span {
2021-06-09 14:37:10 +02:00
span.shrink_to_lo()
}
fn end(&mut self, span: Self::Span) -> Self::Span {
2021-06-09 14:37:10 +02:00
span.shrink_to_hi()
}
2023-05-14 18:30:18 -04:00
fn line(&mut self, span: Self::Span) -> usize {
let loc = self.psess().source_map().lookup_char_pos(span.lo());
2023-05-14 18:30:18 -04:00
loc.line
}
fn column(&mut self, span: Self::Span) -> usize {
let loc = self.psess().source_map().lookup_char_pos(span.lo());
2023-05-14 18:30:18 -04:00
loc.col.to_usize() + 1
}
fn join(&mut self, first: Self::Span, second: Self::Span) -> Option<Self::Span> {
let self_loc = self.psess().source_map().lookup_char_pos(first.lo());
let other_loc = self.psess().source_map().lookup_char_pos(second.lo());
if self_loc.file.name != other_loc.file.name {
return None;
}
Some(first.to(second))
}
fn subspan(
&mut self,
span: Self::Span,
start: Bound<usize>,
end: Bound<usize>,
) -> Option<Self::Span> {
let length = span.hi().to_usize() - span.lo().to_usize();
let start = match start {
Bound::Included(lo) => lo,
Bound::Excluded(lo) => lo.checked_add(1)?,
Bound::Unbounded => 0,
};
let end = match end {
Bound::Included(hi) => hi.checked_add(1)?,
Bound::Excluded(hi) => hi,
Bound::Unbounded => length,
};
// Bounds check the values, preventing addition overflow and OOB spans.
if start > u32::MAX as usize
|| end > u32::MAX as usize
|| (u32::MAX - start as u32) < span.lo().to_u32()
|| (u32::MAX - end as u32) < span.lo().to_u32()
|| start >= end
|| end > length
{
return None;
}
let new_lo = span.lo() + BytePos::from_usize(start);
let new_hi = span.lo() + BytePos::from_usize(end);
Some(span.with_lo(new_lo).with_hi(new_hi))
}
fn resolved_at(&mut self, span: Self::Span, at: Self::Span) -> Self::Span {
span.with_ctxt(at.ctxt())
}
fn source_text(&mut self, span: Self::Span) -> Option<String> {
self.psess().source_map().span_to_snippet(span).ok()
}
Remove `TokenStreamBuilder`. `TokenStreamBuilder` exists to concatenate multiple `TokenStream`s together. This commit removes it, and moves the concatenation functionality directly into `TokenStream`, via two new methods `push_tree` and `push_stream`. This makes things both simpler and faster. `push_tree` is particularly important. `TokenStreamBuilder` only had a single `push` method, which pushed a stream. But in practice most of the time we push a single token tree rather than a stream, and `push_tree` avoids the need to build a token stream with a single entry (which requires two allocations, one for the `Lrc` and one for the `Vec`). The main `push_tree` use arises from a change to one of the `ToInternal` impls in `proc_macro_server.rs`. It now returns a `SmallVec` instead of a `TokenStream`. This return value is then iterated over by `concat_trees`, which does `push_tree` on each element. Furthermore, the use of `SmallVec` avoids more allocations, because there is always only one or two token trees. Note: the removed `TokenStreamBuilder::push` method had some code to deal with a quadratic blowup case from #57735. This commit removes the code. I tried and failed to reproduce the blowup from that PR, before and after this change. Various other changes have happened to `TokenStreamBuilder` in the meantime, so I suspect the original problem is no longer relevant, though I don't have proof of this. Generally speaking, repeatedly extending a `Vec` without pre-determining its capacity is *not* quadratic. It's also incredibly common, within rustc and many other Rust programs, so if there were performance problems there you'd think it would show up in other places, too.
2022-10-05 10:38:15 +11:00
Implement span quoting for proc-macros This PR implements span quoting, allowing proc-macros to produce spans pointing *into their own crate*. This is used by the unstable `proc_macro::quote!` macro, allowing us to get error messages like this: ``` error[E0412]: cannot find type `MissingType` in this scope --> $DIR/auxiliary/span-from-proc-macro.rs:37:20 | LL | pub fn error_from_attribute(_args: TokenStream, _input: TokenStream) -> TokenStream { | ----------------------------------------------------------------------------------- in this expansion of procedural macro `#[error_from_attribute]` ... LL | field: MissingType | ^^^^^^^^^^^ not found in this scope | ::: $DIR/span-from-proc-macro.rs:8:1 | LL | #[error_from_attribute] | ----------------------- in this macro invocation ``` Here, `MissingType` occurs inside the implementation of the proc-macro `#[error_from_attribute]`. Previosuly, this would always result in a span pointing at `#[error_from_attribute]` This will make many proc-macro-related error message much more useful - when a proc-macro generates code containing an error, users will get an error message pointing directly at that code (within the macro definition), instead of always getting a span pointing at the macro invocation site. This is implemented as follows: * When a proc-macro crate is being *compiled*, it causes the `quote!` macro to get run. This saves all of the sapns in the input to `quote!` into the metadata of *the proc-macro-crate* (which we are currently compiling). The `quote!` macro then expands to a call to `proc_macro::Span::recover_proc_macro_span(id)`, where `id` is an opaque identifier for the span in the crate metadata. * When the same proc-macro crate is *run* (e.g. it is loaded from disk and invoked by some consumer crate), the call to `proc_macro::Span::recover_proc_macro_span` causes us to load the span from the proc-macro crate's metadata. The proc-macro then produces a `TokenStream` containing a `Span` pointing into the proc-macro crate itself. The recursive nature of 'quote!' can be difficult to understand at first. The file `src/test/ui/proc-macro/quote-debug.stdout` shows the output of the `quote!` macro, which should make this eaier to understand. This PR also supports custom quoting spans in custom quote macros (e.g. the `quote` crate). All span quoting goes through the `proc_macro::quote_span` method, which can be called by a custom quote macro to perform span quoting. An example of this usage is provided in `src/test/ui/proc-macro/auxiliary/custom-quote.rs` Custom quoting currently has a few limitations: In order to quote a span, we need to generate a call to `proc_macro::Span::recover_proc_macro_span`. However, proc-macros support renaming the `proc_macro` crate, so we can't simply hardcode this path. Previously, the `quote_span` method used the path `crate::Span` - however, this only works when it is called by the builtin `quote!` macro in the same crate. To support being called from arbitrary crates, we need access to the name of the `proc_macro` crate to generate a path. This PR adds an additional argument to `quote_span` to specify the name of the `proc_macro` crate. Howver, this feels kind of hacky, and we may want to change this before stabilizing anything quote-related. Additionally, using `quote_span` currently requires enabling the `proc_macro_internals` feature. The builtin `quote!` macro has an `#[allow_internal_unstable]` attribute, but this won't work for custom quote implementations. This will likely require some additional tricks to apply `allow_internal_unstable` to the span of `proc_macro::Span::recover_proc_macro_span`.
2020-08-02 19:52:16 -04:00
/// Saves the provided span into the metadata of
/// *the crate we are currently compiling*, which must
/// be a proc-macro crate. This id can be passed to
/// `recover_proc_macro_span` when our current crate
/// is *run* as a proc-macro.
///
/// Let's suppose that we have two crates - `my_client`
/// and `my_proc_macro`. The `my_proc_macro` crate
/// contains a procedural macro `my_macro`, which
/// is implemented as: `quote! { "hello" }`
///
/// When we *compile* `my_proc_macro`, we will execute
/// the `quote` proc-macro. This will save the span of
/// "hello" into the metadata of `my_proc_macro`. As a result,
/// the body of `my_proc_macro` (after expansion) will end
/// up containing a call that looks like this:
Implement span quoting for proc-macros This PR implements span quoting, allowing proc-macros to produce spans pointing *into their own crate*. This is used by the unstable `proc_macro::quote!` macro, allowing us to get error messages like this: ``` error[E0412]: cannot find type `MissingType` in this scope --> $DIR/auxiliary/span-from-proc-macro.rs:37:20 | LL | pub fn error_from_attribute(_args: TokenStream, _input: TokenStream) -> TokenStream { | ----------------------------------------------------------------------------------- in this expansion of procedural macro `#[error_from_attribute]` ... LL | field: MissingType | ^^^^^^^^^^^ not found in this scope | ::: $DIR/span-from-proc-macro.rs:8:1 | LL | #[error_from_attribute] | ----------------------- in this macro invocation ``` Here, `MissingType` occurs inside the implementation of the proc-macro `#[error_from_attribute]`. Previosuly, this would always result in a span pointing at `#[error_from_attribute]` This will make many proc-macro-related error message much more useful - when a proc-macro generates code containing an error, users will get an error message pointing directly at that code (within the macro definition), instead of always getting a span pointing at the macro invocation site. This is implemented as follows: * When a proc-macro crate is being *compiled*, it causes the `quote!` macro to get run. This saves all of the sapns in the input to `quote!` into the metadata of *the proc-macro-crate* (which we are currently compiling). The `quote!` macro then expands to a call to `proc_macro::Span::recover_proc_macro_span(id)`, where `id` is an opaque identifier for the span in the crate metadata. * When the same proc-macro crate is *run* (e.g. it is loaded from disk and invoked by some consumer crate), the call to `proc_macro::Span::recover_proc_macro_span` causes us to load the span from the proc-macro crate's metadata. The proc-macro then produces a `TokenStream` containing a `Span` pointing into the proc-macro crate itself. The recursive nature of 'quote!' can be difficult to understand at first. The file `src/test/ui/proc-macro/quote-debug.stdout` shows the output of the `quote!` macro, which should make this eaier to understand. This PR also supports custom quoting spans in custom quote macros (e.g. the `quote` crate). All span quoting goes through the `proc_macro::quote_span` method, which can be called by a custom quote macro to perform span quoting. An example of this usage is provided in `src/test/ui/proc-macro/auxiliary/custom-quote.rs` Custom quoting currently has a few limitations: In order to quote a span, we need to generate a call to `proc_macro::Span::recover_proc_macro_span`. However, proc-macros support renaming the `proc_macro` crate, so we can't simply hardcode this path. Previously, the `quote_span` method used the path `crate::Span` - however, this only works when it is called by the builtin `quote!` macro in the same crate. To support being called from arbitrary crates, we need access to the name of the `proc_macro` crate to generate a path. This PR adds an additional argument to `quote_span` to specify the name of the `proc_macro` crate. Howver, this feels kind of hacky, and we may want to change this before stabilizing anything quote-related. Additionally, using `quote_span` currently requires enabling the `proc_macro_internals` feature. The builtin `quote!` macro has an `#[allow_internal_unstable]` attribute, but this won't work for custom quote implementations. This will likely require some additional tricks to apply `allow_internal_unstable` to the span of `proc_macro::Span::recover_proc_macro_span`.
2020-08-02 19:52:16 -04:00
/// `proc_macro::Ident::new("hello", proc_macro::Span::recover_proc_macro_span(0))`
///
/// where `0` is the id returned by this function.
/// When `my_proc_macro` *executes* (during the compilation of `my_client`),
/// the call to `recover_proc_macro_span` will load the corresponding
/// span from the metadata of `my_proc_macro` (which we have access to,
/// since we've loaded `my_proc_macro` from disk in order to execute it).
/// In this way, we have obtained a span pointing into `my_proc_macro`
fn save_span(&mut self, span: Self::Span) -> usize {
self.psess().save_proc_macro_span(span)
Implement span quoting for proc-macros This PR implements span quoting, allowing proc-macros to produce spans pointing *into their own crate*. This is used by the unstable `proc_macro::quote!` macro, allowing us to get error messages like this: ``` error[E0412]: cannot find type `MissingType` in this scope --> $DIR/auxiliary/span-from-proc-macro.rs:37:20 | LL | pub fn error_from_attribute(_args: TokenStream, _input: TokenStream) -> TokenStream { | ----------------------------------------------------------------------------------- in this expansion of procedural macro `#[error_from_attribute]` ... LL | field: MissingType | ^^^^^^^^^^^ not found in this scope | ::: $DIR/span-from-proc-macro.rs:8:1 | LL | #[error_from_attribute] | ----------------------- in this macro invocation ``` Here, `MissingType` occurs inside the implementation of the proc-macro `#[error_from_attribute]`. Previosuly, this would always result in a span pointing at `#[error_from_attribute]` This will make many proc-macro-related error message much more useful - when a proc-macro generates code containing an error, users will get an error message pointing directly at that code (within the macro definition), instead of always getting a span pointing at the macro invocation site. This is implemented as follows: * When a proc-macro crate is being *compiled*, it causes the `quote!` macro to get run. This saves all of the sapns in the input to `quote!` into the metadata of *the proc-macro-crate* (which we are currently compiling). The `quote!` macro then expands to a call to `proc_macro::Span::recover_proc_macro_span(id)`, where `id` is an opaque identifier for the span in the crate metadata. * When the same proc-macro crate is *run* (e.g. it is loaded from disk and invoked by some consumer crate), the call to `proc_macro::Span::recover_proc_macro_span` causes us to load the span from the proc-macro crate's metadata. The proc-macro then produces a `TokenStream` containing a `Span` pointing into the proc-macro crate itself. The recursive nature of 'quote!' can be difficult to understand at first. The file `src/test/ui/proc-macro/quote-debug.stdout` shows the output of the `quote!` macro, which should make this eaier to understand. This PR also supports custom quoting spans in custom quote macros (e.g. the `quote` crate). All span quoting goes through the `proc_macro::quote_span` method, which can be called by a custom quote macro to perform span quoting. An example of this usage is provided in `src/test/ui/proc-macro/auxiliary/custom-quote.rs` Custom quoting currently has a few limitations: In order to quote a span, we need to generate a call to `proc_macro::Span::recover_proc_macro_span`. However, proc-macros support renaming the `proc_macro` crate, so we can't simply hardcode this path. Previously, the `quote_span` method used the path `crate::Span` - however, this only works when it is called by the builtin `quote!` macro in the same crate. To support being called from arbitrary crates, we need access to the name of the `proc_macro` crate to generate a path. This PR adds an additional argument to `quote_span` to specify the name of the `proc_macro` crate. Howver, this feels kind of hacky, and we may want to change this before stabilizing anything quote-related. Additionally, using `quote_span` currently requires enabling the `proc_macro_internals` feature. The builtin `quote!` macro has an `#[allow_internal_unstable]` attribute, but this won't work for custom quote implementations. This will likely require some additional tricks to apply `allow_internal_unstable` to the span of `proc_macro::Span::recover_proc_macro_span`.
2020-08-02 19:52:16 -04:00
}
Implement span quoting for proc-macros This PR implements span quoting, allowing proc-macros to produce spans pointing *into their own crate*. This is used by the unstable `proc_macro::quote!` macro, allowing us to get error messages like this: ``` error[E0412]: cannot find type `MissingType` in this scope --> $DIR/auxiliary/span-from-proc-macro.rs:37:20 | LL | pub fn error_from_attribute(_args: TokenStream, _input: TokenStream) -> TokenStream { | ----------------------------------------------------------------------------------- in this expansion of procedural macro `#[error_from_attribute]` ... LL | field: MissingType | ^^^^^^^^^^^ not found in this scope | ::: $DIR/span-from-proc-macro.rs:8:1 | LL | #[error_from_attribute] | ----------------------- in this macro invocation ``` Here, `MissingType` occurs inside the implementation of the proc-macro `#[error_from_attribute]`. Previosuly, this would always result in a span pointing at `#[error_from_attribute]` This will make many proc-macro-related error message much more useful - when a proc-macro generates code containing an error, users will get an error message pointing directly at that code (within the macro definition), instead of always getting a span pointing at the macro invocation site. This is implemented as follows: * When a proc-macro crate is being *compiled*, it causes the `quote!` macro to get run. This saves all of the sapns in the input to `quote!` into the metadata of *the proc-macro-crate* (which we are currently compiling). The `quote!` macro then expands to a call to `proc_macro::Span::recover_proc_macro_span(id)`, where `id` is an opaque identifier for the span in the crate metadata. * When the same proc-macro crate is *run* (e.g. it is loaded from disk and invoked by some consumer crate), the call to `proc_macro::Span::recover_proc_macro_span` causes us to load the span from the proc-macro crate's metadata. The proc-macro then produces a `TokenStream` containing a `Span` pointing into the proc-macro crate itself. The recursive nature of 'quote!' can be difficult to understand at first. The file `src/test/ui/proc-macro/quote-debug.stdout` shows the output of the `quote!` macro, which should make this eaier to understand. This PR also supports custom quoting spans in custom quote macros (e.g. the `quote` crate). All span quoting goes through the `proc_macro::quote_span` method, which can be called by a custom quote macro to perform span quoting. An example of this usage is provided in `src/test/ui/proc-macro/auxiliary/custom-quote.rs` Custom quoting currently has a few limitations: In order to quote a span, we need to generate a call to `proc_macro::Span::recover_proc_macro_span`. However, proc-macros support renaming the `proc_macro` crate, so we can't simply hardcode this path. Previously, the `quote_span` method used the path `crate::Span` - however, this only works when it is called by the builtin `quote!` macro in the same crate. To support being called from arbitrary crates, we need access to the name of the `proc_macro` crate to generate a path. This PR adds an additional argument to `quote_span` to specify the name of the `proc_macro` crate. Howver, this feels kind of hacky, and we may want to change this before stabilizing anything quote-related. Additionally, using `quote_span` currently requires enabling the `proc_macro_internals` feature. The builtin `quote!` macro has an `#[allow_internal_unstable]` attribute, but this won't work for custom quote implementations. This will likely require some additional tricks to apply `allow_internal_unstable` to the span of `proc_macro::Span::recover_proc_macro_span`.
2020-08-02 19:52:16 -04:00
fn recover_proc_macro_span(&mut self, id: usize) -> Self::Span {
let (resolver, krate, def_site) = (&*self.ecx.resolver, self.krate, self.def_site);
Implement span quoting for proc-macros This PR implements span quoting, allowing proc-macros to produce spans pointing *into their own crate*. This is used by the unstable `proc_macro::quote!` macro, allowing us to get error messages like this: ``` error[E0412]: cannot find type `MissingType` in this scope --> $DIR/auxiliary/span-from-proc-macro.rs:37:20 | LL | pub fn error_from_attribute(_args: TokenStream, _input: TokenStream) -> TokenStream { | ----------------------------------------------------------------------------------- in this expansion of procedural macro `#[error_from_attribute]` ... LL | field: MissingType | ^^^^^^^^^^^ not found in this scope | ::: $DIR/span-from-proc-macro.rs:8:1 | LL | #[error_from_attribute] | ----------------------- in this macro invocation ``` Here, `MissingType` occurs inside the implementation of the proc-macro `#[error_from_attribute]`. Previosuly, this would always result in a span pointing at `#[error_from_attribute]` This will make many proc-macro-related error message much more useful - when a proc-macro generates code containing an error, users will get an error message pointing directly at that code (within the macro definition), instead of always getting a span pointing at the macro invocation site. This is implemented as follows: * When a proc-macro crate is being *compiled*, it causes the `quote!` macro to get run. This saves all of the sapns in the input to `quote!` into the metadata of *the proc-macro-crate* (which we are currently compiling). The `quote!` macro then expands to a call to `proc_macro::Span::recover_proc_macro_span(id)`, where `id` is an opaque identifier for the span in the crate metadata. * When the same proc-macro crate is *run* (e.g. it is loaded from disk and invoked by some consumer crate), the call to `proc_macro::Span::recover_proc_macro_span` causes us to load the span from the proc-macro crate's metadata. The proc-macro then produces a `TokenStream` containing a `Span` pointing into the proc-macro crate itself. The recursive nature of 'quote!' can be difficult to understand at first. The file `src/test/ui/proc-macro/quote-debug.stdout` shows the output of the `quote!` macro, which should make this eaier to understand. This PR also supports custom quoting spans in custom quote macros (e.g. the `quote` crate). All span quoting goes through the `proc_macro::quote_span` method, which can be called by a custom quote macro to perform span quoting. An example of this usage is provided in `src/test/ui/proc-macro/auxiliary/custom-quote.rs` Custom quoting currently has a few limitations: In order to quote a span, we need to generate a call to `proc_macro::Span::recover_proc_macro_span`. However, proc-macros support renaming the `proc_macro` crate, so we can't simply hardcode this path. Previously, the `quote_span` method used the path `crate::Span` - however, this only works when it is called by the builtin `quote!` macro in the same crate. To support being called from arbitrary crates, we need access to the name of the `proc_macro` crate to generate a path. This PR adds an additional argument to `quote_span` to specify the name of the `proc_macro` crate. Howver, this feels kind of hacky, and we may want to change this before stabilizing anything quote-related. Additionally, using `quote_span` currently requires enabling the `proc_macro_internals` feature. The builtin `quote!` macro has an `#[allow_internal_unstable]` attribute, but this won't work for custom quote implementations. This will likely require some additional tricks to apply `allow_internal_unstable` to the span of `proc_macro::Span::recover_proc_macro_span`.
2020-08-02 19:52:16 -04:00
*self.rebased_spans.entry(id).or_insert_with(|| {
// FIXME: `SyntaxContext` for spans from proc macro crates is lost during encoding,
// replace it with a def-site context until we are encoding it properly.
resolver.get_proc_macro_quoted_span(krate, id).with_ctxt(def_site.ctxt())
Implement span quoting for proc-macros This PR implements span quoting, allowing proc-macros to produce spans pointing *into their own crate*. This is used by the unstable `proc_macro::quote!` macro, allowing us to get error messages like this: ``` error[E0412]: cannot find type `MissingType` in this scope --> $DIR/auxiliary/span-from-proc-macro.rs:37:20 | LL | pub fn error_from_attribute(_args: TokenStream, _input: TokenStream) -> TokenStream { | ----------------------------------------------------------------------------------- in this expansion of procedural macro `#[error_from_attribute]` ... LL | field: MissingType | ^^^^^^^^^^^ not found in this scope | ::: $DIR/span-from-proc-macro.rs:8:1 | LL | #[error_from_attribute] | ----------------------- in this macro invocation ``` Here, `MissingType` occurs inside the implementation of the proc-macro `#[error_from_attribute]`. Previosuly, this would always result in a span pointing at `#[error_from_attribute]` This will make many proc-macro-related error message much more useful - when a proc-macro generates code containing an error, users will get an error message pointing directly at that code (within the macro definition), instead of always getting a span pointing at the macro invocation site. This is implemented as follows: * When a proc-macro crate is being *compiled*, it causes the `quote!` macro to get run. This saves all of the sapns in the input to `quote!` into the metadata of *the proc-macro-crate* (which we are currently compiling). The `quote!` macro then expands to a call to `proc_macro::Span::recover_proc_macro_span(id)`, where `id` is an opaque identifier for the span in the crate metadata. * When the same proc-macro crate is *run* (e.g. it is loaded from disk and invoked by some consumer crate), the call to `proc_macro::Span::recover_proc_macro_span` causes us to load the span from the proc-macro crate's metadata. The proc-macro then produces a `TokenStream` containing a `Span` pointing into the proc-macro crate itself. The recursive nature of 'quote!' can be difficult to understand at first. The file `src/test/ui/proc-macro/quote-debug.stdout` shows the output of the `quote!` macro, which should make this eaier to understand. This PR also supports custom quoting spans in custom quote macros (e.g. the `quote` crate). All span quoting goes through the `proc_macro::quote_span` method, which can be called by a custom quote macro to perform span quoting. An example of this usage is provided in `src/test/ui/proc-macro/auxiliary/custom-quote.rs` Custom quoting currently has a few limitations: In order to quote a span, we need to generate a call to `proc_macro::Span::recover_proc_macro_span`. However, proc-macros support renaming the `proc_macro` crate, so we can't simply hardcode this path. Previously, the `quote_span` method used the path `crate::Span` - however, this only works when it is called by the builtin `quote!` macro in the same crate. To support being called from arbitrary crates, we need access to the name of the `proc_macro` crate to generate a path. This PR adds an additional argument to `quote_span` to specify the name of the `proc_macro` crate. Howver, this feels kind of hacky, and we may want to change this before stabilizing anything quote-related. Additionally, using `quote_span` currently requires enabling the `proc_macro_internals` feature. The builtin `quote!` macro has an `#[allow_internal_unstable]` attribute, but this won't work for custom quote implementations. This will likely require some additional tricks to apply `allow_internal_unstable` to the span of `proc_macro::Span::recover_proc_macro_span`.
2020-08-02 19:52:16 -04:00
})
}
}
impl server::Symbol for Rustc<'_, '_> {
fn normalize_and_validate_ident(&mut self, string: &str) -> Result<Self::Symbol, ()> {
let sym = nfc_normalize(string);
if rustc_lexer::is_ident(sym.as_str()) { Ok(sym) } else { Err(()) }
}
}
impl server::Server for Rustc<'_, '_> {
fn globals(&mut self) -> ExpnGlobals<Self::Span> {
ExpnGlobals {
def_site: self.def_site,
call_site: self.call_site,
mixed_site: self.mixed_site,
}
}
fn intern_symbol(string: &str) -> Self::Symbol {
Symbol::intern(string)
}
fn with_symbol_string(symbol: &Self::Symbol, f: impl FnOnce(&str)) {
f(symbol.as_str())
}
}