2021-07-18 15:53:06 -04:00
|
|
|
use std::ops::{Bound, Range};
|
2024-07-29 08:13:50 +10:00
|
|
|
|
2024-02-13 23:28:27 +00:00
|
|
|
use ast::token::IdentIsRaw;
|
Remove `TokenStreamBuilder`.
`TokenStreamBuilder` exists to concatenate multiple `TokenStream`s
together. This commit removes it, and moves the concatenation
functionality directly into `TokenStream`, via two new methods
`push_tree` and `push_stream`. This makes things both simpler and
faster.
`push_tree` is particularly important. `TokenStreamBuilder` only had a
single `push` method, which pushed a stream. But in practice most of the
time we push a single token tree rather than a stream, and `push_tree`
avoids the need to build a token stream with a single entry (which
requires two allocations, one for the `Lrc` and one for the `Vec`).
The main `push_tree` use arises from a change to one of the `ToInternal`
impls in `proc_macro_server.rs`. It now returns a `SmallVec` instead of
a `TokenStream`. This return value is then iterated over by
`concat_trees`, which does `push_tree` on each element. Furthermore, the
use of `SmallVec` avoids more allocations, because there is always only
one or two token trees.
Note: the removed `TokenStreamBuilder::push` method had some code to
deal with a quadratic blowup case from #57735. This commit removes the
code. I tried and failed to reproduce the blowup from that PR, before
and after this change. Various other changes have happened to
`TokenStreamBuilder` in the meantime, so I suspect the original problem
is no longer relevant, though I don't have proof of this. Generally
speaking, repeatedly extending a `Vec` without pre-determining its
capacity is *not* quadratic. It's also incredibly common, within rustc
and many other Rust programs, so if there were performance problems
there you'd think it would show up in other places, too.
2022-10-05 10:38:15 +11:00
|
|
|
use pm::bridge::{
|
|
|
|
server, DelimSpan, Diagnostic, ExpnGlobals, Group, Ident, LitKind, Literal, Punct, TokenTree,
|
|
|
|
};
|
2023-05-14 17:38:41 -04:00
|
|
|
use pm::{Delimiter, Level};
|
2020-04-27 23:26:11 +05:30
|
|
|
use rustc_ast as ast;
|
2022-01-01 17:15:47 +08:00
|
|
|
use rustc_ast::token;
|
2023-10-12 15:36:14 +11:00
|
|
|
use rustc_ast::tokenstream::{self, DelimSpacing, Spacing, TokenStream};
|
2022-12-05 15:23:27 +11:00
|
|
|
use rustc_ast::util::literal::escape_byte_str_symbol;
|
2020-01-11 17:02:46 +01:00
|
|
|
use rustc_ast_pretty::pprust;
|
Implement span quoting for proc-macros
This PR implements span quoting, allowing proc-macros to produce spans
pointing *into their own crate*. This is used by the unstable
`proc_macro::quote!` macro, allowing us to get error messages like this:
```
error[E0412]: cannot find type `MissingType` in this scope
--> $DIR/auxiliary/span-from-proc-macro.rs:37:20
|
LL | pub fn error_from_attribute(_args: TokenStream, _input: TokenStream) -> TokenStream {
| ----------------------------------------------------------------------------------- in this expansion of procedural macro `#[error_from_attribute]`
...
LL | field: MissingType
| ^^^^^^^^^^^ not found in this scope
|
::: $DIR/span-from-proc-macro.rs:8:1
|
LL | #[error_from_attribute]
| ----------------------- in this macro invocation
```
Here, `MissingType` occurs inside the implementation of the proc-macro
`#[error_from_attribute]`. Previosuly, this would always result in a
span pointing at `#[error_from_attribute]`
This will make many proc-macro-related error message much more useful -
when a proc-macro generates code containing an error, users will get an
error message pointing directly at that code (within the macro
definition), instead of always getting a span pointing at the macro
invocation site.
This is implemented as follows:
* When a proc-macro crate is being *compiled*, it causes the `quote!`
macro to get run. This saves all of the sapns in the input to `quote!`
into the metadata of *the proc-macro-crate* (which we are currently
compiling). The `quote!` macro then expands to a call to
`proc_macro::Span::recover_proc_macro_span(id)`, where `id` is an
opaque identifier for the span in the crate metadata.
* When the same proc-macro crate is *run* (e.g. it is loaded from disk
and invoked by some consumer crate), the call to
`proc_macro::Span::recover_proc_macro_span` causes us to load the span
from the proc-macro crate's metadata. The proc-macro then produces a
`TokenStream` containing a `Span` pointing into the proc-macro crate
itself.
The recursive nature of 'quote!' can be difficult to understand at
first. The file `src/test/ui/proc-macro/quote-debug.stdout` shows
the output of the `quote!` macro, which should make this eaier to
understand.
This PR also supports custom quoting spans in custom quote macros (e.g.
the `quote` crate). All span quoting goes through the
`proc_macro::quote_span` method, which can be called by a custom quote
macro to perform span quoting. An example of this usage is provided in
`src/test/ui/proc-macro/auxiliary/custom-quote.rs`
Custom quoting currently has a few limitations:
In order to quote a span, we need to generate a call to
`proc_macro::Span::recover_proc_macro_span`. However, proc-macros
support renaming the `proc_macro` crate, so we can't simply hardcode
this path. Previously, the `quote_span` method used the path
`crate::Span` - however, this only works when it is called by the
builtin `quote!` macro in the same crate. To support being called from
arbitrary crates, we need access to the name of the `proc_macro` crate
to generate a path. This PR adds an additional argument to `quote_span`
to specify the name of the `proc_macro` crate. Howver, this feels kind
of hacky, and we may want to change this before stabilizing anything
quote-related.
Additionally, using `quote_span` currently requires enabling the
`proc_macro_internals` feature. The builtin `quote!` macro
has an `#[allow_internal_unstable]` attribute, but this won't work for
custom quote implementations. This will likely require some additional
tricks to apply `allow_internal_unstable` to the span of
`proc_macro::Span::recover_proc_macro_span`.
2020-08-02 19:52:16 -04:00
|
|
|
use rustc_data_structures::fx::FxHashMap;
|
2020-01-09 11:18:47 +01:00
|
|
|
use rustc_data_structures::sync::Lrc;
|
2024-02-23 10:20:45 +11:00
|
|
|
use rustc_errors::{Diag, ErrorGuaranteed, MultiSpan, PResult};
|
2019-12-29 19:50:43 +08:00
|
|
|
use rustc_parse::lexer::nfc_normalize;
|
2024-05-31 13:32:54 +10:00
|
|
|
use rustc_parse::parser::Parser;
|
2024-05-31 15:43:18 +10:00
|
|
|
use rustc_parse::{new_parser_from_source_str, source_str_to_stream, unwrap_or_emit_fatal};
|
2020-01-11 15:03:15 +01:00
|
|
|
use rustc_session::parse::ParseSess;
|
Implement span quoting for proc-macros
This PR implements span quoting, allowing proc-macros to produce spans
pointing *into their own crate*. This is used by the unstable
`proc_macro::quote!` macro, allowing us to get error messages like this:
```
error[E0412]: cannot find type `MissingType` in this scope
--> $DIR/auxiliary/span-from-proc-macro.rs:37:20
|
LL | pub fn error_from_attribute(_args: TokenStream, _input: TokenStream) -> TokenStream {
| ----------------------------------------------------------------------------------- in this expansion of procedural macro `#[error_from_attribute]`
...
LL | field: MissingType
| ^^^^^^^^^^^ not found in this scope
|
::: $DIR/span-from-proc-macro.rs:8:1
|
LL | #[error_from_attribute]
| ----------------------- in this macro invocation
```
Here, `MissingType` occurs inside the implementation of the proc-macro
`#[error_from_attribute]`. Previosuly, this would always result in a
span pointing at `#[error_from_attribute]`
This will make many proc-macro-related error message much more useful -
when a proc-macro generates code containing an error, users will get an
error message pointing directly at that code (within the macro
definition), instead of always getting a span pointing at the macro
invocation site.
This is implemented as follows:
* When a proc-macro crate is being *compiled*, it causes the `quote!`
macro to get run. This saves all of the sapns in the input to `quote!`
into the metadata of *the proc-macro-crate* (which we are currently
compiling). The `quote!` macro then expands to a call to
`proc_macro::Span::recover_proc_macro_span(id)`, where `id` is an
opaque identifier for the span in the crate metadata.
* When the same proc-macro crate is *run* (e.g. it is loaded from disk
and invoked by some consumer crate), the call to
`proc_macro::Span::recover_proc_macro_span` causes us to load the span
from the proc-macro crate's metadata. The proc-macro then produces a
`TokenStream` containing a `Span` pointing into the proc-macro crate
itself.
The recursive nature of 'quote!' can be difficult to understand at
first. The file `src/test/ui/proc-macro/quote-debug.stdout` shows
the output of the `quote!` macro, which should make this eaier to
understand.
This PR also supports custom quoting spans in custom quote macros (e.g.
the `quote` crate). All span quoting goes through the
`proc_macro::quote_span` method, which can be called by a custom quote
macro to perform span quoting. An example of this usage is provided in
`src/test/ui/proc-macro/auxiliary/custom-quote.rs`
Custom quoting currently has a few limitations:
In order to quote a span, we need to generate a call to
`proc_macro::Span::recover_proc_macro_span`. However, proc-macros
support renaming the `proc_macro` crate, so we can't simply hardcode
this path. Previously, the `quote_span` method used the path
`crate::Span` - however, this only works when it is called by the
builtin `quote!` macro in the same crate. To support being called from
arbitrary crates, we need access to the name of the `proc_macro` crate
to generate a path. This PR adds an additional argument to `quote_span`
to specify the name of the `proc_macro` crate. Howver, this feels kind
of hacky, and we may want to change this before stabilizing anything
quote-related.
Additionally, using `quote_span` currently requires enabling the
`proc_macro_internals` feature. The builtin `quote!` macro
has an `#[allow_internal_unstable]` attribute, but this won't work for
custom quote implementations. This will likely require some additional
tricks to apply `allow_internal_unstable` to the span of
`proc_macro::Span::recover_proc_macro_span`.
2020-08-02 19:52:16 -04:00
|
|
|
use rustc_span::def_id::CrateNum;
|
2022-06-30 21:05:46 -04:00
|
|
|
use rustc_span::symbol::{self, sym, Symbol};
|
2022-03-24 02:03:04 +00:00
|
|
|
use rustc_span::{BytePos, FileName, Pos, SourceFile, Span};
|
Remove `TokenStreamBuilder`.
`TokenStreamBuilder` exists to concatenate multiple `TokenStream`s
together. This commit removes it, and moves the concatenation
functionality directly into `TokenStream`, via two new methods
`push_tree` and `push_stream`. This makes things both simpler and
faster.
`push_tree` is particularly important. `TokenStreamBuilder` only had a
single `push` method, which pushed a stream. But in practice most of the
time we push a single token tree rather than a stream, and `push_tree`
avoids the need to build a token stream with a single entry (which
requires two allocations, one for the `Lrc` and one for the `Vec`).
The main `push_tree` use arises from a change to one of the `ToInternal`
impls in `proc_macro_server.rs`. It now returns a `SmallVec` instead of
a `TokenStream`. This return value is then iterated over by
`concat_trees`, which does `push_tree` on each element. Furthermore, the
use of `SmallVec` avoids more allocations, because there is always only
one or two token trees.
Note: the removed `TokenStreamBuilder::push` method had some code to
deal with a quadratic blowup case from #57735. This commit removes the
code. I tried and failed to reproduce the blowup from that PR, before
and after this change. Various other changes have happened to
`TokenStreamBuilder` in the meantime, so I suspect the original problem
is no longer relevant, though I don't have proof of this. Generally
speaking, repeatedly extending a `Vec` without pre-determining its
capacity is *not* quadratic. It's also incredibly common, within rustc
and many other Rust programs, so if there were performance problems
there you'd think it would show up in other places, too.
2022-10-05 10:38:15 +11:00
|
|
|
use smallvec::{smallvec, SmallVec};
|
2024-07-29 08:13:50 +10:00
|
|
|
|
2023-03-10 21:16:35 +01:00
|
|
|
use crate::base::ExtCtxt;
|
2018-07-19 15:59:08 +03:00
|
|
|
|
2018-03-20 16:41:14 +02:00
|
|
|
trait FromInternal<T> {
|
|
|
|
fn from_internal(x: T) -> Self;
|
|
|
|
}
|
|
|
|
|
|
|
|
trait ToInternal<T> {
|
|
|
|
fn to_internal(self) -> T;
|
|
|
|
}
|
|
|
|
|
2022-04-26 15:40:14 +03:00
|
|
|
impl FromInternal<token::Delimiter> for Delimiter {
|
|
|
|
fn from_internal(delim: token::Delimiter) -> Delimiter {
|
2018-07-19 15:59:08 +03:00
|
|
|
match delim {
|
2022-04-26 15:40:14 +03:00
|
|
|
token::Delimiter::Parenthesis => Delimiter::Parenthesis,
|
|
|
|
token::Delimiter::Brace => Delimiter::Brace,
|
|
|
|
token::Delimiter::Bracket => Delimiter::Bracket,
|
|
|
|
token::Delimiter::Invisible => Delimiter::None,
|
2018-07-19 15:59:08 +03:00
|
|
|
}
|
|
|
|
}
|
2018-03-20 16:41:14 +02:00
|
|
|
}
|
2018-07-19 15:59:08 +03:00
|
|
|
|
2022-04-26 15:40:14 +03:00
|
|
|
impl ToInternal<token::Delimiter> for Delimiter {
|
|
|
|
fn to_internal(self) -> token::Delimiter {
|
2018-07-19 15:59:08 +03:00
|
|
|
match self {
|
2022-04-26 15:40:14 +03:00
|
|
|
Delimiter::Parenthesis => token::Delimiter::Parenthesis,
|
|
|
|
Delimiter::Brace => token::Delimiter::Brace,
|
|
|
|
Delimiter::Bracket => token::Delimiter::Bracket,
|
|
|
|
Delimiter::None => token::Delimiter::Invisible,
|
2018-07-19 15:59:08 +03:00
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2022-07-03 01:04:31 -04:00
|
|
|
impl FromInternal<token::LitKind> for LitKind {
|
|
|
|
fn from_internal(kind: token::LitKind) -> Self {
|
|
|
|
match kind {
|
|
|
|
token::Byte => LitKind::Byte,
|
|
|
|
token::Char => LitKind::Char,
|
|
|
|
token::Integer => LitKind::Integer,
|
|
|
|
token::Float => LitKind::Float,
|
|
|
|
token::Str => LitKind::Str,
|
|
|
|
token::StrRaw(n) => LitKind::StrRaw(n),
|
|
|
|
token::ByteStr => LitKind::ByteStr,
|
|
|
|
token::ByteStrRaw(n) => LitKind::ByteStrRaw(n),
|
2023-03-06 07:42:04 +00:00
|
|
|
token::CStr => LitKind::CStr,
|
|
|
|
token::CStrRaw(n) => LitKind::CStrRaw(n),
|
2024-02-14 20:12:05 +11:00
|
|
|
token::Err(_guar) => {
|
|
|
|
// This is the only place a `pm::bridge::LitKind::ErrWithGuar`
|
|
|
|
// is constructed. Note that an `ErrorGuaranteed` is available,
|
|
|
|
// as required. See the comment in `to_internal`.
|
|
|
|
LitKind::ErrWithGuar
|
|
|
|
}
|
2022-07-03 01:04:31 -04:00
|
|
|
token::Bool => unreachable!(),
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
impl ToInternal<token::LitKind> for LitKind {
|
|
|
|
fn to_internal(self) -> token::LitKind {
|
|
|
|
match self {
|
|
|
|
LitKind::Byte => token::Byte,
|
|
|
|
LitKind::Char => token::Char,
|
|
|
|
LitKind::Integer => token::Integer,
|
|
|
|
LitKind::Float => token::Float,
|
|
|
|
LitKind::Str => token::Str,
|
|
|
|
LitKind::StrRaw(n) => token::StrRaw(n),
|
|
|
|
LitKind::ByteStr => token::ByteStr,
|
|
|
|
LitKind::ByteStrRaw(n) => token::ByteStrRaw(n),
|
2023-03-06 07:42:04 +00:00
|
|
|
LitKind::CStr => token::CStr,
|
|
|
|
LitKind::CStrRaw(n) => token::CStrRaw(n),
|
2024-02-14 20:12:05 +11:00
|
|
|
LitKind::ErrWithGuar => {
|
|
|
|
// This is annoying but valid. `LitKind::ErrWithGuar` would
|
|
|
|
// have an `ErrorGuaranteed` except that type isn't available
|
|
|
|
// in that crate. So we have to fake one. And we don't want to
|
|
|
|
// use a delayed bug because there might be lots of these,
|
|
|
|
// which would be expensive.
|
|
|
|
#[allow(deprecated)]
|
|
|
|
let guar = ErrorGuaranteed::unchecked_error_guaranteed();
|
|
|
|
token::Err(guar)
|
|
|
|
}
|
2022-07-03 01:04:31 -04:00
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
impl FromInternal<(TokenStream, &mut Rustc<'_, '_>)> for Vec<TokenTree<TokenStream, Span, Symbol>> {
|
2021-07-01 17:36:38 -04:00
|
|
|
fn from_internal((stream, rustc): (TokenStream, &mut Rustc<'_, '_>)) -> Self {
|
2020-02-29 20:37:32 +03:00
|
|
|
use rustc_ast::token::*;
|
2018-07-19 15:59:08 +03:00
|
|
|
|
2022-06-27 20:03:56 -04:00
|
|
|
// Estimate the capacity as `stream.len()` rounded up to the next power
|
|
|
|
// of two to limit the number of required reallocations.
|
|
|
|
let mut trees = Vec::with_capacity(stream.len().next_power_of_two());
|
2023-07-27 11:48:55 +10:00
|
|
|
let mut cursor = stream.trees();
|
2021-07-01 17:36:38 -04:00
|
|
|
|
Remove `TreeAndSpacing`.
A `TokenStream` contains a `Lrc<Vec<(TokenTree, Spacing)>>`. But this is
not quite right. `Spacing` makes sense for `TokenTree::Token`, but does
not make sense for `TokenTree::Delimited`, because a
`TokenTree::Delimited` cannot be joined with another `TokenTree`.
This commit fixes this problem, by adding `Spacing` to `TokenTree::Token`,
changing `TokenStream` to contain a `Lrc<Vec<TokenTree>>`, and removing the
`TreeAndSpacing` typedef.
The commit removes these two impls:
- `impl From<TokenTree> for TokenStream`
- `impl From<TokenTree> for TreeAndSpacing`
These were useful, but also resulted in code with many `.into()` calls
that was hard to read, particularly for anyone not highly familiar with
the relevant types. This commit makes some other changes to compensate:
- `TokenTree::token()` becomes `TokenTree::token_{alone,joint}()`.
- `TokenStream::token_{alone,joint}()` are added.
- `TokenStream::delimited` is added.
This results in things like this:
```rust
TokenTree::token(token::Semi, stmt.span).into()
```
changing to this:
```rust
TokenStream::token_alone(token::Semi, stmt.span)
```
This makes the type of the result, and its spacing, clearer.
These changes also simplifies `Cursor` and `CursorRef`, because they no longer
need to distinguish between `next` and `next_with_spacing`.
2022-07-28 10:31:04 +10:00
|
|
|
while let Some(tree) = cursor.next() {
|
2023-07-27 11:48:55 +10:00
|
|
|
let (Token { kind, span }, joint) = match tree.clone() {
|
2023-10-12 15:36:14 +11:00
|
|
|
tokenstream::TokenTree::Delimited(span, _, delim, tts) => {
|
2021-07-01 17:36:38 -04:00
|
|
|
let delimiter = pm::Delimiter::from_internal(delim);
|
|
|
|
trees.push(TokenTree::Group(Group {
|
|
|
|
delimiter,
|
|
|
|
stream: Some(tts),
|
|
|
|
span: DelimSpan {
|
|
|
|
open: span.open,
|
|
|
|
close: span.close,
|
|
|
|
entire: span.entire(),
|
|
|
|
},
|
|
|
|
}));
|
|
|
|
continue;
|
|
|
|
}
|
2023-08-08 11:43:44 +10:00
|
|
|
tokenstream::TokenTree::Token(token, spacing) => {
|
|
|
|
// Do not be tempted to check here that the `spacing`
|
|
|
|
// values are "correct" w.r.t. the token stream (e.g. that
|
|
|
|
// `Spacing::Joint` is actually followed by a `Punct` token
|
|
|
|
// tree). Because the problem in #76399 was introduced that
|
|
|
|
// way.
|
|
|
|
//
|
|
|
|
// This is where the `Hidden` in `JointHidden` applies,
|
|
|
|
// because the jointness is effectively hidden from proc
|
|
|
|
// macros.
|
|
|
|
let joint = match spacing {
|
|
|
|
Spacing::Alone | Spacing::JointHidden => false,
|
|
|
|
Spacing::Joint => true,
|
|
|
|
};
|
|
|
|
(token, joint)
|
|
|
|
}
|
2018-07-19 15:59:08 +03:00
|
|
|
};
|
|
|
|
|
2022-09-28 10:07:01 +10:00
|
|
|
// Split the operator into one or more `Punct`s, one per character.
|
|
|
|
// The final one inherits the jointness of the original token. Any
|
|
|
|
// before that get `joint = true`.
|
2022-06-27 20:03:56 -04:00
|
|
|
let mut op = |s: &str| {
|
|
|
|
assert!(s.is_ascii());
|
2022-10-03 13:57:47 +11:00
|
|
|
trees.extend(s.bytes().enumerate().map(|(i, ch)| {
|
|
|
|
let is_final = i == s.len() - 1;
|
|
|
|
// Split the token span into single chars. Unless the span
|
|
|
|
// is an unusual one, e.g. due to proc macro expansion. We
|
|
|
|
// determine this by assuming any span with a length that
|
|
|
|
// matches the operator length is a normal one, and any
|
|
|
|
// span with a different length is an unusual one.
|
|
|
|
let span = if (span.hi() - span.lo()).to_usize() == s.len() {
|
|
|
|
let lo = span.lo() + BytePos::from_usize(i);
|
|
|
|
let hi = lo + BytePos::from_usize(1);
|
|
|
|
span.with_lo(lo).with_hi(hi)
|
|
|
|
} else {
|
|
|
|
span
|
|
|
|
};
|
2023-08-08 11:43:44 +10:00
|
|
|
let joint = if is_final { joint } else { true };
|
|
|
|
TokenTree::Punct(Punct { ch, joint, span })
|
2022-06-27 20:03:56 -04:00
|
|
|
}));
|
|
|
|
};
|
2021-07-01 17:36:38 -04:00
|
|
|
|
|
|
|
match kind {
|
2022-06-27 20:03:56 -04:00
|
|
|
Eq => op("="),
|
|
|
|
Lt => op("<"),
|
|
|
|
Le => op("<="),
|
|
|
|
EqEq => op("=="),
|
|
|
|
Ne => op("!="),
|
|
|
|
Ge => op(">="),
|
|
|
|
Gt => op(">"),
|
|
|
|
AndAnd => op("&&"),
|
|
|
|
OrOr => op("||"),
|
|
|
|
Not => op("!"),
|
|
|
|
Tilde => op("~"),
|
|
|
|
BinOp(Plus) => op("+"),
|
|
|
|
BinOp(Minus) => op("-"),
|
|
|
|
BinOp(Star) => op("*"),
|
|
|
|
BinOp(Slash) => op("/"),
|
|
|
|
BinOp(Percent) => op("%"),
|
|
|
|
BinOp(Caret) => op("^"),
|
|
|
|
BinOp(And) => op("&"),
|
|
|
|
BinOp(Or) => op("|"),
|
|
|
|
BinOp(Shl) => op("<<"),
|
|
|
|
BinOp(Shr) => op(">>"),
|
|
|
|
BinOpEq(Plus) => op("+="),
|
|
|
|
BinOpEq(Minus) => op("-="),
|
|
|
|
BinOpEq(Star) => op("*="),
|
|
|
|
BinOpEq(Slash) => op("/="),
|
|
|
|
BinOpEq(Percent) => op("%="),
|
|
|
|
BinOpEq(Caret) => op("^="),
|
|
|
|
BinOpEq(And) => op("&="),
|
|
|
|
BinOpEq(Or) => op("|="),
|
|
|
|
BinOpEq(Shl) => op("<<="),
|
|
|
|
BinOpEq(Shr) => op(">>="),
|
|
|
|
At => op("@"),
|
|
|
|
Dot => op("."),
|
|
|
|
DotDot => op(".."),
|
|
|
|
DotDotDot => op("..."),
|
|
|
|
DotDotEq => op("..="),
|
|
|
|
Comma => op(","),
|
|
|
|
Semi => op(";"),
|
|
|
|
Colon => op(":"),
|
2024-04-04 19:03:32 +02:00
|
|
|
PathSep => op("::"),
|
2022-06-27 20:03:56 -04:00
|
|
|
RArrow => op("->"),
|
|
|
|
LArrow => op("<-"),
|
|
|
|
FatArrow => op("=>"),
|
|
|
|
Pound => op("#"),
|
|
|
|
Dollar => op("$"),
|
|
|
|
Question => op("?"),
|
|
|
|
SingleQuote => op("'"),
|
2021-07-01 17:36:38 -04:00
|
|
|
|
2024-02-13 23:28:27 +00:00
|
|
|
Ident(sym, is_raw) => {
|
|
|
|
trees.push(TokenTree::Ident(Ident { sym, is_raw: is_raw.into(), span }))
|
|
|
|
}
|
2024-04-22 19:46:51 +10:00
|
|
|
NtIdent(ident, is_raw) => trees.push(TokenTree::Ident(Ident {
|
|
|
|
sym: ident.name,
|
|
|
|
is_raw: is_raw.into(),
|
|
|
|
span: ident.span,
|
|
|
|
})),
|
|
|
|
|
2024-09-05 05:43:55 -04:00
|
|
|
Lifetime(name, is_raw) => {
|
2021-07-01 17:36:38 -04:00
|
|
|
let ident = symbol::Ident::new(name, span).without_first_quote();
|
2022-06-27 20:03:56 -04:00
|
|
|
trees.extend([
|
|
|
|
TokenTree::Punct(Punct { ch: b'\'', joint: true, span }),
|
2024-09-05 05:43:55 -04:00
|
|
|
TokenTree::Ident(Ident { sym: ident.name, is_raw: is_raw.into(), span }),
|
2022-06-27 20:03:56 -04:00
|
|
|
]);
|
2018-03-16 01:09:22 +02:00
|
|
|
}
|
2024-09-05 05:43:55 -04:00
|
|
|
NtLifetime(ident, is_raw) => {
|
|
|
|
let stream =
|
|
|
|
TokenStream::token_alone(token::Lifetime(ident.name, is_raw), ident.span);
|
2024-04-22 19:46:51 +10:00
|
|
|
trees.push(TokenTree::Group(Group {
|
|
|
|
delimiter: pm::Delimiter::None,
|
|
|
|
stream: Some(stream),
|
|
|
|
span: DelimSpan::from_single(span),
|
|
|
|
}))
|
|
|
|
}
|
|
|
|
|
2022-07-03 01:04:31 -04:00
|
|
|
Literal(token::Lit { kind, symbol, suffix }) => {
|
|
|
|
trees.push(TokenTree::Literal(self::Literal {
|
|
|
|
kind: FromInternal::from_internal(kind),
|
|
|
|
symbol,
|
|
|
|
suffix,
|
|
|
|
span,
|
|
|
|
}));
|
|
|
|
}
|
2021-07-01 17:36:38 -04:00
|
|
|
DocComment(_, attr_style, data) => {
|
|
|
|
let mut escaped = String::new();
|
|
|
|
for ch in data.as_str().chars() {
|
|
|
|
escaped.extend(ch.escape_debug());
|
|
|
|
}
|
2022-06-27 20:03:56 -04:00
|
|
|
let stream = [
|
2024-02-13 23:28:27 +00:00
|
|
|
Ident(sym::doc, IdentIsRaw::No),
|
2021-07-01 17:36:38 -04:00
|
|
|
Eq,
|
|
|
|
TokenKind::lit(token::Str, Symbol::intern(&escaped), None),
|
|
|
|
]
|
|
|
|
.into_iter()
|
Remove `TreeAndSpacing`.
A `TokenStream` contains a `Lrc<Vec<(TokenTree, Spacing)>>`. But this is
not quite right. `Spacing` makes sense for `TokenTree::Token`, but does
not make sense for `TokenTree::Delimited`, because a
`TokenTree::Delimited` cannot be joined with another `TokenTree`.
This commit fixes this problem, by adding `Spacing` to `TokenTree::Token`,
changing `TokenStream` to contain a `Lrc<Vec<TokenTree>>`, and removing the
`TreeAndSpacing` typedef.
The commit removes these two impls:
- `impl From<TokenTree> for TokenStream`
- `impl From<TokenTree> for TreeAndSpacing`
These were useful, but also resulted in code with many `.into()` calls
that was hard to read, particularly for anyone not highly familiar with
the relevant types. This commit makes some other changes to compensate:
- `TokenTree::token()` becomes `TokenTree::token_{alone,joint}()`.
- `TokenStream::token_{alone,joint}()` are added.
- `TokenStream::delimited` is added.
This results in things like this:
```rust
TokenTree::token(token::Semi, stmt.span).into()
```
changing to this:
```rust
TokenStream::token_alone(token::Semi, stmt.span)
```
This makes the type of the result, and its spacing, clearer.
These changes also simplifies `Cursor` and `CursorRef`, because they no longer
need to distinguish between `next` and `next_with_spacing`.
2022-07-28 10:31:04 +10:00
|
|
|
.map(|kind| tokenstream::TokenTree::token_alone(kind, span))
|
2021-07-01 17:36:38 -04:00
|
|
|
.collect();
|
2022-06-27 20:03:56 -04:00
|
|
|
trees.push(TokenTree::Punct(Punct { ch: b'#', joint: false, span }));
|
2021-07-01 17:36:38 -04:00
|
|
|
if attr_style == ast::AttrStyle::Inner {
|
2022-06-27 20:03:56 -04:00
|
|
|
trees.push(TokenTree::Punct(Punct { ch: b'!', joint: false, span }));
|
2021-07-01 17:36:38 -04:00
|
|
|
}
|
|
|
|
trees.push(TokenTree::Group(Group {
|
|
|
|
delimiter: pm::Delimiter::Bracket,
|
|
|
|
stream: Some(stream),
|
|
|
|
span: DelimSpan::from_single(span),
|
|
|
|
}));
|
2018-07-19 15:59:08 +03:00
|
|
|
}
|
|
|
|
|
2021-07-01 17:36:38 -04:00
|
|
|
Interpolated(nt) => {
|
2024-04-22 16:29:27 +10:00
|
|
|
let stream = TokenStream::from_nonterminal_ast(&nt);
|
2024-05-27 09:00:48 +10:00
|
|
|
// We used to have an alternative behaviour for crates that
|
|
|
|
// needed it: a hack used to pass AST fragments to
|
|
|
|
// attribute and derive macros as a single nonterminal
|
|
|
|
// token instead of a token stream. Such token needs to be
|
|
|
|
// "unwrapped" and not represented as a delimited group. We
|
|
|
|
// had a lint for a long time, but now we just emit a hard
|
|
|
|
// error. Eventually we might remove the special case hard
|
|
|
|
// error check altogether. See #73345.
|
|
|
|
crate::base::nt_pretty_printing_compatibility_hack(&nt, rustc.ecx.sess);
|
|
|
|
trees.push(TokenTree::Group(Group {
|
|
|
|
delimiter: pm::Delimiter::None,
|
|
|
|
stream: Some(stream),
|
|
|
|
span: DelimSpan::from_single(span),
|
|
|
|
}))
|
2021-07-01 17:36:38 -04:00
|
|
|
}
|
|
|
|
|
|
|
|
OpenDelim(..) | CloseDelim(..) => unreachable!(),
|
|
|
|
Eof => unreachable!(),
|
|
|
|
}
|
2018-07-19 15:59:08 +03:00
|
|
|
}
|
2021-07-01 17:36:38 -04:00
|
|
|
trees
|
2018-07-19 15:59:08 +03:00
|
|
|
}
|
2018-03-20 16:41:14 +02:00
|
|
|
}
|
2018-07-19 15:59:08 +03:00
|
|
|
|
Remove `TokenStreamBuilder`.
`TokenStreamBuilder` exists to concatenate multiple `TokenStream`s
together. This commit removes it, and moves the concatenation
functionality directly into `TokenStream`, via two new methods
`push_tree` and `push_stream`. This makes things both simpler and
faster.
`push_tree` is particularly important. `TokenStreamBuilder` only had a
single `push` method, which pushed a stream. But in practice most of the
time we push a single token tree rather than a stream, and `push_tree`
avoids the need to build a token stream with a single entry (which
requires two allocations, one for the `Lrc` and one for the `Vec`).
The main `push_tree` use arises from a change to one of the `ToInternal`
impls in `proc_macro_server.rs`. It now returns a `SmallVec` instead of
a `TokenStream`. This return value is then iterated over by
`concat_trees`, which does `push_tree` on each element. Furthermore, the
use of `SmallVec` avoids more allocations, because there is always only
one or two token trees.
Note: the removed `TokenStreamBuilder::push` method had some code to
deal with a quadratic blowup case from #57735. This commit removes the
code. I tried and failed to reproduce the blowup from that PR, before
and after this change. Various other changes have happened to
`TokenStreamBuilder` in the meantime, so I suspect the original problem
is no longer relevant, though I don't have proof of this. Generally
speaking, repeatedly extending a `Vec` without pre-determining its
capacity is *not* quadratic. It's also incredibly common, within rustc
and many other Rust programs, so if there were performance problems
there you'd think it would show up in other places, too.
2022-10-05 10:38:15 +11:00
|
|
|
// We use a `SmallVec` because the output size is always one or two `TokenTree`s.
|
|
|
|
impl ToInternal<SmallVec<[tokenstream::TokenTree; 2]>>
|
|
|
|
for (TokenTree<TokenStream, Span, Symbol>, &mut Rustc<'_, '_>)
|
|
|
|
{
|
|
|
|
fn to_internal(self) -> SmallVec<[tokenstream::TokenTree; 2]> {
|
2020-02-29 20:37:32 +03:00
|
|
|
use rustc_ast::token::*;
|
2018-03-16 01:09:22 +02:00
|
|
|
|
2023-10-12 15:36:14 +11:00
|
|
|
// The code below is conservative, using `token_alone`/`Spacing::Alone`
|
2024-05-17 09:54:53 +10:00
|
|
|
// in most places. It's hard in general to do better when working at
|
|
|
|
// the token level. When the resulting code is pretty-printed by
|
|
|
|
// `print_tts` the `space_between` function helps avoid a lot of
|
|
|
|
// unnecessary whitespace, so the results aren't too bad.
|
2022-06-30 21:05:46 -04:00
|
|
|
let (tree, rustc) = self;
|
2022-10-05 10:33:49 +11:00
|
|
|
match tree {
|
|
|
|
TokenTree::Punct(Punct { ch, joint, span }) => {
|
|
|
|
let kind = match ch {
|
|
|
|
b'=' => Eq,
|
|
|
|
b'<' => Lt,
|
|
|
|
b'>' => Gt,
|
|
|
|
b'!' => Not,
|
|
|
|
b'~' => Tilde,
|
|
|
|
b'+' => BinOp(Plus),
|
|
|
|
b'-' => BinOp(Minus),
|
|
|
|
b'*' => BinOp(Star),
|
|
|
|
b'/' => BinOp(Slash),
|
|
|
|
b'%' => BinOp(Percent),
|
|
|
|
b'^' => BinOp(Caret),
|
|
|
|
b'&' => BinOp(And),
|
|
|
|
b'|' => BinOp(Or),
|
|
|
|
b'@' => At,
|
|
|
|
b'.' => Dot,
|
|
|
|
b',' => Comma,
|
|
|
|
b';' => Semi,
|
|
|
|
b':' => Colon,
|
|
|
|
b'#' => Pound,
|
|
|
|
b'$' => Dollar,
|
|
|
|
b'?' => Question,
|
|
|
|
b'\'' => SingleQuote,
|
|
|
|
_ => unreachable!(),
|
|
|
|
};
|
2023-08-08 11:43:44 +10:00
|
|
|
// We never produce `token::Spacing::JointHidden` here, which
|
|
|
|
// means the pretty-printing of code produced by proc macros is
|
|
|
|
// ugly, with lots of whitespace between tokens. This is
|
|
|
|
// unavoidable because `proc_macro::Spacing` only applies to
|
|
|
|
// `Punct` token trees.
|
Remove `TokenStreamBuilder`.
`TokenStreamBuilder` exists to concatenate multiple `TokenStream`s
together. This commit removes it, and moves the concatenation
functionality directly into `TokenStream`, via two new methods
`push_tree` and `push_stream`. This makes things both simpler and
faster.
`push_tree` is particularly important. `TokenStreamBuilder` only had a
single `push` method, which pushed a stream. But in practice most of the
time we push a single token tree rather than a stream, and `push_tree`
avoids the need to build a token stream with a single entry (which
requires two allocations, one for the `Lrc` and one for the `Vec`).
The main `push_tree` use arises from a change to one of the `ToInternal`
impls in `proc_macro_server.rs`. It now returns a `SmallVec` instead of
a `TokenStream`. This return value is then iterated over by
`concat_trees`, which does `push_tree` on each element. Furthermore, the
use of `SmallVec` avoids more allocations, because there is always only
one or two token trees.
Note: the removed `TokenStreamBuilder::push` method had some code to
deal with a quadratic blowup case from #57735. This commit removes the
code. I tried and failed to reproduce the blowup from that PR, before
and after this change. Various other changes have happened to
`TokenStreamBuilder` in the meantime, so I suspect the original problem
is no longer relevant, though I don't have proof of this. Generally
speaking, repeatedly extending a `Vec` without pre-determining its
capacity is *not* quadratic. It's also incredibly common, within rustc
and many other Rust programs, so if there were performance problems
there you'd think it would show up in other places, too.
2022-10-05 10:38:15 +11:00
|
|
|
smallvec![if joint {
|
|
|
|
tokenstream::TokenTree::token_joint(kind, span)
|
2022-10-05 10:33:49 +11:00
|
|
|
} else {
|
Remove `TokenStreamBuilder`.
`TokenStreamBuilder` exists to concatenate multiple `TokenStream`s
together. This commit removes it, and moves the concatenation
functionality directly into `TokenStream`, via two new methods
`push_tree` and `push_stream`. This makes things both simpler and
faster.
`push_tree` is particularly important. `TokenStreamBuilder` only had a
single `push` method, which pushed a stream. But in practice most of the
time we push a single token tree rather than a stream, and `push_tree`
avoids the need to build a token stream with a single entry (which
requires two allocations, one for the `Lrc` and one for the `Vec`).
The main `push_tree` use arises from a change to one of the `ToInternal`
impls in `proc_macro_server.rs`. It now returns a `SmallVec` instead of
a `TokenStream`. This return value is then iterated over by
`concat_trees`, which does `push_tree` on each element. Furthermore, the
use of `SmallVec` avoids more allocations, because there is always only
one or two token trees.
Note: the removed `TokenStreamBuilder::push` method had some code to
deal with a quadratic blowup case from #57735. This commit removes the
code. I tried and failed to reproduce the blowup from that PR, before
and after this change. Various other changes have happened to
`TokenStreamBuilder` in the meantime, so I suspect the original problem
is no longer relevant, though I don't have proof of this. Generally
speaking, repeatedly extending a `Vec` without pre-determining its
capacity is *not* quadratic. It's also incredibly common, within rustc
and many other Rust programs, so if there were performance problems
there you'd think it would show up in other places, too.
2022-10-05 10:38:15 +11:00
|
|
|
tokenstream::TokenTree::token_alone(kind, span)
|
|
|
|
}]
|
2022-10-05 10:33:49 +11:00
|
|
|
}
|
2021-07-01 17:36:38 -04:00
|
|
|
TokenTree::Group(Group { delimiter, stream, span: DelimSpan { open, close, .. } }) => {
|
Remove `TokenStreamBuilder`.
`TokenStreamBuilder` exists to concatenate multiple `TokenStream`s
together. This commit removes it, and moves the concatenation
functionality directly into `TokenStream`, via two new methods
`push_tree` and `push_stream`. This makes things both simpler and
faster.
`push_tree` is particularly important. `TokenStreamBuilder` only had a
single `push` method, which pushed a stream. But in practice most of the
time we push a single token tree rather than a stream, and `push_tree`
avoids the need to build a token stream with a single entry (which
requires two allocations, one for the `Lrc` and one for the `Vec`).
The main `push_tree` use arises from a change to one of the `ToInternal`
impls in `proc_macro_server.rs`. It now returns a `SmallVec` instead of
a `TokenStream`. This return value is then iterated over by
`concat_trees`, which does `push_tree` on each element. Furthermore, the
use of `SmallVec` avoids more allocations, because there is always only
one or two token trees.
Note: the removed `TokenStreamBuilder::push` method had some code to
deal with a quadratic blowup case from #57735. This commit removes the
code. I tried and failed to reproduce the blowup from that PR, before
and after this change. Various other changes have happened to
`TokenStreamBuilder` in the meantime, so I suspect the original problem
is no longer relevant, though I don't have proof of this. Generally
speaking, repeatedly extending a `Vec` without pre-determining its
capacity is *not* quadratic. It's also incredibly common, within rustc
and many other Rust programs, so if there were performance problems
there you'd think it would show up in other places, too.
2022-10-05 10:38:15 +11:00
|
|
|
smallvec![tokenstream::TokenTree::Delimited(
|
2021-07-01 17:36:38 -04:00
|
|
|
tokenstream::DelimSpan { open, close },
|
2023-10-12 15:36:14 +11:00
|
|
|
DelimSpacing::new(Spacing::Alone, Spacing::Alone),
|
2021-07-01 17:36:38 -04:00
|
|
|
delimiter.to_internal(),
|
|
|
|
stream.unwrap_or_default(),
|
Remove `TokenStreamBuilder`.
`TokenStreamBuilder` exists to concatenate multiple `TokenStream`s
together. This commit removes it, and moves the concatenation
functionality directly into `TokenStream`, via two new methods
`push_tree` and `push_stream`. This makes things both simpler and
faster.
`push_tree` is particularly important. `TokenStreamBuilder` only had a
single `push` method, which pushed a stream. But in practice most of the
time we push a single token tree rather than a stream, and `push_tree`
avoids the need to build a token stream with a single entry (which
requires two allocations, one for the `Lrc` and one for the `Vec`).
The main `push_tree` use arises from a change to one of the `ToInternal`
impls in `proc_macro_server.rs`. It now returns a `SmallVec` instead of
a `TokenStream`. This return value is then iterated over by
`concat_trees`, which does `push_tree` on each element. Furthermore, the
use of `SmallVec` avoids more allocations, because there is always only
one or two token trees.
Note: the removed `TokenStreamBuilder::push` method had some code to
deal with a quadratic blowup case from #57735. This commit removes the
code. I tried and failed to reproduce the blowup from that PR, before
and after this change. Various other changes have happened to
`TokenStreamBuilder` in the meantime, so I suspect the original problem
is no longer relevant, though I don't have proof of this. Generally
speaking, repeatedly extending a `Vec` without pre-determining its
capacity is *not* quadratic. It's also incredibly common, within rustc
and many other Rust programs, so if there were performance problems
there you'd think it would show up in other places, too.
2022-10-05 10:38:15 +11:00
|
|
|
)]
|
2018-07-19 15:59:08 +03:00
|
|
|
}
|
2018-12-08 21:00:39 +03:00
|
|
|
TokenTree::Ident(self::Ident { sym, is_raw, span }) => {
|
2024-03-04 16:31:49 +11:00
|
|
|
rustc.psess().symbol_gallery.insert(sym, span);
|
2024-02-13 23:28:27 +00:00
|
|
|
smallvec![tokenstream::TokenTree::token_alone(Ident(sym, is_raw.into()), span)]
|
2018-07-19 15:59:08 +03:00
|
|
|
}
|
2018-03-16 01:09:22 +02:00
|
|
|
TokenTree::Literal(self::Literal {
|
2022-07-03 01:04:31 -04:00
|
|
|
kind: self::LitKind::Integer,
|
|
|
|
symbol,
|
|
|
|
suffix,
|
2018-07-19 15:59:08 +03:00
|
|
|
span,
|
2020-02-26 13:03:46 +01:00
|
|
|
}) if symbol.as_str().starts_with('-') => {
|
2018-07-19 15:59:08 +03:00
|
|
|
let minus = BinOp(BinOpToken::Minus);
|
2019-05-19 01:04:26 +03:00
|
|
|
let symbol = Symbol::intern(&symbol.as_str()[1..]);
|
2019-06-04 17:55:23 +03:00
|
|
|
let integer = TokenKind::lit(token::Integer, symbol, suffix);
|
2023-08-08 11:43:44 +10:00
|
|
|
let a = tokenstream::TokenTree::token_joint_hidden(minus, span);
|
Remove `TreeAndSpacing`.
A `TokenStream` contains a `Lrc<Vec<(TokenTree, Spacing)>>`. But this is
not quite right. `Spacing` makes sense for `TokenTree::Token`, but does
not make sense for `TokenTree::Delimited`, because a
`TokenTree::Delimited` cannot be joined with another `TokenTree`.
This commit fixes this problem, by adding `Spacing` to `TokenTree::Token`,
changing `TokenStream` to contain a `Lrc<Vec<TokenTree>>`, and removing the
`TreeAndSpacing` typedef.
The commit removes these two impls:
- `impl From<TokenTree> for TokenStream`
- `impl From<TokenTree> for TreeAndSpacing`
These were useful, but also resulted in code with many `.into()` calls
that was hard to read, particularly for anyone not highly familiar with
the relevant types. This commit makes some other changes to compensate:
- `TokenTree::token()` becomes `TokenTree::token_{alone,joint}()`.
- `TokenStream::token_{alone,joint}()` are added.
- `TokenStream::delimited` is added.
This results in things like this:
```rust
TokenTree::token(token::Semi, stmt.span).into()
```
changing to this:
```rust
TokenStream::token_alone(token::Semi, stmt.span)
```
This makes the type of the result, and its spacing, clearer.
These changes also simplifies `Cursor` and `CursorRef`, because they no longer
need to distinguish between `next` and `next_with_spacing`.
2022-07-28 10:31:04 +10:00
|
|
|
let b = tokenstream::TokenTree::token_alone(integer, span);
|
Remove `TokenStreamBuilder`.
`TokenStreamBuilder` exists to concatenate multiple `TokenStream`s
together. This commit removes it, and moves the concatenation
functionality directly into `TokenStream`, via two new methods
`push_tree` and `push_stream`. This makes things both simpler and
faster.
`push_tree` is particularly important. `TokenStreamBuilder` only had a
single `push` method, which pushed a stream. But in practice most of the
time we push a single token tree rather than a stream, and `push_tree`
avoids the need to build a token stream with a single entry (which
requires two allocations, one for the `Lrc` and one for the `Vec`).
The main `push_tree` use arises from a change to one of the `ToInternal`
impls in `proc_macro_server.rs`. It now returns a `SmallVec` instead of
a `TokenStream`. This return value is then iterated over by
`concat_trees`, which does `push_tree` on each element. Furthermore, the
use of `SmallVec` avoids more allocations, because there is always only
one or two token trees.
Note: the removed `TokenStreamBuilder::push` method had some code to
deal with a quadratic blowup case from #57735. This commit removes the
code. I tried and failed to reproduce the blowup from that PR, before
and after this change. Various other changes have happened to
`TokenStreamBuilder` in the meantime, so I suspect the original problem
is no longer relevant, though I don't have proof of this. Generally
speaking, repeatedly extending a `Vec` without pre-determining its
capacity is *not* quadratic. It's also incredibly common, within rustc
and many other Rust programs, so if there were performance problems
there you'd think it would show up in other places, too.
2022-10-05 10:38:15 +11:00
|
|
|
smallvec![a, b]
|
2018-07-19 15:59:08 +03:00
|
|
|
}
|
2018-03-16 01:09:22 +02:00
|
|
|
TokenTree::Literal(self::Literal {
|
2022-07-03 01:04:31 -04:00
|
|
|
kind: self::LitKind::Float,
|
|
|
|
symbol,
|
|
|
|
suffix,
|
2018-07-19 15:59:08 +03:00
|
|
|
span,
|
2020-02-26 13:03:46 +01:00
|
|
|
}) if symbol.as_str().starts_with('-') => {
|
2018-07-19 15:59:08 +03:00
|
|
|
let minus = BinOp(BinOpToken::Minus);
|
2019-05-19 01:04:26 +03:00
|
|
|
let symbol = Symbol::intern(&symbol.as_str()[1..]);
|
2019-06-04 17:55:23 +03:00
|
|
|
let float = TokenKind::lit(token::Float, symbol, suffix);
|
2023-08-08 11:43:44 +10:00
|
|
|
let a = tokenstream::TokenTree::token_joint_hidden(minus, span);
|
Remove `TreeAndSpacing`.
A `TokenStream` contains a `Lrc<Vec<(TokenTree, Spacing)>>`. But this is
not quite right. `Spacing` makes sense for `TokenTree::Token`, but does
not make sense for `TokenTree::Delimited`, because a
`TokenTree::Delimited` cannot be joined with another `TokenTree`.
This commit fixes this problem, by adding `Spacing` to `TokenTree::Token`,
changing `TokenStream` to contain a `Lrc<Vec<TokenTree>>`, and removing the
`TreeAndSpacing` typedef.
The commit removes these two impls:
- `impl From<TokenTree> for TokenStream`
- `impl From<TokenTree> for TreeAndSpacing`
These were useful, but also resulted in code with many `.into()` calls
that was hard to read, particularly for anyone not highly familiar with
the relevant types. This commit makes some other changes to compensate:
- `TokenTree::token()` becomes `TokenTree::token_{alone,joint}()`.
- `TokenStream::token_{alone,joint}()` are added.
- `TokenStream::delimited` is added.
This results in things like this:
```rust
TokenTree::token(token::Semi, stmt.span).into()
```
changing to this:
```rust
TokenStream::token_alone(token::Semi, stmt.span)
```
This makes the type of the result, and its spacing, clearer.
These changes also simplifies `Cursor` and `CursorRef`, because they no longer
need to distinguish between `next` and `next_with_spacing`.
2022-07-28 10:31:04 +10:00
|
|
|
let b = tokenstream::TokenTree::token_alone(float, span);
|
Remove `TokenStreamBuilder`.
`TokenStreamBuilder` exists to concatenate multiple `TokenStream`s
together. This commit removes it, and moves the concatenation
functionality directly into `TokenStream`, via two new methods
`push_tree` and `push_stream`. This makes things both simpler and
faster.
`push_tree` is particularly important. `TokenStreamBuilder` only had a
single `push` method, which pushed a stream. But in practice most of the
time we push a single token tree rather than a stream, and `push_tree`
avoids the need to build a token stream with a single entry (which
requires two allocations, one for the `Lrc` and one for the `Vec`).
The main `push_tree` use arises from a change to one of the `ToInternal`
impls in `proc_macro_server.rs`. It now returns a `SmallVec` instead of
a `TokenStream`. This return value is then iterated over by
`concat_trees`, which does `push_tree` on each element. Furthermore, the
use of `SmallVec` avoids more allocations, because there is always only
one or two token trees.
Note: the removed `TokenStreamBuilder::push` method had some code to
deal with a quadratic blowup case from #57735. This commit removes the
code. I tried and failed to reproduce the blowup from that PR, before
and after this change. Various other changes have happened to
`TokenStreamBuilder` in the meantime, so I suspect the original problem
is no longer relevant, though I don't have proof of this. Generally
speaking, repeatedly extending a `Vec` without pre-determining its
capacity is *not* quadratic. It's also incredibly common, within rustc
and many other Rust programs, so if there were performance problems
there you'd think it would show up in other places, too.
2022-10-05 10:38:15 +11:00
|
|
|
smallvec![a, b]
|
2018-07-19 15:59:08 +03:00
|
|
|
}
|
2022-07-03 01:04:31 -04:00
|
|
|
TokenTree::Literal(self::Literal { kind, symbol, suffix, span }) => {
|
Remove `TokenStreamBuilder`.
`TokenStreamBuilder` exists to concatenate multiple `TokenStream`s
together. This commit removes it, and moves the concatenation
functionality directly into `TokenStream`, via two new methods
`push_tree` and `push_stream`. This makes things both simpler and
faster.
`push_tree` is particularly important. `TokenStreamBuilder` only had a
single `push` method, which pushed a stream. But in practice most of the
time we push a single token tree rather than a stream, and `push_tree`
avoids the need to build a token stream with a single entry (which
requires two allocations, one for the `Lrc` and one for the `Vec`).
The main `push_tree` use arises from a change to one of the `ToInternal`
impls in `proc_macro_server.rs`. It now returns a `SmallVec` instead of
a `TokenStream`. This return value is then iterated over by
`concat_trees`, which does `push_tree` on each element. Furthermore, the
use of `SmallVec` avoids more allocations, because there is always only
one or two token trees.
Note: the removed `TokenStreamBuilder::push` method had some code to
deal with a quadratic blowup case from #57735. This commit removes the
code. I tried and failed to reproduce the blowup from that PR, before
and after this change. Various other changes have happened to
`TokenStreamBuilder` in the meantime, so I suspect the original problem
is no longer relevant, though I don't have proof of this. Generally
speaking, repeatedly extending a `Vec` without pre-determining its
capacity is *not* quadratic. It's also incredibly common, within rustc
and many other Rust programs, so if there were performance problems
there you'd think it would show up in other places, too.
2022-10-05 10:38:15 +11:00
|
|
|
smallvec![tokenstream::TokenTree::token_alone(
|
2022-07-03 01:04:31 -04:00
|
|
|
TokenKind::lit(kind.to_internal(), symbol, suffix),
|
|
|
|
span,
|
Remove `TokenStreamBuilder`.
`TokenStreamBuilder` exists to concatenate multiple `TokenStream`s
together. This commit removes it, and moves the concatenation
functionality directly into `TokenStream`, via two new methods
`push_tree` and `push_stream`. This makes things both simpler and
faster.
`push_tree` is particularly important. `TokenStreamBuilder` only had a
single `push` method, which pushed a stream. But in practice most of the
time we push a single token tree rather than a stream, and `push_tree`
avoids the need to build a token stream with a single entry (which
requires two allocations, one for the `Lrc` and one for the `Vec`).
The main `push_tree` use arises from a change to one of the `ToInternal`
impls in `proc_macro_server.rs`. It now returns a `SmallVec` instead of
a `TokenStream`. This return value is then iterated over by
`concat_trees`, which does `push_tree` on each element. Furthermore, the
use of `SmallVec` avoids more allocations, because there is always only
one or two token trees.
Note: the removed `TokenStreamBuilder::push` method had some code to
deal with a quadratic blowup case from #57735. This commit removes the
code. I tried and failed to reproduce the blowup from that PR, before
and after this change. Various other changes have happened to
`TokenStreamBuilder` in the meantime, so I suspect the original problem
is no longer relevant, though I don't have proof of this. Generally
speaking, repeatedly extending a `Vec` without pre-determining its
capacity is *not* quadratic. It's also incredibly common, within rustc
and many other Rust programs, so if there were performance problems
there you'd think it would show up in other places, too.
2022-10-05 10:38:15 +11:00
|
|
|
)]
|
2018-07-19 15:59:08 +03:00
|
|
|
}
|
Remove `TreeAndSpacing`.
A `TokenStream` contains a `Lrc<Vec<(TokenTree, Spacing)>>`. But this is
not quite right. `Spacing` makes sense for `TokenTree::Token`, but does
not make sense for `TokenTree::Delimited`, because a
`TokenTree::Delimited` cannot be joined with another `TokenTree`.
This commit fixes this problem, by adding `Spacing` to `TokenTree::Token`,
changing `TokenStream` to contain a `Lrc<Vec<TokenTree>>`, and removing the
`TreeAndSpacing` typedef.
The commit removes these two impls:
- `impl From<TokenTree> for TokenStream`
- `impl From<TokenTree> for TreeAndSpacing`
These were useful, but also resulted in code with many `.into()` calls
that was hard to read, particularly for anyone not highly familiar with
the relevant types. This commit makes some other changes to compensate:
- `TokenTree::token()` becomes `TokenTree::token_{alone,joint}()`.
- `TokenStream::token_{alone,joint}()` are added.
- `TokenStream::delimited` is added.
This results in things like this:
```rust
TokenTree::token(token::Semi, stmt.span).into()
```
changing to this:
```rust
TokenStream::token_alone(token::Semi, stmt.span)
```
This makes the type of the result, and its spacing, clearer.
These changes also simplifies `Cursor` and `CursorRef`, because they no longer
need to distinguish between `next` and `next_with_spacing`.
2022-07-28 10:31:04 +10:00
|
|
|
}
|
2018-07-19 15:59:08 +03:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2020-01-09 11:18:47 +01:00
|
|
|
impl ToInternal<rustc_errors::Level> for Level {
|
|
|
|
fn to_internal(self) -> rustc_errors::Level {
|
2018-07-19 15:59:08 +03:00
|
|
|
match self {
|
2024-01-04 11:44:37 +11:00
|
|
|
Level::Error => rustc_errors::Level::Error,
|
2024-01-09 12:28:45 +11:00
|
|
|
Level::Warning => rustc_errors::Level::Warning,
|
2020-01-09 11:18:47 +01:00
|
|
|
Level::Note => rustc_errors::Level::Note,
|
|
|
|
Level::Help => rustc_errors::Level::Help,
|
2018-03-20 16:41:14 +02:00
|
|
|
_ => unreachable!("unknown proc_macro::Level variant: {:?}", self),
|
2018-07-19 15:59:08 +03:00
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
2018-03-16 01:09:22 +02:00
|
|
|
|
2024-08-27 12:40:38 +10:00
|
|
|
pub(crate) struct FreeFunctions;
|
2020-07-22 21:08:01 +03:00
|
|
|
|
2021-07-18 15:53:06 -04:00
|
|
|
pub(crate) struct Rustc<'a, 'b> {
|
|
|
|
ecx: &'a mut ExtCtxt<'b>,
|
2018-03-19 22:44:24 +02:00
|
|
|
def_site: Span,
|
|
|
|
call_site: Span,
|
2019-09-22 18:38:02 +03:00
|
|
|
mixed_site: Span,
|
Implement span quoting for proc-macros
This PR implements span quoting, allowing proc-macros to produce spans
pointing *into their own crate*. This is used by the unstable
`proc_macro::quote!` macro, allowing us to get error messages like this:
```
error[E0412]: cannot find type `MissingType` in this scope
--> $DIR/auxiliary/span-from-proc-macro.rs:37:20
|
LL | pub fn error_from_attribute(_args: TokenStream, _input: TokenStream) -> TokenStream {
| ----------------------------------------------------------------------------------- in this expansion of procedural macro `#[error_from_attribute]`
...
LL | field: MissingType
| ^^^^^^^^^^^ not found in this scope
|
::: $DIR/span-from-proc-macro.rs:8:1
|
LL | #[error_from_attribute]
| ----------------------- in this macro invocation
```
Here, `MissingType` occurs inside the implementation of the proc-macro
`#[error_from_attribute]`. Previosuly, this would always result in a
span pointing at `#[error_from_attribute]`
This will make many proc-macro-related error message much more useful -
when a proc-macro generates code containing an error, users will get an
error message pointing directly at that code (within the macro
definition), instead of always getting a span pointing at the macro
invocation site.
This is implemented as follows:
* When a proc-macro crate is being *compiled*, it causes the `quote!`
macro to get run. This saves all of the sapns in the input to `quote!`
into the metadata of *the proc-macro-crate* (which we are currently
compiling). The `quote!` macro then expands to a call to
`proc_macro::Span::recover_proc_macro_span(id)`, where `id` is an
opaque identifier for the span in the crate metadata.
* When the same proc-macro crate is *run* (e.g. it is loaded from disk
and invoked by some consumer crate), the call to
`proc_macro::Span::recover_proc_macro_span` causes us to load the span
from the proc-macro crate's metadata. The proc-macro then produces a
`TokenStream` containing a `Span` pointing into the proc-macro crate
itself.
The recursive nature of 'quote!' can be difficult to understand at
first. The file `src/test/ui/proc-macro/quote-debug.stdout` shows
the output of the `quote!` macro, which should make this eaier to
understand.
This PR also supports custom quoting spans in custom quote macros (e.g.
the `quote` crate). All span quoting goes through the
`proc_macro::quote_span` method, which can be called by a custom quote
macro to perform span quoting. An example of this usage is provided in
`src/test/ui/proc-macro/auxiliary/custom-quote.rs`
Custom quoting currently has a few limitations:
In order to quote a span, we need to generate a call to
`proc_macro::Span::recover_proc_macro_span`. However, proc-macros
support renaming the `proc_macro` crate, so we can't simply hardcode
this path. Previously, the `quote_span` method used the path
`crate::Span` - however, this only works when it is called by the
builtin `quote!` macro in the same crate. To support being called from
arbitrary crates, we need access to the name of the `proc_macro` crate
to generate a path. This PR adds an additional argument to `quote_span`
to specify the name of the `proc_macro` crate. Howver, this feels kind
of hacky, and we may want to change this before stabilizing anything
quote-related.
Additionally, using `quote_span` currently requires enabling the
`proc_macro_internals` feature. The builtin `quote!` macro
has an `#[allow_internal_unstable]` attribute, but this won't work for
custom quote implementations. This will likely require some additional
tricks to apply `allow_internal_unstable` to the span of
`proc_macro::Span::recover_proc_macro_span`.
2020-08-02 19:52:16 -04:00
|
|
|
krate: CrateNum,
|
|
|
|
rebased_spans: FxHashMap<usize, Span>,
|
2018-03-19 22:44:24 +02:00
|
|
|
}
|
|
|
|
|
2021-07-18 15:53:06 -04:00
|
|
|
impl<'a, 'b> Rustc<'a, 'b> {
|
2024-08-27 12:40:38 +10:00
|
|
|
pub(crate) fn new(ecx: &'a mut ExtCtxt<'b>) -> Self {
|
2021-07-18 15:53:06 -04:00
|
|
|
let expn_data = ecx.current_expansion.id.expn_data();
|
2018-03-19 22:44:24 +02:00
|
|
|
Rustc {
|
2021-07-18 15:53:06 -04:00
|
|
|
def_site: ecx.with_def_site_ctxt(expn_data.def_site),
|
|
|
|
call_site: ecx.with_call_site_ctxt(expn_data.call_site),
|
|
|
|
mixed_site: ecx.with_mixed_site_ctxt(expn_data.call_site),
|
2021-07-10 23:15:30 +03:00
|
|
|
krate: expn_data.macro_def_id.unwrap().krate,
|
Implement span quoting for proc-macros
This PR implements span quoting, allowing proc-macros to produce spans
pointing *into their own crate*. This is used by the unstable
`proc_macro::quote!` macro, allowing us to get error messages like this:
```
error[E0412]: cannot find type `MissingType` in this scope
--> $DIR/auxiliary/span-from-proc-macro.rs:37:20
|
LL | pub fn error_from_attribute(_args: TokenStream, _input: TokenStream) -> TokenStream {
| ----------------------------------------------------------------------------------- in this expansion of procedural macro `#[error_from_attribute]`
...
LL | field: MissingType
| ^^^^^^^^^^^ not found in this scope
|
::: $DIR/span-from-proc-macro.rs:8:1
|
LL | #[error_from_attribute]
| ----------------------- in this macro invocation
```
Here, `MissingType` occurs inside the implementation of the proc-macro
`#[error_from_attribute]`. Previosuly, this would always result in a
span pointing at `#[error_from_attribute]`
This will make many proc-macro-related error message much more useful -
when a proc-macro generates code containing an error, users will get an
error message pointing directly at that code (within the macro
definition), instead of always getting a span pointing at the macro
invocation site.
This is implemented as follows:
* When a proc-macro crate is being *compiled*, it causes the `quote!`
macro to get run. This saves all of the sapns in the input to `quote!`
into the metadata of *the proc-macro-crate* (which we are currently
compiling). The `quote!` macro then expands to a call to
`proc_macro::Span::recover_proc_macro_span(id)`, where `id` is an
opaque identifier for the span in the crate metadata.
* When the same proc-macro crate is *run* (e.g. it is loaded from disk
and invoked by some consumer crate), the call to
`proc_macro::Span::recover_proc_macro_span` causes us to load the span
from the proc-macro crate's metadata. The proc-macro then produces a
`TokenStream` containing a `Span` pointing into the proc-macro crate
itself.
The recursive nature of 'quote!' can be difficult to understand at
first. The file `src/test/ui/proc-macro/quote-debug.stdout` shows
the output of the `quote!` macro, which should make this eaier to
understand.
This PR also supports custom quoting spans in custom quote macros (e.g.
the `quote` crate). All span quoting goes through the
`proc_macro::quote_span` method, which can be called by a custom quote
macro to perform span quoting. An example of this usage is provided in
`src/test/ui/proc-macro/auxiliary/custom-quote.rs`
Custom quoting currently has a few limitations:
In order to quote a span, we need to generate a call to
`proc_macro::Span::recover_proc_macro_span`. However, proc-macros
support renaming the `proc_macro` crate, so we can't simply hardcode
this path. Previously, the `quote_span` method used the path
`crate::Span` - however, this only works when it is called by the
builtin `quote!` macro in the same crate. To support being called from
arbitrary crates, we need access to the name of the `proc_macro` crate
to generate a path. This PR adds an additional argument to `quote_span`
to specify the name of the `proc_macro` crate. Howver, this feels kind
of hacky, and we may want to change this before stabilizing anything
quote-related.
Additionally, using `quote_span` currently requires enabling the
`proc_macro_internals` feature. The builtin `quote!` macro
has an `#[allow_internal_unstable]` attribute, but this won't work for
custom quote implementations. This will likely require some additional
tricks to apply `allow_internal_unstable` to the span of
`proc_macro::Span::recover_proc_macro_span`.
2020-08-02 19:52:16 -04:00
|
|
|
rebased_spans: FxHashMap::default(),
|
2021-07-18 15:53:06 -04:00
|
|
|
ecx,
|
2018-03-19 22:44:24 +02:00
|
|
|
}
|
|
|
|
}
|
2019-05-19 01:04:26 +03:00
|
|
|
|
2024-03-04 16:31:49 +11:00
|
|
|
fn psess(&self) -> &ParseSess {
|
|
|
|
self.ecx.psess()
|
2021-07-18 15:53:06 -04:00
|
|
|
}
|
2018-03-19 22:44:24 +02:00
|
|
|
}
|
2018-03-16 01:09:22 +02:00
|
|
|
|
2021-07-18 15:53:06 -04:00
|
|
|
impl server::Types for Rustc<'_, '_> {
|
2020-07-22 21:08:01 +03:00
|
|
|
type FreeFunctions = FreeFunctions;
|
2018-03-16 01:09:22 +02:00
|
|
|
type TokenStream = TokenStream;
|
|
|
|
type SourceFile = Lrc<SourceFile>;
|
|
|
|
type Span = Span;
|
2022-06-30 21:05:46 -04:00
|
|
|
type Symbol = Symbol;
|
2018-03-16 01:09:22 +02:00
|
|
|
}
|
|
|
|
|
2021-07-18 15:53:06 -04:00
|
|
|
impl server::FreeFunctions for Rustc<'_, '_> {
|
2023-12-11 16:44:18 +01:00
|
|
|
fn injected_env_var(&mut self, var: &str) -> Option<String> {
|
|
|
|
self.ecx.sess.opts.logical_env.get(var).cloned()
|
|
|
|
}
|
|
|
|
|
2020-07-22 21:08:01 +03:00
|
|
|
fn track_env_var(&mut self, var: &str, value: Option<&str>) {
|
2024-03-04 16:31:49 +11:00
|
|
|
self.psess()
|
2021-07-18 15:53:06 -04:00
|
|
|
.env_depinfo
|
|
|
|
.borrow_mut()
|
|
|
|
.insert((Symbol::intern(var), value.map(Symbol::intern)));
|
2020-07-22 21:08:01 +03:00
|
|
|
}
|
2021-04-09 16:35:40 +02:00
|
|
|
|
|
|
|
fn track_path(&mut self, path: &str) {
|
2024-03-04 16:31:49 +11:00
|
|
|
self.psess().file_depinfo.borrow_mut().insert(Symbol::intern(path));
|
2021-04-09 16:35:40 +02:00
|
|
|
}
|
2022-07-03 01:04:31 -04:00
|
|
|
|
|
|
|
fn literal_from_str(&mut self, s: &str) -> Result<Literal<Self::Span, Self::Symbol>, ()> {
|
|
|
|
let name = FileName::proc_macro_source_code(s);
|
2024-05-31 15:43:18 +10:00
|
|
|
let mut parser =
|
|
|
|
unwrap_or_emit_fatal(new_parser_from_source_str(self.psess(), name, s.to_owned()));
|
2022-07-03 01:04:31 -04:00
|
|
|
|
|
|
|
let first_span = parser.token.span.data();
|
|
|
|
let minus_present = parser.eat(&token::BinOp(token::Minus));
|
|
|
|
|
|
|
|
let lit_span = parser.token.span.data();
|
|
|
|
let token::Literal(mut lit) = parser.token.kind else {
|
|
|
|
return Err(());
|
|
|
|
};
|
|
|
|
|
|
|
|
// Check no comment or whitespace surrounding the (possibly negative)
|
|
|
|
// literal, or more tokens after it.
|
|
|
|
if (lit_span.hi.0 - first_span.lo.0) as usize != s.len() {
|
|
|
|
return Err(());
|
|
|
|
}
|
|
|
|
|
|
|
|
if minus_present {
|
|
|
|
// If minus is present, check no comment or whitespace in between it
|
|
|
|
// and the literal token.
|
|
|
|
if first_span.hi.0 != lit_span.lo.0 {
|
|
|
|
return Err(());
|
|
|
|
}
|
|
|
|
|
|
|
|
// Check literal is a kind we allow to be negated in a proc macro token.
|
|
|
|
match lit.kind {
|
|
|
|
token::LitKind::Bool
|
|
|
|
| token::LitKind::Byte
|
|
|
|
| token::LitKind::Char
|
|
|
|
| token::LitKind::Str
|
|
|
|
| token::LitKind::StrRaw(_)
|
|
|
|
| token::LitKind::ByteStr
|
|
|
|
| token::LitKind::ByteStrRaw(_)
|
2023-03-05 15:03:22 +00:00
|
|
|
| token::LitKind::CStr
|
|
|
|
| token::LitKind::CStrRaw(_)
|
2024-02-14 20:12:05 +11:00
|
|
|
| token::LitKind::Err(_) => return Err(()),
|
2022-07-03 01:04:31 -04:00
|
|
|
token::LitKind::Integer | token::LitKind::Float => {}
|
|
|
|
}
|
|
|
|
|
|
|
|
// Synthesize a new symbol that includes the minus sign.
|
|
|
|
let symbol = Symbol::intern(&s[..1 + lit.symbol.as_str().len()]);
|
|
|
|
lit = token::Lit::new(lit.kind, symbol, lit.suffix);
|
|
|
|
}
|
|
|
|
let token::Lit { kind, symbol, suffix } = lit;
|
|
|
|
Ok(Literal {
|
|
|
|
kind: FromInternal::from_internal(kind),
|
|
|
|
symbol,
|
|
|
|
suffix,
|
|
|
|
span: self.call_site,
|
|
|
|
})
|
|
|
|
}
|
2022-07-25 00:43:33 -04:00
|
|
|
|
|
|
|
fn emit_diagnostic(&mut self, diagnostic: Diagnostic<Self::Span>) {
|
2024-02-29 11:58:51 +11:00
|
|
|
let message = rustc_errors::DiagMessage::from(diagnostic.message);
|
2024-02-23 10:20:45 +11:00
|
|
|
let mut diag: Diag<'_, ()> =
|
2024-06-18 09:43:28 +00:00
|
|
|
Diag::new(self.psess().dcx(), diagnostic.level.to_internal(), message);
|
2023-12-24 09:08:41 +11:00
|
|
|
diag.span(MultiSpan::from_spans(diagnostic.spans));
|
2022-07-25 00:43:33 -04:00
|
|
|
for child in diagnostic.children {
|
2024-02-20 14:12:50 +11:00
|
|
|
// This message comes from another diagnostic, and we are just reconstructing the
|
|
|
|
// diagnostic, so there's no need for translation.
|
|
|
|
#[allow(rustc::untranslatable_diagnostic)]
|
2023-12-21 10:24:44 +11:00
|
|
|
diag.sub(child.level.to_internal(), child.message, MultiSpan::from_spans(child.spans));
|
2022-07-25 00:43:33 -04:00
|
|
|
}
|
Reduce capabilities of `Diagnostic`.
Currently many diagnostic modifier methods are available on both
`Diagnostic` and `DiagnosticBuilder`. This commit removes most of them
from `Diagnostic`. To minimize the diff size, it keeps them within
`diagnostic.rs` but changes the surrounding `impl Diagnostic` block to
`impl DiagnosticBuilder`. (I intend to move things around later, to give
a more sensible code layout.)
`Diagnostic` keeps a few methods that it still needs, like `sub`,
`arg`, and `replace_args`.
The `forward!` macro, which defined two additional methods per call
(e.g. `note` and `with_note`), is replaced by the `with_fn!` macro,
which defines one additional method per call (e.g. `with_note`). It's
now also only used when necessary -- not all modifier methods currently
need a `with_*` form. (New ones can be easily added as necessary.)
All this also requires changing `trait AddToDiagnostic` so its methods
take `DiagnosticBuilder` instead of `Diagnostic`, which leads to many
mechanical changes. `SubdiagnosticMessageOp` gains a type parameter `G`.
There are three subdiagnostics -- `DelayedAtWithoutNewline`,
`DelayedAtWithNewline`, and `InvalidFlushedDelayedDiagnosticLevel` --
that are created within the diagnostics machinery and appended to
external diagnostics. These are handled at the `Diagnostic` level, which
means it's now hard to construct them via `derive(Diagnostic)`, so
instead we construct them by hand. This has no effect on what they look
like when printed.
There are lots of new `allow` markers for `untranslatable_diagnostics`
and `diagnostics_outside_of_impl`. This is because
`#[rustc_lint_diagnostics]` annotations were present on the `Diagnostic`
modifier methods, but missing from the `DiagnosticBuilder` modifier
methods. They're now present.
2024-02-06 16:44:30 +11:00
|
|
|
diag.emit();
|
2022-07-25 00:43:33 -04:00
|
|
|
}
|
2020-07-22 21:08:01 +03:00
|
|
|
}
|
|
|
|
|
2021-07-18 15:53:06 -04:00
|
|
|
impl server::TokenStream for Rustc<'_, '_> {
|
2018-03-16 01:09:22 +02:00
|
|
|
fn is_empty(&mut self, stream: &Self::TokenStream) -> bool {
|
|
|
|
stream.is_empty()
|
|
|
|
}
|
2022-06-20 13:52:48 +10:00
|
|
|
|
2018-03-16 01:09:22 +02:00
|
|
|
fn from_str(&mut self, src: &str) -> Self::TokenStream {
|
2024-05-31 15:43:18 +10:00
|
|
|
unwrap_or_emit_fatal(source_str_to_stream(
|
2024-05-31 14:33:56 +10:00
|
|
|
self.psess(),
|
2019-06-08 19:06:58 +09:00
|
|
|
FileName::proc_macro_source_code(src),
|
2018-03-19 22:44:24 +02:00
|
|
|
src.to_string(),
|
|
|
|
Some(self.call_site),
|
2024-05-31 15:43:18 +10:00
|
|
|
))
|
2018-03-16 01:09:22 +02:00
|
|
|
}
|
2022-06-20 13:52:48 +10:00
|
|
|
|
2018-03-16 01:09:22 +02:00
|
|
|
fn to_string(&mut self, stream: &Self::TokenStream) -> String {
|
2020-06-24 17:45:08 +03:00
|
|
|
pprust::tts_to_string(stream)
|
2018-03-16 01:09:22 +02:00
|
|
|
}
|
2022-06-20 13:52:48 +10:00
|
|
|
|
2021-07-18 15:53:06 -04:00
|
|
|
fn expand_expr(&mut self, stream: &Self::TokenStream) -> Result<Self::TokenStream, ()> {
|
|
|
|
// Parse the expression from our tokenstream.
|
|
|
|
let expr: PResult<'_, _> = try {
|
2024-05-31 13:32:54 +10:00
|
|
|
let mut p = Parser::new(self.psess(), stream.clone(), Some("proc_macro expand expr"));
|
2021-07-18 15:53:06 -04:00
|
|
|
let expr = p.parse_expr()?;
|
|
|
|
if p.token != token::Eof {
|
|
|
|
p.unexpected()?;
|
|
|
|
}
|
|
|
|
expr
|
|
|
|
};
|
Make `DiagnosticBuilder::emit` consuming.
This works for most of its call sites. This is nice, because `emit` very
much makes sense as a consuming operation -- indeed,
`DiagnosticBuilderState` exists to ensure no diagnostic is emitted
twice, but it uses runtime checks.
For the small number of call sites where a consuming emit doesn't work,
the commit adds `DiagnosticBuilder::emit_without_consuming`. (This will
be removed in subsequent commits.)
Likewise, `emit_unless` becomes consuming. And `delay_as_bug` becomes
consuming, while `delay_as_bug_without_consuming` is added (which will
also be removed in subsequent commits.)
All this requires significant changes to `DiagnosticBuilder`'s chaining
methods. Currently `DiagnosticBuilder` method chaining uses a
non-consuming `&mut self -> &mut Self` style, which allows chaining to
be used when the chain ends in `emit()`, like so:
```
struct_err(msg).span(span).emit();
```
But it doesn't work when producing a `DiagnosticBuilder` value,
requiring this:
```
let mut err = self.struct_err(msg);
err.span(span);
err
```
This style of chaining won't work with consuming `emit` though. For
that, we need to use to a `self -> Self` style. That also would allow
`DiagnosticBuilder` production to be chained, e.g.:
```
self.struct_err(msg).span(span)
```
However, removing the `&mut self -> &mut Self` style would require that
individual modifications of a `DiagnosticBuilder` go from this:
```
err.span(span);
```
to this:
```
err = err.span(span);
```
There are *many* such places. I have a high tolerance for tedious
refactorings, but even I gave up after a long time trying to convert
them all.
Instead, this commit has it both ways: the existing `&mut self -> Self`
chaining methods are kept, and new `self -> Self` chaining methods are
added, all of which have a `_mv` suffix (short for "move"). Changes to
the existing `forward!` macro lets this happen with very little
additional boilerplate code. I chose to add the suffix to the new
chaining methods rather than the existing ones, because the number of
changes required is much smaller that way.
This doubled chainging is a bit clumsy, but I think it is worthwhile
because it allows a *lot* of good things to subsequently happen. In this
commit, there are many `mut` qualifiers removed in places where
diagnostics are emitted without being modified. In subsequent commits:
- chaining can be used more, making the code more concise;
- more use of chaining also permits the removal of redundant diagnostic
APIs like `struct_err_with_code`, which can be replaced easily with
`struct_err` + `code_mv`;
- `emit_without_diagnostic` can be removed, which simplifies a lot of
machinery, removing the need for `DiagnosticBuilderState`.
2024-01-03 12:17:35 +11:00
|
|
|
let expr = expr.map_err(|err| {
|
2022-01-27 09:44:25 +00:00
|
|
|
err.emit();
|
|
|
|
})?;
|
2021-07-18 15:53:06 -04:00
|
|
|
|
|
|
|
// Perform eager expansion on the expression.
|
|
|
|
let expr = self
|
|
|
|
.ecx
|
|
|
|
.expander()
|
|
|
|
.fully_expand_fragment(crate::expand::AstFragment::Expr(expr))
|
|
|
|
.make_expr();
|
|
|
|
|
|
|
|
// NOTE: For now, limit `expand_expr` to exclusively expand to literals.
|
|
|
|
// This may be relaxed in the future.
|
2022-05-21 15:50:39 +03:00
|
|
|
// We don't use `TokenStream::from_ast` as the tokenstream currently cannot
|
2021-07-18 15:53:06 -04:00
|
|
|
// be recovered in the general case.
|
|
|
|
match &expr.kind {
|
2022-10-10 13:40:56 +11:00
|
|
|
ast::ExprKind::Lit(token_lit) if token_lit.kind == token::Bool => {
|
2022-08-01 16:46:08 +10:00
|
|
|
Ok(tokenstream::TokenStream::token_alone(
|
2024-02-13 23:28:27 +00:00
|
|
|
token::Ident(token_lit.symbol, IdentIsRaw::No),
|
2022-10-10 13:40:56 +11:00
|
|
|
expr.span,
|
2022-08-01 16:46:08 +10:00
|
|
|
))
|
|
|
|
}
|
2022-10-10 13:40:56 +11:00
|
|
|
ast::ExprKind::Lit(token_lit) => {
|
|
|
|
Ok(tokenstream::TokenStream::token_alone(token::Literal(*token_lit), expr.span))
|
2021-07-18 15:53:06 -04:00
|
|
|
}
|
2022-10-31 18:30:09 +00:00
|
|
|
ast::ExprKind::IncludedBytes(bytes) => {
|
2022-12-05 15:23:27 +11:00
|
|
|
let lit = token::Lit::new(token::ByteStr, escape_byte_str_symbol(bytes), None);
|
2022-11-23 15:39:42 +11:00
|
|
|
Ok(tokenstream::TokenStream::token_alone(token::TokenKind::Literal(lit), expr.span))
|
2022-10-31 18:30:09 +00:00
|
|
|
}
|
2021-07-18 15:53:06 -04:00
|
|
|
ast::ExprKind::Unary(ast::UnOp::Neg, e) => match &e.kind {
|
2022-10-10 13:40:56 +11:00
|
|
|
ast::ExprKind::Lit(token_lit) => match token_lit {
|
2021-07-18 15:53:06 -04:00
|
|
|
token::Lit { kind: token::Integer | token::Float, .. } => {
|
2021-09-03 12:36:33 +02:00
|
|
|
Ok(Self::TokenStream::from_iter([
|
2021-07-18 15:53:06 -04:00
|
|
|
// FIXME: The span of the `-` token is lost when
|
|
|
|
// parsing, so we cannot faithfully recover it here.
|
2023-08-08 11:43:44 +10:00
|
|
|
tokenstream::TokenTree::token_joint_hidden(
|
|
|
|
token::BinOp(token::Minus),
|
|
|
|
e.span,
|
|
|
|
),
|
2022-10-10 13:40:56 +11:00
|
|
|
tokenstream::TokenTree::token_alone(token::Literal(*token_lit), e.span),
|
2021-09-03 12:36:33 +02:00
|
|
|
]))
|
2021-07-18 15:53:06 -04:00
|
|
|
}
|
|
|
|
_ => Err(()),
|
|
|
|
},
|
|
|
|
_ => Err(()),
|
|
|
|
},
|
|
|
|
_ => Err(()),
|
|
|
|
}
|
|
|
|
}
|
2022-06-20 13:52:48 +10:00
|
|
|
|
2018-03-16 01:09:22 +02:00
|
|
|
fn from_token_tree(
|
|
|
|
&mut self,
|
2022-07-03 01:04:31 -04:00
|
|
|
tree: TokenTree<Self::TokenStream, Self::Span, Self::Symbol>,
|
2018-03-16 01:09:22 +02:00
|
|
|
) -> Self::TokenStream {
|
Remove `TokenStreamBuilder`.
`TokenStreamBuilder` exists to concatenate multiple `TokenStream`s
together. This commit removes it, and moves the concatenation
functionality directly into `TokenStream`, via two new methods
`push_tree` and `push_stream`. This makes things both simpler and
faster.
`push_tree` is particularly important. `TokenStreamBuilder` only had a
single `push` method, which pushed a stream. But in practice most of the
time we push a single token tree rather than a stream, and `push_tree`
avoids the need to build a token stream with a single entry (which
requires two allocations, one for the `Lrc` and one for the `Vec`).
The main `push_tree` use arises from a change to one of the `ToInternal`
impls in `proc_macro_server.rs`. It now returns a `SmallVec` instead of
a `TokenStream`. This return value is then iterated over by
`concat_trees`, which does `push_tree` on each element. Furthermore, the
use of `SmallVec` avoids more allocations, because there is always only
one or two token trees.
Note: the removed `TokenStreamBuilder::push` method had some code to
deal with a quadratic blowup case from #57735. This commit removes the
code. I tried and failed to reproduce the blowup from that PR, before
and after this change. Various other changes have happened to
`TokenStreamBuilder` in the meantime, so I suspect the original problem
is no longer relevant, though I don't have proof of this. Generally
speaking, repeatedly extending a `Vec` without pre-determining its
capacity is *not* quadratic. It's also incredibly common, within rustc
and many other Rust programs, so if there were performance problems
there you'd think it would show up in other places, too.
2022-10-05 10:38:15 +11:00
|
|
|
Self::TokenStream::new((tree, &mut *self).to_internal().into_iter().collect::<Vec<_>>())
|
2018-03-16 01:09:22 +02:00
|
|
|
}
|
2022-06-20 13:52:48 +10:00
|
|
|
|
2021-07-01 15:03:51 -04:00
|
|
|
fn concat_trees(
|
|
|
|
&mut self,
|
|
|
|
base: Option<Self::TokenStream>,
|
2022-07-03 01:04:31 -04:00
|
|
|
trees: Vec<TokenTree<Self::TokenStream, Self::Span, Self::Symbol>>,
|
2021-07-01 15:03:51 -04:00
|
|
|
) -> Self::TokenStream {
|
Remove `TokenStreamBuilder`.
`TokenStreamBuilder` exists to concatenate multiple `TokenStream`s
together. This commit removes it, and moves the concatenation
functionality directly into `TokenStream`, via two new methods
`push_tree` and `push_stream`. This makes things both simpler and
faster.
`push_tree` is particularly important. `TokenStreamBuilder` only had a
single `push` method, which pushed a stream. But in practice most of the
time we push a single token tree rather than a stream, and `push_tree`
avoids the need to build a token stream with a single entry (which
requires two allocations, one for the `Lrc` and one for the `Vec`).
The main `push_tree` use arises from a change to one of the `ToInternal`
impls in `proc_macro_server.rs`. It now returns a `SmallVec` instead of
a `TokenStream`. This return value is then iterated over by
`concat_trees`, which does `push_tree` on each element. Furthermore, the
use of `SmallVec` avoids more allocations, because there is always only
one or two token trees.
Note: the removed `TokenStreamBuilder::push` method had some code to
deal with a quadratic blowup case from #57735. This commit removes the
code. I tried and failed to reproduce the blowup from that PR, before
and after this change. Various other changes have happened to
`TokenStreamBuilder` in the meantime, so I suspect the original problem
is no longer relevant, though I don't have proof of this. Generally
speaking, repeatedly extending a `Vec` without pre-determining its
capacity is *not* quadratic. It's also incredibly common, within rustc
and many other Rust programs, so if there were performance problems
there you'd think it would show up in other places, too.
2022-10-05 10:38:15 +11:00
|
|
|
let mut stream =
|
|
|
|
if let Some(base) = base { base } else { tokenstream::TokenStream::default() };
|
2021-07-01 15:03:51 -04:00
|
|
|
for tree in trees {
|
Remove `TokenStreamBuilder`.
`TokenStreamBuilder` exists to concatenate multiple `TokenStream`s
together. This commit removes it, and moves the concatenation
functionality directly into `TokenStream`, via two new methods
`push_tree` and `push_stream`. This makes things both simpler and
faster.
`push_tree` is particularly important. `TokenStreamBuilder` only had a
single `push` method, which pushed a stream. But in practice most of the
time we push a single token tree rather than a stream, and `push_tree`
avoids the need to build a token stream with a single entry (which
requires two allocations, one for the `Lrc` and one for the `Vec`).
The main `push_tree` use arises from a change to one of the `ToInternal`
impls in `proc_macro_server.rs`. It now returns a `SmallVec` instead of
a `TokenStream`. This return value is then iterated over by
`concat_trees`, which does `push_tree` on each element. Furthermore, the
use of `SmallVec` avoids more allocations, because there is always only
one or two token trees.
Note: the removed `TokenStreamBuilder::push` method had some code to
deal with a quadratic blowup case from #57735. This commit removes the
code. I tried and failed to reproduce the blowup from that PR, before
and after this change. Various other changes have happened to
`TokenStreamBuilder` in the meantime, so I suspect the original problem
is no longer relevant, though I don't have proof of this. Generally
speaking, repeatedly extending a `Vec` without pre-determining its
capacity is *not* quadratic. It's also incredibly common, within rustc
and many other Rust programs, so if there were performance problems
there you'd think it would show up in other places, too.
2022-10-05 10:38:15 +11:00
|
|
|
for tt in (tree, &mut *self).to_internal() {
|
|
|
|
stream.push_tree(tt);
|
|
|
|
}
|
2021-07-01 15:03:51 -04:00
|
|
|
}
|
Remove `TokenStreamBuilder`.
`TokenStreamBuilder` exists to concatenate multiple `TokenStream`s
together. This commit removes it, and moves the concatenation
functionality directly into `TokenStream`, via two new methods
`push_tree` and `push_stream`. This makes things both simpler and
faster.
`push_tree` is particularly important. `TokenStreamBuilder` only had a
single `push` method, which pushed a stream. But in practice most of the
time we push a single token tree rather than a stream, and `push_tree`
avoids the need to build a token stream with a single entry (which
requires two allocations, one for the `Lrc` and one for the `Vec`).
The main `push_tree` use arises from a change to one of the `ToInternal`
impls in `proc_macro_server.rs`. It now returns a `SmallVec` instead of
a `TokenStream`. This return value is then iterated over by
`concat_trees`, which does `push_tree` on each element. Furthermore, the
use of `SmallVec` avoids more allocations, because there is always only
one or two token trees.
Note: the removed `TokenStreamBuilder::push` method had some code to
deal with a quadratic blowup case from #57735. This commit removes the
code. I tried and failed to reproduce the blowup from that PR, before
and after this change. Various other changes have happened to
`TokenStreamBuilder` in the meantime, so I suspect the original problem
is no longer relevant, though I don't have proof of this. Generally
speaking, repeatedly extending a `Vec` without pre-determining its
capacity is *not* quadratic. It's also incredibly common, within rustc
and many other Rust programs, so if there were performance problems
there you'd think it would show up in other places, too.
2022-10-05 10:38:15 +11:00
|
|
|
stream
|
2018-03-16 01:09:22 +02:00
|
|
|
}
|
2022-06-20 13:52:48 +10:00
|
|
|
|
2021-07-01 15:03:51 -04:00
|
|
|
fn concat_streams(
|
|
|
|
&mut self,
|
|
|
|
base: Option<Self::TokenStream>,
|
|
|
|
streams: Vec<Self::TokenStream>,
|
|
|
|
) -> Self::TokenStream {
|
Remove `TokenStreamBuilder`.
`TokenStreamBuilder` exists to concatenate multiple `TokenStream`s
together. This commit removes it, and moves the concatenation
functionality directly into `TokenStream`, via two new methods
`push_tree` and `push_stream`. This makes things both simpler and
faster.
`push_tree` is particularly important. `TokenStreamBuilder` only had a
single `push` method, which pushed a stream. But in practice most of the
time we push a single token tree rather than a stream, and `push_tree`
avoids the need to build a token stream with a single entry (which
requires two allocations, one for the `Lrc` and one for the `Vec`).
The main `push_tree` use arises from a change to one of the `ToInternal`
impls in `proc_macro_server.rs`. It now returns a `SmallVec` instead of
a `TokenStream`. This return value is then iterated over by
`concat_trees`, which does `push_tree` on each element. Furthermore, the
use of `SmallVec` avoids more allocations, because there is always only
one or two token trees.
Note: the removed `TokenStreamBuilder::push` method had some code to
deal with a quadratic blowup case from #57735. This commit removes the
code. I tried and failed to reproduce the blowup from that PR, before
and after this change. Various other changes have happened to
`TokenStreamBuilder` in the meantime, so I suspect the original problem
is no longer relevant, though I don't have proof of this. Generally
speaking, repeatedly extending a `Vec` without pre-determining its
capacity is *not* quadratic. It's also incredibly common, within rustc
and many other Rust programs, so if there were performance problems
there you'd think it would show up in other places, too.
2022-10-05 10:38:15 +11:00
|
|
|
let mut stream =
|
|
|
|
if let Some(base) = base { base } else { tokenstream::TokenStream::default() };
|
|
|
|
for s in streams {
|
|
|
|
stream.push_stream(s);
|
2021-07-01 15:03:51 -04:00
|
|
|
}
|
Remove `TokenStreamBuilder`.
`TokenStreamBuilder` exists to concatenate multiple `TokenStream`s
together. This commit removes it, and moves the concatenation
functionality directly into `TokenStream`, via two new methods
`push_tree` and `push_stream`. This makes things both simpler and
faster.
`push_tree` is particularly important. `TokenStreamBuilder` only had a
single `push` method, which pushed a stream. But in practice most of the
time we push a single token tree rather than a stream, and `push_tree`
avoids the need to build a token stream with a single entry (which
requires two allocations, one for the `Lrc` and one for the `Vec`).
The main `push_tree` use arises from a change to one of the `ToInternal`
impls in `proc_macro_server.rs`. It now returns a `SmallVec` instead of
a `TokenStream`. This return value is then iterated over by
`concat_trees`, which does `push_tree` on each element. Furthermore, the
use of `SmallVec` avoids more allocations, because there is always only
one or two token trees.
Note: the removed `TokenStreamBuilder::push` method had some code to
deal with a quadratic blowup case from #57735. This commit removes the
code. I tried and failed to reproduce the blowup from that PR, before
and after this change. Various other changes have happened to
`TokenStreamBuilder` in the meantime, so I suspect the original problem
is no longer relevant, though I don't have proof of this. Generally
speaking, repeatedly extending a `Vec` without pre-determining its
capacity is *not* quadratic. It's also incredibly common, within rustc
and many other Rust programs, so if there were performance problems
there you'd think it would show up in other places, too.
2022-10-05 10:38:15 +11:00
|
|
|
stream
|
2018-03-16 01:09:22 +02:00
|
|
|
}
|
2022-06-20 13:52:48 +10:00
|
|
|
|
2022-06-17 22:10:07 -04:00
|
|
|
fn into_trees(
|
2018-03-16 01:09:22 +02:00
|
|
|
&mut self,
|
2021-07-01 15:03:51 -04:00
|
|
|
stream: Self::TokenStream,
|
2022-07-03 01:04:31 -04:00
|
|
|
) -> Vec<TokenTree<Self::TokenStream, Self::Span, Self::Symbol>> {
|
2021-07-01 17:36:38 -04:00
|
|
|
FromInternal::from_internal((stream, self))
|
2018-03-16 01:09:22 +02:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2021-07-18 15:53:06 -04:00
|
|
|
impl server::SourceFile for Rustc<'_, '_> {
|
2018-03-16 01:09:22 +02:00
|
|
|
fn eq(&mut self, file1: &Self::SourceFile, file2: &Self::SourceFile) -> bool {
|
|
|
|
Lrc::ptr_eq(file1, file2)
|
|
|
|
}
|
2022-06-20 13:52:48 +10:00
|
|
|
|
2018-03-16 01:09:22 +02:00
|
|
|
fn path(&mut self, file: &Self::SourceFile) -> String {
|
2022-12-23 17:34:23 +00:00
|
|
|
match &file.name {
|
|
|
|
FileName::Real(name) => name
|
2020-05-29 11:31:55 -04:00
|
|
|
.local_path()
|
2021-04-09 00:54:51 +01:00
|
|
|
.expect("attempting to get a file path in an imported file in `proc_macro::SourceFile::path`")
|
2018-03-16 01:09:22 +02:00
|
|
|
.to_str()
|
|
|
|
.expect("non-UTF8 file path in `proc_macro::SourceFile::path`")
|
|
|
|
.to_string(),
|
2021-04-19 23:27:02 +01:00
|
|
|
_ => file.name.prefer_local().to_string(),
|
2018-03-16 01:09:22 +02:00
|
|
|
}
|
|
|
|
}
|
2022-06-20 13:52:48 +10:00
|
|
|
|
2018-03-16 01:09:22 +02:00
|
|
|
fn is_real(&mut self, file: &Self::SourceFile) -> bool {
|
|
|
|
file.is_real_file()
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2021-07-18 15:53:06 -04:00
|
|
|
impl server::Span for Rustc<'_, '_> {
|
2018-03-16 01:09:22 +02:00
|
|
|
fn debug(&mut self, span: Self::Span) -> String {
|
2021-07-18 15:53:06 -04:00
|
|
|
if self.ecx.ecfg.span_debug {
|
2023-07-25 22:00:13 +02:00
|
|
|
format!("{span:?}")
|
2020-05-31 16:20:50 -04:00
|
|
|
} else {
|
|
|
|
format!("{:?} bytes({}..{})", span.ctxt(), span.lo().0, span.hi().0)
|
|
|
|
}
|
2018-03-16 01:09:22 +02:00
|
|
|
}
|
2022-06-20 13:52:48 +10:00
|
|
|
|
2018-03-16 01:09:22 +02:00
|
|
|
fn source_file(&mut self, span: Self::Span) -> Self::SourceFile {
|
2024-03-04 16:31:49 +11:00
|
|
|
self.psess().source_map().lookup_char_pos(span.lo()).file
|
2018-03-16 01:09:22 +02:00
|
|
|
}
|
2022-06-20 13:52:48 +10:00
|
|
|
|
2018-03-16 01:09:22 +02:00
|
|
|
fn parent(&mut self, span: Self::Span) -> Option<Self::Span> {
|
2021-04-18 14:27:04 +02:00
|
|
|
span.parent_callsite()
|
2018-03-16 01:09:22 +02:00
|
|
|
}
|
2022-06-20 13:52:48 +10:00
|
|
|
|
2018-03-16 01:09:22 +02:00
|
|
|
fn source(&mut self, span: Self::Span) -> Self::Span {
|
|
|
|
span.source_callsite()
|
|
|
|
}
|
2022-06-20 13:52:48 +10:00
|
|
|
|
2023-03-11 12:14:06 +01:00
|
|
|
fn byte_range(&mut self, span: Self::Span) -> Range<usize> {
|
2024-03-04 16:31:49 +11:00
|
|
|
let source_map = self.psess().source_map();
|
2023-04-19 13:02:30 +02:00
|
|
|
|
|
|
|
let relative_start_pos = source_map.lookup_byte_offset(span.lo()).pos;
|
|
|
|
let relative_end_pos = source_map.lookup_byte_offset(span.hi()).pos;
|
|
|
|
|
2023-04-19 13:04:01 +02:00
|
|
|
Range { start: relative_start_pos.0 as usize, end: relative_end_pos.0 as usize }
|
2023-03-10 21:16:35 +01:00
|
|
|
}
|
2023-05-14 18:11:27 -04:00
|
|
|
fn start(&mut self, span: Self::Span) -> Self::Span {
|
2021-06-09 14:37:10 +02:00
|
|
|
span.shrink_to_lo()
|
|
|
|
}
|
2022-06-20 13:52:48 +10:00
|
|
|
|
2023-05-14 18:11:27 -04:00
|
|
|
fn end(&mut self, span: Self::Span) -> Self::Span {
|
2021-06-09 14:37:10 +02:00
|
|
|
span.shrink_to_hi()
|
|
|
|
}
|
2022-06-20 13:52:48 +10:00
|
|
|
|
2023-05-14 18:30:18 -04:00
|
|
|
fn line(&mut self, span: Self::Span) -> usize {
|
2024-03-04 16:31:49 +11:00
|
|
|
let loc = self.psess().source_map().lookup_char_pos(span.lo());
|
2023-05-14 18:30:18 -04:00
|
|
|
loc.line
|
|
|
|
}
|
|
|
|
|
|
|
|
fn column(&mut self, span: Self::Span) -> usize {
|
2024-03-04 16:31:49 +11:00
|
|
|
let loc = self.psess().source_map().lookup_char_pos(span.lo());
|
2023-05-14 18:30:18 -04:00
|
|
|
loc.col.to_usize() + 1
|
|
|
|
}
|
|
|
|
|
2018-03-16 01:09:22 +02:00
|
|
|
fn join(&mut self, first: Self::Span, second: Self::Span) -> Option<Self::Span> {
|
2024-03-04 16:31:49 +11:00
|
|
|
let self_loc = self.psess().source_map().lookup_char_pos(first.lo());
|
|
|
|
let other_loc = self.psess().source_map().lookup_char_pos(second.lo());
|
2018-03-16 01:09:22 +02:00
|
|
|
|
|
|
|
if self_loc.file.name != other_loc.file.name {
|
|
|
|
return None;
|
|
|
|
}
|
|
|
|
|
|
|
|
Some(first.to(second))
|
|
|
|
}
|
2022-06-20 13:52:48 +10:00
|
|
|
|
2022-07-09 11:34:06 -04:00
|
|
|
fn subspan(
|
|
|
|
&mut self,
|
|
|
|
span: Self::Span,
|
|
|
|
start: Bound<usize>,
|
|
|
|
end: Bound<usize>,
|
|
|
|
) -> Option<Self::Span> {
|
|
|
|
let length = span.hi().to_usize() - span.lo().to_usize();
|
|
|
|
|
|
|
|
let start = match start {
|
|
|
|
Bound::Included(lo) => lo,
|
|
|
|
Bound::Excluded(lo) => lo.checked_add(1)?,
|
|
|
|
Bound::Unbounded => 0,
|
|
|
|
};
|
|
|
|
|
|
|
|
let end = match end {
|
|
|
|
Bound::Included(hi) => hi.checked_add(1)?,
|
|
|
|
Bound::Excluded(hi) => hi,
|
|
|
|
Bound::Unbounded => length,
|
|
|
|
};
|
|
|
|
|
|
|
|
// Bounds check the values, preventing addition overflow and OOB spans.
|
|
|
|
if start > u32::MAX as usize
|
|
|
|
|| end > u32::MAX as usize
|
|
|
|
|| (u32::MAX - start as u32) < span.lo().to_u32()
|
|
|
|
|| (u32::MAX - end as u32) < span.lo().to_u32()
|
|
|
|
|| start >= end
|
|
|
|
|| end > length
|
|
|
|
{
|
|
|
|
return None;
|
|
|
|
}
|
|
|
|
|
|
|
|
let new_lo = span.lo() + BytePos::from_usize(start);
|
|
|
|
let new_hi = span.lo() + BytePos::from_usize(end);
|
|
|
|
Some(span.with_lo(new_lo).with_hi(new_hi))
|
|
|
|
}
|
|
|
|
|
2018-03-16 01:09:22 +02:00
|
|
|
fn resolved_at(&mut self, span: Self::Span, at: Self::Span) -> Self::Span {
|
|
|
|
span.with_ctxt(at.ctxt())
|
|
|
|
}
|
2022-06-20 13:52:48 +10:00
|
|
|
|
2018-11-08 10:07:02 +01:00
|
|
|
fn source_text(&mut self, span: Self::Span) -> Option<String> {
|
2024-03-04 16:31:49 +11:00
|
|
|
self.psess().source_map().span_to_snippet(span).ok()
|
2018-11-08 10:07:02 +01:00
|
|
|
}
|
Remove `TokenStreamBuilder`.
`TokenStreamBuilder` exists to concatenate multiple `TokenStream`s
together. This commit removes it, and moves the concatenation
functionality directly into `TokenStream`, via two new methods
`push_tree` and `push_stream`. This makes things both simpler and
faster.
`push_tree` is particularly important. `TokenStreamBuilder` only had a
single `push` method, which pushed a stream. But in practice most of the
time we push a single token tree rather than a stream, and `push_tree`
avoids the need to build a token stream with a single entry (which
requires two allocations, one for the `Lrc` and one for the `Vec`).
The main `push_tree` use arises from a change to one of the `ToInternal`
impls in `proc_macro_server.rs`. It now returns a `SmallVec` instead of
a `TokenStream`. This return value is then iterated over by
`concat_trees`, which does `push_tree` on each element. Furthermore, the
use of `SmallVec` avoids more allocations, because there is always only
one or two token trees.
Note: the removed `TokenStreamBuilder::push` method had some code to
deal with a quadratic blowup case from #57735. This commit removes the
code. I tried and failed to reproduce the blowup from that PR, before
and after this change. Various other changes have happened to
`TokenStreamBuilder` in the meantime, so I suspect the original problem
is no longer relevant, though I don't have proof of this. Generally
speaking, repeatedly extending a `Vec` without pre-determining its
capacity is *not* quadratic. It's also incredibly common, within rustc
and many other Rust programs, so if there were performance problems
there you'd think it would show up in other places, too.
2022-10-05 10:38:15 +11:00
|
|
|
|
Implement span quoting for proc-macros
This PR implements span quoting, allowing proc-macros to produce spans
pointing *into their own crate*. This is used by the unstable
`proc_macro::quote!` macro, allowing us to get error messages like this:
```
error[E0412]: cannot find type `MissingType` in this scope
--> $DIR/auxiliary/span-from-proc-macro.rs:37:20
|
LL | pub fn error_from_attribute(_args: TokenStream, _input: TokenStream) -> TokenStream {
| ----------------------------------------------------------------------------------- in this expansion of procedural macro `#[error_from_attribute]`
...
LL | field: MissingType
| ^^^^^^^^^^^ not found in this scope
|
::: $DIR/span-from-proc-macro.rs:8:1
|
LL | #[error_from_attribute]
| ----------------------- in this macro invocation
```
Here, `MissingType` occurs inside the implementation of the proc-macro
`#[error_from_attribute]`. Previosuly, this would always result in a
span pointing at `#[error_from_attribute]`
This will make many proc-macro-related error message much more useful -
when a proc-macro generates code containing an error, users will get an
error message pointing directly at that code (within the macro
definition), instead of always getting a span pointing at the macro
invocation site.
This is implemented as follows:
* When a proc-macro crate is being *compiled*, it causes the `quote!`
macro to get run. This saves all of the sapns in the input to `quote!`
into the metadata of *the proc-macro-crate* (which we are currently
compiling). The `quote!` macro then expands to a call to
`proc_macro::Span::recover_proc_macro_span(id)`, where `id` is an
opaque identifier for the span in the crate metadata.
* When the same proc-macro crate is *run* (e.g. it is loaded from disk
and invoked by some consumer crate), the call to
`proc_macro::Span::recover_proc_macro_span` causes us to load the span
from the proc-macro crate's metadata. The proc-macro then produces a
`TokenStream` containing a `Span` pointing into the proc-macro crate
itself.
The recursive nature of 'quote!' can be difficult to understand at
first. The file `src/test/ui/proc-macro/quote-debug.stdout` shows
the output of the `quote!` macro, which should make this eaier to
understand.
This PR also supports custom quoting spans in custom quote macros (e.g.
the `quote` crate). All span quoting goes through the
`proc_macro::quote_span` method, which can be called by a custom quote
macro to perform span quoting. An example of this usage is provided in
`src/test/ui/proc-macro/auxiliary/custom-quote.rs`
Custom quoting currently has a few limitations:
In order to quote a span, we need to generate a call to
`proc_macro::Span::recover_proc_macro_span`. However, proc-macros
support renaming the `proc_macro` crate, so we can't simply hardcode
this path. Previously, the `quote_span` method used the path
`crate::Span` - however, this only works when it is called by the
builtin `quote!` macro in the same crate. To support being called from
arbitrary crates, we need access to the name of the `proc_macro` crate
to generate a path. This PR adds an additional argument to `quote_span`
to specify the name of the `proc_macro` crate. Howver, this feels kind
of hacky, and we may want to change this before stabilizing anything
quote-related.
Additionally, using `quote_span` currently requires enabling the
`proc_macro_internals` feature. The builtin `quote!` macro
has an `#[allow_internal_unstable]` attribute, but this won't work for
custom quote implementations. This will likely require some additional
tricks to apply `allow_internal_unstable` to the span of
`proc_macro::Span::recover_proc_macro_span`.
2020-08-02 19:52:16 -04:00
|
|
|
/// Saves the provided span into the metadata of
|
|
|
|
/// *the crate we are currently compiling*, which must
|
|
|
|
/// be a proc-macro crate. This id can be passed to
|
|
|
|
/// `recover_proc_macro_span` when our current crate
|
|
|
|
/// is *run* as a proc-macro.
|
|
|
|
///
|
|
|
|
/// Let's suppose that we have two crates - `my_client`
|
|
|
|
/// and `my_proc_macro`. The `my_proc_macro` crate
|
|
|
|
/// contains a procedural macro `my_macro`, which
|
|
|
|
/// is implemented as: `quote! { "hello" }`
|
|
|
|
///
|
|
|
|
/// When we *compile* `my_proc_macro`, we will execute
|
|
|
|
/// the `quote` proc-macro. This will save the span of
|
|
|
|
/// "hello" into the metadata of `my_proc_macro`. As a result,
|
|
|
|
/// the body of `my_proc_macro` (after expansion) will end
|
2022-03-03 19:47:23 +08:00
|
|
|
/// up containing a call that looks like this:
|
Implement span quoting for proc-macros
This PR implements span quoting, allowing proc-macros to produce spans
pointing *into their own crate*. This is used by the unstable
`proc_macro::quote!` macro, allowing us to get error messages like this:
```
error[E0412]: cannot find type `MissingType` in this scope
--> $DIR/auxiliary/span-from-proc-macro.rs:37:20
|
LL | pub fn error_from_attribute(_args: TokenStream, _input: TokenStream) -> TokenStream {
| ----------------------------------------------------------------------------------- in this expansion of procedural macro `#[error_from_attribute]`
...
LL | field: MissingType
| ^^^^^^^^^^^ not found in this scope
|
::: $DIR/span-from-proc-macro.rs:8:1
|
LL | #[error_from_attribute]
| ----------------------- in this macro invocation
```
Here, `MissingType` occurs inside the implementation of the proc-macro
`#[error_from_attribute]`. Previosuly, this would always result in a
span pointing at `#[error_from_attribute]`
This will make many proc-macro-related error message much more useful -
when a proc-macro generates code containing an error, users will get an
error message pointing directly at that code (within the macro
definition), instead of always getting a span pointing at the macro
invocation site.
This is implemented as follows:
* When a proc-macro crate is being *compiled*, it causes the `quote!`
macro to get run. This saves all of the sapns in the input to `quote!`
into the metadata of *the proc-macro-crate* (which we are currently
compiling). The `quote!` macro then expands to a call to
`proc_macro::Span::recover_proc_macro_span(id)`, where `id` is an
opaque identifier for the span in the crate metadata.
* When the same proc-macro crate is *run* (e.g. it is loaded from disk
and invoked by some consumer crate), the call to
`proc_macro::Span::recover_proc_macro_span` causes us to load the span
from the proc-macro crate's metadata. The proc-macro then produces a
`TokenStream` containing a `Span` pointing into the proc-macro crate
itself.
The recursive nature of 'quote!' can be difficult to understand at
first. The file `src/test/ui/proc-macro/quote-debug.stdout` shows
the output of the `quote!` macro, which should make this eaier to
understand.
This PR also supports custom quoting spans in custom quote macros (e.g.
the `quote` crate). All span quoting goes through the
`proc_macro::quote_span` method, which can be called by a custom quote
macro to perform span quoting. An example of this usage is provided in
`src/test/ui/proc-macro/auxiliary/custom-quote.rs`
Custom quoting currently has a few limitations:
In order to quote a span, we need to generate a call to
`proc_macro::Span::recover_proc_macro_span`. However, proc-macros
support renaming the `proc_macro` crate, so we can't simply hardcode
this path. Previously, the `quote_span` method used the path
`crate::Span` - however, this only works when it is called by the
builtin `quote!` macro in the same crate. To support being called from
arbitrary crates, we need access to the name of the `proc_macro` crate
to generate a path. This PR adds an additional argument to `quote_span`
to specify the name of the `proc_macro` crate. Howver, this feels kind
of hacky, and we may want to change this before stabilizing anything
quote-related.
Additionally, using `quote_span` currently requires enabling the
`proc_macro_internals` feature. The builtin `quote!` macro
has an `#[allow_internal_unstable]` attribute, but this won't work for
custom quote implementations. This will likely require some additional
tricks to apply `allow_internal_unstable` to the span of
`proc_macro::Span::recover_proc_macro_span`.
2020-08-02 19:52:16 -04:00
|
|
|
/// `proc_macro::Ident::new("hello", proc_macro::Span::recover_proc_macro_span(0))`
|
|
|
|
///
|
|
|
|
/// where `0` is the id returned by this function.
|
|
|
|
/// When `my_proc_macro` *executes* (during the compilation of `my_client`),
|
|
|
|
/// the call to `recover_proc_macro_span` will load the corresponding
|
|
|
|
/// span from the metadata of `my_proc_macro` (which we have access to,
|
|
|
|
/// since we've loaded `my_proc_macro` from disk in order to execute it).
|
|
|
|
/// In this way, we have obtained a span pointing into `my_proc_macro`
|
2021-07-10 23:44:22 +03:00
|
|
|
fn save_span(&mut self, span: Self::Span) -> usize {
|
2024-03-04 16:31:49 +11:00
|
|
|
self.psess().save_proc_macro_span(span)
|
Implement span quoting for proc-macros
This PR implements span quoting, allowing proc-macros to produce spans
pointing *into their own crate*. This is used by the unstable
`proc_macro::quote!` macro, allowing us to get error messages like this:
```
error[E0412]: cannot find type `MissingType` in this scope
--> $DIR/auxiliary/span-from-proc-macro.rs:37:20
|
LL | pub fn error_from_attribute(_args: TokenStream, _input: TokenStream) -> TokenStream {
| ----------------------------------------------------------------------------------- in this expansion of procedural macro `#[error_from_attribute]`
...
LL | field: MissingType
| ^^^^^^^^^^^ not found in this scope
|
::: $DIR/span-from-proc-macro.rs:8:1
|
LL | #[error_from_attribute]
| ----------------------- in this macro invocation
```
Here, `MissingType` occurs inside the implementation of the proc-macro
`#[error_from_attribute]`. Previosuly, this would always result in a
span pointing at `#[error_from_attribute]`
This will make many proc-macro-related error message much more useful -
when a proc-macro generates code containing an error, users will get an
error message pointing directly at that code (within the macro
definition), instead of always getting a span pointing at the macro
invocation site.
This is implemented as follows:
* When a proc-macro crate is being *compiled*, it causes the `quote!`
macro to get run. This saves all of the sapns in the input to `quote!`
into the metadata of *the proc-macro-crate* (which we are currently
compiling). The `quote!` macro then expands to a call to
`proc_macro::Span::recover_proc_macro_span(id)`, where `id` is an
opaque identifier for the span in the crate metadata.
* When the same proc-macro crate is *run* (e.g. it is loaded from disk
and invoked by some consumer crate), the call to
`proc_macro::Span::recover_proc_macro_span` causes us to load the span
from the proc-macro crate's metadata. The proc-macro then produces a
`TokenStream` containing a `Span` pointing into the proc-macro crate
itself.
The recursive nature of 'quote!' can be difficult to understand at
first. The file `src/test/ui/proc-macro/quote-debug.stdout` shows
the output of the `quote!` macro, which should make this eaier to
understand.
This PR also supports custom quoting spans in custom quote macros (e.g.
the `quote` crate). All span quoting goes through the
`proc_macro::quote_span` method, which can be called by a custom quote
macro to perform span quoting. An example of this usage is provided in
`src/test/ui/proc-macro/auxiliary/custom-quote.rs`
Custom quoting currently has a few limitations:
In order to quote a span, we need to generate a call to
`proc_macro::Span::recover_proc_macro_span`. However, proc-macros
support renaming the `proc_macro` crate, so we can't simply hardcode
this path. Previously, the `quote_span` method used the path
`crate::Span` - however, this only works when it is called by the
builtin `quote!` macro in the same crate. To support being called from
arbitrary crates, we need access to the name of the `proc_macro` crate
to generate a path. This PR adds an additional argument to `quote_span`
to specify the name of the `proc_macro` crate. Howver, this feels kind
of hacky, and we may want to change this before stabilizing anything
quote-related.
Additionally, using `quote_span` currently requires enabling the
`proc_macro_internals` feature. The builtin `quote!` macro
has an `#[allow_internal_unstable]` attribute, but this won't work for
custom quote implementations. This will likely require some additional
tricks to apply `allow_internal_unstable` to the span of
`proc_macro::Span::recover_proc_macro_span`.
2020-08-02 19:52:16 -04:00
|
|
|
}
|
2022-06-20 13:52:48 +10:00
|
|
|
|
Implement span quoting for proc-macros
This PR implements span quoting, allowing proc-macros to produce spans
pointing *into their own crate*. This is used by the unstable
`proc_macro::quote!` macro, allowing us to get error messages like this:
```
error[E0412]: cannot find type `MissingType` in this scope
--> $DIR/auxiliary/span-from-proc-macro.rs:37:20
|
LL | pub fn error_from_attribute(_args: TokenStream, _input: TokenStream) -> TokenStream {
| ----------------------------------------------------------------------------------- in this expansion of procedural macro `#[error_from_attribute]`
...
LL | field: MissingType
| ^^^^^^^^^^^ not found in this scope
|
::: $DIR/span-from-proc-macro.rs:8:1
|
LL | #[error_from_attribute]
| ----------------------- in this macro invocation
```
Here, `MissingType` occurs inside the implementation of the proc-macro
`#[error_from_attribute]`. Previosuly, this would always result in a
span pointing at `#[error_from_attribute]`
This will make many proc-macro-related error message much more useful -
when a proc-macro generates code containing an error, users will get an
error message pointing directly at that code (within the macro
definition), instead of always getting a span pointing at the macro
invocation site.
This is implemented as follows:
* When a proc-macro crate is being *compiled*, it causes the `quote!`
macro to get run. This saves all of the sapns in the input to `quote!`
into the metadata of *the proc-macro-crate* (which we are currently
compiling). The `quote!` macro then expands to a call to
`proc_macro::Span::recover_proc_macro_span(id)`, where `id` is an
opaque identifier for the span in the crate metadata.
* When the same proc-macro crate is *run* (e.g. it is loaded from disk
and invoked by some consumer crate), the call to
`proc_macro::Span::recover_proc_macro_span` causes us to load the span
from the proc-macro crate's metadata. The proc-macro then produces a
`TokenStream` containing a `Span` pointing into the proc-macro crate
itself.
The recursive nature of 'quote!' can be difficult to understand at
first. The file `src/test/ui/proc-macro/quote-debug.stdout` shows
the output of the `quote!` macro, which should make this eaier to
understand.
This PR also supports custom quoting spans in custom quote macros (e.g.
the `quote` crate). All span quoting goes through the
`proc_macro::quote_span` method, which can be called by a custom quote
macro to perform span quoting. An example of this usage is provided in
`src/test/ui/proc-macro/auxiliary/custom-quote.rs`
Custom quoting currently has a few limitations:
In order to quote a span, we need to generate a call to
`proc_macro::Span::recover_proc_macro_span`. However, proc-macros
support renaming the `proc_macro` crate, so we can't simply hardcode
this path. Previously, the `quote_span` method used the path
`crate::Span` - however, this only works when it is called by the
builtin `quote!` macro in the same crate. To support being called from
arbitrary crates, we need access to the name of the `proc_macro` crate
to generate a path. This PR adds an additional argument to `quote_span`
to specify the name of the `proc_macro` crate. Howver, this feels kind
of hacky, and we may want to change this before stabilizing anything
quote-related.
Additionally, using `quote_span` currently requires enabling the
`proc_macro_internals` feature. The builtin `quote!` macro
has an `#[allow_internal_unstable]` attribute, but this won't work for
custom quote implementations. This will likely require some additional
tricks to apply `allow_internal_unstable` to the span of
`proc_macro::Span::recover_proc_macro_span`.
2020-08-02 19:52:16 -04:00
|
|
|
fn recover_proc_macro_span(&mut self, id: usize) -> Self::Span {
|
2021-07-18 15:53:06 -04:00
|
|
|
let (resolver, krate, def_site) = (&*self.ecx.resolver, self.krate, self.def_site);
|
Implement span quoting for proc-macros
This PR implements span quoting, allowing proc-macros to produce spans
pointing *into their own crate*. This is used by the unstable
`proc_macro::quote!` macro, allowing us to get error messages like this:
```
error[E0412]: cannot find type `MissingType` in this scope
--> $DIR/auxiliary/span-from-proc-macro.rs:37:20
|
LL | pub fn error_from_attribute(_args: TokenStream, _input: TokenStream) -> TokenStream {
| ----------------------------------------------------------------------------------- in this expansion of procedural macro `#[error_from_attribute]`
...
LL | field: MissingType
| ^^^^^^^^^^^ not found in this scope
|
::: $DIR/span-from-proc-macro.rs:8:1
|
LL | #[error_from_attribute]
| ----------------------- in this macro invocation
```
Here, `MissingType` occurs inside the implementation of the proc-macro
`#[error_from_attribute]`. Previosuly, this would always result in a
span pointing at `#[error_from_attribute]`
This will make many proc-macro-related error message much more useful -
when a proc-macro generates code containing an error, users will get an
error message pointing directly at that code (within the macro
definition), instead of always getting a span pointing at the macro
invocation site.
This is implemented as follows:
* When a proc-macro crate is being *compiled*, it causes the `quote!`
macro to get run. This saves all of the sapns in the input to `quote!`
into the metadata of *the proc-macro-crate* (which we are currently
compiling). The `quote!` macro then expands to a call to
`proc_macro::Span::recover_proc_macro_span(id)`, where `id` is an
opaque identifier for the span in the crate metadata.
* When the same proc-macro crate is *run* (e.g. it is loaded from disk
and invoked by some consumer crate), the call to
`proc_macro::Span::recover_proc_macro_span` causes us to load the span
from the proc-macro crate's metadata. The proc-macro then produces a
`TokenStream` containing a `Span` pointing into the proc-macro crate
itself.
The recursive nature of 'quote!' can be difficult to understand at
first. The file `src/test/ui/proc-macro/quote-debug.stdout` shows
the output of the `quote!` macro, which should make this eaier to
understand.
This PR also supports custom quoting spans in custom quote macros (e.g.
the `quote` crate). All span quoting goes through the
`proc_macro::quote_span` method, which can be called by a custom quote
macro to perform span quoting. An example of this usage is provided in
`src/test/ui/proc-macro/auxiliary/custom-quote.rs`
Custom quoting currently has a few limitations:
In order to quote a span, we need to generate a call to
`proc_macro::Span::recover_proc_macro_span`. However, proc-macros
support renaming the `proc_macro` crate, so we can't simply hardcode
this path. Previously, the `quote_span` method used the path
`crate::Span` - however, this only works when it is called by the
builtin `quote!` macro in the same crate. To support being called from
arbitrary crates, we need access to the name of the `proc_macro` crate
to generate a path. This PR adds an additional argument to `quote_span`
to specify the name of the `proc_macro` crate. Howver, this feels kind
of hacky, and we may want to change this before stabilizing anything
quote-related.
Additionally, using `quote_span` currently requires enabling the
`proc_macro_internals` feature. The builtin `quote!` macro
has an `#[allow_internal_unstable]` attribute, but this won't work for
custom quote implementations. This will likely require some additional
tricks to apply `allow_internal_unstable` to the span of
`proc_macro::Span::recover_proc_macro_span`.
2020-08-02 19:52:16 -04:00
|
|
|
*self.rebased_spans.entry(id).or_insert_with(|| {
|
2021-07-10 23:44:22 +03:00
|
|
|
// FIXME: `SyntaxContext` for spans from proc macro crates is lost during encoding,
|
|
|
|
// replace it with a def-site context until we are encoding it properly.
|
|
|
|
resolver.get_proc_macro_quoted_span(krate, id).with_ctxt(def_site.ctxt())
|
Implement span quoting for proc-macros
This PR implements span quoting, allowing proc-macros to produce spans
pointing *into their own crate*. This is used by the unstable
`proc_macro::quote!` macro, allowing us to get error messages like this:
```
error[E0412]: cannot find type `MissingType` in this scope
--> $DIR/auxiliary/span-from-proc-macro.rs:37:20
|
LL | pub fn error_from_attribute(_args: TokenStream, _input: TokenStream) -> TokenStream {
| ----------------------------------------------------------------------------------- in this expansion of procedural macro `#[error_from_attribute]`
...
LL | field: MissingType
| ^^^^^^^^^^^ not found in this scope
|
::: $DIR/span-from-proc-macro.rs:8:1
|
LL | #[error_from_attribute]
| ----------------------- in this macro invocation
```
Here, `MissingType` occurs inside the implementation of the proc-macro
`#[error_from_attribute]`. Previosuly, this would always result in a
span pointing at `#[error_from_attribute]`
This will make many proc-macro-related error message much more useful -
when a proc-macro generates code containing an error, users will get an
error message pointing directly at that code (within the macro
definition), instead of always getting a span pointing at the macro
invocation site.
This is implemented as follows:
* When a proc-macro crate is being *compiled*, it causes the `quote!`
macro to get run. This saves all of the sapns in the input to `quote!`
into the metadata of *the proc-macro-crate* (which we are currently
compiling). The `quote!` macro then expands to a call to
`proc_macro::Span::recover_proc_macro_span(id)`, where `id` is an
opaque identifier for the span in the crate metadata.
* When the same proc-macro crate is *run* (e.g. it is loaded from disk
and invoked by some consumer crate), the call to
`proc_macro::Span::recover_proc_macro_span` causes us to load the span
from the proc-macro crate's metadata. The proc-macro then produces a
`TokenStream` containing a `Span` pointing into the proc-macro crate
itself.
The recursive nature of 'quote!' can be difficult to understand at
first. The file `src/test/ui/proc-macro/quote-debug.stdout` shows
the output of the `quote!` macro, which should make this eaier to
understand.
This PR also supports custom quoting spans in custom quote macros (e.g.
the `quote` crate). All span quoting goes through the
`proc_macro::quote_span` method, which can be called by a custom quote
macro to perform span quoting. An example of this usage is provided in
`src/test/ui/proc-macro/auxiliary/custom-quote.rs`
Custom quoting currently has a few limitations:
In order to quote a span, we need to generate a call to
`proc_macro::Span::recover_proc_macro_span`. However, proc-macros
support renaming the `proc_macro` crate, so we can't simply hardcode
this path. Previously, the `quote_span` method used the path
`crate::Span` - however, this only works when it is called by the
builtin `quote!` macro in the same crate. To support being called from
arbitrary crates, we need access to the name of the `proc_macro` crate
to generate a path. This PR adds an additional argument to `quote_span`
to specify the name of the `proc_macro` crate. Howver, this feels kind
of hacky, and we may want to change this before stabilizing anything
quote-related.
Additionally, using `quote_span` currently requires enabling the
`proc_macro_internals` feature. The builtin `quote!` macro
has an `#[allow_internal_unstable]` attribute, but this won't work for
custom quote implementations. This will likely require some additional
tricks to apply `allow_internal_unstable` to the span of
`proc_macro::Span::recover_proc_macro_span`.
2020-08-02 19:52:16 -04:00
|
|
|
})
|
|
|
|
}
|
2018-03-16 01:09:22 +02:00
|
|
|
}
|
2021-06-29 14:49:54 -04:00
|
|
|
|
2022-06-30 21:05:46 -04:00
|
|
|
impl server::Symbol for Rustc<'_, '_> {
|
|
|
|
fn normalize_and_validate_ident(&mut self, string: &str) -> Result<Self::Symbol, ()> {
|
|
|
|
let sym = nfc_normalize(string);
|
|
|
|
if rustc_lexer::is_ident(sym.as_str()) { Ok(sym) } else { Err(()) }
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2022-06-25 12:11:44 -04:00
|
|
|
impl server::Server for Rustc<'_, '_> {
|
2022-06-26 12:48:33 -04:00
|
|
|
fn globals(&mut self) -> ExpnGlobals<Self::Span> {
|
|
|
|
ExpnGlobals {
|
|
|
|
def_site: self.def_site,
|
|
|
|
call_site: self.call_site,
|
|
|
|
mixed_site: self.mixed_site,
|
|
|
|
}
|
2021-06-29 14:49:54 -04:00
|
|
|
}
|
2022-06-30 21:05:46 -04:00
|
|
|
|
|
|
|
fn intern_symbol(string: &str) -> Self::Symbol {
|
|
|
|
Symbol::intern(string)
|
|
|
|
}
|
|
|
|
|
|
|
|
fn with_symbol_string(symbol: &Self::Symbol, f: impl FnOnce(&str)) {
|
2023-11-21 20:07:32 +01:00
|
|
|
f(symbol.as_str())
|
2022-06-30 21:05:46 -04:00
|
|
|
}
|
2021-06-29 14:49:54 -04:00
|
|
|
}
|