Remove TokenStream::Tree variant.

`TokenStream::Stream` can represent a token stream containing any number
of token trees. `TokenStream::Tree` is the special case representing a
single token tree. The latter doesn't occur all that often dynamically,
so this commit removes it, which simplifies the code quite a bit.

This change has mixed performance effects.

- The size of `TokenStream` drops from 32 bytes to 8 bytes, and there
  is one less case for all the match statements.

- The conversion of a `TokenTree` to a `TokenStream` now requires two
  allocations, for the creation of a single element Lrc<Vec<_>>. (But a
  subsequent commit in this PR will reduce the main source of such
  conversions.)
This commit is contained in:
Nicholas Nethercote 2019-01-09 15:20:56 +11:00
parent 2cf736f765
commit 28966e1a7a
2 changed files with 6 additions and 41 deletions

View file

@ -269,7 +269,7 @@ impl ToInternal<TokenStream> for TokenTree<Group, Punct, Ident, Literal> {
};
let tree = tokenstream::TokenTree::Token(span, token);
TokenStream::Tree(tree, if joint { Joint } else { NonJoint })
TokenStream::new(vec![(tree, if joint { Joint } else { NonJoint })])
}
}