Instead of trying to collect tokens at each depth, we 'flatten' the
stream as we go allong, pushing open/close delimiters to our buffer
just like regular tokens. One capturing is complete, we reconstruct a
nested `TokenTree::Delimited` structure, producing a normal
`TokenStream`.
The reconstructed `TokenStream` is not created immediately - instead, it is
produced on-demand by a closure (wrapped in a new `LazyTokenStream` type). This
closure stores a clone of the original `TokenCursor`, plus a record of the
number of calls to `next()/next_desugared()`. This is sufficient to reconstruct
the tokenstream seen by the callback without storing any additional state. If
the tokenstream is never used (e.g. when a captured `macro_rules!` argument is
never passed to a proc macro), we never actually create a `TokenStream`.
This implementation has a number of advantages over the previous one:
* It is significantly simpler, with no edge cases around capturing the
start/end of a delimited group.
* It can be easily extended to allow replacing tokens an an arbitrary
'depth' by just using `Vec::splice` at the proper position. This is
important for PR #76130, which requires us to track information about
attributes along with tokens.
* The lazy approach to `TokenStream` construction allows us to easily
parse an AST struct, and then decide after the fact whether we need a
`TokenStream`. This will be useful when we start collecting tokens for
`Attribute` - we can discard the `LazyTokenStream` if the parsed
attribute doesn't need tokens (e.g. is a builtin attribute).
The performance impact seems to be neglibile (see
https://github.com/rust-lang/rust/pull/77250#issuecomment-703960604). There is a
small slowdown on a few benchmarks, but it only rises above 1% for incremental
builds, where it represents a larger fraction of the much smaller instruction
count. There a ~1% speedup on a few other incremental benchmarks - my guess is
that the speedups and slowdowns will usually cancel out in practice.
Move `slice::check_range` to `RangeBounds`
Since this method doesn't take a slice anymore (#76662), it makes more sense to define it on `RangeBounds`.
Questions:
- Should the new method be `assert_len` or `assert_length`?
normalize substs while inlining
fixes#68347 or more precisely, this fixes the same ICE in rust analyser as veloren is pinned to a specific nightly
and had an error with the current one.
I didn't look into creating an MVCE here as that seems fairly annoying, will spend a few minutes doing so rn. (failed)
r? `@eddyb` cc `@bjorn3`
Make sure arenas don't allocate bigger than HUGE_PAGE
Right now, arenas allocate based on the size of the last chunk. It is possible for a `grow` call to allocate a chunk that is not a multiple of `PAGE`, and this size is doubled for each subsequent allocation. This means, instead of `HUGE_PAGE`, the biggest page possible is actually unknown.
This change fixes this, and also removes an unnecessary checked multiplication. It is still possible to allocate bigger than `HUGE_PAGE` pages, but this will only happen as many times as absolutely necessary.
Make set_span take mut self
This was a mistake in https://github.com/rust-lang/rust/pull/77614
It's not a _huge_ deal, because backends can always implement this with interior mutability, but it's nice to avoid interior mutability when possible. For context, the `set_source_location` method, called alongside `set_span`, also takes `&mut self`.
r? `@eddyb`
Add std:🧵:available_concurrency
This PR adds a counterpart to [C++'s `std:🧵:hardware_concurrency`](https://en.cppreference.com/w/cpp/thread/thread/hardware_concurrency) to Rust, tracking issue https://github.com/rust-lang/rust/issues/74479.
cc/ `@rust-lang/libs`
## Motivation
Being able to know how many hardware threads a platform supports is a core part of building multi-threaded code. In C++ 11 this has become available through the [`std:🧵:hardware_concurrency`](https://en.cppreference.com/w/cpp/thread/thread/hardware_concurrency) API. Currently in Rust most of the ecosystem depends on the [`num_cpus` crate](https://docs.rs/num_cpus/1.13.0/num_cpus/) ([no.35 in top 500 crates](https://docs.google.com/spreadsheets/d/1wwahRMHG3buvnfHjmPQFU4Kyfq15oTwbfsuZpwHUKc4/edit#gid=1253069234)) to provide this functionality. This PR proposes an API to provide access to the number of hardware threads available on a given platform.
__edit (2020-07-24):__ The purpose of this PR is to provide a hint for how many threads to spawn to saturate the processor. There's value in introducing APIs for NUMA and Windows processor groups, but those are intentionally out of scope for this PR. See: https://github.com/rust-lang/rust/pull/74480#issuecomment-662116186.
## Naming
Discussing the naming of the API on Zulip surfaced two options:
- `std:🧵:hardware_concurrency`
- `std:🧵:hardware_threads`
Both options seemed acceptable, but overall people seem to gravitate the most towards `hardware_threads`. Additionally `@jonas-schievink` pointed out that the "hardware threads" terminology is well-established and is used in among other the [RISC-V specification](https://riscv.org/specifications/isa-spec-pdf/) (page 20):
> A component is termed a core if it contains an independent instruction fetch unit. A RISC-V-compatible core might support multiple RISC-V-compatible __hardware threads__, or harts, through multithreading.
It's also worth noting that [the original paper introducing C++'s `std::thread` submodule](http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2007/n2320.html) unfortunately doesn't feature any discussion on the naming of `hardware_concurrency`, so we can't use that to help inform our decision here.
## Return type
An important consideration `@joshtriplett` brought up is that we don't want to default to `1` for platforms where the number of available threads cannot be retrieved. Instead we want to inform the users of the fact that we don't know and allow them to handle that case. Which is why this PR uses `Option<NonZeroUsize>` as its return type, where `None` is returned on platforms where we don't know the number of hardware threads available.
The reasoning for `NonZeroUsize` vs `usize` is that if the number of threads for a platform are known, they'll always be at least 1. As evidenced by the example the `NonZero*` family of APIs may currently not be the most ergonomic to use, but improving the ergonomics of them is something that I think we can address separately.
## Implementation
`@Mark-Simulacrum` pointed out that most of the code we wanted to expose here was already available under `libtest`. So this PR mostly moves the internal code of libtest into a public API.
Add some more regression tests
This is another round of #77741. Tested with `debug-assertions=true` and it passed on my local.
Closes#70877Closes#70944Closes#71659Closes#74816Closes#75707Closes#75983
(Skipped #63355 because I'm not sure about the error.)
Rollup of 7 pull requests
Successful merges:
- #75802 (resolve: Do not put nonexistent crate `meta` into prelude)
- #76607 (Modify executable checking to be more universal)
- #77851 (BTreeMap: refactor Entry out of map.rs into its own file)
- #78043 (Fix grammar in note for orphan-rule error [E0210])
- #78048 (Suggest correct place to add `self` parameter when inside closure)
- #78050 (Small CSS cleanup)
- #78059 (Set `MDBOOK_OUTPUT__HTML__INPUT_404` on linkchecker)
Failed merges:
r? `@ghost`
Suggest correct place to add `self` parameter when inside closure
It would incorrectly suggest adding it as a parameter to the closure instead of the containing function.
[For example](https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=1936bcd1e5f981573386e0cee985c3c0):
```
help: add a `self` receiver parameter to make the associated `fn` a method
|
5 | let _ = || self&self;
| ^^^^^
```
`DiagnosticMetadata.current_function` is only used for these messages so tweaking its behavior should be ok.
Fix grammar in note for orphan-rule error [E0210]
Fixes the grammar in the error note for [E0210] from:
_"= note: implementing a foreign trait is only possible if at least one of the types for which **is it** implemented is local"_
to:
_"= note: implementing a foreign trait is only possible if at least one of the types for which **it is** implemented is local"_
The content of this commit is the result of running the following command at the repository root:
`find . \( -type d -name .git -prune \) -o -type f -print0 | xargs -0 sed -i 's/which is it implemented/which it is implemented/g'`
BTreeMap: refactor Entry out of map.rs into its own file
btree/map.rs is approaching the 3000 line mark, splitting out the entry
code buys about 500 lines of headroom.
I've created this PR because the changes I've made in #77438 will push `map.rs` over the 3000 line limit and cause tidy to complain.
I picked `Entry` to factor out because it feels less tightly coupled to the rest of `BTreeMap` than the various iterator implementations.
Related: #60302
Modify executable checking to be more universal
This uses a dummy file to check if the filesystem being used supports the executable bit in general.
Supersedes #74753.
resolve: Do not put nonexistent crate `meta` into prelude
Before the 2018 edition release there was some vague suggestion about adding a crate named `meta` to the standard distribution.
On this basis the name `meta` was "partially reserved" by putting `meta` into extern prelude (this means importing something named `meta` will result in an ambiguity error, for example).
This only caused confusion so far, and two years later there are no specific plans to add such crate.
If some standard crate (named `meta` or not) is added in the future, then cargo will hopefully already have ability to put it into extern prelude explicitly through `Cargo.toml`.
Otherwise, it could be added to extern prelude by the compiler at edition boundary.
Closes https://github.com/rust-lang/rust/issues/73948
