Fix UB from misalignment and provenance widening in `std::sys::windows`
This fixes two types of UB:
1. Reading past the end of a reference in types like `&c::REPARSE_DATA_BUFFER` (see https://github.com/rust-lang/unsafe-code-guidelines/issues/256). This is fixed by using `addr_of!`. I think there are probably a couple more cases where we do this for other structures, and will look into it in a bit.
2. Failing to ensure that a `[u8; N]` on the stack is sufficiently aligned to convert to a `REPARSE_DATA_BUFFER`. ~~This was done by introducing a new `AlignedAs` struct that allows aligning one type to the alignment of another type. I expect there are other places where we have this issue too, or I wouldn't introduce this type, but will get to them after this lands.~~
~~Worth noting, it *is* implemented in a way that can cause problems depending on how we fix#81996, but this would be caught by the test I added (and presumably if we decide to fix that in a way that would break this code, we'd also introduce a `#[repr(simple)]` or `#[repr(linear)]` as a replacement for this usage of `#[repr(C)]`).~~
Edit: None of that is still in the code, I just went with a `Align8` since that's all we'll need for almost everything we want to call.
These are more or less "potential UB" since it's likely at the moment everything works fine, although the alignment not causing issues might just be down to luck (and x86 being forgiving).
~~NB: I've only ensured this check builds, but will run tests soon.~~ All tests pass, including stage2 compiler tests.
r? ``@ChrisDenton``
Use getentropy when possible on all Apple platforms
As the current code comments say, `SecRandomCopyBytes` is very heavyweight (regardless of purpose) compared to just asking the kernel directly for bytes from its own CSPRNG. We were not previously making an attempt to use the more efficient `getentropy` call on other Apple targets, instead solely using it on macOS. As the function is available on newer versions of Apple's different OSes, this changes the random filling to always attempt it first everywhere, only falling back to the less ideal alternatives after. This also cleans up the multiple Apple `imp` blocks into one.
It also should give a perf improvement, even if its likely unnoticeably small.
Refed XCode header for `getentropy` in the SDK:
```h
int getentropy(void* buffer, size_t size) __OSX_AVAILABLE(10.12) __IOS_AVAILABLE(10.0) __TVOS_AVAILABLE(10.0) __WATCHOS_AVAILABLE(3.0);
```
r? ``@thomcc``
Reinstate preloading of some dll imports
I've now come around to the conclusion that there is a justification for pre-loading the synchronization functions `WaitOnAddress` and `WakeByAddressSingle`. I've found this to have a particularly impact in testing frameworks that may have short lived processes which immediately spawn lots of threads.
Also, because pre-main initializers imply a single-threaded environment, we can switch back to using relaxed atomics which might be a minor perf improvement on some platforms (though I doubt it's particularly notable).
r? ``@Mark-Simulacrum`` and sorry for the churn here.
For convenience I'll summarise previous issues with preloading and the solutions that are included in this PR (if any):
**Issue:** User pre-main initializers may be run before std's
**Solution:** The std now uses initializers that are guaranteed to run earlier than the old initializers. A note is also added that users should not copy std's behaviour if they want to ensure they run their initializers after std.
**Issue:** Miri does not understand pre-main initializers.
**Solution:** For miri only, run the function loading lazily instead.
**Issue:** We should ideally use `LoadLibrary` to get "api-ms-win-core-synch-l1-2-0". Only "ntdll" and "kernel32" are guaranteed to always be loaded.
**Solution:** None. We can't use `LoadLibrary` pre-main. However, in the past `GetModuleHandle` has always worked in practice so this should hopefully not be a problem.
If/when Windows 7 support is dropped, we can finally remove all this for good and just use normal imports.
Avoid zeroing large stack buffers in stdio on Windows
Does what it says on the tin, using `[MaybeUninit<u16>; N]` instead of `[0u16; N]`. These buffers seem to be around 8kb, which is big enough that this is likely to be a very nice perf boost to stdio-heavy windows code.
r? ``@ChrisDenton``
*(Note: this PR also has a commit that adds windows to CI, but as it mentions I'll revert that after it comes out green -- I can only do a check build on the machine I'm typing this on)*
Revert let_chains stabilization
This is the revert against master, the beta revert was already done in #100538.
Bumps the stage0 compiler which already has it reverted.
Rollup of 7 pull requests
Successful merges:
- #100898 (Do not report too many expr field candidates)
- #101056 (Add the syntax of references to their documentation summary.)
- #101106 (Rustdoc-Json: Retain Stripped Modules when they are imported, not when they have items)
- #101131 (CTFE: exposing pointers and calling extern fn is just impossible)
- #101141 (Simplify `get_trait_ref` fn used for `virtual_function_elimination`)
- #101146 (Various changes to logging of borrowck-related code)
- #101156 (Remove `Sync` requirement from lint pass objects)
Failed merges:
r? `@ghost`
`@rustbot` modify labels: rollup
Add the syntax of references to their documentation summary.
Without this change, in <https://doc.rust-lang.org/1.63.0/std/#primitives>, `reference` is the only entry in that list which does not contain the syntax by which the type is named in source code. With this change, it contains them, in roughly the same way as the `pointer` entry does.
Make use of `[wrapping_]byte_{add,sub}`
These new methods trivially replace old `.cast().wrapping_offset().cast()` & similar code.
Note that [`arith_offset`](https://doc.rust-lang.org/std/intrinsics/fn.arith_offset.html) and `wrapping_offset` are the same thing.
r? ``@scottmcm``
_split off from #100746_
Stabilize `std::io::read_to_string`
Closes#80218. 🎉
This PR stabilizes the `std::io::read_to_string` function, with the following public API:
```rust
pub fn read_to_string<R: Read>(reader: R) -> Result<String>;
```
It's analogous to `std::fs::read_to_string` for files, but it works on anything that implements `io::Read`, including `io::stdin()`.
See the tracking issue (#80218) or documentation for details.
Add a `File::create_new` constructor
We have `File::create` for creating a file or opening an existing file,
but the secure way to guarantee creating a new file requires a longhand
invocation via `OpenOptions`.
Add `File::create_new` to handle this case, to make it easier for people
to do secure file creation.
Use posix_spawn for absolute paths on macOS
Currently, on macOS, Rust never uses the fast posix_spawn path if a
directory change is requested, due to a bug in Apple's libc. However, the
bug is only triggered if the program is a relative path.
This PR makes it so that the fast path continues to work if the program
is an absolute path or a lone filename.
This was an alternative proposed in https://github.com/rust-lang/rust/pull/80537#issue-776674009, and it makes a measurable performance difference in some of my code that spawns thousands of processes.
Support parsing IP addresses from a byte string
Fixes#94821
The goal is to be able to parse addresses from a byte string without requiring to do any utf8 validation. Since internally the parser already works on byte strings, this should be possible and I personally already needed this in the past too.
~~I used the proposed approach from the issue by implementing `TryFrom<&'a [u8]>` for all 6 address types (3 ip address types and 3 socket address types). I believe implementing stable traits for stable types is insta-stable so this will probably need an FCP?~~
Switched to an unstable inherent method approach called `parse_ascii` as requested.
cc ``````@jyn514``````
Currently, on macOS, Rust never uses the fast posix_spawn path if a
directory change is requested due to a bug in Apple's libc. However, the
bug is only triggered if the program is a relative path.
This PR makes it so that the fast path continues to work if the program
is an absolute path or a lone filename.
This was an alternative proposed in
https://github.com/rust-lang/rust/pull/80537#issue-776674009, and it
makes a measurable performance difference in some of my code that spawns
thousands of processes.
Add next_up and next_down for f32/f64 - take 2
This is a revival of https://github.com/rust-lang/rust/pull/88728 which staled due to inactivity of the original author. I've address the last review comment.
---
This is a pull request implementing the features described at https://github.com/rust-lang/rfcs/pull/3173.
`@rustbot` label +T-libs-api -T-libs
r? `@scottmcm`
cc `@orlp`
std: use realstd fast key when building tests
Under `cfg(test)`, the `std` crate is not the actual standard library, just any old crate we are testing. It imports the real standard library as `realstd`, and then does some careful `cfg` magic so that the crate built for testing uses the `realstd` global state rather than having its own copy of that.
However, this was not done for all global state hidden in std: the 'fast' version of thread-local keys, at least on some platforms, also involves some global state. Specifically its macOS version has this [`static REGISTERED`](bc63d5a26a/library/std/src/sys/unix/thread_local_dtor.rs (L62)) that would get duplicated. So this PR imports the 'fast' key type from `realstd` rather than using the local copy, to ensure its internal state (and that of the functions it calls) does not get duplicated.
I also noticed that the `__OsLocalKeyInner` is unused under `cfg(target_thread_local)`, so I removed it for that configuration. There was a comment saying macOS picks between `__OsLocalKeyInner` and `__FastLocalKeyInner` at runtime, but I think that comment is outdated -- I found no trace of such a runtime switching mechanism, and the library still check-builds on apple targets with this PR. (I don't have a Mac so I cannot actually run it.)
Some papercuts on error::Error
Renames the chain method, since I chain could mean anything and doesn't refer to a chain of sources (cc #58520) (and adds a comment explaining why sources is not a provided method on Error). Renames arguments to the request method from `req` to `demand` since the type is `Demand` rather than Request or Requisition.
r? ``@yaahc``
Add mention of `BufReader` in `Read::bytes` docs
There is a general paragraph about `BufRead` in the `Read` trait's docs, however using `bytes` without `BufRead` *always* has a large impact, due to reads of size 1.
`@rustbot` label +A-docs
Add standard C error function aliases to last_os_error
This aids the discoverability of `io::Error::last_os_error()` by linking to commonly used error number functions from C/C++.
I've seen a few people not realize this exists, so hopefully this helps draw attention to the API to encourage using it over integer error codes.
std::io: migrate ReadBuf to BorrowBuf/BorrowCursor
This PR replaces `ReadBuf` (used by the `Read::read_buf` family of methods) with `BorrowBuf` and `BorrowCursor`.
The general idea is to split `ReadBuf` because its API is large and confusing. `BorrowBuf` represents a borrowed buffer which is mostly read-only and (other than for construction) deals only with filled vs unfilled segments. a `BorrowCursor` is a mostly write-only view of the unfilled part of a `BorrowBuf` which distinguishes between initialized and uninitialized segments. For `Read::read_buf`, the caller would create a `BorrowBuf`, then pass a `BorrowCursor` to `read_buf`.
In addition to the major API split, I've made the following smaller changes:
* Removed some methods entirely from the API (mostly the functionality can be replicated with two calls rather than a single one)
* Unified naming, e.g., by replacing initialized with init and assume_init with set_init
* Added an easy way to get the number of bytes written to a cursor (`written` method)
As well as simplifying the API (IMO), this approach has the following advantages:
* Since we pass the cursor by value, we remove the 'unsoundness footgun' where a malicious `read_buf` could swap out the `ReadBuf`.
* Since `read_buf` cannot write into the filled part of the buffer, we prevent the filled part shrinking or changing which could cause underflow for the caller or unexpected behaviour.
## Outline
```rust
pub struct BorrowBuf<'a>
impl Debug for BorrowBuf<'_>
impl<'a> From<&'a mut [u8]> for BorrowBuf<'a>
impl<'a> From<&'a mut [MaybeUninit<u8>]> for BorrowBuf<'a>
impl<'a> BorrowBuf<'a> {
pub fn capacity(&self) -> usize
pub fn len(&self) -> usize
pub fn init_len(&self) -> usize
pub fn filled(&self) -> &[u8]
pub fn unfilled<'this>(&'this mut self) -> BorrowCursor<'this, 'a>
pub fn clear(&mut self) -> &mut Self
pub unsafe fn set_init(&mut self, n: usize) -> &mut Self
}
pub struct BorrowCursor<'buf, 'data>
impl<'buf, 'data> BorrowCursor<'buf, 'data> {
pub fn clone<'this>(&'this mut self) -> BorrowCursor<'this, 'data>
pub fn capacity(&self) -> usize
pub fn written(&self) -> usize
pub fn init_ref(&self) -> &[u8]
pub fn init_mut(&mut self) -> &mut [u8]
pub fn uninit_mut(&mut self) -> &mut [MaybeUninit<u8>]
pub unsafe fn as_mut(&mut self) -> &mut [MaybeUninit<u8>]
pub unsafe fn advance(&mut self, n: usize) -> &mut Self
pub fn ensure_init(&mut self) -> &mut Self
pub unsafe fn set_init(&mut self, n: usize) -> &mut Self
pub fn append(&mut self, buf: &[u8])
}
```
## TODO
* ~~Migrate non-unix libs and tests~~
* ~~Naming~~
* ~~`BorrowBuf` or `BorrowedBuf` or `SliceBuf`? (We might want an owned equivalent for the async IO traits)~~
* ~~Should we rename the `readbuf` module? We might keep the name indicate it includes both the buf and cursor variations and someday the owned version too. Or we could change it. It is not publicly exposed, so it is not that important~~.
* ~~`read_buf` method: we read into the cursor now, so the `_buf` suffix is a bit weird.~~
* ~~Documentation~~
* Tests are incomplete (I adjusted existing tests, but did not add new ones).
cc https://github.com/rust-lang/rust/issues/78485, https://github.com/rust-lang/rust/issues/94741
supersedes: https://github.com/rust-lang/rust/pull/95770, https://github.com/rust-lang/rust/pull/93359fixes#93305
Move EH personality functions to std
These were previously in the panic_unwind crate with dummy stubs in the
panic_abort crate. However it turns out that this is insufficient: we
still need a proper personality function even with -C panic=abort to
handle the following cases:
1) `extern "C-unwind"` still needs to catch foreign exceptions with -C
panic=abort to turn them into aborts. This requires landing pads and a
personality function.
2) ARM EHABI uses the personality function when creating backtraces.
The dummy personality function in panic_abort was causing backtrace
generation to get stuck in a loop since the personality function is
responsible for advancing the unwind state to the next frame.
Fixes#41004
Without this change, in <https://doc.rust-lang.org/1.63.0/std/#primitives>,
`reference` is the only entry in that list which does not contain the
syntax by which the type is named in source code. With this change, it
contains them, in roughly the same way as the `pointer` entry does.
Extra documentation for new formatting feature
Documentation of this feature was added in #90473 and released in Rust 1.58. However, high traffic macros did not receive new examples. Namely `println!()` and `format!()`.
The doc comments included in Rust are super important to the community- especially newcomers. I have met several other newbies like myself who are unaware of this recent (well about 7 months old now) update to the language allowing for convenient intra-string identifiers.
Bringing small examples of this feature to the doc comments of `println!()` and `format!()` would be helpful to everyone learning the language.
[Blog Post Announcing Feature](https://blog.rust-lang.org/2022/01/13/Rust-1.58.0.html)
[Feature PR](https://github.com/rust-lang/rust/pull/90473) - includes several instances of documentation of the feature- minus the macros in question for this PR
*This is my first time contributing to a project this large. Feedback would mean the world to me 😄*
---
*Recreated; I violated the [No-Merge Policy](https://rustc-dev-guide.rust-lang.org/git.html#no-merge-policy)*
Optimize `Wtf8Buf::into_string` for the case where it contains UTF-8.
Add a `is_known_utf8` flag to `Wtf8Buf`, which tracks whether the
string is known to contain UTF-8. This is efficiently computed in many
common situations, such as when a `Wtf8Buf` is constructed from a `String`
or `&str`, or with `Wtf8Buf::from_wide` which is already doing UTF-16
decoding and already checking for surrogates.
This makes `OsString::into_string` O(1) rather than O(N) on Windows in
common cases.
And, it eliminates the need to scan through the string for surrogates in
`Args::next` and `Vars::next`, because the strings are already being
translated with `Wtf8Buf::from_wide`.
Many things on Windows construct `OsString`s with `Wtf8Buf::from_wide`,
such as `DirEntry::file_name` and `fs::read_link`, so with this patch,
users of those functions can subsequently call `.into_string()` without
paying for an extra scan through the string for surrogates.
r? `@ghost`
Move Error trait into core
This PR moves the error trait from the standard library into a new unstable `error` module within the core library. The goal of this PR is to help unify error reporting across the std and no_std ecosystems, as well as open the door to integrating the error trait into the panic reporting system when reporting panics whose source is an errors (such as via `expect`).
This PR is a rewrite of https://github.com/rust-lang/rust/pull/90328 using new compiler features that have been added to support error in core.
These were previously in the panic_unwind crate with dummy stubs in the
panic_abort crate. However it turns out that this is insufficient: we
still need a proper personality function even with -C panic=abort to
handle the following cases:
1) `extern "C-unwind"` still needs to catch foreign exceptions with -C
panic=abort to turn them into aborts. This requires landing pads and a
personality function.
2) ARM EHABI uses the personality function when creating backtraces.
The dummy personality function in panic_abort was causing backtrace
generation to get stuck in a loop since the personality function is
responsible for advancing the unwind state to the next frame.
