Stabilize async closures (RFC 3668)
# Async Closures Stabilization Report
This report proposes the stabilization of `#![feature(async_closure)]` ([RFC 3668](https://rust-lang.github.io/rfcs/3668-async-closures.html)). This is a long-awaited feature that increases the expressiveness of the Rust language and fills a pressing gap in the async ecosystem.
## Stabilization summary
* You can write async closures like `async || {}` which return futures that can borrow from their captures and can be higher-ranked in their argument lifetimes.
* You can express trait bounds for these async closures using the `AsyncFn` family of traits, analogous to the `Fn` family.
```rust
async fn takes_an_async_fn(f: impl AsyncFn(&str)) {
futures::join(f("hello"), f("world")).await;
}
takes_an_async_fn(async |s| { other_fn(s).await }).await;
```
## Motivation
Without this feature, users hit two major obstacles when writing async code that uses closures and `Fn` trait bounds:
- The inability to express higher-ranked async function signatures.
- That closures cannot return futures that borrow from the closure captures.
That is, for the first, we cannot write:
```rust
// We cannot express higher-ranked async function signatures.
async fn f<Fut>(_: impl for<'a> Fn(&'a u8) -> Fut)
where
Fut: Future<Output = ()>,
{ todo!() }
async fn main() {
async fn g(_: &u8) { todo!() }
f(g).await;
//~^ ERROR mismatched types
//~| ERROR one type is more general than the other
}
```
And for the second, we cannot write:
```rust
// Closures cannot return futures that borrow closure captures.
async fn f<Fut: Future<Output = ()>>(_: impl FnMut() -> Fut)
{ todo!() }
async fn main() {
let mut xs = vec![];
f(|| async {
async fn g() -> u8 { todo!() }
xs.push(g().await);
});
//~^ ERROR captured variable cannot escape `FnMut` closure body
}
```
Async closures provide a first-class solution to these problems.
For further background, please refer to the [motivation section](https://rust-lang.github.io/rfcs/3668-async-closures.html#motivation) of the RFC.
## Major design decisions since RFC
The RFC had left open the question of whether we would spell the bounds syntax for async closures...
```rust
// ...as this...
fn f() -> impl AsyncFn() -> u8 { todo!() }
// ...or as this:
fn f() -> impl async Fn() -> u8 { todo!() }
```
We've decided to spell this as `AsyncFn{,Mut,Once}`.
The `Fn` family of traits is special in many ways. We had originally argued that, due to this specialness, that perhaps the `async Fn` syntax could be adopted without having to decide whether a general `async Trait` mechanism would ever be adopted. However, concerns have been raised that we may not want to use `async Fn` syntax unless we would pursue more general trait modifiers. Since there remain substantial open questions on those -- and we don't want to rush any design work there -- it makes sense to ship this needed feature using the `AsyncFn`-style bounds syntax.
Since we would, in no case, be shipping a generalized trait modifier system anytime soon, we'll be continuing to see `AsyncFoo` traits appear across the ecosystem regardless. If we were to ever later ship some general mechanism, we could at that time manage the migration from `AsyncFn` to `async Fn`, just as we'd be enabling and managing the migration of many other traits.
Note that, as specified in RFC 3668, the details of the `AsyncFn*` traits are not exposed and they can only be named via the "parentheses sugar". That is, we can write `T: AsyncFn() -> u8` but not `T: AsyncFn<Output = u8>`.
Unlike the `Fn` traits, we cannot project to the `Output` associated type of the `AsyncFn` traits. That is, while we can write...
```rust
fn f<F: Fn() -> u8>(_: F::Output) {}
```
...we cannot write:
```rust
fn f<F: AsyncFn() -> u8>(_: F::Output) {}
//~^ ERROR
```
The choice of `AsyncFn{,Mut,Once}` bounds syntax obviates, for our purposes here, another question decided after that RFC, which was how to order bound modifiers such as `for<'a> async Fn()`.
Other than answering the open question in the RFC on syntax, nothing has changed about the design of this feature between RFC 3668 and this stabilization.
## What is stabilized
For those interested in the technical details, please see [the dev guide section](https://rustc-dev-guide.rust-lang.org/coroutine-closures.html) I authored.
#### Async closures
Other than in how they solve the problems described above, async closures act similarly to closures that return async blocks, and can have parts of their signatures specified:
```rust
// They can have arguments annotated with types:
let _ = async |_: u8| { todo!() };
// They can have their return types annotated:
let _ = async || -> u8 { todo!() };
// They can be higher-ranked:
let _ = async |_: &str| { todo!() };
// They can capture values by move:
let x = String::from("hello, world");
let _ = async move || do_something(&x).await };
```
When called, they return an anonymous future type corresponding to the (not-yet-executed) body of the closure. These can be awaited like any other future.
What distinguishes async closures is that, unlike closures that return async blocks, the futures returned from the async closure can capture state from the async closure. For example:
```rust
let vec: Vec<String> = vec![];
let closure = async || {
vec.push(ready(String::from("")).await);
};
```
The async closure captures `vec` with some `&'closure mut Vec<String>` which lives until the closure is dropped. Every call to `closure()` returns a future which reborrows that mutable reference `&'call mut Vec<String>` which lives until the future is dropped (e.g. it is `await`ed).
As another example:
```rust
let string: String = "Hello, world".into();
let closure = async move || {
ready(&string).await;
};
```
The closure is marked with `move`, which means it takes ownership of the string by *value*. The future that is returned by calling `closure()` returns a future which borrows a reference `&'call String` which lives until the future is dropped (e.g. it is `await`ed).
#### Async fn trait family
To support the lending capability of async closures, and to provide a first-class way to express higher-ranked async closures, we introduce the `AsyncFn*` family of traits. See the [corresponding section](https://rust-lang.github.io/rfcs/3668-async-closures.html#asyncfn) of the RFC.
We stabilize naming `AsyncFn*` via the "parenthesized sugar" syntax that normal `Fn*` traits can be named. The `AsyncFn*` trait can be used anywhere a `Fn*` trait bound is allowed, such as:
```rust
/// In return-position impl trait:
fn closure() -> impl AsyncFn() { async || {} }
/// In trait bounds:
trait Foo<F>: Sized
where
F: AsyncFn()
{
fn new(f: F) -> Self;
}
/// in GATs:
trait Gat {
type AsyncHasher<T>: AsyncFn(T) -> i32;
}
```
Other than using them in trait bounds, the definitions of these traits are not directly observable, but certain aspects of their behavior can be indirectly observed such as the fact that:
* `AsyncFn::async_call` and `AsyncFnMut::async_call_mut` return a future which is *lending*, and therefore borrows the `&self` lifetime of the callee.
```rust
fn by_ref_call(c: impl AsyncFn()) {
let fut = c();
drop(c);
// ^ Cannot drop `c` since it is borrowed by `fut`.
}
```
* `AsyncFnOnce::async_call_once` returns a future that takes ownership of the callee.
```rust
fn by_ref_call(c: impl AsyncFnOnce()) {
let fut = c();
let _ = c();
// ^ Cannot call `c` since calling it takes ownership the callee.
}
```
* All currently-stable callable types (i.e., closures, function items, function pointers, and `dyn Fn*` trait objects) automatically implement `AsyncFn*() -> T` if they implement `Fn*() -> Fut` for some output type `Fut`, and `Fut` implements `Future<Output = T>`.
* This is to make sure that `AsyncFn*()` trait bounds have maximum compatibility with existing callable types which return futures, such as async function items and closures which return boxed futures.
* For now, this only works currently for *concrete* callable types -- for example, a argument-position impl trait like `impl Fn() -> impl Future<Output = ()>` does not implement `AsyncFn()`, due to the fact that a `AsyncFn`-if-`Fn` blanket impl does not exist in reality. This may be relaxed in the future. Users can work around this by wrapping their type in an async closure and calling it. I expect this to not matter much in practice, as users are encouraged to write `AsyncFn` bounds directly.
```rust
fn is_async_fn(_: impl AsyncFn(&str)) {}
async fn async_fn_item(s: &str) { todo!() }
is_async_fn(s);
// ^^^ This works.
fn generic(f: impl Fn() -> impl Future<Output = ()>) {
is_async_fn(f);
// ^^^ This does not work (yet).
}
```
#### The by-move future
When async closures are called with `AsyncFn`/`AsyncFnMut`, they return a coroutine that borrows from the closure. However, when they are called via `AsyncFnOnce`, we consume that closure, and cannot return a coroutine that borrows from data that is now dropped.
To work around around this limitation, we synthesize a separate future type for calling the async closure via `AsyncFnOnce`.
This future executes identically to the by-ref future returned from calling the async closure, except for the fact that it has a different set of captures, since we must *move* the captures from the parent async into the child future.
#### Interactions between async closures and the `Fn*` family of traits
Async closures always implement `FnOnce`, since they always can be called once. They may also implement `Fn` or `FnMut` if their body is compatible with the calling mode (i.e. if they do not mutate their captures, or they do not capture their captures, respectively) and if the future returned by the async closure is not *lending*.
```rust
let id = String::new();
let mapped: Vec</* impl Future */> =
[/* elements */]
.into_iter()
// `Iterator::map` takes an `impl FnMut`
.map(async |element| {
do_something(&id, element).await;
})
.collect();
```
See [the dev guide](https://rustc-dev-guide.rust-lang.org/coroutine-closures.html#follow-up-when-do-async-closures-implement-the-regular-fn-traits) for a detailed explanation for the situations where this may not be possible due to the lending nature of async closures.
#### Other notable features of async closures shared with synchronous closures
* Async closures are `Copy` and/or `Clone` if their captures are `Copy`/`Clone`.
* Async closures do closure signature inference: If an async closure is passed to a function with a `AsyncFn` or `Fn` trait bound, we can eagerly infer the argument types of the closure. More details are provided in [the dev guide](https://rustc-dev-guide.rust-lang.org/coroutine-closures.html#closure-signature-inference).
#### Lints
This PR also stabilizes the `CLOSURE_RETURNING_ASYNC_BLOCK` lint as an `allow` lint. This lints on "old-style" async closures:
```rust
#![warn(closure_returning_async_block)]
let c = |x: &str| async {};
```
We should encourage users to use `async || {}` where possible. This lint remains `allow` and may be refined in the future because it has a few false positives (namely, see: "Where do we expect rewriting `|| async {}` into `async || {}` to fail?")
An alternative that could be made at the time of stabilization is to put this lint behind another gate, so we can decide to stabilize it later.
## What isn't stabilized (aka, potential future work)
#### `async Fn*()` bound syntax
We decided to stabilize async closures without the `async Fn*()` bound modifier syntax. The general direction of this syntax and how it fits is still being considered by T-lang (e.g. in [RFC 3710](https://github.com/rust-lang/rfcs/pull/3710)).
#### Naming the futures returned by async closures
This stabilization PR does not provide a way of naming the futures returned by calling `AsyncFn*`.
Exposing a stable way to refer to these futures is important for building async-closure-aware combinators, and will be an important future step.
#### Return type notation-style bounds for async closures
The RFC described an RTN-like syntax for putting bounds on the future returned by an async closure:
```rust
async fn foo(x: F) -> Result<()>
where
F: AsyncFn(&str) -> Result<()>,
// The future from calling `F` is `Send` and `'static`.
F(..): Send + 'static,
{}
```
This stabilization PR does not stabilize that syntax yet, which remains unimplemented (though will be soon).
#### `dyn AsyncFn*()`
`AsyncFn*` are not dyn-compatible yet. This will likely be implemented in the future along with the dyn-compatibility of async fn in trait, since the same issue (dealing with the future returned by a call) applies there.
## Tests
Tests exist for this feature in [`tests/ui/async-await/async-closures`](5b54286640/tests/ui/async-await/async-closures).
<details>
<summary>A selected set of tests:</summary>
* Lending behavior of async closures
* `tests/ui/async-await/async-closures/mutate.rs`
* `tests/ui/async-await/async-closures/captures.rs`
* `tests/ui/async-await/async-closures/precise-captures.rs`
* `tests/ui/async-await/async-closures/no-borrow-from-env.rs`
* Async closures may be higher-ranked
* `tests/ui/async-await/async-closures/higher-ranked.rs`
* `tests/ui/async-await/async-closures/higher-ranked-return.rs`
* Async closures may implement `Fn*` traits
* `tests/ui/async-await/async-closures/is-fn.rs`
* `tests/ui/async-await/async-closures/implements-fnmut.rs`
* Async closures may be cloned
* `tests/ui/async-await/async-closures/clone-closure.rs`
* Ownership of the upvars when `AsyncFnOnce` is called
* `tests/ui/async-await/async-closures/drop.rs`
* `tests/ui/async-await/async-closures/move-is-async-fn.rs`
* `tests/ui/async-await/async-closures/force-move-due-to-inferred-kind.rs`
* `tests/ui/async-await/async-closures/force-move-due-to-actually-fnonce.rs`
* Closure signature inference
* `tests/ui/async-await/async-closures/signature-deduction.rs`
* `tests/ui/async-await/async-closures/sig-from-bare-fn.rs`
* `tests/ui/async-await/async-closures/signature-inference-from-two-part-bound.rs`
</details>
## Remaining bugs and open issues
* https://github.com/rust-lang/rust/issues/120694 tracks moving onto more general `LendingFn*` traits. No action needed, since it's not observable.
* https://github.com/rust-lang/rust/issues/124020 - Polymorphization ICE. Polymorphization needs to be heavily reworked. No action needed.
* https://github.com/rust-lang/rust/issues/127227 - Tracking reworking the way that rustdoc re-sugars bounds.
* The part relevant to to `AsyncFn` is fixed by https://github.com/rust-lang/rust/pull/132697.
## Where do we expect rewriting `|| async {}` into `async || {}` to fail?
* Fn pointer coercions
* Currently, it is not possible to coerce an async closure to an fn pointer like regular closures can be. This functionality may be implemented in the future.
```rust
let x: fn() -> _ = async || {};
```
* Argument capture
* Like async functions, async closures always capture their input arguments. This is in contrast to something like `|t: T| async {}`, which doesn't capture `t` unless it is used in the async block. This may affect the `Send`-ness of the future or affect its outlives.
```rust
fn needs_send_future(_: impl Fn(NotSendArg) -> Fut)
where
Fut: Future<Output = ()>,
{}
needs_send_future(async |_| {});
```
## History
#### Important feature history
- https://github.com/rust-lang/rust/pull/51580
- https://github.com/rust-lang/rust/pull/62292
- https://github.com/rust-lang/rust/pull/120361
- https://github.com/rust-lang/rust/pull/120712
- https://github.com/rust-lang/rust/pull/121857
- https://github.com/rust-lang/rust/pull/123660
- https://github.com/rust-lang/rust/pull/125259
- https://github.com/rust-lang/rust/pull/128506
- https://github.com/rust-lang/rust/pull/127482
## Acknowledgements
Thanks to `@oli-obk` for reviewing the bulk of the work for this feature. Thanks to `@nikomatsakis` for his design blog posts which generated interest for this feature, `@traviscross` for feedback and additions to this stabilization report. All errors are my own.
r? `@ghost`
In this new version of Arbitrary Self Types, we no longer use the Deref trait
exclusively when working out which self types are valid. Instead, we follow a
chain of Receiver traits. This enables methods to be called on smart pointer
types which fundamentally cannot support Deref (for instance because they are
wrappers for pointers that don't follow Rust's aliasing rules).
This includes:
* Changes to tests appropriately
* New tests for:
* The basics of the feature
* Ensuring lifetime elision works properly
* Generic Receivers
* A copy of the method subst test enhanced with Receiver
This is really the heart of the 'arbitrary self types v2' feature, and
is the most critical commit in the current PR.
Subsequent commits are focused on:
* Detecting "shadowing" problems, where a smart pointer type can hide
methods in the pointee.
* Diagnostics and cleanup.
Naming: in this commit, the "Autoderef" type is modified so that it no
longer solely focuses on the "Deref" trait, but can now consider the
"Receiver" trait instead. Should it be renamed, to something like
"TraitFollower"? This was considered, but rejected, because
* even in the Receiver case, it still considers built-in derefs
* the name Autoderef is short and snappy.
Corrected a grammatical error in the explanation for E0751. Changed "exists" to "exist" to improve clarity and ensure proper grammar in the error message.
Approved in [ACP 491](https://github.com/rust-lang/libs-team/issues/491).
Remove the `unsafe` on `core::intrinsics::breakpoint()`, since it's a
safe intrinsic to call and has no prerequisites.
(Thanks to @zachs18 for figuring out the `bootstrap`/`not(bootstrap)`
logic.)
The RFC for arbitrary self types v2 declares that we should reject
"generic" self types. This commit does so.
The definition of "generic" was unclear in the RFC, but has been
explored in
https://github.com/rust-lang/rust/issues/129147
and the conclusion is that "generic" means any `self` type which
is a type parameter defined on the method itself, or references
to such a type.
This approach was chosen because other definitions of "generic"
don't work. Specifically,
* we can't filter out generic type _arguments_, because that would
filter out Rc<Self> and all the other types of smart pointer
we want to support;
* we can't filter out all type params, because Self itself is a
type param, and because existing Rust code depends on other
type params declared on the type (as opposed to the method).
This PR decides to make a new error code for this case, instead of
reusing the existing E0307 error. This makes the code a
bit more complex, but it seems we have an opportunity to provide
specific diagnostics for this case so we should do so.
This PR filters out generic self types whether or not the
'arbitrary self types' feature is enabled. However, it's believed
that it can't have any effect on code which uses stable Rust, since
there are no stable traits which can be used to indicate a valid
generic receiver type, and thus it would have been impossible to
write code which could trigger this new error case.
It is however possible that this could break existing code which
uses either of the unstable `arbitrary_self_types` or
`receiver_trait` features. This breakage is intentional; as
we move arbitrary self types towards stabilization we don't want
to continue to support generic such types.
This PR adds lots of extra tests to arbitrary-self-from-method-substs.
Most of these are ways to trigger a "type mismatch" error which
9b82580c73/compiler/rustc_hir_typeck/src/method/confirm.rs (L519)
hopes can be minimized by filtering out generics in this way.
We remove a FIXME from confirm.rs suggesting that we make this change.
It's still possible to cause type mismatch errors, and a subsequent
PR may be able to improve diagnostics in this area, but it's harder
to cause these errors without contrived uses of the turbofish.
This is a part of the arbitrary self types v2 project,
https://github.com/rust-lang/rfcs/pull/3519https://github.com/rust-lang/rust/issues/44874
r? @wesleywiser
`#[naked]`: report incompatible attributes
tracking issue: https://github.com/rust-lang/rust/issues/90957
this is a re-implementation of https://github.com/rust-lang/rust/pull/93809 by ``@bstrie`` which was closed 2 years ago due to inactivity.
This PR takes some of the final comments into account, specifically providing a little more context in error messages, and using an allow list to determine which attributes are compatible with `#[naked]`.
Notable attributes that are incompatible with `#[naked]` are:
* `#[inline]`
* `#[track_caller]`
* ~~`#[target_feature]`~~ (this is now allowed, see PR discussion)
* `#[test]`, `#[ignore]`, `#[should_panic]`
These attributes just directly conflict with what `#[naked]` should do.
Naked functions are still important for systems programming, embedded, and operating systems, so I'd like to move them forward.
- merge error codes
- use attribute name that is incompatible in error message
- add test for conditional incompatible attribute
- add `linkage` to the allowlist
Support lists and stylings in more places for `rustc --explain`
Adds support for `*foo*`, stylings not immediately following whitespace e.g. ``(`Foo`)`` and lists starting with whitespace:
```md
* previously supported
```
```md
* now also supported
```
These are fairly common in the existing error docs, some before/after examples:
### E0460
### E0059
Forbid borrows and unsized types from being used as the type of a const generic under `adt_const_params`
Fixes#112219Fixes#112124Fixes#112125
### Motivation
Currently the `adt_const_params` feature allows writing `Foo<const N: [u8]>` this is entirely useless as it is not possible to write an expression which evaluates to a type that is not `Sized`. In order to actually use unsized types in const generics they are typically written as `const N: &[u8]` which *is* possible to provide a value of.
Unfortunately allowing the types of const parameters to contain references is non trivial (#120961) as it introduces a number of difficult questions about how equality of references in the type system should behave. References in the types of const generics is largely only useful for using unsized types in const generics.
This PR introduces a new feature gate `unsized_const_parameters` and moves support for `const N: [u8]` and `const N: &...` from `adt_const_params` into it. The goal here hopefully is to experiment with allowing `const N: [u8]` to work without references and then eventually completely forbid references in const generics.
Splitting this out into a new feature gate means that stabilization of `adt_const_params` does not have to resolve#120961 which is the only remaining "big" blocker for the feature. Remaining issues after this are a few ICEs and naming bikeshed for `ConstParamTy`.
### Implementation
The implementation is slightly subtle here as we would like to ensure that a stabilization of `adt_const_params` is forwards compatible with any outcome of `unsized_const_parameters`. This is inherently tricky as we do not support unstable trait implementations and we determine whether a type is valid as the type of a const parameter via a trait bound.
There are a few constraints here:
- We would like to *allow for the possibility* of adding a `Sized` supertrait to `ConstParamTy` in the event that we wind up opting to not support unsized types and instead requiring people to write the 'sized version', e.g. `const N: [u8; M]` instead of `const N: [u8]`.
- Crates should be able to enable `unsized_const_parameters` and write trait implementations of `ConstParamTy` for `!Sized` types without downstream crates that only enable `adt_const_params` being able to observe this (required for std to be able to `impl<T> ConstParamTy for [T]`
Ultimately the way this is accomplished is via having two traits (sad), `ConstParamTy` and `UnsizedConstParamTy`. Depending on whether `unsized_const_parameters` is enabled or not we change which trait is used to check whether a type is allowed to be a const parameter.
Long term (when stabilizing `UnsizedConstParamTy`) it should be possible to completely merge these traits (and derive macros), only having a single `trait ConstParamTy` and `macro ConstParamTy`.
Under `adt_const_params` it is now illegal to directly refer to `ConstParamTy` it is only used as an internal impl detail by `derive(ConstParamTy)` and checking const parameters are well formed. This is necessary in order to ensure forwards compatibility with all possible future directions for `feature(unsized_const_parameters)`.
Generally the intuition here should be that `ConstParamTy` is the stable trait that everything uses, and `UnsizedConstParamTy` is that plus unstable implementations (well, I suppose `ConstParamTy` isn't stable yet :P).
`C-cmse-nonsecure-call`: improved error messages
tracking issue: #81391
issue for the error messages (partially implemented by this PR): #81347
related, in that it also deals with CMSE: https://github.com/rust-lang/rust/pull/127766
When using the `C-cmse-nonsecure-call` ABI, both the arguments and return value must be passed via registers. Previously, when violating this constraint, an ugly LLVM error would be shown. Now, the rust compiler itself will print a pretty message and link to more information.
when the `C-cmse-nonsecure-call` ABI is used, arguments and return values must be passed via registers. Failing to do so (i.e. spilling to the stack) causes an LLVM error down the line, but now rustc will properly emit an error a bit earlier in the chain
