Do not ignore uninhabited types for function-call ABI purposes. (Remove BackendRepr::Uninhabited)
Accepted MCP: https://github.com/rust-lang/compiler-team/issues/832Fixes#135802
Do not consider the inhabitedness of a type for function call ABI purposes.
* Remove the [`rustc_abi::BackendRepr::Uninhabited`](https://doc.rust-lang.org/nightly/nightly-rustc/rustc_abi/enum.BackendRepr.html) variant
* Instead calculate the `BackendRepr` of uninhabited types "normally" (as though they were not uninhabited "at the top level", but still considering inhabitedness of variants to determine enum layout, etc)
* Add an `uninhabited: bool` field to [`rustc_abi::LayoutData`](https://doc.rust-lang.org/nightly/nightly-rustc/rustc_abi/struct.LayoutData.html) so inhabitedness of a `LayoutData` can still be queried when necessary (e.g. when determining if an enum variant needs a tag value allocated to it).
This should not affect type layouts (size/align/field offset); this should only affect function call ABI, and only of uninhabited types.
cc ``@RalfJung``
Pass through of target features to llvm-bitcode-linker and handling them
When using the llvm-bitcode-linker (`linker-flavor=llbc`) target-features are not passed through and are not handled by it.
The llvm-bitcode-linker is mainly used as a self contained linker to link llvm bitcode for the nvptx64 target. It uses `llvm-link`, `opt` and `llc` internally. To produce a `.ptx` file of a specific ptx-version it is necessary to pass the version to llc with the `--mattr` option. Without explicitly setting it, the emitted `.ptx`-version is the minimum supported version of the `--target-cpu`.
I would like to be able to explicitly set the ptx version as [some llvm problems only occur in earlier `.ptx`-versions](https://github.com/llvm/llvm-project/issues/112998).
Therefore this pull request adds support for passing target features to llvm-bitcode-linker and handling them.
I was not quite sure if adding these features to `rustc_target/src/target_features.rs` is necessary or not. If so I will gladly add these.
r? ``@kjetilkjeka``
Create a generic AVR target: avr-none
This commit removes the `avr-unknown-gnu-atmega328` target and replaces it with a more generic `avr-none` variant that must be specialized using `-C target-cpu` (e.g. `-C target-cpu=atmega328p`).
Seizing the day, I'm adding myself as the maintainer of this target - I've been already fixing the bugs anyway, might as well make it official 🙂
Related discussions:
- https://github.com/rust-lang/rust/pull/131171
- https://github.com/rust-lang/compiler-team/issues/800
try-job: x86_64-gnu-debug
Fix codegen of uninhabited PassMode::Indirect return types.
Add codegen test for uninhabited PassMode::Indirect return types.
Enable optimizations for uninhabited return type codegen test
- For shifts this shrinks the IR by no longer needing an `assume` while still providing the UB information
- Having this on the `i8`→`i1` truncations will hopefully help with some places that have to load `i8`s or pass those in LLVM structs without range information
This commit removes the `avr-unknown-gnu-atmega328` target and replaces
it with a more generic `avr-none` variant that must be specialized with
the `-C target-cpu` flag (e.g. `-C target-cpu=atmega328p`).
Remove `rustc_middle::mir::tcx` module.
This is a really weird module. For example, what does `tcx` in `rustc_middle::mir::tcx::PlaceTy` mean? The answer is "not much".
The top-level module comment says:
> Methods for the various MIR types. These are intended for use after
> building is complete.
Awfully broad for a module that has a handful of impl blocks for some MIR types, none of which really relates to `TyCtxt`. `git blame` indicates the comment is ancient, from 2015, and made sense then.
This module is now vestigial. This commit removes it and moves all the code within into `rustc_middle::mir::statement`. Some specifics:
- `Place`, `PlaceRef`, `Rvalue`, `Operand`, `BorrowKind`: they all have `impl` blocks in both the `tcx` and `statement` modules. The commit merges the former into the latter.
- `BinOp`, `UnOp`: they only have `impl` blocks in `tcx`. The commit moves these into `statement`.
- `PlaceTy`, `RvalueInitializationState`: they are defined in `tcx`. This commit moves them into `statement` *and* makes them available in `mir::*`, like many other MIR types.
r? `@tmandry`
This is a really weird module. For example, what does `tcx` in
`rustc_middle::mir::tcx::PlaceTy` mean? The answer is "not much".
The top-level module comment says:
> Methods for the various MIR types. These are intended for use after
> building is complete.
Awfully broad for a module that has a handful of impl blocks for some
MIR types, none of which really relates to `TyCtxt`. `git blame`
indicates the comment is ancient, from 2015, and made sense then.
This module is now vestigial. This commit removes it and moves all the
code within into `rustc_middle::mir::statement`. Some specifics:
- `Place`, `PlaceRef`, `Rvalue`, `Operand`, `BorrowKind`: they all have `impl`
blocks in both the `tcx` and `statement` modules. The commit merges
the former into the latter.
- `BinOp`, `UnOp`: they only have `impl` blocks in `tcx`. The commit
moves these into `statement`.
- `PlaceTy`, `RvalueInitializationState`: they are defined in `tcx`.
This commit moves them into `statement` *and* makes them available in
`mir::*`, like many other MIR types.
improve cold_path()
#120370 added a new instrinsic `cold_path()` and used it to fix `likely` and `unlikely`
However, in order to limit scope, the information about cold code paths is only used in 2-target switch instructions. This is sufficient for `likely` and `unlikely`, but limits usefulness of `cold_path` for idiomatic rust. For example, code like this:
```
if let Some(x) = y { ... }
```
may generate 3-target switch:
```
switch y.discriminator:
0 => true branch
1 = > false branch
_ => unreachable
```
and therefore marking a branch as cold will have no effect.
This PR improves `cold_path()` to work with arbitrary switch instructions.
Note that for 2-target switches, we can use `llvm.expect`, but for multiple targets we need to manually emit branch weights. I checked Clang and it also emits weights in this situation. The Clang's weight calculation is more complex that this PR, which I believe is mainly because `switch` in `C/C++` can have multiple cases going to the same target.
Rollup of 7 pull requests
Successful merges:
- #137095 (Replace some u64 hashes with Hash64)
- #137100 (HIR analysis: Remove unnecessary abstraction over list of clauses)
- #137105 (Restrict DerefPure for Cow<T> impl to T = impl Clone, [impl Clone], str.)
- #137120 (Enable `relative-path-include-bytes-132203` rustdoc-ui test on Windows)
- #137125 (Re-add missing empty lines in the releases notes)
- #137145 (use add-core-stubs / minicore for a few more tests)
- #137149 (Remove SSE ABI from i586-pc-windows-msvc)
r? `@ghost`
`@rustbot` modify labels: rollup
Replace some u64 hashes with Hash64
I introduced the Hash64 and Hash128 types in https://github.com/rust-lang/rust/pull/110083, essentially as a mechanism to prevent hashes from landing in our leb128 encoding paths. If you just have a u64 or u128 field in a struct then derive Encodable/Decodable, that number gets leb128 encoding. So if you need to store a hash or some other value which behaves very close to a hash, don't store it as a u64.
This reverts part of https://github.com/rust-lang/rust/pull/117603, which turned an encoded Hash64 into a u64.
Based on https://github.com/rust-lang/rust/pull/110083, I don't expect this to be perf-sensitive on its own, though I expect that it may help stabilize some of the small rmeta size fluctuations we currently see in perf reports.
The end goal is to eliminate `Map` altogether.
I added a `hir_` prefix to all of them, that seemed simplest. The
exceptions are `module_items` which became `hir_module_free_items` because
there was already a `hir_module_items`, and `items` which became
`hir_free_items` for consistency with `hir_module_free_items`.
The .ptx version produced by llc can be specified by passing it with --mattr. Currently it is not possible to specify the .ptx version with -Ctarget-feature because these are not passed through to llvm-bitcode-linker and handled by it. This commit adds both.
--target-feature and -mattr are passed with equals to mitigate issues when the value starts with a - (minus).
Bitcode linkers like llvm-bitcode-linker or bpf linker hand over the target features to llvm during link stage. During link stage the `TyCtxt` is already gone so it is not possible to create a query for the global backend features any longer. The features preserved in `Session.target_features` only incorporate target features known to rustc. This would contradict with the behaviour during codegen stage which also passes target features to llvm which are unknown to rustc.
This commit adds target features as a field to the `CrateInfo` struct and queries the target features in its new function. This way the target features are preserved beyond tcx and available at link stage.
To make sure the `global_backend_features` query is always registered even if the CodegenBackend does not register it, this registration is added to the `provide`function of the `rustc_codegen_ssa` crate.
Export kernel descriptor for amdgpu kernels
The host runtime (HIP or HSA) expects a kernel descriptor object for each kernel in the ELF file. The amdgpu LLVM backend generates the object. It is created as a symbol with the name of the kernel plus a `.kd` suffix.
Add it to the exported symbols in the linker script, so that it can be found.
For reference, the symbol is created here in LLVM: d5457e4c16/llvm/lib/Target/AMDGPU/MCTargetDesc/AMDGPUTargetStreamer.cpp (L966)
I wrote [a test](6a9115b121) for this as well, I’ll add that once the target is merged and working.
With this, all PRs to get working code for amdgpu are open (this + the target + the two patches adding addrspacecasts for alloca and global variables).
Tracking issue: #135024
r? `@workingjubilee`
Set both `nuw` and `nsw` in slice size calculation
There's an old note in the code to do this, and now that [LLVM-C has an API for it](f0b8ff1251/llvm/include/llvm-c/Core.h (L4403-L4408)), we might as well. And it's been there since what looks like LLVM 17 de9b6aa341 so doesn't even need to be conditional.
(There's other places, like `RawVecInner` or `Layout`, that might want to do things like this too, but I'll leave those for a future PR.)
`transmute` should also assume non-null pointers
Previously it only did integer-ABI things, but this way it does data pointers too. That gives more information in general to the backend, and allows slightly simplifying one of the helpers in slice iterators.
Previously it only did integer-ABI things, but this way it does data pointers too. That gives more information in general to the backend, and allows slightly simplifying one of the helpers in slice iterators.
Stabilize target_feature_11
# Stabilization report
This is an updated version of https://github.com/rust-lang/rust/pull/116114, which is itself a redo of https://github.com/rust-lang/rust/pull/99767. Most of this commit and report were copied from those PRs. Thanks ```@LeSeulArtichaut``` and ```@calebzulawski!```
## Summary
Allows for safe functions to be marked with `#[target_feature]` attributes.
Functions marked with `#[target_feature]` are generally considered as unsafe functions: they are unsafe to call, cannot *generally* be assigned to safe function pointers, and don't implement the `Fn*` traits.
However, calling them from other `#[target_feature]` functions with a superset of features is safe.
```rust
// Demonstration function
#[target_feature(enable = "avx2")]
fn avx2() {}
fn foo() {
// Calling `avx2` here is unsafe, as we must ensure
// that AVX is available first.
unsafe {
avx2();
}
}
#[target_feature(enable = "avx2")]
fn bar() {
// Calling `avx2` here is safe.
avx2();
}
```
Moreover, once https://github.com/rust-lang/rust/pull/135504 is merged, they can be converted to safe function pointers in a context in which calling them is safe:
```rust
// Demonstration function
#[target_feature(enable = "avx2")]
fn avx2() {}
fn foo() -> fn() {
// Converting `avx2` to fn() is a compilation error here.
avx2
}
#[target_feature(enable = "avx2")]
fn bar() -> fn() {
// `avx2` coerces to fn() here
avx2
}
```
See the section "Closures" below for justification of this behaviour.
## Test cases
Tests for this feature can be found in [`tests/ui/target_feature/`](f6cb952dc1/tests/ui/target-feature).
## Edge cases
### Closures
* [target-feature 1.1: should closures inherit target-feature annotations? #73631](https://github.com/rust-lang/rust/issues/73631)
Closures defined inside functions marked with #[target_feature] inherit the target features of their parent function. They can still be assigned to safe function pointers and implement the appropriate `Fn*` traits.
```rust
#[target_feature(enable = "avx2")]
fn qux() {
let my_closure = || avx2(); // this call to `avx2` is safe
let f: fn() = my_closure;
}
```
This means that in order to call a function with #[target_feature], you must guarantee that the target-feature is available while the function, any closures defined inside it, as well as any safe function pointers obtained from target-feature functions inside it, execute.
This is usually ensured because target features are assumed to never disappear, and:
- on any unsafe call to a `#[target_feature]` function, presence of the target feature is guaranteed by the programmer through the safety requirements of the unsafe call.
- on any safe call, this is guaranteed recursively by the caller.
If you work in an environment where target features can be disabled, it is your responsibility to ensure that no code inside a target feature function (including inside a closure) runs after this (until the feature is enabled again).
**Note:** this has an effect on existing code, as nowadays closures do not inherit features from the enclosing function, and thus this strengthens a safety requirement. It was originally proposed in #73631 to solve this by adding a new type of UB: “taking a target feature away from your process after having run code that uses that target feature is UB” .
This was motivated by userspace code already assuming in a few places that CPU features never disappear from a program during execution (see i.e. 2e29bdf908/crates/std_detect/src/detect/arch/x86.rs); however, concerns were raised in the context of the Linux kernel; thus, we propose to relax that requirement to "causing the set of usable features to be reduced is unsafe; when doing so, the programmer is required to ensure that no closures or safe fn pointers that use removed features are still in scope".
* [Fix #[inline(always)] on closures with target feature 1.1 #111836](https://github.com/rust-lang/rust/pull/111836)
Closures accept `#[inline(always)]`, even within functions marked with `#[target_feature]`. Since these attributes conflict, `#[inline(always)]` wins out to maintain compatibility.
### ABI concerns
* [The extern "C" ABI of SIMD vector types depends on target features #116558](https://github.com/rust-lang/rust/issues/116558)
The ABI of some types can change when compiling a function with different target features. This could have introduced unsoundness with target_feature_11, but recent fixes (#133102, #132173) either make those situations invalid or make the ABI no longer dependent on features. Thus, those issues should no longer occur.
### Special functions
The `#[target_feature]` attribute is forbidden from a variety of special functions, such as main, current and future lang items (e.g. `#[start]`, `#[panic_handler]`), safe default trait implementations and safe trait methods.
This was not disallowed at the time of the first stabilization PR for target_features_11, and resulted in the following issues/PRs:
* [`#[target_feature]` is allowed on `main` #108645](https://github.com/rust-lang/rust/issues/108645)
* [`#[target_feature]` is allowed on default implementations #108646](https://github.com/rust-lang/rust/issues/108646)
* [#[target_feature] is allowed on #[panic_handler] with target_feature 1.1 #109411](https://github.com/rust-lang/rust/issues/109411)
* [Prevent using `#[target_feature]` on lang item functions #115910](https://github.com/rust-lang/rust/pull/115910)
## Documentation
* Reference: [Document the `target_feature_11` feature reference#1181](https://github.com/rust-lang/reference/pull/1181)
---
cc tracking issue https://github.com/rust-lang/rust/issues/69098
cc ```@workingjubilee```
cc ```@RalfJung```
r? ```@rust-lang/lang```
Rename rustc_middle::Ty::is_unsafe_ptr to is_raw_ptr
The wording unsafe pointer is less common and not mentioned in a lot of places, instead this is usually called a "raw pointer". For the sake of uniformity, we rename this method.
This came up during the review of
https://github.com/rust-lang/rust/pull/134424.
r? `@Noratrieb`
The host runtime (HIP or HSA) expects a kernel descriptor object for
each kernel in the ELF file. The amdgpu LLVM backend generates the
object. It is created as a symbol with the name of the kernel plus a
`.kd` suffix.
Add it to the exported symbols in the linker script, so that it can be
found.
compiler: gate `extern "{abi}"` in ast_lowering
I don't believe low-level crates like `rustc_abi` should have to know or care about higher-level concerns like whether the ABI string is stable for users. These implementation details can be made less open to public inspection. This way the code that governs stability is near the code that enforces stability, and compiled together.
It also abstracts away certain error messages instead of constantly repeating them.
A few error messages are simply deleted outright, instead of made uniform, because they are either too dated to be useful or redundant with other diagnostic improvements we could make. These can be pursued in followups: my first concern was making sure there wasn't unnecessary diagnostics-related code in `rustc_abi`, which is not well-positioned to understand what kind of errors are going to be generated based on how it is used.
r? ``@ghost``
adding autodiff tests
I'd like to get started with upstreaming some tests, even though I'm still waiting for an answer on how to best integrate the enzyme pass. Can we therefore temporarily support the -Z llvm-plugins here without too much effort? And in that case, how would that work? I saw you can do remapping, e.g. `rust-src-base`, but I don't think that will give me the path to libEnzyme.so. Do you have another suggestion?
Other than that this test simply checks that the derivative of `x*x` is `2.0 * x`, which in this case is computed as
`%0 = fadd fast double %x.0.val, %x.0.val`
(I'll add a few more tests and move it to an autodiff folder if we can use the -Z flag)
r? ``@jieyouxu``
Locally at least `-Zllvm-plugins=${PWD}/build/x86_64-unknown-linux-gnu/enzyme/build/Enzyme/libEnzyme-19.so` seems to work if I copy the command I get from x.py test and run it manually. However, running x.py test itself fails.
Tracking:
- https://github.com/rust-lang/rust/issues/124509
Zulip discussion: https://rust-lang.zulipchat.com/#narrow/channel/326414-t-infra.2Fbootstrap/topic/Enzyme.20build.20changes
The wording unsafe pointer is less common and not mentioned in a lot of
places, instead this is usually called a "raw pointer". For the sake of
uniformity, we rename this method.
This came up during the review of
https://github.com/rust-lang/rust/pull/134424.
Update bootstrap compiler and rustfmt
The rustfmt version we previously used formats things differently from what the latest nightly rustfmt does. This causes issues for subtrees that get formatted both in-tree and in their own repo. Updating the rustfmt used in-tree solves those issues. Also bumped the bootstrap compiler as the stage0 update command always updates both at the same
time.
compiler: mostly-finish `rustc_abi` updates
This almost-finishes all the updates in the compiler to use `rustc_abi` and removes some of the reexports of `rustc_abi` items in `rustc_target` that were previously available.
r? ```@compiler-errors```
Generate correct terminate block under Wasm EH
This fixes failing LLVM assertions during insnsel.
Improves #135665.
r? bjorn3
^ you reviewed the PR bringing Wasm EH in, I assume this is within your area of expertise?
