bjoernager/rust
bjoernager/rust
1
Fork 0
Code Activity

Rollup merge of #117458 - kjetilkjeka:embedded-linker, r=petrochenkov

Browse source 
LLVM Bitcode Linker: A self contained linker for nvptx and other targets

This PR introduces a new linker named `llvm-bitcode-linker`. It is a `self-contained` linker that can be used to link programs in `llbc` before optimizing and compiling to native code. It will first be used internally in the Rust compiler to enable tests for the `nvptx64-nvidia-cuda` target as the original `rust-ptx-linker` is deprecated. It will then be provided to users of the `nvptx64-nvidia-cuda` target with the purpose of linking ptx. More targets than nvptx will also be supported eventually.

The PR introduces a new unstable `LinkerFlavor` for the compiler. The compiler will also not be shipped with rustc but most likely instead be shipped in it's own unstable component (a follow up PR will be opened for this). This means that merging this PR should not add any stability guarantees.

When more details of `self-contained` is implemented it will only be possible to use the linker when `-Clink-self-contained=+linker` is passed.

<details>
  <summary>Original Description</summary>

**When this PR was created it was focused a bit differently. The original text is preserved here in case there's some interests in it**

I have experimenting with approaches to replace the ptx-linker and enable the nvptx target tests again. I think it's time to get some feedback on the approach.

### The problem
The only useful linker for the nvptx target is [this crate](https://github.com/denzp/rust-ptx-linker). Since this linker performs linking on llvm bitcode it needs to track the llvm version of rustc and use the same format. It has not been maintained for 3+ years and must be considered abandoned. Over the years rust have upgraded LLVM while the linker has been left to bitrot. It is no longer in a usable state.

Due to the difficulty of keeping the ptx-linker up to date outside of tree the nvptx tests was [disabled a long time ago](f8f9a2869c). It was [previously discussed](https://github.com/rust-lang/rust/pull/96842#issuecomment-1146470177) if adding the ptx-linker to the rust repo would be a possibility. My efforts in doing this stopped at getting an answered if the license would prohibit it from inclusion in the [Rust repo](https://github.com/rust-lang/rust/pull/96842#issuecomment-1148397554). I therefore concluded that a re-write would be necessary.

### The possible solution presented here
The llvm tools know perfectly well how to link and optimize llvm bitcode. Each of them only perform a single task, and are therefore a bit cumbersome to call with the current linker approach rustc takes.

This PR adds a simple tool (current name `embedded-linker`) which can link self contained (often embedded) programs in llvm bitcode before compiling to the target format. Optimization will also be performed if lto is enabled. The rust compiler will make a single invocation to this tool, while the tool will orchestrate the many calls to the llvm tools.

### The questions
 - Is having control over the nvptx linking and therefore also tests worth it to add such tool? or should the tool live outside the rust repo?
 - Is the approach of calling llvm tools acceptable? Or would we want to keep the ptx-linker approach of using the llvm library? The tools seems to provide more simplicity and stability, but more intermediate files are being written. Perhaps there also are some performance penalty for the calling tools approach.
 - What is the process for adding such tool? MCP?
 - Does adding `llvm-link` to the llvm-tool component require any process?
 - Does it require some sort of FCP to remove ptx-linker as the default linker for ptx? Or is it sufficient that using the upstream ptx-linker is broken in its current state. it is possible to use a somewhat patched version of ptx-linker.
</details>
This commit is contained in:
Jubilee 2024-03-11 09:29:32 -07:00 • committed by GitHub
commit e1ceadcdfe
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
32 changed files with 529 additions and 25 deletions
Download patch file Download diff file Expand all files Collapse all files

22
compiler/rustc_target/src/spec/mod.rs
View file

@ -123,6 +123,8 @@ pub enum LinkerFlavor {
Bpf,
/// Linker tool for Nvidia PTX.
Ptx,
/// LLVM bitcode linker that can be used as a `self-contained` linker
Llbc,
}
/// Linker flavors available externally through command line (`-Clinker-flavor`)
@ -141,6 +143,7 @@ pub enum LinkerFlavorCli {
EmCc,
Bpf,
Ptx,
Llbc,
// Legacy stable values
Gcc,
@ -160,6 +163,7 @@ impl LinkerFlavorCli {
| LinkerFlavorCli::Msvc(Lld::Yes)
| LinkerFlavorCli::EmCc
| LinkerFlavorCli::Bpf
| LinkerFlavorCli::Llbc
| LinkerFlavorCli::Ptx => true,
LinkerFlavorCli::Gcc
| LinkerFlavorCli::Ld
@ -219,6 +223,7 @@ impl LinkerFlavor {
LinkerFlavorCli::Msvc(lld) => LinkerFlavor::Msvc(lld),
LinkerFlavorCli::EmCc => LinkerFlavor::EmCc,
LinkerFlavorCli::Bpf => LinkerFlavor::Bpf,
LinkerFlavorCli::Llbc => LinkerFlavor::Llbc,
LinkerFlavorCli::Ptx => LinkerFlavor::Ptx,
// Below: legacy stable values
@ -258,6 +263,7 @@ impl LinkerFlavor {
LinkerFlavor::Msvc(..) => LinkerFlavorCli::Msvc(Lld::No),
LinkerFlavor::EmCc => LinkerFlavorCli::Em,
LinkerFlavor::Bpf => LinkerFlavorCli::Bpf,
LinkerFlavor::Llbc => LinkerFlavorCli::Llbc,
LinkerFlavor::Ptx => LinkerFlavorCli::Ptx,
}
}
@ -272,6 +278,7 @@ impl LinkerFlavor {
LinkerFlavor::Msvc(lld) => LinkerFlavorCli::Msvc(lld),
LinkerFlavor::EmCc => LinkerFlavorCli::EmCc,
LinkerFlavor::Bpf => LinkerFlavorCli::Bpf,
LinkerFlavor::Llbc => LinkerFlavorCli::Llbc,
LinkerFlavor::Ptx => LinkerFlavorCli::Ptx,
}
}
@ -286,6 +293,7 @@ impl LinkerFlavor {
LinkerFlavorCli::Msvc(lld) => (Some(Cc::No), Some(lld)),
LinkerFlavorCli::EmCc => (Some(Cc::Yes), Some(Lld::Yes)),
LinkerFlavorCli::Bpf | LinkerFlavorCli::Ptx => (None, None),
LinkerFlavorCli::Llbc => (None, None),
// Below: legacy stable values
LinkerFlavorCli::Gcc => (Some(Cc::Yes), None),
@ -340,7 +348,7 @@ impl LinkerFlavor {
LinkerFlavor::WasmLld(cc) => LinkerFlavor::WasmLld(cc_hint.unwrap_or(cc)),
LinkerFlavor::Unix(cc) => LinkerFlavor::Unix(cc_hint.unwrap_or(cc)),
LinkerFlavor::Msvc(lld) => LinkerFlavor::Msvc(lld_hint.unwrap_or(lld)),
LinkerFlavor::EmCc | LinkerFlavor::Bpf | LinkerFlavor::Ptx => self,
LinkerFlavor::EmCc | LinkerFlavor::Bpf | LinkerFlavor::Llbc | LinkerFlavor::Ptx => self,
}
}
@ -355,8 +363,8 @@ impl LinkerFlavor {
pub fn check_compatibility(self, cli: LinkerFlavorCli) -> Option<String> {
let compatible = |cli| {
// The CLI flavor should be compatible with the target if:
// 1. they are counterparts: they have the same principal flavor.
match (self, cli) {
// 1. they are counterparts: they have the same principal flavor.
(LinkerFlavor::Gnu(..), LinkerFlavorCli::Gnu(..))
| (LinkerFlavor::Darwin(..), LinkerFlavorCli::Darwin(..))
| (LinkerFlavor::WasmLld(..), LinkerFlavorCli::WasmLld(..))
@ -364,11 +372,14 @@ impl LinkerFlavor {
| (LinkerFlavor::Msvc(..), LinkerFlavorCli::Msvc(..))
| (LinkerFlavor::EmCc, LinkerFlavorCli::EmCc)
| (LinkerFlavor::Bpf, LinkerFlavorCli::Bpf)
| (LinkerFlavor::Llbc, LinkerFlavorCli::Llbc)
| (LinkerFlavor::Ptx, LinkerFlavorCli::Ptx) => return true,
// 2. The linker flavor is independent of target and compatible
(LinkerFlavor::Ptx, LinkerFlavorCli::Llbc) => return true,
_ => {}
}
// 2. or, the flavor is legacy and survives this roundtrip.
// 3. or, the flavor is legacy and survives this roundtrip.
cli == self.with_cli_hints(cli).to_cli()
};
(!compatible(cli)).then(|| {
@ -387,6 +398,7 @@ impl LinkerFlavor {
| LinkerFlavor::Unix(..)
| LinkerFlavor::EmCc
| LinkerFlavor::Bpf
| LinkerFlavor::Llbc
| LinkerFlavor::Ptx => LldFlavor::Ld,
LinkerFlavor::Darwin(..) => LldFlavor::Ld64,
LinkerFlavor::WasmLld(..) => LldFlavor::Wasm,
@ -412,6 +424,7 @@ impl LinkerFlavor {
| LinkerFlavor::Msvc(_)
| LinkerFlavor::Unix(_)
| LinkerFlavor::Bpf
| LinkerFlavor::Llbc
| LinkerFlavor::Ptx => false,
}
}
@ -431,6 +444,7 @@ impl LinkerFlavor {
| LinkerFlavor::Msvc(_)
| LinkerFlavor::Unix(_)
| LinkerFlavor::Bpf
| LinkerFlavor::Llbc
| LinkerFlavor::Ptx => false,
}
}
@ -480,6 +494,7 @@ linker_flavor_cli_impls! {
(LinkerFlavorCli::Msvc(Lld::No)) "msvc"
(LinkerFlavorCli::EmCc) "em-cc"
(LinkerFlavorCli::Bpf) "bpf"
(LinkerFlavorCli::Llbc) "llbc"
(LinkerFlavorCli::Ptx) "ptx"
// Legacy stable flavors
@ -2228,6 +2243,7 @@ fn add_link_args_iter(
| LinkerFlavor::Unix(..)
| LinkerFlavor::EmCc
| LinkerFlavor::Bpf
| LinkerFlavor::Llbc
| LinkerFlavor::Ptx => {}
}
}

4
compiler/rustc_target/src/spec/targets/nvptx64_nvidia_cuda.rs
View file

@ -1,3 +1,4 @@
use crate::spec::LinkSelfContainedDefault;
use crate::spec::{LinkerFlavor, MergeFunctions, PanicStrategy, Target, TargetOptions};
pub fn target() -> Target {
@ -52,6 +53,9 @@ pub fn target() -> Target {
// The LLVM backend does not support stack canaries for this target
supports_stack_protector: false,
// Support using `self-contained` linkers like the llvm-bitcode-linker
link_self_contained: LinkSelfContainedDefault::True,
..Default::default()
},
}

5
compiler/rustc_target/src/spec/tests/tests_impl.rs
View file

@ -56,7 +56,10 @@ impl Target {
LinkerFlavor::Msvc(..) => {
assert_matches!(flavor, LinkerFlavor::Msvc(..))
}
LinkerFlavor::EmCc | LinkerFlavor::Bpf | LinkerFlavor::Ptx => {
LinkerFlavor::EmCc
| LinkerFlavor::Bpf
| LinkerFlavor::Ptx
| LinkerFlavor::Llbc => {
assert_eq!(flavor, self.linker_flavor)
}
}