Autodiff batching
Enzyme supports batching, which is especially known from the ML side when training neural networks.
There we would normally have a training loop, where in each iteration we would pass in some data (e.g. an image), and a target vector. Based on how close we are with our prediction we compute our loss, and then use backpropagation to compute the gradients and update our weights.
That's quite inefficient, so what you normally do is passing in a batch of 8/16/.. images and targets, and compute the gradients for those all at once, allowing better optimizations.
Enzyme supports batching in two ways, the first one (which I implemented here) just accepts a Batch size,
and then each Dual/Duplicated argument has not one, but N shadow arguments. So instead of
```rs
for i in 0..100 {
df(x[i], y[i], 1234);
}
```
You can now do
```rs
for i in 0..100.step_by(4) {
df(x[i+0],x[i+1],x[i+2],x[i+3], y[i+0], y[i+1], y[i+2], y[i+3], 1234);
}
```
which will give the same results, but allows better compiler optimizations. See the testcase for details.
There is a second variant, where we can mark certain arguments and instead of having to pass in N shadow arguments, Enzyme assumes that the argument is N times longer. I.e. instead of accepting 4 slices with 12 floats each, we would accept one slice with 48 floats. I'll implement this over the next days.
I will also add more tests for both modes.
For any one preferring some more interactive explanation, here's a video of Tim's llvm dev talk, where he presents his work. https://www.youtube.com/watch?v=edvaLAL5RqU
I'll also add some other docs to the dev guide and user docs in another PR.
r? ghost
Tracking:
- https://github.com/rust-lang/rust/issues/124509
- https://github.com/rust-lang/rust/issues/135283
We also have to remove the LLVM argument in cast-target-abi.rs for LLVM
21. I'm not really sure what the best approach here is since that test
already uses revisions. We could also fork the test into a copy for LLVM
19-20 and another for LLVM 21, but what I did for now was drop the
lint-abort-on-error flag to LLVM figuring that some coverage was better
than none, but I'm happy to change this if that was a bad direction.
The above also applies for ffi-out-of-bounds-loads.rs.
r? dianqk
@rustbot label llvm-main
The embedded bitcode should always be prepared for LTO/ThinLTO
Fixes#115344. Fixes#117220.
There are currently two methods for generating bitcode that used for LTO. One method involves using `-C linker-plugin-lto` to emit object files as bitcode, which is the typical setting used by cargo. The other method is through `-C embed-bitcode=yes`.
When using with `-C embed-bitcode=yes -C lto=no`, we run a complete non-LTO LLVM pipeline to obtain bitcode, then the bitcode is used for LTO. We run the Call Graph Profile Pass twice on the same module.
This PR is doing something similar to LLVM's `buildFatLTODefaultPipeline`, obtaining the bitcode for embedding after running `buildThinLTOPreLinkDefaultPipeline`.
r? nikic
Emit getelementptr inbounds nuw for pointer::add()
Lower pointer::add (via intrinsic::offset with unsigned offset) to getelementptr inbounds nuw on LLVM versions that support it. This lets LLVM make use of the pre-condition that the offset addition does not wrap in an unsigned sense. Together with inbounds, this also implies that the offset is non-negative.
Fixes https://github.com/rust-lang/rust/issues/137217.
The formatting of the command line arguments has been moved to the
frontend in:
e190d074a0
However, the Rust logic introduced in
ad0ecebf43
did not replicate the previous argument quoting behavior.
adding autodiff tests
I'd like to get started with upstreaming some tests, even though I'm still waiting for an answer on how to best integrate the enzyme pass. Can we therefore temporarily support the -Z llvm-plugins here without too much effort? And in that case, how would that work? I saw you can do remapping, e.g. `rust-src-base`, but I don't think that will give me the path to libEnzyme.so. Do you have another suggestion?
Other than that this test simply checks that the derivative of `x*x` is `2.0 * x`, which in this case is computed as
`%0 = fadd fast double %x.0.val, %x.0.val`
(I'll add a few more tests and move it to an autodiff folder if we can use the -Z flag)
r? ``@jieyouxu``
Locally at least `-Zllvm-plugins=${PWD}/build/x86_64-unknown-linux-gnu/enzyme/build/Enzyme/libEnzyme-19.so` seems to work if I copy the command I get from x.py test and run it manually. However, running x.py test itself fails.
Tracking:
- https://github.com/rust-lang/rust/issues/124509
Zulip discussion: https://rust-lang.zulipchat.com/#narrow/channel/326414-t-infra.2Fbootstrap/topic/Enzyme.20build.20changes
Replace our `LLVMRustDIBuilderRef` with LLVM-C's `LLVMDIBuilderRef`
Inspired by trying to split #134009 into smaller steps that are easier to review individually.
This makes it possible to start incrementally replacing our debuginfo bindings with the ones in the LLVM-C API, all of which operate on `LLVMDIBuilderRef`.
There should be no change to compiler behaviour.
This makes it possible to start incrementally replacing our debuginfo bindings
with the ones in the LLVM-C API, all of which operate on `LLVMDIBuilderRef`.
See llvm/llvm-project#121851
For LLVM 20+, this function (`renameModuleForThinLTO`) has no return
value. For prior versions of LLVM, this never failed, but had a
signature which allowed an error value people were handling.
[Debuginfo] Force enum `DISCR_*` to `static const u64` to allow for inspection via LLDB
see [here](486614878) for more info.
This change mainly helps `*-msvc` debugged with LLDB. Currently, LLDB cannot inspect `static` struct fields, so the intended visualization for enums is only borderline functional, and niche enums with ranges of discriminant cannot be determined at all .
LLDB *can* inspect `static const` values (though for whatever reason, non-enum/non-u64 consts don't work).
This change adds the `LLVMRustDIBuilderCreateQualifiedType` to the rust FFI layer to wrap the discr type with a `const` modifier, as well as forcing all generated integer enum `DISCR_*` values to be u64's. Those values will only ever be used by debugger visualizers anyway, so it shouldn't be a huge deal, but I left a fixme comment for it just in case.. The `tag` also still properly reflects the discriminant type, so no information is lost.
Pass end position of span through inline ASM cookie
Before this PR, only the start position of the span was passed though the inline ASM cookie to diagnostics. LLVM 19 has full support for 64-bit inline ASM cookies; this PR uses that to pass the end position of the span in the upper 32 bits, meaning inline ASM diagnostics now point at the entire line the error occurred on, not just the first character of it.
Allow disabling ASan instrumentation for globals
AddressSanitizer adds instrumentation to global variables unless the [`no_sanitize_address`](https://llvm.org/docs/LangRef.html#global-attributes) attribute is set on them.
This commit extends the existing `#[no_sanitize(address)]` attribute to set this; previously it only had the desired effect on functions.
(cc https://github.com/rust-lang/rust/issues/39699)
As described here UseOdrIndicator should be disabled on Windows
since link.exe does not support duplicate weak definitions
(https://reviews.llvm.org/D137227).
Co-Authored-By: Bastian Kersting <bkersting@google.com>
LLVM does not expect to ever see multiple dbg_declares for the same variable at the same
location with different values. proc-macros make it possible for arbitrary code,
including multiple calls that get inlined, to happen at any given location in the source
code. Add discriminators when that happens so these locations are different to LLVM.
This may interfere with the AddDiscriminators pass in LLVM, which is added by the
unstable flag -Zdebug-info-for-profiling.
Fixes#131944
Trim and tidy includes in `rustc_llvm`
These includes tend to accumulate over time, and are usually only removed when something breaks in a new LLVM version, so it's nice to clean them up manually once in a while.
General strategy used for this PR:
- Remove all includes from `LLVMWrapper.h` that aren't needed by the header itself, transplanting them to individual source files as necessary.
- For each source file, temporarily remove each include if doing so doesn't cause a compile error.
- If a “required” include looks like it shouldn't be needed, try replacing it with its sub-includes, then trim that list.
- After doing all of the above, go back and re-add any removed include if the file does actually use things defined in that header, even if the header happens to also be included by something else.
