coverage bug fixes and optimization support
Adjusted LLVM codegen for code compiled with `-Zinstrument-coverage` to address multiple, somewhat related issues. Fixed a significant flaw in prior coverage solution: Every counter generated a new counter variable, but there should have only been one counter variable per function. This appears to have bloated .profraw files significantly. (For a small program, it increased the size by about 40%. I have not tested large programs, but there is anecdotal evidence that profraw files were way too large. This is a good fix, regardless, but hopefully it also addresses related issues. Fixes: #82144 Invalid LLVM coverage data produced when compiled with -C opt-level=1 Existing tests now work up to at least `opt-level=3`. This required a detailed analysis of the LLVM IR, comparisons with Clang C++ LLVM IR when compiled with coverage, and a lot of trial and error with codegen adjustments. The biggest hurdle was figuring out how to continue to support coverage results for unused functions and generics. Rust's coverage results have three advantages over Clang's coverage results: 1. Rust's coverage map does not include any overlapping code regions, making coverage counting unambiguous. 2. Rust generates coverage results (showing zero counts) for all unused functions, including generics. (Clang does not generate coverage for uninstantiated template functions.) 3. Rust's unused functions produce minimal stubbed functions in LLVM IR, sufficient for including in the coverage results; while Clang must generate the complete LLVM IR for each unused function, even though it will never be called. This PR removes the previous hack of attempting to inject coverage into some other existing function instance, and generates dedicated instances for each unused function. This change, and a few other adjustments (similar to what is required for `-C link-dead-code`, but with lower impact), makes it possible to support LLVM optimizations. Fixes: #79651 Coverage report: "Unexecuted instantiation:..." for a generic function from multiple crates Fixed by removing the aforementioned hack. Some "Unexecuted instantiation" notices are unavoidable, as explained in the `used_crate.rs` test, but `-Zinstrument-coverage` has new options to back off support for either unused generics, or all unused functions, which avoids the notice, at the cost of less coverage of unused functions. Fixes: #82875 Invalid LLVM coverage data produced with crate brotli_decompressor Fixed by disabling the LLVM function attribute that forces inlining, if `-Z instrument-coverage` is enabled. This attribute is applied to Rust functions with `#[inline(always)], and in some cases, the forced inlining breaks coverage instrumentation and reports.
This commit is contained in:
Rich Kadel
parent
cebc8fef5f
commit
bcf755562a
62 changed files with 3061 additions and 342 deletions
|
|
@ -1,5 +1,6 @@
|
|||
use crate::llvm;
|
||||
|
||||
use crate::abi::{Abi, FnAbi};
|
||||
use crate::builder::Builder;
|
||||
use crate::common::CodegenCx;
|
||||
|
||||
|
|
@ -7,33 +8,47 @@ use libc::c_uint;
|
|||
use llvm::coverageinfo::CounterMappingRegion;
|
||||
use rustc_codegen_ssa::coverageinfo::map::{CounterExpression, FunctionCoverage};
|
||||
use rustc_codegen_ssa::traits::{
|
||||
BaseTypeMethods, CoverageInfoBuilderMethods, CoverageInfoMethods, MiscMethods, StaticMethods,
|
||||
BaseTypeMethods, BuilderMethods, ConstMethods, CoverageInfoBuilderMethods, CoverageInfoMethods,
|
||||
MiscMethods, StaticMethods,
|
||||
};
|
||||
use rustc_data_structures::fx::FxHashMap;
|
||||
use rustc_hir as hir;
|
||||
use rustc_hir::def_id::DefId;
|
||||
use rustc_llvm::RustString;
|
||||
use rustc_middle::bug;
|
||||
use rustc_middle::mir::coverage::{
|
||||
CodeRegion, CounterValueReference, ExpressionOperandId, InjectedExpressionId, Op,
|
||||
};
|
||||
use rustc_middle::ty;
|
||||
use rustc_middle::ty::layout::FnAbiExt;
|
||||
use rustc_middle::ty::subst::InternalSubsts;
|
||||
use rustc_middle::ty::Instance;
|
||||
|
||||
use std::cell::RefCell;
|
||||
use std::ffi::CString;
|
||||
|
||||
use std::iter;
|
||||
use tracing::debug;
|
||||
|
||||
pub mod mapgen;
|
||||
|
||||
const UNUSED_FUNCTION_COUNTER_ID: CounterValueReference = CounterValueReference::START;
|
||||
|
||||
const VAR_ALIGN_BYTES: usize = 8;
|
||||
|
||||
/// A context object for maintaining all state needed by the coverageinfo module.
|
||||
pub struct CrateCoverageContext<'tcx> {
|
||||
pub struct CrateCoverageContext<'ll, 'tcx> {
|
||||
// Coverage data for each instrumented function identified by DefId.
|
||||
pub(crate) function_coverage_map: RefCell<FxHashMap<Instance<'tcx>, FunctionCoverage<'tcx>>>,
|
||||
pub(crate) pgo_func_name_var_map: RefCell<FxHashMap<Instance<'tcx>, &'ll llvm::Value>>,
|
||||
}
|
||||
|
||||
impl<'tcx> CrateCoverageContext<'tcx> {
|
||||
impl<'ll, 'tcx> CrateCoverageContext<'ll, 'tcx> {
|
||||
pub fn new() -> Self {
|
||||
Self { function_coverage_map: Default::default() }
|
||||
Self {
|
||||
function_coverage_map: Default::default(),
|
||||
pgo_func_name_var_map: Default::default(),
|
||||
}
|
||||
}
|
||||
|
||||
pub fn take_function_coverage_map(&self) -> FxHashMap<Instance<'tcx>, FunctionCoverage<'tcx>> {
|
||||
|
|
@ -41,23 +56,44 @@ impl<'tcx> CrateCoverageContext<'tcx> {
|
|||
}
|
||||
}
|
||||
|
||||
impl CoverageInfoMethods for CodegenCx<'ll, 'tcx> {
|
||||
impl CoverageInfoMethods<'tcx> for CodegenCx<'ll, 'tcx> {
|
||||
fn coverageinfo_finalize(&self) {
|
||||
mapgen::finalize(self)
|
||||
}
|
||||
|
||||
fn get_pgo_func_name_var(&self, instance: Instance<'tcx>) -> &'ll llvm::Value {
|
||||
if let Some(coverage_context) = self.coverage_context() {
|
||||
debug!("getting pgo_func_name_var for instance={:?}", instance);
|
||||
let mut pgo_func_name_var_map = coverage_context.pgo_func_name_var_map.borrow_mut();
|
||||
pgo_func_name_var_map
|
||||
.entry(instance)
|
||||
.or_insert_with(|| self.create_pgo_func_name_var(instance))
|
||||
} else {
|
||||
bug!("Could not get the `coverage_context`");
|
||||
}
|
||||
}
|
||||
|
||||
/// Calls llvm::createPGOFuncNameVar() with the given function instance's
|
||||
/// mangled function name. The LLVM API returns an llvm::GlobalVariable
|
||||
/// containing the function name, with the specific variable name and
|
||||
/// linkage required by LLVM InstrProf source-based coverage
|
||||
/// instrumentation. Use `bx.get_pgo_func_name_var()` to ensure the variable
|
||||
/// is only created once per `Instance`.
|
||||
fn create_pgo_func_name_var(&self, instance: Instance<'tcx>) -> &'ll llvm::Value {
|
||||
let mangled_fn_name = CString::new(self.tcx.symbol_name(instance).name)
|
||||
.expect("error converting function name to C string");
|
||||
let llfn = self.get_fn(instance);
|
||||
unsafe { llvm::LLVMRustCoverageCreatePGOFuncNameVar(llfn, mangled_fn_name.as_ptr()) }
|
||||
}
|
||||
|
||||
fn define_unused_fn(&self, def_id: DefId) {
|
||||
let instance = declare_unused_fn(self, &def_id);
|
||||
codegen_unused_fn_and_counter(self, instance);
|
||||
add_function_coverage(self, instance, def_id);
|
||||
}
|
||||
}
|
||||
|
||||
impl CoverageInfoBuilderMethods<'tcx> for Builder<'a, 'll, 'tcx> {
|
||||
/// Calls llvm::createPGOFuncNameVar() with the given function instance's mangled function name.
|
||||
/// The LLVM API returns an llvm::GlobalVariable containing the function name, with the specific
|
||||
/// variable name and linkage required by LLVM InstrProf source-based coverage instrumentation.
|
||||
fn create_pgo_func_name_var(&self, instance: Instance<'tcx>) -> Self::Value {
|
||||
let llfn = self.cx.get_fn(instance);
|
||||
let mangled_fn_name = CString::new(self.tcx.symbol_name(instance).name)
|
||||
.expect("error converting function name to C string");
|
||||
unsafe { llvm::LLVMRustCoverageCreatePGOFuncNameVar(llfn, mangled_fn_name.as_ptr()) }
|
||||
}
|
||||
|
||||
fn set_function_source_hash(
|
||||
&mut self,
|
||||
instance: Instance<'tcx>,
|
||||
|
|
@ -145,6 +181,86 @@ impl CoverageInfoBuilderMethods<'tcx> for Builder<'a, 'll, 'tcx> {
|
|||
}
|
||||
}
|
||||
|
||||
fn declare_unused_fn(cx: &CodegenCx<'ll, 'tcx>, def_id: &DefId) -> Instance<'tcx> {
|
||||
let tcx = cx.tcx;
|
||||
|
||||
let instance = Instance::new(
|
||||
*def_id,
|
||||
InternalSubsts::for_item(tcx, *def_id, |param, _| {
|
||||
if let ty::GenericParamDefKind::Lifetime = param.kind {
|
||||
tcx.lifetimes.re_erased.into()
|
||||
} else {
|
||||
tcx.mk_param_from_def(param)
|
||||
}
|
||||
}),
|
||||
);
|
||||
|
||||
let llfn = cx.declare_fn(
|
||||
&tcx.symbol_name(instance).name,
|
||||
&FnAbi::of_fn_ptr(
|
||||
cx,
|
||||
ty::Binder::dummy(tcx.mk_fn_sig(
|
||||
iter::once(tcx.mk_unit()),
|
||||
tcx.mk_unit(),
|
||||
false,
|
||||
hir::Unsafety::Unsafe,
|
||||
Abi::Rust,
|
||||
)),
|
||||
&[],
|
||||
),
|
||||
);
|
||||
|
||||
unsafe {
|
||||
llvm::LLVMRustSetLinkage(llfn, llvm::Linkage::ExternalLinkage);
|
||||
llvm::LLVMRustSetVisibility(llfn, llvm::Visibility::Hidden);
|
||||
}
|
||||
|
||||
cx.instances.borrow_mut().insert(instance, llfn);
|
||||
|
||||
instance
|
||||
}
|
||||
|
||||
fn codegen_unused_fn_and_counter(cx: &CodegenCx<'ll, 'tcx>, instance: Instance<'tcx>) {
|
||||
let llfn = cx.get_fn(instance);
|
||||
let mut bx = Builder::new_block(cx, llfn, "unused_function");
|
||||
let fn_name = bx.get_pgo_func_name_var(instance);
|
||||
let hash = bx.const_u64(0);
|
||||
let num_counters = bx.const_u32(1);
|
||||
let index = bx.const_u32(u32::from(UNUSED_FUNCTION_COUNTER_ID));
|
||||
debug!(
|
||||
"codegen intrinsic instrprof.increment(fn_name={:?}, hash={:?}, num_counters={:?},
|
||||
index={:?}) for unused function: {:?}",
|
||||
fn_name, hash, num_counters, index, instance
|
||||
);
|
||||
bx.instrprof_increment(fn_name, hash, num_counters, index);
|
||||
bx.ret_void();
|
||||
}
|
||||
|
||||
fn add_function_coverage(cx: &CodegenCx<'ll, 'tcx>, instance: Instance<'tcx>, def_id: DefId) {
|
||||
let tcx = cx.tcx;
|
||||
|
||||
let mut function_coverage = FunctionCoverage::unused(tcx, instance);
|
||||
for (index, &code_region) in tcx.covered_code_regions(def_id).iter().enumerate() {
|
||||
if index == 0 {
|
||||
// Insert at least one real counter so the LLVM CoverageMappingReader will find expected
|
||||
// definitions.
|
||||
function_coverage.add_counter(UNUSED_FUNCTION_COUNTER_ID, code_region.clone());
|
||||
}
|
||||
// Add a Zero Counter for every code region.
|
||||
//
|
||||
// Even though the first coverage region already has an actual Counter, `llvm-cov` will not
|
||||
// always report it. Re-adding an unreachable region (zero counter) for the same region
|
||||
// seems to help produce the expected coverage.
|
||||
function_coverage.add_unreachable_region(code_region.clone());
|
||||
}
|
||||
|
||||
if let Some(coverage_context) = cx.coverage_context() {
|
||||
coverage_context.function_coverage_map.borrow_mut().insert(instance, function_coverage);
|
||||
} else {
|
||||
bug!("Could not get the `coverage_context`");
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) fn write_filenames_section_to_buffer<'a>(
|
||||
filenames: impl IntoIterator<Item = &'a CString>,
|
||||
buffer: &RustString,
|
||||
|
|
@ -177,6 +293,7 @@ pub(crate) fn write_mapping_to_buffer(
|
|||
);
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) fn hash_str(strval: &str) -> u64 {
|
||||
let strval = CString::new(strval).expect("null error converting hashable str to C string");
|
||||
unsafe { llvm::LLVMRustCoverageHashCString(strval.as_ptr()) }
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue