Auto merge of #136593 - lukas-code:ty-value-perf, r=oli-obk

valtree performance tuning

Summary: This PR makes type checking of code with many type-level constants faster.

After https://github.com/rust-lang/rust/pull/136180 was merged, we observed a small perf regression (https://github.com/rust-lang/rust/pull/136318#issuecomment-2635562821). This happened because that PR introduced additional copies in the fast reject code path for consts, which is very hot for certain crates: 6c1d960d88/compiler/rustc_type_ir/src/fast_reject.rs (L486-L487)

This PR improves the performance again by properly interning the valtrees so that copying and comparing them becomes faster. This will become especially useful with `feature(adt_const_params)`, so the fast reject code doesn't have to do a deep compare of the valtrees.

Note that we can't just compare the interned consts themselves in the fast reject, because sometimes `'static` lifetimes in the type are be replaced with inference variables (due to canonicalization) on one side but not the other.

A less invasive alternative that I considered is simply avoiding copies introduced by https://github.com/rust-lang/rust/pull/136180 and comparing the valtrees it in-place (see commit: 9e91e50ac5 / perf results: https://github.com/rust-lang/rust/pull/136593#issuecomment-2642303245), however that was still measurably slower than interning.

There are some minor regressions in secondary benchmarks: These happen due to changes in memory allocations and seem acceptable to me. The crates that make heavy use of valtrees show no significant changes in memory usage.

compiler/rustc_const_eval/src/const_eval/valtrees.rs

@@ -2,7 +2,7 @@ use rustc_abi::{BackendRepr, VariantIdx};
 use rustc_data_structures::stack::ensure_sufficient_stack;
 use rustc_middle::mir::interpret::{EvalToValTreeResult, GlobalId, ReportedErrorInfo};
 use rustc_middle::ty::layout::{LayoutCx, LayoutOf, TyAndLayout};
-use rustc_middle::ty::{self, ScalarInt, Ty, TyCtxt};
+use rustc_middle::ty::{self, Ty, TyCtxt};
 use rustc_middle::{bug, mir};
 use rustc_span::DUMMY_SP;
 use tracing::{debug, instrument, trace};
@@ -21,7 +21,7 @@ use crate::interpret::{
 fn branches<'tcx>(
     ecx: &CompileTimeInterpCx<'tcx>,
     place: &MPlaceTy<'tcx>,
-    n: usize,
+    field_count: usize,
     variant: Option<VariantIdx>,
     num_nodes: &mut usize,
 ) -> ValTreeCreationResult<'tcx> {
@@ -29,30 +29,28 @@ fn branches<'tcx>(
         Some(variant) => ecx.project_downcast(place, variant).unwrap(),
         None => place.clone(),
     };
-    let variant = variant.map(|variant| Some(ty::ValTree::Leaf(ScalarInt::from(variant.as_u32()))));
+    debug!(?place);
-    debug!(?place, ?variant);
 
-    let mut fields = Vec::with_capacity(n);
+    let mut branches = Vec::with_capacity(field_count + variant.is_some() as usize);
-    for i in 0..n {
-        let field = ecx.project_field(&place, i).unwrap();
-        let valtree = const_to_valtree_inner(ecx, &field, num_nodes)?;
-        fields.push(Some(valtree));
-    }
 
     // For enums, we prepend their variant index before the variant's fields so we can figure out
     // the variant again when just seeing a valtree.
-    let branches = variant
+    if let Some(variant) = variant {
-        .into_iter()
+        branches.push(ty::ValTree::from_scalar_int(*ecx.tcx, variant.as_u32().into()));
-        .chain(fields.into_iter())
+    }
-        .collect::<Option<Vec<_>>>()
+
-        .expect("should have already checked for errors in ValTree creation");
+    for i in 0..field_count {
+        let field = ecx.project_field(&place, i).unwrap();
+        let valtree = const_to_valtree_inner(ecx, &field, num_nodes)?;
+        branches.push(valtree);
+    }
 
     // Have to account for ZSTs here
     if branches.len() == 0 {
         *num_nodes += 1;
     }
 
-    Ok(ty::ValTree::Branch(ecx.tcx.arena.alloc_from_iter(branches)))
+    Ok(ty::ValTree::from_branches(*ecx.tcx, branches))
 }
 
 #[instrument(skip(ecx), level = "debug")]
@@ -70,7 +68,7 @@ fn slice_branches<'tcx>(
         elems.push(valtree);
     }
 
-    Ok(ty::ValTree::Branch(ecx.tcx.arena.alloc_from_iter(elems)))
+    Ok(ty::ValTree::from_branches(*ecx.tcx, elems))
 }
 
 #[instrument(skip(ecx), level = "debug")]
@@ -79,6 +77,7 @@ fn const_to_valtree_inner<'tcx>(
     place: &MPlaceTy<'tcx>,
     num_nodes: &mut usize,
 ) -> ValTreeCreationResult<'tcx> {
+    let tcx = *ecx.tcx;
     let ty = place.layout.ty;
     debug!("ty kind: {:?}", ty.kind());
 
@@ -89,14 +88,14 @@ fn const_to_valtree_inner<'tcx>(
     match ty.kind() {
         ty::FnDef(..) => {
             *num_nodes += 1;
-            Ok(ty::ValTree::zst())
+            Ok(ty::ValTree::zst(tcx))
         }
         ty::Bool | ty::Int(_) | ty::Uint(_) | ty::Float(_) | ty::Char => {
             let val = ecx.read_immediate(place).unwrap();
             let val = val.to_scalar_int().unwrap();
             *num_nodes += 1;
 
-            Ok(ty::ValTree::Leaf(val))
+            Ok(ty::ValTree::from_scalar_int(tcx, val))
         }
 
         ty::Pat(base, ..) => {
@@ -127,7 +126,7 @@ fn const_to_valtree_inner<'tcx>(
                 return Err(ValTreeCreationError::NonSupportedType(ty));
             };
             // It's just a ScalarInt!
-            Ok(ty::ValTree::Leaf(val))
+            Ok(ty::ValTree::from_scalar_int(tcx, val))
         }
 
         // Technically we could allow function pointers (represented as `ty::Instance`), but this is not guaranteed to
@@ -287,16 +286,11 @@ pub fn valtree_to_const_value<'tcx>(
     // FIXME: Does this need an example?
     match *cv.ty.kind() {
         ty::FnDef(..) => {
-            assert!(cv.valtree.unwrap_branch().is_empty());
+            assert!(cv.valtree.is_zst());
             mir::ConstValue::ZeroSized
         }
         ty::Bool | ty::Int(_) | ty::Uint(_) | ty::Float(_) | ty::Char | ty::RawPtr(_, _) => {
-            match cv.valtree {
+            mir::ConstValue::Scalar(Scalar::Int(cv.valtree.unwrap_leaf()))
-                ty::ValTree::Leaf(scalar_int) => mir::ConstValue::Scalar(Scalar::Int(scalar_int)),
-                ty::ValTree::Branch(_) => bug!(
-                    "ValTrees for Bool, Int, Uint, Float, Char or RawPtr should have the form ValTree::Leaf"
-                ),
-            }
         }
         ty::Pat(ty, _) => {
             let cv = ty::Value { valtree: cv.valtree, ty };

compiler/rustc_hir_analysis/src/hir_ty_lowering/mod.rs

@@ -2161,7 +2161,7 @@ impl<'tcx> dyn HirTyLowerer<'tcx> + '_ {
                     did,
                     path.segments.last().unwrap(),
                 );
-                ty::Const::new_value(tcx, ty::ValTree::zst(), Ty::new_fn_def(tcx, did, args))
+                ty::Const::zero_sized(tcx, Ty::new_fn_def(tcx, did, args))
             }
 
             // Exhaustive match to be clear about what exactly we're considering to be

compiler/rustc_middle/src/arena.rs

@@ -90,6 +90,7 @@ macro_rules! arena_types {
             [] autodiff_item: rustc_ast::expand::autodiff_attrs::AutoDiffItem,
             [] ordered_name_set: rustc_data_structures::fx::FxIndexSet<rustc_span::Symbol>,
             [] pats: rustc_middle::ty::PatternKind<'tcx>,
+            [] valtree: rustc_middle::ty::ValTreeKind<'tcx>,
 
             // Note that this deliberately duplicates items in the `rustc_hir::arena`,
             // since we need to allocate this type on both the `rustc_hir` arena

compiler/rustc_middle/src/ty/codec.rs

@@ -146,6 +146,12 @@ impl<'tcx, E: TyEncoder<I = TyCtxt<'tcx>>> Encodable<E> for ty::Pattern<'tcx> {
     }
 }
 
+impl<'tcx, E: TyEncoder<I = TyCtxt<'tcx>>> Encodable<E> for ty::ValTree<'tcx> {
+    fn encode(&self, e: &mut E) {
+        self.0.0.encode(e);
+    }
+}
+
 impl<'tcx, E: TyEncoder<I = TyCtxt<'tcx>>> Encodable<E> for ConstAllocation<'tcx> {
     fn encode(&self, e: &mut E) {
         self.inner().encode(e)
@@ -355,12 +361,9 @@ impl<'tcx, D: TyDecoder<I = TyCtxt<'tcx>>> Decodable<D> for ty::Pattern<'tcx> {
     }
 }
 
-impl<'tcx, D: TyDecoder<I = TyCtxt<'tcx>>> RefDecodable<'tcx, D> for [ty::ValTree<'tcx>] {
+impl<'tcx, D: TyDecoder<I = TyCtxt<'tcx>>> Decodable<D> for ty::ValTree<'tcx> {
-    fn decode(decoder: &mut D) -> &'tcx Self {
+    fn decode(decoder: &mut D) -> Self {
-        decoder
+        decoder.interner().intern_valtree(Decodable::decode(decoder))
-            .interner()
-            .arena
-            .alloc_from_iter((0..decoder.read_usize()).map(|_| Decodable::decode(decoder)))
     }
 }
 

compiler/rustc_middle/src/ty/consts.rs

@@ -20,7 +20,7 @@ pub type ConstKind<'tcx> = ir::ConstKind<TyCtxt<'tcx>>;
 pub type UnevaluatedConst<'tcx> = ir::UnevaluatedConst<TyCtxt<'tcx>>;
 
 #[cfg(target_pointer_width = "64")]
-rustc_data_structures::static_assert_size!(ConstKind<'_>, 32);
+rustc_data_structures::static_assert_size!(ConstKind<'_>, 24);
 
 #[derive(Copy, Clone, PartialEq, Eq, Hash, HashStable)]
 #[rustc_pass_by_value]
@@ -190,7 +190,7 @@ impl<'tcx> Const<'tcx> {
             .size;
         ty::Const::new_value(
             tcx,
-            ty::ValTree::from_scalar_int(ScalarInt::try_from_uint(bits, size).unwrap()),
+            ty::ValTree::from_scalar_int(tcx, ScalarInt::try_from_uint(bits, size).unwrap()),
             ty,
         )
     }
@@ -198,7 +198,7 @@ impl<'tcx> Const<'tcx> {
     #[inline]
     /// Creates an interned zst constant.
     pub fn zero_sized(tcx: TyCtxt<'tcx>, ty: Ty<'tcx>) -> Self {
-        ty::Const::new_value(tcx, ty::ValTree::zst(), ty)
+        ty::Const::new_value(tcx, ty::ValTree::zst(tcx), ty)
     }
 
     #[inline]

compiler/rustc_middle/src/ty/consts/valtree.rs

@@ -1,4 +1,8 @@
-use rustc_macros::{HashStable, TyDecodable, TyEncodable, TypeFoldable, TypeVisitable};
+use std::fmt;
+use std::ops::Deref;
+
+use rustc_data_structures::intern::Interned;
+use rustc_macros::{HashStable, Lift, TyDecodable, TyEncodable, TypeFoldable, TypeVisitable};
 
 use super::ScalarInt;
 use crate::mir::interpret::Scalar;
@@ -16,9 +20,9 @@ use crate::ty::{self, Ty, TyCtxt};
 ///
 /// `ValTree` does not have this problem with representation, as it only contains integers or
 /// lists of (nested) `ValTree`.
-#[derive(Copy, Clone, Debug, Hash, Eq, PartialEq)]
+#[derive(Clone, Debug, Hash, Eq, PartialEq)]
 #[derive(HashStable, TyEncodable, TyDecodable)]
-pub enum ValTree<'tcx> {
+pub enum ValTreeKind<'tcx> {
     /// integers, `bool`, `char` are represented as scalars.
     /// See the `ScalarInt` documentation for how `ScalarInt` guarantees that equal values
     /// of these types have the same representation.
@@ -33,58 +37,98 @@ pub enum ValTree<'tcx> {
     /// the fields of the variant.
     ///
     /// ZST types are represented as an empty slice.
-    Branch(&'tcx [ValTree<'tcx>]),
+    Branch(Box<[ValTree<'tcx>]>),
 }
 
-impl<'tcx> ValTree<'tcx> {
+impl<'tcx> ValTreeKind<'tcx> {
-    pub fn zst() -> Self {
-        Self::Branch(&[])
-    }
-
     #[inline]
-    pub fn unwrap_leaf(self) -> ScalarInt {
+    pub fn unwrap_leaf(&self) -> ScalarInt {
         match self {
-            Self::Leaf(s) => s,
+            Self::Leaf(s) => *s,
             _ => bug!("expected leaf, got {:?}", self),
         }
     }
 
     #[inline]
-    pub fn unwrap_branch(self) -> &'tcx [Self] {
+    pub fn unwrap_branch(&self) -> &[ValTree<'tcx>] {
         match self {
-            Self::Branch(branch) => branch,
+            Self::Branch(branch) => &**branch,
             _ => bug!("expected branch, got {:?}", self),
         }
     }
 
-    pub fn from_raw_bytes<'a>(tcx: TyCtxt<'tcx>, bytes: &'a [u8]) -> Self {
+    pub fn try_to_scalar(&self) -> Option<Scalar> {
-        let branches = bytes.iter().map(|b| Self::Leaf(ScalarInt::from(*b)));
-        let interned = tcx.arena.alloc_from_iter(branches);
-
-        Self::Branch(interned)
-    }
-
-    pub fn from_scalar_int(i: ScalarInt) -> Self {
-        Self::Leaf(i)
-    }
-
-    pub fn try_to_scalar(self) -> Option<Scalar> {
         self.try_to_scalar_int().map(Scalar::Int)
     }
 
-    pub fn try_to_scalar_int(self) -> Option<ScalarInt> {
+    pub fn try_to_scalar_int(&self) -> Option<ScalarInt> {
         match self {
-            Self::Leaf(s) => Some(s),
+            Self::Leaf(s) => Some(*s),
             Self::Branch(_) => None,
         }
     }
+
+    pub fn try_to_branch(&self) -> Option<&[ValTree<'tcx>]> {
+        match self {
+            Self::Branch(branch) => Some(&**branch),
+            Self::Leaf(_) => None,
+        }
+    }
+}
+
+/// An interned valtree. Use this rather than `ValTreeKind`, whenever possible.
+///
+/// See the docs of [`ValTreeKind`] or the [dev guide] for an explanation of this type.
+///
+/// [dev guide]: https://rustc-dev-guide.rust-lang.org/mir/index.html#valtrees
+#[derive(Copy, Clone, Hash, Eq, PartialEq)]
+#[derive(HashStable)]
+pub struct ValTree<'tcx>(pub(crate) Interned<'tcx, ValTreeKind<'tcx>>);
+
+impl<'tcx> ValTree<'tcx> {
+    /// Returns the zero-sized valtree: `Branch([])`.
+    pub fn zst(tcx: TyCtxt<'tcx>) -> Self {
+        tcx.consts.valtree_zst
+    }
+
+    pub fn is_zst(self) -> bool {
+        matches!(*self, ValTreeKind::Branch(box []))
+    }
+
+    pub fn from_raw_bytes(tcx: TyCtxt<'tcx>, bytes: &[u8]) -> Self {
+        let branches = bytes.iter().map(|&b| Self::from_scalar_int(tcx, b.into()));
+        Self::from_branches(tcx, branches)
+    }
+
+    pub fn from_branches(tcx: TyCtxt<'tcx>, branches: impl IntoIterator<Item = Self>) -> Self {
+        tcx.intern_valtree(ValTreeKind::Branch(branches.into_iter().collect()))
+    }
+
+    pub fn from_scalar_int(tcx: TyCtxt<'tcx>, i: ScalarInt) -> Self {
+        tcx.intern_valtree(ValTreeKind::Leaf(i))
+    }
+}
+
+impl<'tcx> Deref for ValTree<'tcx> {
+    type Target = &'tcx ValTreeKind<'tcx>;
+
+    #[inline]
+    fn deref(&self) -> &&'tcx ValTreeKind<'tcx> {
+        &self.0.0
+    }
+}
+
+impl fmt::Debug for ValTree<'_> {
+    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+        (**self).fmt(f)
+    }
 }
 
 /// A type-level constant value.
 ///
 /// Represents a typed, fully evaluated constant.
 #[derive(Copy, Clone, Debug, Hash, Eq, PartialEq)]
-#[derive(HashStable, TyEncodable, TyDecodable, TypeFoldable, TypeVisitable)]
+#[derive(HashStable, TyEncodable, TyDecodable, TypeFoldable, TypeVisitable, Lift)]
 pub struct Value<'tcx> {
     pub ty: Ty<'tcx>,
     pub valtree: ValTree<'tcx>,

compiler/rustc_middle/src/ty/context.rs

@@ -80,7 +80,7 @@ use crate::ty::{
     GenericArgsRef, GenericParamDefKind, List, ListWithCachedTypeInfo, ParamConst, ParamTy,
     Pattern, PatternKind, PolyExistentialPredicate, PolyFnSig, Predicate, PredicateKind,
     PredicatePolarity, Region, RegionKind, ReprOptions, TraitObjectVisitor, Ty, TyKind, TyVid,
-    Visibility,
+    ValTree, ValTreeKind, Visibility,
 };
 
 #[allow(rustc::usage_of_ty_tykind)]
@@ -807,6 +807,7 @@ pub struct CtxtInterners<'tcx> {
     local_def_ids: InternedSet<'tcx, List<LocalDefId>>,
     captures: InternedSet<'tcx, List<&'tcx ty::CapturedPlace<'tcx>>>,
     offset_of: InternedSet<'tcx, List<(VariantIdx, FieldIdx)>>,
+    valtree: InternedSet<'tcx, ty::ValTreeKind<'tcx>>,
 }
 
 impl<'tcx> CtxtInterners<'tcx> {
@@ -836,6 +837,7 @@ impl<'tcx> CtxtInterners<'tcx> {
             local_def_ids: Default::default(),
             captures: Default::default(),
             offset_of: Default::default(),
+            valtree: Default::default(),
         }
     }
 
@@ -1027,6 +1029,8 @@ pub struct CommonConsts<'tcx> {
     pub unit: Const<'tcx>,
     pub true_: Const<'tcx>,
     pub false_: Const<'tcx>,
+    /// Use [`ty::ValTree::zst`] instead.
+    pub(crate) valtree_zst: ValTree<'tcx>,
 }
 
 impl<'tcx> CommonTypes<'tcx> {
@@ -1130,19 +1134,30 @@ impl<'tcx> CommonConsts<'tcx> {
             )
         };
 
+        let mk_valtree = |v| {
+            ty::ValTree(Interned::new_unchecked(
+                interners.valtree.intern(v, |v| InternedInSet(interners.arena.alloc(v))).0,
+            ))
+        };
+
+        let valtree_zst = mk_valtree(ty::ValTreeKind::Branch(Box::default()));
+        let valtree_true = mk_valtree(ty::ValTreeKind::Leaf(ty::ScalarInt::TRUE));
+        let valtree_false = mk_valtree(ty::ValTreeKind::Leaf(ty::ScalarInt::FALSE));
+
         CommonConsts {
             unit: mk_const(ty::ConstKind::Value(ty::Value {
                 ty: types.unit,
-                valtree: ty::ValTree::zst(),
+                valtree: valtree_zst,
             })),
             true_: mk_const(ty::ConstKind::Value(ty::Value {
                 ty: types.bool,
-                valtree: ty::ValTree::Leaf(ty::ScalarInt::TRUE),
+                valtree: valtree_true,
             })),
             false_: mk_const(ty::ConstKind::Value(ty::Value {
                 ty: types.bool,
-                valtree: ty::ValTree::Leaf(ty::ScalarInt::FALSE),
+                valtree: valtree_false,
             })),
+            valtree_zst,
         }
     }
 }
@@ -2260,6 +2275,7 @@ nop_lift! { const_allocation; ConstAllocation<'a> => ConstAllocation<'tcx> }
 nop_lift! { predicate; Predicate<'a> => Predicate<'tcx> }
 nop_lift! { predicate; Clause<'a> => Clause<'tcx> }
 nop_lift! { layout; Layout<'a> => Layout<'tcx> }
+nop_lift! { valtree; ValTree<'a> => ValTree<'tcx> }
 
 nop_list_lift! { type_lists; Ty<'a> => Ty<'tcx> }
 nop_list_lift! {
@@ -2270,26 +2286,6 @@ nop_list_lift! { bound_variable_kinds; ty::BoundVariableKind => ty::BoundVariabl
 // This is the impl for `&'a GenericArgs<'a>`.
 nop_list_lift! { args; GenericArg<'a> => GenericArg<'tcx> }
 
-macro_rules! nop_slice_lift {
-    ($ty:ty => $lifted:ty) => {
-        impl<'a, 'tcx> Lift<TyCtxt<'tcx>> for &'a [$ty] {
-            type Lifted = &'tcx [$lifted];
-            fn lift_to_interner(self, tcx: TyCtxt<'tcx>) -> Option<Self::Lifted> {
-                if self.is_empty() {
-                    return Some(&[]);
-                }
-                tcx.interners
-                    .arena
-                    .dropless
-                    .contains_slice(self)
-                    .then(|| unsafe { mem::transmute(self) })
-            }
-        }
-    };
-}
-
-nop_slice_lift! { ty::ValTree<'a> => ty::ValTree<'tcx> }
-
 macro_rules! sty_debug_print {
     ($fmt: expr, $ctxt: expr, $($variant: ident),*) => {{
         // Curious inner module to allow variant names to be used as
@@ -2534,6 +2530,7 @@ macro_rules! direct_interners {
 // crate only, and have a corresponding `mk_` function.
 direct_interners! {
     region: pub(crate) intern_region(RegionKind<'tcx>): Region -> Region<'tcx>,
+    valtree: pub(crate) intern_valtree(ValTreeKind<'tcx>): ValTree -> ValTree<'tcx>,
     pat: pub mk_pat(PatternKind<'tcx>): Pattern -> Pattern<'tcx>,
     const_allocation: pub mk_const_alloc(Allocation): ConstAllocation -> ConstAllocation<'tcx>,
     layout: pub mk_layout(LayoutData<FieldIdx, VariantIdx>): Layout -> Layout<'tcx>,

compiler/rustc_middle/src/ty/mod.rs

@@ -60,7 +60,8 @@ pub use self::closure::{
     place_to_string_for_capture,
 };
 pub use self::consts::{
-    Const, ConstInt, ConstKind, Expr, ExprKind, ScalarInt, UnevaluatedConst, ValTree, Value,
+    Const, ConstInt, ConstKind, Expr, ExprKind, ScalarInt, UnevaluatedConst, ValTree, ValTreeKind,
+    Value,
 };
 pub use self::context::{
     CtxtInterners, CurrentGcx, DeducedParamAttrs, Feed, FreeRegionInfo, GlobalCtxt, Lift, TyCtxt,

compiler/rustc_middle/src/ty/print/pretty.rs

@@ -1639,14 +1639,12 @@ pub trait PrettyPrinter<'tcx>: Printer<'tcx> + fmt::Write {
         match ty.kind() {
             // Byte strings (&[u8; N])
             ty::Ref(_, inner, _) => {
-                if let ty::Array(elem, len) = inner.kind()
+                if let ty::Array(elem, ct_len) = inner.kind()
                     && let ty::Uint(ty::UintTy::U8) = elem.kind()
-                    && let ty::ConstKind::Value(cv) = len.kind()
+                    && let Some(len) = ct_len.try_to_target_usize(self.tcx())
-                    && let ty::ValTree::Leaf(int) = cv.valtree
                 {
                     match self.tcx().try_get_global_alloc(prov.alloc_id()) {
                         Some(GlobalAlloc::Memory(alloc)) => {
-                            let len = int.to_bits(self.tcx().data_layout.pointer_size);
                             let range = AllocRange { start: offset, size: Size::from_bytes(len) };
                             if let Ok(byte_str) =
                                 alloc.inner().get_bytes_strip_provenance(&self.tcx(), range)
@@ -1800,8 +1798,8 @@ pub trait PrettyPrinter<'tcx>: Printer<'tcx> + fmt::Write {
         }
 
         let u8_type = self.tcx().types.u8;
-        match (cv.valtree, *cv.ty.kind()) {
+        match (*cv.valtree, *cv.ty.kind()) {
-            (ty::ValTree::Branch(_), ty::Ref(_, inner_ty, _)) => match inner_ty.kind() {
+            (ty::ValTreeKind::Branch(_), ty::Ref(_, inner_ty, _)) => match inner_ty.kind() {
                 ty::Slice(t) if *t == u8_type => {
                     let bytes = cv.try_to_raw_bytes(self.tcx()).unwrap_or_else(|| {
                         bug!(
@@ -1826,7 +1824,7 @@ pub trait PrettyPrinter<'tcx>: Printer<'tcx> + fmt::Write {
                     return Ok(());
                 }
             },
-            (ty::ValTree::Branch(_), ty::Array(t, _)) if t == u8_type => {
+            (ty::ValTreeKind::Branch(_), ty::Array(t, _)) if t == u8_type => {
                 let bytes = cv.try_to_raw_bytes(self.tcx()).unwrap_or_else(|| {
                     bug!("expected to convert valtree to raw bytes for type {:?}", t)
                 });
@@ -1835,7 +1833,7 @@ pub trait PrettyPrinter<'tcx>: Printer<'tcx> + fmt::Write {
                 return Ok(());
             }
             // Aggregates, printed as array/tuple/struct/variant construction syntax.
-            (ty::ValTree::Branch(_), ty::Array(..) | ty::Tuple(..) | ty::Adt(..)) => {
+            (ty::ValTreeKind::Branch(_), ty::Array(..) | ty::Tuple(..) | ty::Adt(..)) => {
                 let contents = self.tcx().destructure_const(ty::Const::new_value(
                     self.tcx(),
                     cv.valtree,
@@ -1891,12 +1889,12 @@ pub trait PrettyPrinter<'tcx>: Printer<'tcx> + fmt::Write {
                 }
                 return Ok(());
             }
-            (ty::ValTree::Leaf(leaf), ty::Ref(_, inner_ty, _)) => {
+            (ty::ValTreeKind::Leaf(leaf), ty::Ref(_, inner_ty, _)) => {
                 p!(write("&"));
-                return self.pretty_print_const_scalar_int(leaf, inner_ty, print_ty);
+                return self.pretty_print_const_scalar_int(*leaf, inner_ty, print_ty);
             }
-            (ty::ValTree::Leaf(leaf), _) => {
+            (ty::ValTreeKind::Leaf(leaf), _) => {
-                return self.pretty_print_const_scalar_int(leaf, cv.ty, print_ty);
+                return self.pretty_print_const_scalar_int(*leaf, cv.ty, print_ty);
             }
             (_, ty::FnDef(def_id, args)) => {
                 // Never allowed today, but we still encounter them in invalid const args.
@@ -1909,7 +1907,7 @@ pub trait PrettyPrinter<'tcx>: Printer<'tcx> + fmt::Write {
         }
 
         // fallback
-        if cv.valtree == ty::ValTree::zst() {
+        if cv.valtree.is_zst() {
             p!(write("<ZST>"));
         } else {
             p!(write("{:?}", cv.valtree));

compiler/rustc_middle/src/ty/structural_impls.rs

@@ -165,13 +165,9 @@ impl<'tcx> fmt::Debug for ty::consts::Expr<'tcx> {
 impl<'tcx> fmt::Debug for ty::Const<'tcx> {
     fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
         // If this is a value, we spend some effort to make it look nice.
-        if let ConstKind::Value(_) = self.kind() {
+        if let ConstKind::Value(cv) = self.kind() {
             return ty::tls::with(move |tcx| {
-                // ValTrees aren't interned, so we lift the entire constant.
+                let cv = tcx.lift(cv).unwrap();
-                let lifted = tcx.lift(*self).unwrap();
-                let ConstKind::Value(cv) = lifted.kind() else {
-                    bug!("we checked that this is a valtree")
-                };
                 let mut cx = FmtPrinter::new(tcx, Namespace::ValueNS);
                 cx.pretty_print_const_valtree(cv, /*print_ty*/ true)?;
                 f.write_str(&cx.into_buffer())

compiler/rustc_mir_build/src/thir/constant.rs

@@ -48,7 +48,7 @@ pub(crate) fn lit_to_const<'tcx>(
             ty::ValTree::from_raw_bytes(tcx, bytes)
         }
         (ast::LitKind::Byte(n), ty::Uint(ty::UintTy::U8)) => {
-            ty::ValTree::from_scalar_int((*n).into())
+            ty::ValTree::from_scalar_int(tcx, (*n).into())
         }
         (ast::LitKind::CStr(data, _), ty::Ref(_, inner_ty, _)) if matches!(inner_ty.kind(), ty::Adt(def, _) if tcx.is_lang_item(def.did(), LangItem::CStr)) =>
         {
@@ -57,23 +57,23 @@ pub(crate) fn lit_to_const<'tcx>(
         }
         (ast::LitKind::Int(n, _), ty::Uint(ui)) if !neg => {
             let scalar_int = trunc(n.get(), *ui);
-            ty::ValTree::from_scalar_int(scalar_int)
+            ty::ValTree::from_scalar_int(tcx, scalar_int)
         }
         (ast::LitKind::Int(n, _), ty::Int(i)) => {
             let scalar_int = trunc(
                 if neg { (n.get() as i128).overflowing_neg().0 as u128 } else { n.get() },
                 i.to_unsigned(),
             );
-            ty::ValTree::from_scalar_int(scalar_int)
+            ty::ValTree::from_scalar_int(tcx, scalar_int)
         }
-        (ast::LitKind::Bool(b), ty::Bool) => ty::ValTree::from_scalar_int((*b).into()),
+        (ast::LitKind::Bool(b), ty::Bool) => ty::ValTree::from_scalar_int(tcx, (*b).into()),
         (ast::LitKind::Float(n, _), ty::Float(fty)) => {
             let bits = parse_float_into_scalar(*n, *fty, neg).unwrap_or_else(|| {
                 tcx.dcx().bug(format!("couldn't parse float literal: {:?}", lit_input.lit))
             });
-            ty::ValTree::from_scalar_int(bits)
+            ty::ValTree::from_scalar_int(tcx, bits)
         }
-        (ast::LitKind::Char(c), ty::Char) => ty::ValTree::from_scalar_int((*c).into()),
+        (ast::LitKind::Char(c), ty::Char) => ty::ValTree::from_scalar_int(tcx, (*c).into()),
         (ast::LitKind::Err(guar), _) => return ty::Const::new_error(tcx, *guar),
         _ => return ty::Const::new_misc_error(tcx),
     };

compiler/rustc_smir/src/rustc_smir/context.rs

@@ -494,7 +494,7 @@ impl<'tcx> Context for TablesWrapper<'tcx> {
         let scalar = ScalarInt::try_from_uint(value, size).ok_or_else(|| {
             Error::new(format!("Value overflow: cannot convert `{value}` to `{ty}`."))
         })?;
-        Ok(ty::Const::new_value(tables.tcx, ValTree::from_scalar_int(scalar), ty)
+        Ok(ty::Const::new_value(tcx, ValTree::from_scalar_int(tcx, scalar), ty)
             .stable(&mut *tables))
     }
 

compiler/rustc_ty_utils/src/consts.rs

@@ -26,10 +26,7 @@ fn destructure_const<'tcx>(
         bug!("cannot destructure constant {:?}", const_)
     };
 
-    let branches = match cv.valtree {
+    let branches = cv.valtree.unwrap_branch();
-        ty::ValTree::Branch(b) => b,
-        _ => bug!("cannot destructure constant {:?}", const_),
-    };
 
     let (fields, variant) = match cv.ty.kind() {
         ty::Array(inner_ty, _) | ty::Slice(inner_ty) => {
@@ -126,13 +123,10 @@ fn recurse_build<'tcx>(
             tcx.at(sp).lit_to_const(LitToConstInput { lit: &lit.node, ty: node.ty, neg })
         }
         &ExprKind::NonHirLiteral { lit, user_ty: _ } => {
-            let val = ty::ValTree::from_scalar_int(lit);
+            let val = ty::ValTree::from_scalar_int(tcx, lit);
-            ty::Const::new_value(tcx, val, node.ty)
-        }
-        &ExprKind::ZstLiteral { user_ty: _ } => {
-            let val = ty::ValTree::zst();
             ty::Const::new_value(tcx, val, node.ty)
         }
+        &ExprKind::ZstLiteral { user_ty: _ } => ty::Const::zero_sized(tcx, node.ty),
         &ExprKind::NamedConst { def_id, args, user_ty: _ } => {
             let uneval = ty::UnevaluatedConst::new(def_id, args);
             ty::Const::new_unevaluated(tcx, uneval)

src/doc/rustc-dev-guide/src/mir/index.md

@@ -304,9 +304,9 @@ The most important rule for
 this representation is that every value must be uniquely represented. In other
 words: a specific value must only be representable in one specific way. For example: there is only
 one way to represent an array of two integers as a `ValTree`:
-`ValTree::Branch(&[ValTree::Leaf(first_int), ValTree::Leaf(second_int)])`.
+`Branch([Leaf(first_int), Leaf(second_int)])`.
 Even though theoretically a `[u32; 2]` could be encoded in a `u64` and thus just be a
-`ValTree::Leaf(bits_of_two_u32)`, that is not a legal construction of `ValTree`
+`Leaf(bits_of_two_u32)`, that is not a legal construction of `ValTree`
 (and is very complex to do, so it is unlikely anyone is tempted to do so).
 
 These rules also mean that some values are not representable. There can be no `union`s in type

src/tools/clippy/clippy_lints/src/non_copy_const.rs

@@ -179,8 +179,8 @@ impl<'tcx> NonCopyConst<'tcx> {
     }
 
     fn is_value_unfrozen_raw_inner(cx: &LateContext<'tcx>, val: ty::ValTree<'tcx>, ty: Ty<'tcx>) -> bool {
-        // No branch that we check (yet) should continue if val isn't a ValTree::Branch
+        // No branch that we check (yet) should continue if val isn't a branch
-        let ty::ValTree::Branch(val) = val else { return false };
+        let Some(val) = val.try_to_branch() else { return false };
         match *ty.kind() {
             // the fact that we have to dig into every structs to search enums
             // leads us to the point checking `UnsafeCell` directly is the only option.
@@ -192,9 +192,10 @@ impl<'tcx> NonCopyConst<'tcx> {
                 .iter()
                 .any(|field| Self::is_value_unfrozen_raw_inner(cx, *field, ty)),
             ty::Adt(def, args) if def.is_enum() => {
-                let Some((&ty::ValTree::Leaf(variant_index), fields)) = val.split_first() else {
+                let Some((&variant_valtree, fields)) = val.split_first() else {
                     return false;
                 };
+                let variant_index = variant_valtree.unwrap_leaf();
                 let variant_index = VariantIdx::from_u32(variant_index.to_u32());
                 fields
                     .iter()