bjoernager/rust
bjoernager/rust
1
Fork 0
Code Activity

Rollup merge of #124663 - Kobzol:docker-local-download, r=Mark-Simulacrum

Browse source 
Enable reusing CI Docker cache when running CI images locally

When running a CI image locally, e.g. using `DEPLOY=1 src/ci/docker/run.sh dist-x86_64-linux`, it can take a long time until the Docker image is built, which is annoying.

Since we now use proper Docker caching on CI, it should be possible to just `docker pull` the prebuilt image to reuse the cache. We didn't want to do this on CI, since our caching key isn't perfect and it's possible that we can miss some changes, but I think that for local usage it is fine (we could introduce some env. var. to force disable the image download, if needed).

r? `@Mark-Simulacrum`
This commit is contained in:
Matthias Krüger 2024-05-04 22:27:31 +02:00 • committed by GitHub
commit f4d0776b4f
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
1 changed files with 34 additions and 30 deletions
Download patch file Download diff file Expand all files Collapse all files

64
src/ci/docker/run.sh
View file

@ -50,39 +50,35 @@ fi
CACHE_DOMAIN="${CACHE_DOMAIN:-ci-caches.rust-lang.org}"
if [ -f "$docker_dir/$image/Dockerfile" ]; then
if isCI; then
hash_key=/tmp/.docker-hash-key.txt
rm -f "${hash_key}"
echo $image >> $hash_key
hash_key=/tmp/.docker-hash-key.txt
rm -f "${hash_key}"
echo $image >> $hash_key
cat "$docker_dir/$image/Dockerfile" >> $hash_key
# Look for all source files involves in the COPY command
copied_files=/tmp/.docker-copied-files.txt
rm -f "$copied_files"
for i in $(sed -n -e '/^COPY --from=/! s/^COPY \(.*\) .*$/\1/p' \
"$docker_dir/$image/Dockerfile"); do
# List the file names
find "$script_dir/$i" -type f >> $copied_files
done
# Sort the file names and cat the content into the hash key
sort $copied_files | xargs cat >> $hash_key
cat "$docker_dir/$image/Dockerfile" >> $hash_key
# Look for all source files involves in the COPY command
copied_files=/tmp/.docker-copied-files.txt
rm -f "$copied_files"
for i in $(sed -n -e '/^COPY --from=/! s/^COPY \(.*\) .*$/\1/p' \
"$docker_dir/$image/Dockerfile"); do
# List the file names
find "$script_dir/$i" -type f >> $copied_files
done
# Sort the file names and cat the content into the hash key
sort $copied_files | xargs cat >> $hash_key
# Include the architecture in the hash key, since our Linux CI does not
# only run in x86_64 machines.
uname -m >> $hash_key
# Include the architecture in the hash key, since our Linux CI does not
# only run in x86_64 machines.
uname -m >> $hash_key
docker --version >> $hash_key
# Include cache version. Can be used to manually bust the Docker cache.
echo "2" >> $hash_key
# Include cache version. Can be used to manually bust the Docker cache.
echo "2" >> $hash_key
echo "Image input"
cat $hash_key
echo "Image input"
cat $hash_key
cksum=$(sha512sum $hash_key | \
awk '{print $1}')
echo "Image input checksum ${cksum}"
fi
cksum=$(sha512sum $hash_key | \
awk '{print $1}')
echo "Image input checksum ${cksum}"
dockerfile="$docker_dir/$image/Dockerfile"
if [ -x /usr/bin/cygpath ]; then
@ -105,10 +101,18 @@ if [ -f "$docker_dir/$image/Dockerfile" ]; then
# It seems that it cannot be the same as $IMAGE_TAG, otherwise it overwrites the cache
CACHE_IMAGE_TAG=${REGISTRY}/${REGISTRY_USERNAME}/rust-ci-cache:${cksum}
# On non-CI jobs, we don't do any caching.
# On non-CI jobs, we try to download a pre-built image from the rust-lang-ci
# ghcr.io registry. If it is not possible, we fall back to building the image
# locally.
if ! isCI;
then
retry docker build --rm -t rust-ci -f "$dockerfile" "$context"
if docker pull "${IMAGE_TAG}"; then
echo "Downloaded Docker image from CI"
docker tag "${IMAGE_TAG}" rust-ci
else
echo "Building local Docker image"
retry docker build --rm -t rust-ci -f "$dockerfile" "$context"
fi
# On PR CI jobs, we don't have permissions to write to the registry cache,
# but we can still read from it.
elif [[ "$PR_CI_JOB" == "1" ]];