1
Fork 0
Commit graph

295 commits

Author SHA1 Message Date
Joshua Nelson
345eb14f6c Remove vendoring support when building from git sources
This is difficult to support without submodule handling in bootstrap.py, because cargo will refuse
to vendor sources unless it knows the Cargo.toml files of all tools in tree. Moving vendor support
to rustbuild means that rustbuild will be built without vendoring.

Rather than trying to solve this, just remove support altogether and require
people to use `rustc-src` if they want vendoring (or run `cargo vendor` manually).
2022-06-21 23:42:56 -05:00
Joshua Nelson
9cde0f7877 Fully remove submodule handling from bootstrap.py
These submodules were previously updated in python because Cargo gives a hard error if toml files
are missing from the workspace:

```
error: failed to load manifest for workspace member `/home/jnelson/rust-lang/rust/src/tools/rls`

Caused by:
  failed to read `/home/jnelson/rust-lang/rust/src/tools/rls/Cargo.toml`

Caused by:
  No such file or directory (os error 2)
failed to run: /home/jnelson/rust-lang/rust/build/x86_64-unknown-linux-gnu/stage0/bin/cargo build --manifest-path /home/jnelson/rust-lang/rust/src/bootstrap/Cargo.toml
```

However, bootstrap doesn't actually need to be part of the workspace.
Remove it so we can move submodule handling fully to Rust, avoiding duplicate code between Rust and Python.

Note that this does break `cargo run`; it has to be `cd src/bootstrap && cargo run` now.
Given that we're planning to make the main entrypoint a shell script (or rust binary),
I think this is a good tradeoff for reduced complexity in bootstrap.py.
2022-06-21 22:55:43 -05:00
Pietro Albini
c1a0f49e9f
keep the same config values in stage0 between invocations
This commit allows users to change the contents of the "config" key in
src/stage0.json without having it overridden the next time the
bump-stage0 tool is executed.
2022-06-09 17:54:11 +02:00
Joshua Nelson
81f511cc2b Move beta rustfmt downloads to rustbuild 2022-06-07 10:16:55 -05:00
bors
6dadfc06fe Auto merge of #93717 - pietroalbini:pa-ci-profiler, r=Mark-Simulacrum
Add build metrics to rustbuild

This PR adds a new module of rustbuild, `ci_profiler`, whose job is to gather as much information as possible about the CI build as possible and store it in a JSON file uploaded to `ci-artifacts`. Right now for each step it collects:

* Type name and debug representation of the `Step` object.
* Duration of the step (excluding child steps).
* Systemwide CPU stats for the duration of the step (both single core and all cores).
* Which child steps were executed.

This is capable of replacing both the scripts to collect CPU stats and the `[TIMING]` lines in build logs (not yet removed, until we port our tooling to use the CI profiler). The format is also extensible to be able in the future to collect more information.

r? `@Mark-Simulacrum`
2022-06-05 01:35:03 +00:00
binggh
c0f18f9412 Re-add help_on_error for download-ci-llvm
Remove dead code

Missing }

./x.py fmt

Remove duplicate check

Recursively remove all usage of help_on_error
2022-05-29 23:16:10 +08:00
Joshua Nelson
00bb4df1fb Remove download-rustc handling from bootstrap.py 2022-05-25 17:32:53 -05:00
Joshua Nelson
53bf24c82b Move download-rustc from bootstrap.py to rustbuild
- Remove download-rustc handling from bootstrap.py
- Allow a custom `pattern` in `builder.unpack()`
- Only download rustc once another part of bootstrap depends on it.

  This is somewhat necessary since the download functions rely on having a full
  `Builder`, which isn't available until after config parsing finishes.
2022-05-25 17:32:31 -05:00
Yuki Okushi
9845a41233
Rollup merge of #97290 - jyn514:fast-submodules, r=Mark-Simulacrum
Turn on `fast_submodules` unconditionally

I don't know why anyone would turn this off; doing so makes builds much slower (nearly a 60x slowdown according to #49057).
Remove the option to do so, which makes bootstrap a little easier to maintain.

Bootstrap continues to allow you to manage submodules manually by setting `submodules = false`.
2022-05-24 12:18:32 +09:00
Joshua Nelson
5df276eef5 Turn on fast_submodules unconditionally
I don't know why anyone would turn this off; doing so makes builds much slower (nearly a 60x slowdown according to #49057).
Remove the option to do so, which makes bootstrap a little easier to maintain.

Bootstrap continues to allow you to manage submodules manually by setting `submodules = false`.
2022-05-23 21:17:09 -05:00
Jack Huey
b4c17d43a6
Rollup merge of #97277 - jyn514:no-unstable-for-bootstrap, r=Mark-Simulacrum
Avoid accidentally enabling unstable features in compilers (take 2)

This allows rustbuild to control whether crates can use nightly features or not.
It also prevents rustbuild from using nightly features itself.

This is #92261, but I fixed the CI error.
2022-05-22 11:37:42 -04:00
Joshua Nelson
b0ea4e74cb Avoid accidentally enabling unstable features in compilers (take 2)
This allows rustbuild to control whether crates can use nightly features or not.
It also prevents rustbuild from using nightly features itself.
2022-05-22 08:31:50 -05:00
bors
0be8768323 Auto merge of #96602 - TApplencourt:patch-1, r=Mark-Simulacrum
boostrap.py use curl by default

Fixes #61611
2022-05-15 04:52:16 +00:00
bors
30046ce1fe Auto merge of #96457 - yungkneez:fix-bootstrap, r=Mark-Simulacrum
Initialize rust-analyzer submodule on bootstrap

Fixes #96456
2022-05-08 06:22:21 +00:00
Pietro Albini
53965d3daf
add build metrics, to gather ci stats from x.py
This tool will generate a JSON file with statistics about each
individual step to disk. It will be used in rust-lang/rust's CI to
replace the mix of scripts and log scraping we currently have to gather
this data.
2022-05-02 21:33:01 +02:00
Thomas Applencourt
ad7dbe18da Use curl by default 2022-05-01 21:43:12 -05:00
yungkneez
2878f98672 Initialize rust-analyzer submodule on bootstrap 2022-05-01 17:36:39 -07:00
Joshua Nelson
93c1a941bb Move download-ci-llvm to rustbuild
This attempts to keep the logic as close to the original python as possible.
`probably_large` has been removed, since it was always `True`, and UTF-8 paths are no longer supported when patching files for NixOS.
I can readd UTF-8 support if desired.

Note that this required making `llvm_link_shared` computed on-demand,
since we don't know whether it will be static or dynamic until we download LLVM from CI.
2022-04-24 19:40:13 -05:00
Dylan DPC
9a2d14c39a
Rollup merge of #95809 - ytmimi:llvm_stamp_typo, r=Mark-Simulacrum
Fix typo in bootstrap.py
2022-04-19 14:43:15 +02:00
Dylan DPC
d449a63e93
Rollup merge of #95885 - gimbles:patch-1, r=Mark-Simulacrum
Improve error message in case of missing checksum

# Fixes
#94217
2022-04-13 17:35:32 +02:00
gimbles
71ad003bf6 Improve error message when there's no checksum 2022-04-13 13:54:22 +05:30
fee1-dead
ae068e78c0
Rollup merge of #95671 - gimbles:master, r=Mark-Simulacrum
feat: Allow usage of sudo [while not accessing root] in x.py

# Fixes
This PR should fix #93344
# Info
Allows usage of sudo (while not accessing root) in x.py
2022-04-12 22:44:41 +10:00
bors
90ca44752a Auto merge of #95796 - bzEq:bzEq/curl-redirect, r=Dylan-DPC
[bootstrap.py] Instruct curl to follow redirect

Some mirror RUSTUP_DIST_SERVER (like https://mirrors.sjtug.sjtu.edu.cn/rust-static) perform redirection when downloading
stage0 compiler. Curl should be able to follow that.
2022-04-11 18:59:40 +00:00
gimbles
386ca6acc3 Allow usage of sudo while not accessing root 2022-04-10 20:51:11 +05:30
Eric Huss
ca9ef27ed1 Check for git submodules in non-git source tree. 2022-04-09 08:10:34 -07:00
Yacin Tmimi
1433bfd745 Fix typo in bootstrap.py 2022-04-08 16:47:34 -04:00
Kai Luo
95b1d71a1b [bootstrap.py] Instruct curl to follow redirect
Some RUSTUP_DIST_SERVER perform redirection when downloading
stage0 compiler. Curl should be able to follow that.
2022-04-08 14:51:05 +08:00
Benjamin Herr
76756ccb3a bootstrap.py: nixos check in /etc/os-release with quotes
Per https://www.freedesktop.org/software/systemd/man/os-release.html,

> Variable assignment values must be enclosed in double or single quotes
> if they include spaces, semicolons or other special characters outside
> of A–Z, a–z, 0–9. (Assignments that do not include these special
> characters may be enclosed in quotes too, but this is optional.)

So, past `ID=nixos`, let's also check for `ID='nixos'` and `ID="nixos"`.

One of these is necessary between nixos/nixpkgs#162168 and
nixos/nixpkgs#164068, but this seems more correct either way.
2022-03-23 13:00:38 -07:00
Joshua Nelson
4d56f09650 Fallback to top-level config.toml if not present in current directory
This also preserves the behavior where x.py will only give a hard error on a missing config file
if it was configured through `--config` or RUST_BOOTSTRAP_CONFIG.
It also removes the top-level fallback for everything except the default path.
2022-03-09 22:35:43 -06:00
Joshua Nelson
62b522ec3a Don't depend on python for RUST_BOOTSTRAP_CONFIG 2022-03-07 17:06:11 -05:00
Joshua Nelson
240f288329 Move some more bootstrap logic from python to rust
Same rationale as https://github.com/rust-lang/rust/pull/76544;
it would be nice to make python entirely optional at some point.

This also removes $ROOT as an option for the build directory; I haven't been using it, and like Alex
said in https://github.com/rust-lang/rust/pull/76544#discussion_r488248930 it seems like a
misfeature.

This allows running `cargo run` from src/bootstrap, although that still gives
lots of compile errors if you don't use the beta toolchain.
2022-03-07 17:06:11 -05:00
Badel2
410145e0a7 Suggest disabling download-ci-llvm option if url fails to download 2022-02-12 14:23:35 +01:00
Muhammad Falak R Wani
95054de501
bootstrap: prefer using '--config' over 'RUST_BOOTSTRAP_CONFIG'
Signed-off-by: Muhammad Falak R Wani <falakreyaz@gmail.com>
2022-02-07 18:26:25 +05:30
Maxwase
a7092f91a6 Typos fix 2022-01-14 00:17:11 +03:00
Chris Denton
014f22abaf
Break the loop
A missing break statement lead to an infinite loop in bootstrap.py.
2022-01-05 17:59:13 +00:00
Nicholas Nethercote
10e3d92306 Label more build steps.
Currently the output of a command like `./x.py build --stage 0 library/std` is this:
```
Updating only changed submodules
Submodules updated in 0.02 seconds
extracting [...]
   Compiling [...]
    Finished dev [unoptimized] target(s) in 17.53s
Building stage0 std artifacts (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
   Compling [...]
    Finished release [optimized + debuginfo] target(s) in 21.99s
Copying stage0 std from stage0 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-unknown-linux-gnu)
Build completed successfully in 0:00:51
```
I find the part before the "Building stage0 std artifacts" a bit confusing.

After this commit, it looks like this:
```
Updating only changed submodules
  Submodules updated in 0.02 seconds
extracting [...]
Building rustbuild
   Compiling [...]
    Finished dev [unoptimized] target(s) in 17.53s
Building stage0 std artifacts (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
   Compling [...]
    Finished release [optimized + debuginfo] target(s) in 21.99s
Copying stage0 std from stage0 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-unknown-linux-gnu)
Build completed successfully in 0:00:51
```
The "Building rustbuild" label makes it clear what the first cargo build
invocation is for. The indentation of the "Submodules updated" line
indicates it is a sub-step of a parent task.
2022-01-04 09:24:10 +11:00
Matthias Krüger
5a5c9282e0
Rollup merge of #88310 - worldeva:bootstrap-locking, r=Mark-Simulacrum
Lock bootstrap (x.py) build directory

Closes #76661,  closes #80849,
`x.py` creates a lock file at `project_root/lock.db`

r? `@jyn514` , because he was one that told me about this~
2022-01-01 10:48:53 +01:00
worldeva
1326bd6484 Lock x.py build state
Prevent spurious build failures and other bugs caused by parallel runs of
x.py. We back the lock with sqlite, so that we have a cross-platform locking
strategy, and which can be done nearly first in the build process (from Python),
which helps move the lock as early as possible.
2021-12-31 10:31:13 -05:00
Joshua Nelson
ae6f5fbc3f If --verbose is passed, print a warning in bootstrap.py when download-rustc is ignored 2021-12-09 11:30:38 -06:00
Joshua Nelson
8a6f54fc3a Use more accurate wording in bootstrap.py logging 2021-12-09 11:30:38 -06:00
bors
87dce6e8df Auto merge of #91284 - t6:freebsd-riscv64, r=Amanieu
Add support for riscv64gc-unknown-freebsd

For https://doc.rust-lang.org/nightly/rustc/target-tier-policy.html#tier-3-target-policy:

* A tier 3 target must have a designated developer or developers (the "target maintainers") on record to be CCed when issues arise regarding the target. (The mechanism to track and CC such developers may evolve over time.)

For all Rust targets on FreeBSD, it's [rust@FreeBSD.org](mailto:rust@FreeBSD.org).

* Targets must use naming consistent with any existing targets; for instance, a target for the same CPU or OS as an existing Rust target should use the same name for that CPU or OS. Targets should normally use the same names and naming conventions as used elsewhere in the broader ecosystem beyond Rust (such as in other toolchains), unless they have a very good reason to diverge. Changing the name of a target can be highly disruptive, especially once the target reaches a higher tier, so getting the name right is important even for a tier 3 target.

Done.

* Target names should not introduce undue confusion or ambiguity unless absolutely necessary to maintain ecosystem compatibility. For example, if the name of the target makes people extremely likely to form incorrect beliefs about what it targets, the name should be changed or augmented to disambiguate it.

Done

* Tier 3 targets may have unusual requirements to build or use, but must not create legal issues or impose onerous legal terms for the Rust project or for Rust developers or users.

Done.

* The target must not introduce license incompatibilities.

Done.

* Anything added to the Rust repository must be under the standard Rust license (MIT OR Apache-2.0).

Fine with me.

* The target must not cause the Rust tools or libraries built for any other host (even when supporting cross-compilation to the target) to depend on any new dependency less permissive than the Rust licensing policy. This applies whether the dependency is a Rust crate that would require adding new license exceptions (as specified by the tidy tool in the rust-lang/rust repository), or whether the dependency is a native library or binary. In other words, the introduction of the target must not cause a user installing or running a version of Rust or the Rust tools to be subject to any new license requirements.

Done.

* If the target supports building host tools (such as rustc or cargo), those host tools must not depend on proprietary (non-FOSS) libraries, other than ordinary runtime libraries supplied by the platform and commonly used by other binaries built for the target. For instance, rustc built for the target may depend on a common proprietary C runtime library or console output library, but must not depend on a proprietary code generation library or code optimization library. Rust's license permits such combinations, but the Rust project has no interest in maintaining such combinations within the scope of Rust itself, even at tier 3.

Done.

* Targets should not require proprietary (non-FOSS) components to link a functional binary or library.

Done.

* "onerous" here is an intentionally subjective term. At a minimum, "onerous" legal/licensing terms include but are not limited to: non-disclosure requirements, non-compete requirements, contributor license agreements (CLAs) or equivalent, "non-commercial"/"research-only"/etc terms, requirements conditional on the employer or employment of any particular Rust developers, revocable terms, any requirements that create liability for the Rust project or its developers or users, or any requirements that adversely affect the livelihood or prospects of the Rust project or its developers or users.

Fine with me.

* Neither this policy nor any decisions made regarding targets shall create any binding agreement or estoppel by any party. If any member of an approving Rust team serves as one of the maintainers of a target, or has any legal or employment requirement (explicit or implicit) that might affect their decisions regarding a target, they must recuse themselves from any approval decisions regarding the target's tier status, though they may otherwise participate in discussions.

Ok.

* This requirement does not prevent part or all of this policy from being cited in an explicit contract or work agreement (e.g. to implement or maintain support for a target). This requirement exists to ensure that a developer or team responsible for reviewing and approving a target does not face any legal threats or obligations that would prevent them from freely exercising their judgment in such approval, even if such judgment involves subjective matters or goes beyond the letter of these requirements.

Ok.

* Tier 3 targets should attempt to implement as much of the standard libraries as possible and appropriate (core for most targets, alloc for targets that can support dynamic memory allocation, std for targets with an operating system or equivalent layer of system-provided functionality), but may leave some code unimplemented (either unavailable or stubbed out as appropriate), whether because the target makes it impossible to implement or challenging to implement. The authors of pull requests are not obligated to avoid calling any portions of the standard library on the basis of a tier 3 target not implementing those portions.

std is implemented.

* The target must provide documentation for the Rust community explaining how to build for the target, using cross-compilation if possible. If the target supports running tests (even if they do not pass), the documentation must explain how to run tests for the target, using emulation if possible or dedicated hardware if necessary.

Building is possible the same way as other Rust on FreeBSD targets.

* Tier 3 targets must not impose burden on the authors of pull requests, or other developers in the community, to maintain the target. In particular, do not post comments (automated or manual) on a PR that derail or suggest a block on the PR based on a tier 3 target. Do not send automated messages or notifications (via any medium, including via `@)` to a PR author or others involved with a PR regarding a tier 3 target, unless they have opted into such messages.

Ok.

* Backlinks such as those generated by the issue/PR tracker when linking to an issue or PR are not considered a violation of this policy, within reason. However, such messages (even on a separate repository) must not generate notifications to anyone involved with a PR who has not requested such notifications.

Ok.

* Patches adding or updating tier 3 targets must not break any existing tier 2 or tier 1 target, and must not knowingly break another tier 3 target without approval of either the compiler team or the maintainers of the other tier 3 target.

Ok.

* In particular, this may come up when working on closely related targets, such as variations of the same architecture with different features. Avoid introducing unconditional uses of features that another variation of the target may not have; use conditional compilation or runtime detection, as appropriate, to let each target run code supported by that target.

Ok.
2021-12-06 03:51:05 +00:00
Tobias Kortkamp
47474f1055
Add riscv64gc-unknown-freebsd 2021-11-27 07:24:18 +01:00
Alberto Planas
afd9dfa89d bootstap: create .cargo/config only if not present
In some situations we should want on influence into the .cargo/config
when we use vendored source.  One example is #90764, when we want to
workaround some references to crates forked and living in git, that are
missing in the vendor/ directory.

This commit will create the .cargo/config file only when the .cargo/
directory needs to be created.
2021-11-14 17:04:32 +01:00
Matthias Krüger
b7be4fc99f
Rollup merge of #89929 - yuvaldolev:handle-submodule-checkout-more-gracefully, r=Mark-Simulacrum
Handling submodule update failures more gracefully from x.py

Addresses #80498

Handling the case where x.py can't check out the right commit of a submodule, because the submodule has local edits that would be overwritten by the checkout, more gracefully.
The error is printed in detail, with some hints on how to revert the local changes to the submodule.
2021-11-01 03:33:04 +01:00
Matthias Krüger
b902aa98e5
Rollup merge of #89950 - infinity0:master, r=Mark-Simulacrum
bootstrap: tweak verbosity settings

Currently the verbosity settings are:
- 2: RUSTC-SHIM envvars get spammed on every invocation, O(30) lines
     cargo is passed -v which outputs CLI invocations, O(5) lines
- 3: cargo is passed -vv which outputs build script output, O(0-10) lines

This commit changes it to:

- 1: cargo is passed -v, O(5) lines
- 2: cargo is passed -vv, O(10) lines
- 3: RUSTC-SHIM envvars get spammed, O(30) lines
2021-10-18 08:13:25 +02:00
Ximin Luo
d78559ac11 bootstrap: tweak verbosity settings
Currently the verbosity settings are:
- 2: RUSTC-SHIM envvars get spammed on every invocation, O(30) lines
     cargo is passed -v which outputs CLI invocations, O(5) lines
- 3: cargo is passed -vv which outputs build script output, O(0-10) lines

This commit changes it to:

- 1: cargo is passed -v, O(5) lines
- 2: cargo is passed -vv, O(10) lines
- 3: RUSTC-SHIM envvars get spammed, O(30) lines
2021-10-16 13:30:24 +01:00
Yuval Dolev
95ae8687df Handling submodule update failures more gracefully from x.py 2021-10-16 01:38:57 +03:00
Hans Kratz
ac298c9181 Make rust.download-ci-llvm="if-available" work for tier 2 platforms. 2021-10-15 07:45:27 +02:00
Matthias Krüger
9f0ef184b8
Rollup merge of #89757 - jyn514:submodule, r=Mark-Simulacrum
Use shallow clones for submodules

This reduces the amount of git history downloaded for submodules from ~67M to ~11M. For comparison, a shallow clone of rust-lang/rust is 103M and a deep clone is 740M, so this almost halves the amount of history necessary if you made a shallow clone to start, and it's a significant reduction even if not.

Closes https://github.com/rust-lang/rust/issues/63978. r? `@Mark-Simulacrum`
2021-10-13 22:51:02 +02:00
Matthias Krüger
1be64f3431
Rollup merge of #89756 - jyn514:bootstrap-times, r=Mark-Simulacrum
Greatly reduce amount of debuginfo compiled for bootstrap itself

Rather than compiling rustbuild and all its dependencies with
`debuginfo=2`, this compiles dependencies without debuginfo and
rustbuild with `debuginfo=1`. On my laptop, this brings compile times
down from ~1:20 to ~1:05.

See also 254847594.

r? ``@Mark-Simulacrum``
2021-10-11 23:45:51 +02:00