rustbuild: Compile rustc twice, not thrice #38631

Merged
merged 1 commit into from Dec 30, 2016

Projects

None yet

7 participants

@alexcrichton
Member
alexcrichton commented Dec 27, 2016 edited

This commit switches the rustbuild build system to compiling the
compiler twice for a normal bootstrap rather than the historical three
times.

Rust is a bootstrapped language which means that a previous version of
the compiler is used to build the next version of the compiler. Over
time, however, we change many parts of compiler artifacts such as the
metadata format, symbol names, etc. These changes make artifacts from
one compiler incompatible from another compiler. Consequently if a
compiler wants to be able to use some artifacts then it itself must have
compiled the artifacts.

Historically the rustc build system has achieved this by compiling the
compiler three times:

  • An older compiler (stage0) is downloaded to kick off the chain.
  • This compiler now compiles a new compiler (stage1)
  • The stage1 compiler then compiles another compiler (stage2)
  • Finally, the stage2 compiler needs libraries to link against, so it
    compiles all the libraries again.

This entire process amounts in compiling the compiler three times.
Additionally, this process always guarantees that the Rust source tree
can compile itself because the stage2 compiler (created by a freshly
created compiler) would successfully compile itself again. This
property, ensuring Rust can compile itself, is quite important!

In general, though, this third compilation is not required for general
purpose development on the compiler. The third compiler (stage2) can
reuse the libraries that were created during the second compile. In
other words, the second compilation can produce both a compiler and the
libraries that compiler will use. These artifacts must be compatible
due to the way plugins work today anyway, and they were created by the
same source code so they should be compatible as well.

So given all that, this commit switches the default build process to
only compile the compiler two times, avoiding this third compilation
by copying artifacts from the previous one. Along the way a new entry in
the Travis matrix was also added to ensure that our full bootstrap can
succeed. This entry does not run tests, though, as it should not be
necessary.

To restore the old behavior of a full bootstrap (three compiles) you can
either pass:

./configure --enable-full-bootstrap

or if you're using config.toml:

[build]
full-bootstrap = true

Overall this will hopefully be an easy 33% win in build times of the
compiler. If we do 33% less work we should be 33% faster! This in turn
should affect cycle times and such on Travis and AppVeyor positively as
well as making it easier to work on the compiler itself.

@rust-highfive
Collaborator

r? @nikomatsakis

(rust_highfive has picked a reviewer for you, use r? to override)

@alexcrichton
Member

r? @brson

cc #38531

@rust-highfive rust-highfive assigned brson and unassigned nikomatsakis Dec 27, 2016
+ --enable-full-bootstrap
+ENV RUST_CHECK_TARGET ""
+RUN mkdir /tmp/obj
+RUN chmod 777 /tmp/obj
@petrochenkov
petrochenkov Dec 27, 2016 Contributor

Nit: ccache and /tmp/obj are not removed (cc #38632)

@alexcrichton
alexcrichton Dec 27, 2016 Member

Oops, indeed!

@alexcrichton alexcrichton rustbuild: Compile rustc twice, not thrice
This commit switches the rustbuild build system to compiling the
compiler twice for a normal bootstrap rather than the historical three
times.

Rust is a bootstrapped language which means that a previous version of
the compiler is used to build the next version of the compiler. Over
time, however, we change many parts of compiler artifacts such as the
metadata format, symbol names, etc. These changes make artifacts from
one compiler incompatible from another compiler. Consequently if a
compiler wants to be able to use some artifacts then it itself must have
compiled the artifacts.

Historically the rustc build system has achieved this by compiling the
compiler three times:

* An older compiler (stage0) is downloaded to kick off the chain.
* This compiler now compiles a new compiler (stage1)
* The stage1 compiler then compiles another compiler (stage2)
* Finally, the stage2 compiler needs libraries to link against, so it
  compiles all the libraries again.

This entire process amounts in compiling the compiler three times.
Additionally, this process always guarantees that the Rust source tree
can compile itself because the stage2 compiler (created by a freshly
created compiler) would successfully compile itself again. This
property, ensuring Rust can compile itself, is quite important!

In general, though, this third compilation is not required for general
purpose development on the compiler. The third compiler (stage2) can
reuse the libraries that were created during the second compile. In
other words, the second compilation can produce both a compiler and the
libraries that compiler will use. These artifacts *must* be compatible
due to the way plugins work today anyway, and they were created by the
same source code so they *should* be compatible as well.

So given all that, this commit switches the default build process to
only compile the compiler three times, avoiding this third compilation
by copying artifacts from the previous one. Along the way a new entry in
the Travis matrix was also added to ensure that our full bootstrap can
succeed. This entry does not run tests, though, as it should not be
necessary.

To restore the old behavior of a full bootstrap (three compiles) you can
either pass:

    ./configure --enable-full-bootstrap

or if you're using config.toml:

    [build]
    full-bootstrap = true

Overall this will hopefully be an easy 33% win in build times of the
compiler. If we do 33% less work we should be 33% faster! This in turn
should affect cycle times and such on Travis and AppVeyor positively as
well as making it easier to work on the compiler itself.
7046fea
@brson
Contributor
brson commented Dec 29, 2016

@bors r+

@bors
Contributor
bors commented Dec 29, 2016

📌 Commit 7046fea has been approved by brson

@xen0n
Contributor
xen0n commented Dec 29, 2016

A small PR message nit:

[...] this commit switches the default build process to
only compile the compiler three times, [...]

three -> two

@alexcrichton
Member

Oops, indeed!

@alexcrichton alexcrichton added a commit to alexcrichton/rust that referenced this pull request Dec 30, 2016
@alexcrichton alexcrichton Rollup merge of #38631 - alexcrichton:supafast, r=brson
rustbuild: Compile rustc twice, not thrice

This commit switches the rustbuild build system to compiling the
compiler twice for a normal bootstrap rather than the historical three
times.

Rust is a bootstrapped language which means that a previous version of
the compiler is used to build the next version of the compiler. Over
time, however, we change many parts of compiler artifacts such as the
metadata format, symbol names, etc. These changes make artifacts from
one compiler incompatible from another compiler. Consequently if a
compiler wants to be able to use some artifacts then it itself must have
compiled the artifacts.

Historically the rustc build system has achieved this by compiling the
compiler three times:

* An older compiler (stage0) is downloaded to kick off the chain.
* This compiler now compiles a new compiler (stage1)
* The stage1 compiler then compiles another compiler (stage2)
* Finally, the stage2 compiler needs libraries to link against, so it
  compiles all the libraries again.

This entire process amounts in compiling the compiler three times.
Additionally, this process always guarantees that the Rust source tree
can compile itself because the stage2 compiler (created by a freshly
created compiler) would successfully compile itself again. This
property, ensuring Rust can compile itself, is quite important!

In general, though, this third compilation is not required for general
purpose development on the compiler. The third compiler (stage2) can
reuse the libraries that were created during the second compile. In
other words, the second compilation can produce both a compiler and the
libraries that compiler will use. These artifacts *must* be compatible
due to the way plugins work today anyway, and they were created by the
same source code so they *should* be compatible as well.

So given all that, this commit switches the default build process to
only compile the compiler two times, avoiding this third compilation
by copying artifacts from the previous one. Along the way a new entry in
the Travis matrix was also added to ensure that our full bootstrap can
succeed. This entry does not run tests, though, as it should not be
necessary.

To restore the old behavior of a full bootstrap (three compiles) you can
either pass:

    ./configure --enable-full-bootstrap

or if you're using config.toml:

    [build]
    full-bootstrap = true

Overall this will hopefully be an easy 33% win in build times of the
compiler. If we do 33% less work we should be 33% faster! This in turn
should affect cycle times and such on Travis and AppVeyor positively as
well as making it easier to work on the compiler itself.
9bb3543
@bors bors added a commit that referenced this pull request Dec 30, 2016
@bors bors Auto merge of #38697 - alexcrichton:rollup, r=alexcrichton
Rollup of 25 pull requests

- Successful merges: #37149, #38491, #38517, #38559, #38587, #38609, #38611, #38622, #38628, #38630, #38631, #38632, #38635, #38647, #38649, #38655, #38659, #38660, #38662, #38665, #38671, #38674, #38676, #38693, #38695
- Failed merges: #38657, #38680
4a68076
@bors bors added a commit that referenced this pull request Dec 30, 2016
@bors bors Auto merge of #38697 - alexcrichton:rollup, r=alexcrichton
Rollup of 25 pull requests

- Successful merges: #37149, #38491, #38517, #38559, #38587, #38609, #38611, #38622, #38628, #38630, #38631, #38632, #38635, #38647, #38649, #38655, #38659, #38660, #38662, #38665, #38671, #38674, #38676, #38693, #38695
- Failed merges: #38657, #38680
4536280
@bors bors added a commit that referenced this pull request Dec 30, 2016
@bors bors Auto merge of #38697 - alexcrichton:rollup, r=alexcrichton
Rollup of 25 pull requests

- Successful merges: #37149, #38491, #38517, #38559, #38587, #38609, #38611, #38622, #38628, #38630, #38631, #38632, #38635, #38647, #38649, #38655, #38659, #38660, #38662, #38665, #38671, #38674, #38676, #38693, #38695
- Failed merges: #38657, #38680
c4908ce
@bors bors added a commit that referenced this pull request Dec 30, 2016
@bors bors Auto merge of #38697 - alexcrichton:rollup, r=alexcrichton
Rollup of 25 pull requests

- Successful merges: #37149, #38491, #38517, #38559, #38587, #38609, #38611, #38622, #38628, #38630, #38631, #38632, #38635, #38647, #38649, #38655, #38659, #38660, #38662, #38665, #38671, #38674, #38676, #38693, #38695
- Failed merges: #38657, #38680
7f2d2af
@bors bors merged commit 7046fea into rust-lang:master Dec 30, 2016

1 check passed

Details continuous-integration/travis-ci/pr The Travis CI build passed
@alexcrichton alexcrichton deleted the alexcrichton:supafast branch Dec 30, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment