Try Server usage

Overview

The try server lets committers try uncommitted patches on multiple platform in an automated way.

Contents

  1. 1 Overview
  2. 2 Workflow
  3. 3 How to specify a subset of tests to run
    1. 3.1 Examples
  4. 4 Troubleshooting failed try jobs
  5. 5 FAQ
    1. 5.1 I'm modifying a DEPS, will it work?
    2. 5.2 I'm modifying a GYP file, will it work?
    3. 5.3 I'm modifying gyp scripts under src/tools/gyp, will it work?
    4. 5.4 Get detailed command-line help
    5. 5.5 Disable automatic try on gcl upload
    6. 5.6 I have an awesome patch for depot_tools!
    7. 5.7 How can I see what will happen before I submit the job?
    8. 5.8 apply_issue failed
    9. 5.9 compiling failed
    10. 5.10 My patch includes updated webkit baselines (binary files) and it's not working
    11. 5.11 I want to cancel my job, should I press the 'Stop' button?
    12. 5.12 Run the try job only on one platform
    13. 5.13 Run the try job for "real" Chrome OS
    14. 5.14 Submit a try job for another person (e.g. non-committer)
    15. 5.15 I need to hack on a try slave for a few hours?
    16. 5.16 What is the difference between 'git try' and 'git cl try'?
    17. 5.17 Where can I find the results of webkit layout tests for a try bot build?
  • The try server is a standard buildbot plus a set of slaves, currently Linux (Ubuntu), Mac, and Windows.
  • The try server can be used with git cl try (preferred), or gcl try and git-try, both are front ends for trychange.py.
  • It can also be used via the web interface by clicking on the "Try more bots" link in Rietveld.
  • Any committer can use the try server.
    • Non-committers with try job access may also run try jobs. See become-a-commiter#TOC-try-job-access for getting access.
    • External contributors can ask a committer to try the job for them and have the result email directly into their inbox.
    • If you do not have sufficient permissions, attempts to use try servers (such as via the web interface) will result in immediate failure.
  • File limitations are:
    • Patches must be less than 20MB.
    • Binary files are not supported.
    • Text files with CRLF will have issues too.
  • Try server waterfalls: gpu linux mac win
  • See the Design doc: Try server.

Workflow

  • The developer explicitly requests a job to be run with git cl try, git try,  or gcl try.
  • The try server will schedule one job per platform that you can follow live on the waterfall.
  • The job revision will default to http://chromium-status.appspot.com/lkgr. This can be overridden with the tools with --revision.
  • On each job end
    • An email will be sent to the email specified in the job. It is usually specified implicitly.
    • The corresponding review on http://codereview.chromium.org/ will have its try job status updated live.
  • On gcl commit / git cl dcommit (Not always), a PRESUBMIT.py will enforce that the job succeeded.

How to specify a subset of tests to run

-b, --bot to specify the bot
-b, --bot is additive, except that if you don't specify it at all you'll get default bots

Allowed uses:
Specify multiple builders using the default tests:
  -b buildername1 -b buildername2 -b buildername3

-m, --master to specify a tryserver master.

For Linux and Android builders, the master is tryserver.chromium.linux, for Mac and iOS - tryserver.chromium.mac, and for Windows - tryserver.chromium.win. Note, that all the builders specified with -b will be launched on the same master. Issue separate git cl try command for each master.

Examples

git cl try -b win_chromium_rel -m tryserver.chromium.win

git cl try -b linux_blink_rel -m tryserver.blink

Troubleshooting failed try jobs

Please, before contacting maruel, please look at these items:
  • Did it fail at the Update step?
    • Did your patch contain CRLF files? You can look at the 'patch' link in the 'update' step.
      • Too bad for you. Create a separate commit to fix the file in the tree and try again with --revision 123
    • Did you get GYP failures in the update 'stdio'?
      • Maybe you screwed the GYP file.
    • Did you get "HUNK", "fuzzing" or "offset" messages in the update 'stdio'?
      • The files you are modifying have been modified on the tree. Maybe LKGR is more recent or older than your checkout. You can look at the build link in the email in the Build Properties section, search for the number beside revision. Maybe try specifying --revision on your try.
  • Did it fail at the Compile step?
    • Maybe your code is broken?
      • Yes, that is a possibility. Sorry 'bout that.
    • Maybe someone else broke the slave?
      • That happens. Especially on Windows. Please try again and alert me with the build status url. You can use the --bot win flag to not try needlessly on other platforms.
    • Maybe a clobber is required. There are command-line options to request clobber (most likely just -c).
  • Did it fail an unrelated test?
    • Maybe it was broken at that revision on the main waterfall. Please look up the revision and take a look on the main waterfall.
    • Maybe the tree slave is broken?
      • That happens. Please try again and alert me with the build status url.
  • I get no email!
    • Did you get a warning message when sending your job? By default, it sends it to your checkout credential.
  • 'gcl try' fails
    • Send an email to <tryserver at chromium dot 0rg>.
  • The try server is soo slow
    • When a clobber is needed, a full rebuild takes time.
    • Maybe someone changed a GYP and forgot to update the svn:ignore?
      • If you feel like getting a peer bonus, please fix the property accordingly, otherwise ping <tryserver at chromium dot 0rg>.
  • The try server needs a clobber
    • That happens, usually ask to someone knowledgeable on irc to fix it for you or ping maruel if no answer.
    • You can also request a clobber using command-line flags (usually -c).
  • error message: 'hostname nor servname provided, or not known'
    • If you're using the try server from outside of Google, it will fail to look up the hostname and should then fall back to using svn to enqueue the try job.  If svn fails, unfortunately, you'll only see the error message about the hostname, so you won't realize that svn is the problem.  
    • To fix, try this first: svn ls svn://svn.chromium.org/chrome-try
    • If that last step didn't work, maybe you're using git try on Windows?  In that case, you may have a mismatch where the tools are using a mix of depot_tools svn and cygwin svn.  In that case, also type /usr/bin/svn ls svn://svn.chromium.org/chrome-try
  • Anything else?
    • Please update this guide.
Requesting new bots

File a bug to get a bot request going.  Do this as early as possible and provide as many details as you can.

FAQ

I'm modifying a DEPS, will it work?

Yes.

I'm modifying a GYP file, will it work?

Yes.

I'm modifying gyp scripts under src/tools/gyp, will it work?

Yes, but you have to use  --no_search, otherwise patch will fail saying it cannot find files to patch.

Get detailed command-line help

Use: git try -h   (git will redirect --help to man which fails for git-try)
Or: trychange.py --help

Disable automatic try on gcl upload

To do this for 'gcl try': create a file at  '~/.gcl_upload_no_try'
For 'git try' : Edit src/codereview.settings thusly: "TRY_ON_UPLOAD: False"

I have an awesome patch for depot_tools!

Use the same way to contribute code that for chromium in general, e.g. gcl or git-cl.

How can I see what will happen before I submit the job?

Use the --dry_run flag (which implies --verbose) to (a) see what your patch is and (b) not actually submit the job when using git try or gcl try.

apply_issue failed

By default the trybot will patch your change against HEAD, so there might be differences in the files you are modifying between the revision you've been working on and HEAD. The easiest way to fix this is check which revision you were using, and then pass it to the try job with --revision, e.g., git cl try -rHEAD. 

If this still doesn't work and:

  1. you are using git
  2. master is not upstream of the branch you are trying
then its possible there are uncommitted changes between your branch and master, causing the apply_issue to fail.  In this case, use  git try  instead of  git cl try  as follows:

    git try --upstream_branch=origin/master

Keep in mind that this means your branch cannot be committed until all upstream branches are committed first, even if the trybots succeed.

compiling failed

Possible issues:
  • Incremental build failure, slightly less probable but still happens occasionally. Try with -c, --clobber
  • The slave has a broken checkout and needs a complete checkout removal, rarely happens but still does. Reply to the try job email.
  • Your patch is broken. Please don't completely rule this possibility out yet. :)

My patch includes updated webkit baselines (binary files) and it's not working

Correct! The try servers do not currently support binary patches. 

I want to cancel my job, should I press the 'Stop' button?

No! DON'T EVER DO THAT. This button is for maintainers and is quite tricky to use correctly. Just let it go.

Why! you ask. You may stop it during the update step and leave the svn checkout locked, breaking the following tries. You may stop it during compilation, corrupting the PDB database, breaking the following jobs. You may stop it during ui_tests, leaving zombies around. You may stop it during unit_tests, leaving temp files around.

Run the try job only on one platform

Use --bot win or whatever 'builder' name you want to use from the waterfalls.

Run the try job for "real" Chrome OS

As of June 2014 the Chrome try server provides chromium.chromeos builders, which are 64-bit Linux builds of Chromium with OS_CHROMEOS=1. This means the CQ and try server will not compile or test 32-bit Intel or ARM Chrome OS builds. But you can try your changes on the Chrome OS side with cbuildbot. See How to patch in a Reitveld CL in the Chromium OS remote trybot docs.

Submit a try job for another person (e.g. non-committer)

Use Rietvield WebUI with the "Try more bots" link. Alternatively, you can use:

git cl patch -b new-branch-name Rietveld-issue-id
git cl issue Rietveld-issue-id
git cl try

Where new-branch-name and Rietveld-issue-id should be replaced accordingly (give it a new branch name and link it with the id for the non-committer's CL on Rietveld).

I need to hack on a try slave for a few hours?

Sorry, only Googlers can do this for now. You can get more information at the internal link.

What is the difference between 'git try' and 'git cl try'?

  • git try runs the try job on your local changes. This causes a diff file to be committed in a special repository. This file is read by the Try Server and the slave applies it during the update step.
  • git cl try runs the try job on your changes already uploaded to the codereview tool. The Try Server instructs the slave to download the patch by itself in the apply_issue step.

Where can I find the results of webkit layout tests for a try bot build?

Under the builder's build results page, look for the step entitled archive_webkit_tests_results. A zip archive of the entire results directory is available the (zip) link under this step.

Subpages (1): Design doc: Try server
Comments