How to conditionally fail a build in a pull requests with TFS/VSTS

By terje

When your TFS/VSTS CI build is also used for a pull request, you often want to enable more checks before you let this go into the master (or any target) branch, and fail the build if these raise warnings.

It can be extra tests you want to run, code quality checks,  process checks, and  you might want to block the pull request if you have warnings from these.

Setting a single test task to fail could be done, but that only works well if you have a single step, with multiple tasks you will have build stops for one and every step, so it also reduces the error reporting granularity.  And, it is nice to be able to see what is an absolutely blocker, red, and what is quality issues, yellow.

You can achieve this by adding a standard command line task that fails given these conditions.  It is very easy to set up.

In the build shown below, there are seven tasks that may give rise to warnings (green checkmarks).  Note that some are ordinary unit tests, the tests are also split into pure L0 class based unit tests, and L1 integration tests, see these two posts.  There are also checks that verify that we have more than a set number of tests, and finally a Resharper code quality check task.  They are checked using the red circled command line task below them.

Build task setup for conditional fail

  1. Give it a good name like: "If above is partially succeeded, and we have a PR , then fail"
  2. Use the command "echo"
  3. The arguments are "1>&2".  This mean we redirect the standard output to the error output
  4. Set to fail on Standard Error.
  5. Set to custom conditions
  6. Set to: and(eq(variables['Agent.JobStatus'], 'SucceededWithIssues'), eq(variables['Build.Reason'], 'PullRequest'))
Reusing a CI build in a pull request saves you an extra build.  The PR build which you enable in the Pull Request policy, runs on the merged code of your branch and the target branch, normally the master.  It makes a lot of sense to make this build stricter than the ordinary CI build.  Note in the build definition above that the L1 tests are also conditionals, they only run for a pull request started build. The L1 tests are normally slower than the L0, and it makes sense to only use them as an extra quality step.

Tip 1: Note that everything above where you place the command line will be affected, anything below will not be affected.

Tip 2: This has to be used in the build definition proper, it doesn’t work in a task group, due to the custom conditional which is not available in task groups.

Tip 3: Examples of other useful conditions, and the condition syntax can  be found in this msdn article