Web Development

Automated Testing in Azure DevOps with Cypress

June 08, 2020 5 min reading time

“Azure DevOps sucks.”

At least that’s what I was thinking while I struggled to add automated testing to a pipeline in Azure DevOps (ADO). In reality, I’ve really grown to like ADO, specifically its continuous integration capabilities.

Setting up pipelines is usually pretty intuitive; however, sometimes setting up newer bleeding edge tools into it can be a bit of a pain. The speed at which these tools change in their early days often results in outdated or incorrect documentation.

We’ve been implementing end-to-end testing in many of our projects using Cypress, a relatively new JavaScript end-to-end testing framework. In my opinion, Cypress is much easier to set up and run locally, compared to something like Selenium. It currently supports Chrome-family browsers (including Edge) and Firefox, so if you need to test IE11 or any other browsers, you may want to look for another solution. If it aligns with your supported browsers, the ease with which you can get started will have you writing tests in no time.

End-to-end testing is a great method to start with if you’re looking to add tests to an existing app that currently has no or few tests. The barrier for entry is lower, and it can even help with onboarding new developers to a project. It’s a great way to become familiar with a new codebase.

It’s also useful when resolving existing bugs. When a bug is reported, a developer will add a test to verify and replicate it, then implement a fix. Once the test (or tests) pass, you’ll have higher confidence that the bug won’t resurface in the future.

What follows is a pipeline to run a single page app on an agent and run Cypress tests against that app. If any test fails, the pipeline stops; otherwise, an artifact is created that can then be incorporated into a release.

The first step is to set up Cypress to output your test results as an xml file and use the junit reporter. Once you’ve installed Cypress (if you haven’t worked with Cypress before, check out the getting started section here: https://docs.cypress.io/guides/getting-started/installing-cypress.html), create a cypress.json file at the root of your project and add the following lines:

"reporter": "junit",
"reporterOptions": {
    "mochaFile": "test/results/e2e-tests-[hash].xml"
}

This provides test results that ADO can publish on the test tab of the build.

Next, we’ll look at a sample pipeline’s YAML file and explain what each block does.

trigger:
    -develop

The pipeline will run when a commit is pushed or merged into the “develop” branch.

pool:
    vmImage: 'ubuntu-16.04'

This line is important, and what caused most of my struggles setting up the pipeline. Currently only the ubuntu-16.04 image has the dependencies necessary to run Cypress. Without specifying this version of Ubuntu the “npm run cy:verify” command fails. This has since been updated in the Azure example referenced in the Cypress documentation.

steps:
    -task: [email protected]
     inputs:
        versionSpec: '12.x'
     displayName: 'Install Node.js'

Install Node 12. Cypress supports Node 8 and above.

 - task: [email protected]
   inputs:
        key: npm | $(Agent.OS) | package-lock.json
        path: /home/vsts/.npm
        restoreKeys: npm | $(Agent.OS) | package-lock.json
   displayName: Cache NPM Packages
 - task: [email protected]
   inputs:
        key: cypress | $(Agent.OS) | package-lock.json
        path: /home/vsts/.cache/Cypress
        restoreKeys: cypress | $(Agent.OS) | package-lock.json
   displayName: Cache Cypress binary

Use pipeline caching for both the NPM packages and the Cypress binary.

  • “path” indicates where to store the cached files.
  • “Key” is a unique string used as an identifier for the cache.
  • “restoreKeys” will be used if the key is not found. If an exact match for a key is not found, restoreKeys will partial match. If we just put “cypress” here, it will search for cypress*. For our purposes we can just use the key line. If you need to update or “clear” the cache, you can update the key from Cypress to Cypress2, etc.
-script: |
    npm install
    npm run build
 displayName: 'npm install and build'

Here we install our NPM dependencies and build our App’s JS bundle.

-script: npm run cy:verify
 displayName: 'Cypress Verify'

This command verifies that Cypress is installed and is executable. It runs the cy:verify script in package.json, which is set to: “cypress verify”.

-script: |
    npx print-env AGENT
    npx print-env BUILD
    npx print-env SYSTEM
    npm run start & npm run cy:test
 displayName: 'Run Cypress tests'
 env:
    # avoid warnings about terminal
    TERM: xterm

Finally, we start our app and run our cypress tests. It runs the cy:test script in package.json, which is set to: “cypress run”.

-task: [email protected]
 condition: succeededOrFailed()
 inputs:
    testRunner: JUnit
    testResultsFiles: '$(Build.SourcesDirectory)/test/results/*.xml'

Now we use the PublishTestResults task to grab our test results and publish them to the “Tests” tab in the Pipeline summary. The “succeededOrFailed()” condition ensures the task runs every time the pipeline is run, regardless of if the tests fail.

-task: [email protected]
 inputs:
    PathtoPublish: '$(Build.SourcesDirectory)/build'
    ArtifactName: 'drop'
    publishLocation: 'Container'

Finally, we publish our build artifacts to be used in a pipeline release. This task will run only if all previous tasks complete successfully. If any tests fail or the pipeline encounters any other errors, this task will not run.

Automated e2e testing is not a silver bullet to make your code completely free of defects, but it is a great tool to make it more resilient to change. While it can add a little more effort to the initial set up and development, it can help prevent future defects.

Written by Cory Gugler, Director, Interactive Engineering