# Test Coverage

The Test coverage analyzer accepts the coverage report file and creates metrics and issues around code coverage.

This section covers configuration specific to the test-coverage analyzer. Please make sure to read the general configuration guide first.

# Configuration - .deepsource.toml

# name

name = "test-coverage"

# enabled

enabled = true

# Configuration - Artifacts

Refer 'report' command of CLI documentation for command usage.

Key Value format
python XML(Cobertura)
go cover.out (Generated)
javascript XML(Cobertura)
ruby JSON(SimpleCov)
java XML (Jacoco/Clover)

TIP

The maximum allowed size for the coverage report file is 5 MB.

# Examples

# JavaScript

# JavaScript: Jest

Currently, only the cobertura-XML format is supported by DeepSource.

Here are the steps to report the coverage data.

# Make sure `jest` is installed. If not, install it using `yarn`
yarn add --dev jest
# OR `npm`
npm install --save-dev jest

# Add the following options to the jest config to get a cobertura report
"collectCoverage": true,
"coverageReporters": ["cobertura"]

# Or, add these options as flags to test script in package.json
{
  "scripts": {
    "test": "jest --coverage=true --coverageReporters=cobertura"
  }
}

# Run the tests
yarn test
# OR
npm test

# Install 'deepsource CLI'
curl https://deepsource.io/cli | sh

# Set DEEPSOURCE_DSN env variable from repository settings page
export DEEPSOURCE_DSN=https://sampledsn@deepsource.io

# From the root directory, run the report coverage command
./bin/deepsource report --analyzer test-coverage --key javascript --value-file ./coverage/cobertura-coverage.xml

TIP

If the coverage file is generated at a path other than the root directory, pass that path to the value-file flag of the last command.

Example :

./bin/deepsource report --analyzer test-coverage --key javascript --value-file ./src/app/coverage/cobertura-coverage.xml

# Go

# Run your tests and generate coverage report
go test -coverprofile=cover.out

# Install 'deepsource CLI'
curl https://deepsource.io/cli | sh

# Set DEEPSOURCE_DSN env variable from repository settings page
export DEEPSOURCE_DSN=https://sampledsn@deepsource.io

# From the root directory, run the report coverage command
./bin/deepsource report --analyzer test-coverage --key go --value-file ./cover.out

# Note: Tests for different go packages

If you are running tests and generating coverage report for different packages separately, you need to combine all generated coverage reports before reporting it to the test-coverage analyzer.

This is how you can combine the coverage artifacts before reporting it:

# Run tests for a package
go test -coverprofile=cover.out ./somePackage

# Append coverage data in a seperate file
cat cover.out >> coverage.out

# Run tests for another package
go test -coverprofile=cover.out ./someOtherPackage

# Combine the coverage data in the same way
cat cover.out >> coverage.out

# Install 'deepsource CLI'
curl https://deepsource.io/cli | sh

# Set DEEPSOURCE_DSN env variable from repository settings page
export DEEPSOURCE_DSN=https://sampledsn@deepsource.io


# From the root directory, run the report coverage command
./bin/deepsource report --analyzer test-coverage --key go --value-file ./coverage.out

# Python

# Python: unittests

# Install coverage.py pacakage from pip
pip install coverage

# Run coverage
coverage run tests.py

# Generate coverage report in xml format
coverage xml

# Install 'deepsource CLI'
curl https://deepsource.io/cli | sh

# Set DEEPSOURCE_DSN env variable from repository settings page
export DEEPSOURCE_DSN=https://sampledsn@deepsource.io

# From the root directory, run the report coverage command
./bin/deepsource report --analyzer test-coverage --key python --value-file ./coverage.xml

# Python: pytest

# Install pytest and pytest-cov pacakages from pip
pip install pytest pytest-cov

# Run pytest with --cov and --cov-report flags
pytest --cov=./ --cov-report xml

# Install deepsource CLI
curl https://deepsource.io/cli | sh

# Set DEEPSOURCE_DSN env variable from repository settings page
export DEEPSOURCE_DSN=https://sampledsn@deepsource.io

# From the root directory, run the report coverage command
./bin/deepsource report --analyzer test-coverage --key python --value-file ./coverage.xml

Reference: https://pypi.org/project/pytest-cov/ (opens new window)

# Python: nose2

# Install nose2 package from pip
pip install nose2[coverage_plugin]>=0.6.5

# Run nose with --with-coverage and --coverage-report flags
nose2 --with-coverage --coverage-report xml

# Install deepsource CLI
curl https://deepsource.io/cli | sh

# Set DEEPSOURCE_DSN env variable from repository settings page
export DEEPSOURCE_DSN=https://sampledsn@deepsource.io

# From the root directory, run the report coverage command
./bin/deepsource report --analyzer test-coverage --key python --value-file ./coverage.xml

Reference: https://docs.nose2.io/en/latest/plugins/coverage.html (opens new window)

# NOTE: Usage with tox

If you are using tox.readthedocs.io (opens new window) then make sure you omit .tox/* and env/* in your .coveragerc file because when running tests with tox the coverage report will contain test report files other than your source e.g site-packages etc.

example .coveragerc

[run]
branch = True
source = src
omit =
    .tox/*
    env/*

and with multiple python versions, combine the coverage data files before reporting.

example tox.ini

[tox]
envlist = cov-init,py27,py36,py37,cov-report
skipsdist=True
skip_missing_interpreters=True


[testenv]
setenv =
    COVERAGE_FILE = .coverage.{envname}
deps =
    pytest
    pytest-cov
    coverage
commands =
    pytest --cov


[testenv:cov-init]
skipsdist = True
setenv =
    COVERAGE_FILE = .coverage
deps = coverage
commands =
    coverage erase


[testenv:cov-report]
skipsdist = True
setenv =
    COVERAGE_FILE = .coverage
deps = coverage
commands =
    coverage combine
    coverage report
    coverage xml

# Ruby

# SimpleCov

First, install simplecov if not already installed.

gem install simplecov

Then, follow the steps below to generate test coverage.

  1. Add the following to spec_helper.rb file inside the tests folder:
# frozen_string_literal: true

require 'simplecov'

SimpleCov.start
  1. Add --require spec_helper.rb to .rspec file.
  2. Run rspec using bundle exec rake rspec to generate coverage.
  3. The coverage results will be available inside the coverage folder created.

Now, you can do the following to report the coverage to DeepSource:

# Install deepsource CLI
curl https://deepsource.io/cli | sh

# Set DEEPSOURCE_DSN env variable from repository settings page
export DEEPSOURCE_DSN=https://sampledsn@deepsource.io

# From the root directory, run the report coverage command
./bin/deepsource report --analyzer test-coverage --key ruby --value-file ./coverage/.resultset.json

TIP

SimpleCov generates the results in JSON format to .resultset.json. Please report the same to DeepSource.

# Java

The test coverage analyzer supports test coverage metrics for Jacoco and Clover XML reports.

# Jacoco

Setting up test coverage differs with each type of build system (Maven, Gradle, etc.). Here's an example of the configuration needed to run Jacoco on a maven repo:

<!-- Within pom.xml -->
...
    <plugin>
        <groupId>org.jacoco</groupId>
        <artifactId>jacoco-maven-plugin</artifactId>
        <version>0.8.2</version>
        <executions>
            <execution>
                <goals>
                    <goal>prepare-agent</goal>
                </goals>
            </execution>
            <!-- attached to Maven test phase -->
            <execution>
                <id>report</id>
                <phase>test</phase>
                <goals>
                    <goal>report</goal>
                </goals>
            </execution>
        </executions>
    </plugin>

...

Once you've added Jacoco to your project's pom.xml file, you should be able to run tests and generate the coverage report. The default location of the coverage report is target/site/jacoco/jacocoTestReport.xml. In case your project has multiple modules, you will need to use the jacoco:report-aggregate goal to merge all reports together.

mvn test

After you have the XML test report, you can upload it to DeepSource using the cli:

# Install deepsource CLI
curl https://deepsource.io/cli | sh

# Set the DEEPSOURCE_DSN env variable from the reporting tab of 
# your repository's DeepSource settings page.
export DEEPSOURCE_DSN=https://sampledsn@deepsource.io

# From the project's root directory, run the report coverage command
./bin/deepsource report --analyzer test-coverage --key java --value-file target/site/jacoco/jacocoTestReport.xml

You should be able to proceed similarly with other build systems such as Ant or Gradle.

Reference: Jacoco documentation (opens new window)

# Clover

DeepSource also supports reports generated using Atlassian's Clover coverage framework. Here's an example of using it with Maven:

    ...

    <build>
        <plugins>
    
            ...

            <plugin>
                <groupId>org.openclover</groupId>
                <artifactId>clover-maven-plugin</artifactId>
                <version>4.4.1</version>
                <configuration>
                </configuration>
                <executions>
                    <execution>
                        <phase>verify</phase>
                        <goals>
                            <goal>instrument</goal>
                            <goal>aggregate</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>

    ...

    <reporting>
        <plugins>
            <plugin>
                <groupId>org.openclover</groupId>
                <artifactId>clover-maven-plugin</artifactId>
                <version>4.4.1</version>
            </plugin>
        </plugins>
    </reporting>

    ...

Once you have added these elements to your project's pom.xml file, you'll be able to instrument and run tests:

mvn clean clover:setup test clover:aggregate clover:clover

This will run tests and in the case of a multimodule project, aggregate the results into a single clover report.

The default output directory for the report is target/site/clover/clover.xml.

After you have the XML test report, you can upload it to DeepSource using the cli:

# Install deepsource CLI
curl https://deepsource.io/cli | sh

# Set the DEEPSOURCE_DSN env variable from the reporting tab of 
# your repository's DeepSource settings page.
export DEEPSOURCE_DSN=https://sampledsn@deepsource.io

# From the project's root directory, run the report coverage command
./bin/deepsource report --analyzer test-coverage --key java --value-file target/site/clover/clover.xml

You should be able to proceed similarly with other build systems such as Ant or Gradle.

Reference: Clover documentation (opens new window)

# Tracking Coverage with CI

Last Updated: 4/17/2021, 6:00:13 AM

← Secrets →