2017-04-11

Grails 3 code coverage

Intro

Last Greach I had the oportunity to talk to @sdelamo and @jeffbrown about the behavior of existing code coverage tools with Grails 3.

Tools

It’s sad but the truth is there’re not so many open source code coverage tools out there. I guess the most popular in the JVM world are cobertura and jacoco.

Cobertura

Cobertura is a free Java tool that calculates the percentage of code accessed by tests. It can be used to identify which parts of your Java program are lacking test coverage. It is based on jcoverage.

— http://cobertura.github.io/cobertura/

Jacoco

JaCoCo is a free code coverage library for Java, which has been created by the EclEmma team based on the lessons learned from using and integration existing libraries for many years.

— http://www.jacoco.org/jacoco/

Project

The structure of the project used as example is a gradle multimodule with the following modules:

api: Grails 3.2.4 application
common: Groovy 2.4.10 library for common utilities

Initial configuration

First you have to add both plugins to Gradle configuration in your file build.gradle:

cobertura

apply plugin: 'net.saliman.cobertura'

jacoco

apply plugin: 'jacoco'

Jacoco is bundled with Gradle, and it has a default version. You can change the Jacoco version. Check oficial Jacoco/Gradle documentation at https://docs.gradle.org/current/userguide/jacoco_plugin.html

Now you can execute cobertura and jacoco.

execute cobertura

./gradlew cobertura

execute jacoco

./gradlew check jacocoTestReport

In order to save time and execute both tools one after the other there’s a task (thanks to @sdelamo) that execute both sequentially:

codeCoverage task

Therefore from now on, I’ll do:

execute both

./gradlew codeCoverage

Table 1. Initial state
cobertura	jacoco

Reducing the noise

Sometimes when creating a new project there are classes that are part of the framework that we rarerly consider to test. But if we don’t test them they could ruin our coverage reports, even though we’re sure they won’t affect the application behavior.

Please note that’s under your responsibility to decide which classes are eligible to be tested or not. As a rule of thumb I would say that if you have doubts about it, you should test it.

Omiting classes

In order to get rid of classes we know for sure we don’t want to touch, or we think it doesn’t make sense to test, we can tell cobertura and jacoco to omit them from their reports.

Omiting classes with cobertura

Before showing how to omit those clases, I need to mention that all cobertura related configuration will be included in the cobertura { } configuration:

Cobertura configuration

cobertura {
  // cobertura configuration here
}

Ok, now, in order to tell cobertura to omit classes like Application or UrlMappings…etc we use the property coverageExcludes:

Adding classes exclusions

If we execute the cobertura report again, we’ll see that the excluded classes don’t appear anymore.

Executing cobertura

./gradlew codeCoverage

Omiting classes with jacoco

Well if you take a look at the documentation, you will find that the configuration attribute excludes can exclude classes from jacoco. But it seems what you’re really doing is excluding the class from the report but not from the analysis.

This part has been taken from this post: https://liviutudor.com/2016/02/11/jacoco-gradle-excluding-source-files-and-classes

In order to exclude a give class from both the analysis and the report you need to add the following to your jacoco configuration:

Inside the jacocoReport configuration:

Jacoco report configuration

jacocoTestReport {
  //...
}

Add inside the following:

Excluding classes

afterEvaluate {
    classDirectories = files(classDirectories.files.collect {
        fileTree(dir: it,
                 exclude: ['**/Application**',
                           '**/BootStrap**',
                           '**/UrlMappings**'])
    })
}

And then execute code coverage again:

./gradlew codeCoverage

Now I got rid of the noise and I can focus on the classes that are really important to my application.

Table 2. Reducing class noise
cobertura	jacoco

A simple controller

This is a simple greetings controller. It receives a name and an age from a PersonCommand command object, and eventually it will return a greetings message.

Greetings controller

package api

class GreetingsController {
    def index(PersonCommand person) {
        if (person.hasErrors()) {
            response.status = 400
            respond(message: "Please check name and age")
            return
        }

        if (person.name == 'mario') {
            respond(message: "You're awesome!")
        } else {
            respond(message: "You're not so awesome ${person.name} you look older than ${person.age}")
        }
    }
}

Next I’ll create a simple test checking one possible successful condition, in order to get some code coverage metrics.

Greetings controller Spec

void 'check a successful greetings message for mario'() {
    given: 'a proper person information'
    params.name = 'mario'
    params.age = 20

    when: 'invoking the controller'
    controller.index()

    then: 'we should get the expected result'
    response.json.message == "You're awesome!"

    and: 'the correct status code'
    response.status == 200
}

Now I’m running both cobertura and jacoco to see how covered my code is:

./gradlew codeCoverage

Table 3. Naive test coverage
cobertura	jacoco

Lets see the metrics.

Table 4. line coverage over GreetingsController.groovy
_	line coverage	branch coverage
cobertura	42% covered	18% covered
jacoco	missed 4/7 ⇒ covered 3/7 (42%)	25% covered (based on instructions)

Line coverage: If we check both reports it seems that in both reports line coverage remains the same. But due the fact that Jacoco shows percentages based on instructions makes harder to get line coverage metrics.
Branch coverage: Both differ mainly because I think Jacoco considers instructions instead of full statements.

The number in parenthesis is the number of checks done by the coverage tool in that line

Table 5. branch coverage
code	cobertura	jacoco
`if (person.hasErrors())`	50% (1/2 covered)	50% (1/2 missed)
`response.status = 400`	-	-
`respond(errors)`	-	-
`return`	0% (0/2 covered)	0% (2/2 missed)
`if (person.name == 'mario')`	100% (2/2 covered)	100% (0/2 missed)
`respond(message: "You’re awesome")`	50% (1/2 covered)	50% (1/2 missed)
`respond(message: ….)`	12% (1/8 covered)	12% (7/8 missed)

I really encourage everybody to take a look at the section: Coverage counters to have a hint on how Jacoco create its metrics.

All sum up 16 possible checks. Although it seems both tools are considering the same branches both try to show opposite views of the same event. Whereas cobertura highlights the branches covered, jacoco informs about the missing covered branches. What do I prefer ? As long as I understand what they’re describing I’m fine with both approaches because so far, both tools are giving me the same information about my code coverage.

Still:

One thing I’m missing from Jacoco report is that in Cobertura’s class detailed view you have a progress bar that gives you a quick hint about the whole class. Jacoco doesn’t have that.
Jacoco is showing all Grails generated code and takes it under consideration for the general report. That’s a lot of code I don’t need to know about, and definitely I don’t want to be testing.

While Cobertura gives you an easy way to omit certain code at method level Jacoco still doesn’t have that. You can find the discussion about this at https://github.com/jacoco/jacoco/wiki/FilteringOptions

Next step: `Nail line coverage`

I really encourage everybody to take a look at the section: Line coverage explained.

Conclusions

TODO