Measure Code Coverage by Integration Tests with Sonar

By on October 27, 2010 » tags , ,

You probably know already that JaCoCo is the most performant code coverage engine. But you might not know that you can now combine it with Sonar to assess the code coverage by integration tests. This was the most voted Sonar issue (SONAR-613) and the latest version of the Sonar JaCoCo Plugin solves it. I am now going to explain how.

Let’s first come back to the needs :

  • There should be an easy way to assess code coverage by integration / functional / acceptance / UI tests (let’s say integration tests), whatever system is used to execute those tests : Maven plugin (maven-surefire-plugin, maven-osgi-test-plugin, maven-failsafe-plugin…), Ant script, GreenPepper, Selenium… and also whatever the package format : jar, ear, war… This can really be tricky and even a nightmare with coverage engines that use source code instrumentation (Clover) or offline bytecode instrumentation (Cobertura or Emma). Indeed an application is usually made of multiples java libraries and doing static instrumentation of each java libraries is often too complex to automate. The only suitable approach for this usage is to do on-the-fly byte code instrumentation like JaCoCo does.
  • Code coverage by unit tests and code coverage by integration tests must not be mixed as the information provided by each one is not the same. For instance, having a very good code coverage by integration tests but a poor code coverage by unit tests is a good start but not really a good news for the productivity of the development team. On the other side, having a very good code coverage by unit tests but a poor code coverage by integration tests will not prevent regressions to appear in production environment. This is why there should be a new set of dedicated metrics for integration tests.
  • For multi-modules Maven projects, there should be a way to compute overall code coverage by unit tests across modules. This case clearly lies between unit tests and integration tests but this is widely used and a lot of users have requested the feature.

The Sonar JaCoCo plugin covers the 3 needs in two steps :

  • Launch your integration tests after having configured the JaCoCo agent to be launched with the integration tests and to dump a Jacoco result file at the end of the execution
  • Configure and launch Sonar to reuse the JaCoCo result file. The Sonar plugin will extract only the required code coverage information

Let’s take an example with a Maven project containing three modules A, B and C. The C module is only used to execute integration tests with the Maven Failsafe Plugin and we would like to get the code coverage on module A and B by integration tests contained in module C. The following line must be added to the configuration of the Maven Failsafe Plugin in the pom.xml file of the C module :

<argLine>-javaagent:${jacoco.agent.path}=destfile=${jacoco.file.path}</argLine>

The integration tests on module C must then be launched with the following command line :

mvn -Djacoco.agent.path="PATH_TO_AGENT" -Djacoco.file.path="PATH_TO_DUMP" -Prun-its clean install

After having run integration tests you can execute Sonar analyzes on the overall project. Note that you still can use Clover, Cobertura or Emma to assess code coverage by unit tests together with JaCoCo for integration tests.
mvn -Dsonar.jacoco.itReportPath="PATH_TO_DUMP" sonar:sonar

And here is what you will get in Sonar in addition to code coverage measures by unit tests :

Of course, it’s possible to drilldown to the source code in order to see which lines are covered or uncovered by integration tests :

To get the code coverage by unit tests across all Maven modules, the configuration steps are pretty similar :

  • Add the argline property to your Maven Surefire Plugin configuration
  • execute mvn clean install -Djacoco.agent.path=”PATH_TO_AGENT” -Djacoco.file.path=”PATH_TO_DUMP”
  • execute mvn -Dsonar.jacoco.itReportPath=”PATH_TO_DUMP” sonar:sonar”

It is now time for you to give a try to new version of Sonar JaCoCo Plugin. We are waiting for your feedback !

  • Thanks man. It works. How long I have been waiting for this. Thanks a lot.

    By Gábor Lipták on October 28, 2010 at 2:38 pm

  • You’re welcome!

    By Evgeny Mandrikov on October 28, 2010 at 2:55 pm

  • Thanks a lot!

    I have configured a profile to test with emma (for unit tests) and jacoco (for integration tests) coverage and reuse results in sonar goal.

    Now I’m able to got daily analyze reports without redundant test run & instrumentation calls using following command on the CI server:

    mvn -Psonar clean install sonar:sonar

    Following properties was added to settings.xml:
    D:\Java\jacoco-0.4.1\lib\jacocoagent.jar
    ${user.home}
    jacoco.bin
    http://localhost:9000

    To the parent POM:
    …

    org.sonatype.maven.plugin
    emma-maven-plugin
    1.2

    …

    sonar

    reuseReports
    ${jacoco.file.directory}\${jacoco.file.name}

    To the modules with unit tests:

    sonar

    ${project.build.directory}/emma

    org.sonatype.maven.plugin
    emma-maven-plugin

    process-classes

    instrument

    ${project.build.directory}/emma/coverage.em
    ${project.build.directory}/emma

    org.apache.maven.plugins
    maven-surefire-plugin
    true

    ${project.build.directory}/emma/classes

    -Xms128m -Xmx1024m -XX:MaxPermSize=256m -Demma.coverage.out.file=${project.build.directory}/emma/coverage-0.ec

    To module with integration tests:

    sonar

    maven-clean-plugin

    ${jacoco.file.directory}\

    ${jacoco.file.name}

    maven-failsafe-plugin
    true

    -Xms128m -Xmx1024m -XX:MaxPermSize=256m -javaagent:${jacoco.agent.path}=destfile=${jacoco.file.directory}\${jacoco.file.name}

    By Andriy Plokhotnyuk on November 17, 2010 at 7:37 pm

  • Sorry…
    Xml tags was filtered out when posting…

    Please suggest how to escape xml tags when posting to see them in comments then I will repost.

    By Andriy Plokhotnyuk on November 17, 2010 at 7:43 pm

  • Hi Andriy,
    please switch to the user mailing list for such questions : http://www.sonarsource.org/support/support/

    By Evgeny Mandrikov on November 17, 2010 at 8:01 pm

  • It seems you are actually telling the failsafe plug-in to instrument the unit testing classes.

    This seems to work well when the integration tests are executing directly the code that is being instrumented.

    However my project is a web based project so we use Jetty to start a container to do our integration testing. I had to add the javaagent to jetty through the MAVEN_OPTS command to get it to instrument the code that I was actually measuring.

    Also because you have to have access to the jacocoagent.jar this doesn’t work great in build environments like Jenkins. We really need a jacoco plugin to handle the jacoco integration. I think I would like to help with that!

    By Eric on February 23, 2011 at 7:40 pm

  • Great Job!

    I was waiting for something like this…

    I will configure it with my selenium integration tests running in tomcat.

    Thanks again

    By Rafael on March 14, 2011 at 8:13 pm

  • Hi,

    Is there any maven plug-in for JaCoCo? If not, I’m looking for some guidelines to be able to develop one.

    Thanks,
    Gabi

    By gabi on March 22, 2011 at 9:25 pm

  • Hello,

    The approach here works well for library code that you can test in process. However, it is common for enterprise applications to be distributed, and the integration testing phases thus cannot happen in-process with maven.

    In my case we do a standard maven compile and unit-test, then distribute the code to a cluster of machines, start a number of server processes on each machine, and launch some unit tests with maven-surefire.

    The coverage figures need to an amalgam of
    1) client code being exercised in-process with the integration test
    2) code running across different functional server processes (data cache processes, cache query processes, business application processes, etc.
    3) multiple instances of the same processes running across a cluster of machines.

    As far as i can see, this sonar plugin can only process one dump file, if i deploy the JaCoCo agent to the client and all server processes, is there a way to have sonar combine all the dump files generated?

    Thanks,
    Dan.

    By dan on March 31, 2011 at 5:29 pm

  • Hi Dan, this is not yet possible to “aggregate” some results based on several jacoco dump files but feel free to create a JIRA ticket. Thanks, freddy.

    By Freddy Mallet on April 6, 2011 at 3:34 pm

  • Evgeny,

    Thank you for all the great work you do for the community.

    You start this blog post by saying “This was the most voted Sonar issue (SONAR-613) and the latest version of the Sonar JaCoCo Plugin solves it”. However I’m unclear how?

    Issue SONAR-613 is “Combining code coverage results of integration and unit tests for a module”. Can you please explain how your solution does the ‘combining’? I have tried using JaCoCo for my integration tests and it works very well, but everything it produces is separate to my unit test coverage.

    I would dearly love to have combined code coverage of integration tests and unit tests!

    Regards,

    Richard.

    By Richard Kennard on April 7, 2011 at 12:11 pm

  • Hi Richard,

    First of all – thanks for feedback.

    Second – better to use users mailing list for questions.

    Finally – I suppose that you talk about following issue for Sonar JaCoCo Plugin : http://jira.codehaus.org/browse/SONARPLUGINS-773 so you can vote ;)

    By Evgeny Mandrikov on April 7, 2011 at 12:21 pm

  • Hi,
    First thanks for your article.

    According to this article, “you still can use Clover, Cobertura or Emma to assess code coverage by unit tests together with JaCoCo for integration tests.”

    And with Sonar too ?

    The Sonar Emma and JaCoCo Plugin tell you to define [sonar.core.codeCoveragePlugin] in [system\settings\core]. So we have to choose one or an other.

    So how can we have both of them ?

    Thanks in advance.

    By mada on June 29, 2011 at 4:22 pm

  • Hi Mada,

    First of all – as was stated several times on this page : better to use users mailing list for questions.

    But answering on your question – see http://jira.codehaus.org/browse/SONARPLUGINS-1089

    P.S. My advice – forget other tools and use only JaCoCo ;)

    By Evgeny Mandrikov on June 29, 2011 at 11:00 pm

  • This works great indeed.

    It was a little tricky for me to find the new widget. In case anyone has the same problem, to make it visible do the following:
    - Log in as administrator.
    - Open the dashboard of your project in Sonar.
    - Click on “Configure widgets” in the top right corner.
    - Add the IT Coverage widget.

    By Martin on July 14, 2011 at 10:49 am

  • Everyone, is there any way to see a combined coverage figure from the unit and integration tests?

    By Bartosz on December 2, 2011 at 2:22 pm

  • i allways get “Coverage information was not collected. Perhaps you forget to include debug information into compiled classes?” warning on sonar jacoco sensor, but debug informations are present (checked it with javap).
    the html report states also 0% coverage.

    how does jacoco measure coverage what is 100%? in a case of separate module how can jacoco differentiate between my code that should be covered (all of it = 100%) and external libraries that are used in runtime? in my case jacoco “considers” simply all classes that are used in a session (including classes from “module a” that should be covered) but it seems that no class accounts for 100% coverage?!

    By tante anni on January 12, 2012 at 12:00 pm

  • Please use only the Sonar User mailing list to get some support tante anni. Thanks

    By Freddy Malllet on January 12, 2012 at 7:19 pm

  • I noticed that classes that have 100% coverage disappear from the list. Example: You have an integration test on the same class with less than 100%, and you click on that class, then the Coverage tab indicates that unit tests are 100% and integration tests for example 60%. But you can’t view the class metrics from the unit test perspective, because it’s not there. Remove a test from the unit tests, and here it is again.

    By Daniel on January 31, 2012 at 4:12 pm

  • Concerning the behaviour of not showing 100% coverage, I found clarification here:
    http://sonar-user.15.n6.nabble.com/it-coverage-jacoco-shows-inconsistent-wrong-data-or-misunderstanding-of-data-td3666799.html

    By Daniel on January 31, 2012 at 4:21 pm

  • Is there an issue to have this supported for cobertura?

    In my environment, I can get a cobertura report for both my unit tests and integration tests. I would like to import both sets of information to analyze separately.

    I don’t need the sonar to execute the tests, I would want to do as just providing a data report.

    Also, how is this reported and displayable as metrics. I only see one metric called Coverage. I don’t see Unit Test Coverage and Integration Coverage as separate metrics.

    By Daniel Holmes on February 16, 2012 at 5:23 pm

  • Hi Daniel, please ask your question to the Sonar User mailing list. Thanks.

    By Freddy Mallet on February 27, 2012 at 10:04 pm

  • I want to use Integration test coverage dashboard for QE test case effectiveness. i.e. I want to measure the code coverage, when QE executes manual functional test cases.
    I plan to provide instrumented build to QE engineers and QE engineers will test the application by executing manual test cases. Once the test execution is complete, i want to upload the code coverage files to Sonar and would like to see the code coverage data in Integration Test Coverage dashboard.

    By Avinash on June 6, 2012 at 7:57 am

  • Avinash, please ask your question to the Sonar User mailing list. Thanks.

    By Olivier Gaudin on March 16, 2013 at 8:02 am

Leave a reply


four × 1 =

Previous :
Next :