Pick your code coverage tool in Sonar 2.2
By default, Sonar embarks two tools to calculate code coverage by unit tests on java projects : Cobertura and Clover. But last week, we also released plugins for two other coverage tools : Emma and JaCoCo. Both plugins are available in the Sonar Plugin Library. So I thought it would be a good time to compare all of them and share the results with the community.
| Clover | Cobertura | Emma | JaCoCo | |
|---|---|---|---|---|
| License | Commercial | GNU GPL | CPL | EPL |
| Latest stable release | 3.0.2 (13 April 2010) |
1.9.4.1 (3 March 2010) |
2.0.5312 (13 June 2005) |
0.4.0 (4 June 2010) |
| Type of instrumentation | Source code instrumentation | Offline bytecode instrumentation | Offline bytecode instrumentation | On-The-Fly bytecode instrumentation |
| Java | 1.4+ | 1.3+ | 1.2+ | 1.5+ |
| Line hits | yes | yes | yes | yes |
| Branch coverage | yes | yes | no | no (but planned) |
| Process within Sonar | Instrumentation Compilation Execution Report generation Report parsing |
Instrumentation Execution Report generation Report parsing |
Instrumentation Execution Data reading |
Execution Data reading |
To compare results and performance of those tools I’ve used following projects:
| Lines | Statements | Lines of code | Classes | Tests | |
|---|---|---|---|---|---|
| Sonar LDAP Plugin 0.1 | 925 | 201 | 459 | 8 | 9 |
| Struts 1.3.9 | 114621 | 21896 | 50080 | 518 | 323 |
| Commons Collections 3.3RC1 | 64447 | 12402 | 26558 | 412 | 13023 |
And here is code coverage results:
| Clover 2.6.3 | Clover 3.0.2 | Cobertura 1.9.4.1 | Emma 2.0.5312 | JaCoCo 0.4.0 | |
|---|---|---|---|---|---|
| Sonar LDAP Plugin 0.1 | |||||
| Line coverage | 91.9 | 91.9 | 88.8 | 86.7 | 88.0 |
| Branch coverage | 73.4 | 73.4 | 75.0 | N/A | N/A |
| Struts 1.3.9 | |||||
| Line coverage | 15.7 | 15.7 | 15.4 | 14.8 | 15.4 |
| Branch coverage | 14.6 | 14.6 | 12.8 | N/A | N/A |
| Commons Collections 3.3RC1 | |||||
| Line coverage | 82.5 | 82.9 | 82.1 | 81.1 | 82.4 |
| Branch coverage | 78.7 | 78.8 | 78.6 | N/A | N/A |
And finally I’ve made three measures of time of Sonar analysis for each combination of project and tool with clean Sonar database (which means 54 builds on my Intel(R) Core(TM) i3 CPU M 330 @ 2.13GHz). For this I’ve used Maven 2.2.1 with Java 1.6.0_20 and Sonar 2.2.
so now we know why lazy people pay for clover.
By raveman on October 28, 2010 at 4:16 pm
A very interesting comparison.
What do you make of the consistently lesser results from the non-clover plugins?
Which Line and Branch coverage numbers are correct?
By Simon on September 1, 2011 at 12:12 pm
Hi Simon,
> What do you make of the consistently lesser results from the non-clover plugins?
Not sure that I understand this question.
> Which Line and Branch coverage numbers are correct?
Well, you are right – precision of results was not included into this review. Maybe it’s time to do a new comparison, since one year has passed ;)
But difference can be explained by the fact that those tools use different ways to instrument and to analyse code. Indeed – results of JaCoCo, Cobertura and Emma are near, because all they use bytecode instrumentation, whereas Clover – sourcecode instrumentation. And most probably sourcecode instrumentation provides more accurate results, however comes with other limitations, which doesn’t exist in JaCoCo due to online bytecode instrumentation.
By Evgeny Mandrikov on September 5, 2011 at 7:21 pm
Is there a way to get the code coverage to report on all tests, we have junit and non-junit tests …. system and regression. We have some code that is not amenable to junit testing, but would like the coverage reports.
By Steve Ray on October 3, 2011 at 3:42 pm
Is there a way to get the dump of jacoco output file in XML format. I would like to know how to extract the code coverage output from the Jacoco output file.
By Sridharan on January 31, 2012 at 11:52 pm