Is Code Coverage a Good Idea?

Is Code Coverage a Good Idea?

These days, code coverage is really popular when it comes to assessing code quality. There are many other ways to improve code quality.  Some of them are simple to add to existing development practice, while others require more radical changes to the development process, such as test-driven development. Code coverage is one of those “simple to add” approaches. It is simple because it is completely automated, requiring a developer to configure it to collect code coverage data during the unit testing process, and voila, you have a code coverage report! You can even set up an automated email to let yourself know that coverage numbers are too low.  However, code coverage may not be enough to ensure code quality. The fact that code was covered is only a simple measure of code quality; it does not mean that it is executed correctly.  The code coverage percentage threshold can easily be used as the integration acceptance criteria, which is why many clients of ours like it. In this blog, we explore how code coverage may help us, and try to determine if it’s enough as a measure of code quality (hint: it is not! :-)).

Artemis Consulting uses tools like Sonarqube to allow us to automate testing of code by integrating it into the development process. For example, in a Java Maven project, we can configure the pom.xml to collect test coverage during test execution with JaCoCo  plug-in, and upload the coverage information to the Sonarqube server using a SonarScanner. At this point, the CI can be added into Sonarqube’s integration cycle, which will collect code analysis results. Then, you can use the Sonarqube web UI to check if your code failed the acceptance criteria and why.

So, what do you do if your unit test code failed the criteria?  Sonarqube‘s web UI has a rich set of views and graphs to demonstrate the analysis results. The web UI illustrates the unit test coverage and static analysis data, using plenty of different views, graphs, diagrams, etc. This UI allows you to see what is not covered and many, many other problems with your code that you may not be aware of. That’s because Sonarqube does much more than just test coverage. It will search through your code to find anti-patterns (called “code smells”), bugs, vulnerabilities and duplicates.    

Additionally, when it comes to the code coverage improvement, one metric that is especially useful is “Overall uncovered lines.” It shows the least covered source files. This information can be used to prioritize your effort, so you can concentrate your attention on the poorest covered files.

Do it locally

Lint tools and plugins are good during coding, but installing the Sonarqube server locally on a developer workstation or laptop is recommended to get the most accurate numbers before you check in your code. Running Sonarqube locally will save you time by getting coverage figures before code goes to CI. For the best results, you can import rules from the main Sonarqube server to your local server, enabling the same results during local unit testing and avoiding any surprises after you push your code. It’s also a good idea to use the same version of Sonarqube locally as the one on the server. 

Don’t test smelly code

Sonarqube, as well as many other tools, allows you to automatically detect problems in your code.  If you use non-compiled languages such as Python or PHP, you can still use Sonarqube.  From compiler warnings to IDEs to dedicated linters, tools are available for most languages and they can be plugged into the build process of your project. Many IDEs have integrated code analysis. Some even have assisted editing that will show you problems and sometimes possible solutions as you type. These tools are great, and they all perform what is called ‘static code analysis.’  Static analysis doesn’t need to execute the code to analyze it. It only needs the code.

Suggestions from those tools will make it easier to test and write your code. Different tools can catch different set of problems. For example, the analyzer integrated into IntelliJ IDEA is really good, and it will find many problems, but Sonarqube can find others.

It’s a good idea to listen to static analysis tools. For example, when an analyzer suggests using Java 8 style list filtering rather than explicit loops, it is best to do so. Not only is this code ugly and error prone, it introduces “for” and “if” statements, each of which create a control flow branch that is supposed to be covered. Fixing the “smells” before testing your code can make it easier in the long run. 

Still not enough

Let’s say you had a good set of unit tests that cover all business logic. You used Sonarqube to pinpoint uncovered lines and improved your tests. Now, you’ve covered every possible decision. You have a perfect data set to provide realistic input.  You assert and verify everything important. And, still you are a couple of percentages short. In this case, you will have to put extra effort to boost it up, which means you have to write an extra test just for the sake of coverage.

Primitive code

Languages, platforms and frameworks (Enterprise Java, Spring, etc.) are getting more and more advanced, and this simplifies business application development. Your algorithms now contain less technical code and more high-level business logic. This creates an interesting effect. Not that long ago, big chunks of project code were technical (persistence, data conversion and transfer, custom frameworks, etc.) This code was quite complex. Nowadays, most of this technical code has become a part of standard and popular frameworks. Consequently, projects tend to have fewer algorithms. In this context, scaffolding code (models, wrappers, exceptions, generated code, etc.) begins to take a bigger part of the code base. This code doesn’t deserve to be tested, but still counts in code coverage. In these cases, you will have to write primitive unit tests just to boost the coverage and call some methods (getter, setters, constructor, toString, equals, hashCodes, etc.). There are some tools to help with that. For example, OpenPOJO, steps into every setter and getter of all models.

Other tricks

If possible, avoid adding “if” statements to your code. Use collection streams. It will help to avoid “for” loops, filtering loops, nested loops, and searching loops. Use third party libraries, even for small things, such as null checks. Apache Commons has plenty of helpers, such as CollectionUtils.emptyIfNull(), BooleanUtils.isFalse(), etc.

Example 1: 


List<String> toList = null;<if(fromList != null) { toList = new ArrayList<>();for(String s: fromList) { if(s.startsWith("a")) { toList.add(s);} }} else { toList = new ArrayList<>();}


toList = ListUtils.emptyIfNull(fromList).stream().filter(s -> s.startsWith("a")).collect(Collectors.toList());

Example 2:

Boolean flag = null;

This will throw NPE

​if(flag) { // Do something}

This will work, but will create an uncovered “flag != null” decision

if (flag != null && flag) {    // Do something}

This will work

if (BooleanUtils.isFalse(flag)) {    // Do something}



As “simple” as it is, code coverage can be challenging to improve in satisfying the acceptance criteria. Even though code coverage is easy to automate, it still does not give us a full picture. We do not know if the code was covered in the right circumstances, if execution produced the expected results, or if it was optimal execution and so on. Using code coverage as a criterion of the code quality can be viewed as the first line of defense, but it should not be relied on solely.   Code quality should be a continuous aspect of the development cycle.  We need to make sure we have the following:

  • Code organized in testable fashion
  • Proper unit tests with smart test of data, verifications and assertions
  • Well-defined practice and discipline of code reviews

In conclusion, code coverage is always a good idea since it is easy to setup, easy to understand, relatively easy to improve, and in conjunction with other code analysis, it can give quick feedback on bad code before you go deeper into testing.  Code coverage, and more generally, static code analysis, properly integrated into development process will definitely improve the code quality.