Where 100% Code Coverage is not sufficient

[This was originally posted at http://timstall.dotnetdevelopersjournal.com/where_100_code_coverage_is_not_sufficient.htm]

Sometimes I wish development were as easy as telling junior guys to "follow this one metric", and then they write perfect code. However, it's not. One example is how to know when you've written enough unit tests. Code Coverage is the obvious metric, and therefore "100% Code Coverage" sounds great. But there are plenty of cases where even 100% coverage doesn't do the job.

Case 1: Regular expressions

Take an email validator, something like so:

public static bool IsEmail(string s)
{
    return System.Text.RegularExpressions.Regex.IsMatch(s,
        @"\b[A-Z0-9._%+-]+@[A-Z0-9.-]+\.[A-Z]{2,4}\b");
}

A single test would give 100% coverage, but obviously there's a lot of other paths to check. Ironically, because regular expressions are often used to validate input data, and it's an in-memory operation (no databases or external files to hit), it's a prime candidate for lots of unit tests to catch all the boundary conditions - as opposed to just the 1 test needed to reach 100% coverage.

Case 2: Single-line expressions

Similar to the previous case, merely calling this method with one set of inputs (say the "less-than" path, such as i1=5 and i2=10), will get 100% coverage. But that wouldn't test the "greater-than" and "equal to" conditions.

public static bool IsGreater(int i1, int i2)
{
    return (i1 > i2);
}

Case 3: Missing Asserts (bad logic)

Even with 100% coverage, it doesn't guarantee that the method logic is correct.

For example, say you've got a CSV-parsing method:

public static string[] ParseCsvString(string strLine)
{
    string[] astr = strLine.Split(',');
    return astr;
}
 

That is tested by:

[TestMethod]
public void ParseCsvString()
{
    string[] astr = Foo.ParseCsvString("a, b, c");
}

This will give 100% coverage. However, there are no asserts, so it's really just showing that the method didn't throw an exception. Even if a developer adds an assert, they need to make sure it's asserting the right thing. Say, adding an assert that the returned array is not null, or has a length of 3, misses the logic that trims the white space after each comma. For example, we want elements like "b" (no whitespace), not " b". In other words, we'd want the ParseCsvString method to loop through each item and Trim() it.

Case 4: Mocking giving a false sense of security

Mocking Frameworks, like TypeMock, are very powerful tools for increasing unit test coverage. These tools allow you to "mock out" a method call within code, such as that database or logging call that would be hard to run in a test method.

While this is great for testing legacy code, it can easily be abused. If every line is mocked out, there's nothing real that's left to test. So while it does get high coverage, if used incorrectly, it becomes meaningless.