Unit Testing Struggles

So, over the last 1.5 weeks or so I've been having a bit of an identity crisis with unit tests. Sure, I got the basic idea of writing tests for each method in college but like most never used it much while in school. Now that I'm at OCLC and unit testing is expected, I'm trying to develop my own philosophy on the matter and get a feel for it.

I felt like I should bring code coverage up on the project I was working on, but for purely religious reasons. Now, granted, this project is heavy in IO (it's the same project I mentioned previously). So perhaps I'd see less benefit from unit tests on this project than others, but despite the fact I now have >70% coverage (up from 0%), I haven't found a single bug by using unit tests, only with integration tests. The results might also have been different had I used TDD or my home-brewed syncretic TOD approach.

Despite these facts, I don't argue that Unit Testing used with CI can help as a tool for preventing software regressions and as up-to-date documentation for the code (though I still don't find it a very natural read except for BDD approaches like EasyB). It's also useful for finding the root cause of a problem. Whereas an integration test might only be able to say "something blew up" a unit test might be able to tell "here is what blew up". However, the value of the unit tests I've created I think remains to be seen. Meanwhile, it did deliver value as a learning platform.

Groovy has served as an excellent testing platform for me. This particular project was written in Groovy, but I think this would work well with Java projects as well (there is, however, the slight overhead of the additional dependency). I was able to do everything I wanted (still a few kinks to work out) with stubbing using the wonderful, magical, ExpandoMetaClass. There are a few tests I have yet to do where I may have to use their mocking framework.

A couple gotchas:

// getters cannot be overridden using just the property name, even though they can be called that way
class Foo {
  int bar
}

Foo.metaClass.getBar {->
  return 44
}
foo1 = new Foo()
assert foo1.bar == 44  // this passes

GroovySystem.metaClassRegistry.removeMetaClass Foo
Foo.metaClass.bar {->
  return 42
}
Foo foo1 = new Foo()
assert foo1.bar == 42  // this does not

I wanted to send some precanned input to a method that uses BufferedReader to get its input. The constructor from it eventually constructs a File to get the data. I can't extend File or create a new interface with all the File stuff for a test, because that would require modifying BufferedReader and Reader classes to match these changes. I've not found a way around this.

Another problem was a method that takes a String path of a file that contains filepaths of input files to process and adds the string and a BufferedReader to that file to a collection (not sure why that decision was made). So, I tried to mock out eachline(). But there is a problem...

// you cannot metaclass constructors, therefore, this code doesn't work. I've still got to figure out a way of faking a file, since File cannot use map coercion because it has no default constructor
String aPath = "a/path/to/file"
String fakeData = "some\nfake\nstuff\n"

File.metaClass.init {String filePath ->
  def mock = [eachLine: {return "${fakeData}"}, exists: {return true}] as File
  return mock
}

File f = new File(aPath)  // doesn't work

There are still some larger questions I have that maybe I'll never get THE ANSWER to. One question in particular I've been struggling with is "How simple is too simple to break?" The JUnit authors suggest this is a never-ending source of pain:

becomeTimidAndTestEverything
while writingTheSameThingOverAndOverAgain
    becomeMoreAggressive
    writeFewerTests
    writeTestsForMoreInterestingCases
    if getBurnedByStupidDefect
        feelStupid
        becomeTimidAndTestEverything
    end
end

And it's still very easy for me to lose sight of what I'm actually testing in the midst of all the mocking, stubbing, and so forth. More than a few times this last week I've looked down and realized that what I've written is so paranoid that it is really testing stuff that can only fail if the compiler or JVM fails or cosmic rays come down and change my data.

Just as important as making sure your tests pass is making sure they fail. I struggled with this the most when I started this process. I thought "wonderful, everything works." When it turns out that the code didn't work quite the way I thought it did and my tests were actually written in such a way that they would NEVER fail. All those green bars might not actually mean much. That't not to say they're worthless, just maybe not as valuable as you might initially think.

And this leads to my greatest fear: How do you know when something is thoroughly tested? And can some sort of confidence be associated with your tests? Clearly, code coverage doesn't cut it. I'm still new to all this, but I'm not taking much comfort from unit tests. I feel a bit better when integration tests return exactly the result I'm expecting and I test several possible scenarios. Still, even with this, you cannot test all possible scenarios and when do you know you've got enough? I guess when something blows up and you didn't find it. (>_<)

P.S. by 'Mein Kampf' I just meant the literal 'my struggle' it has nothing to do with Hitler or his work.