Don't Bother Testing for Null?

This appeared on Rainsberger's blog a couple of days ago

Unless I’m publishing an API for general use, I don’t worry about testing method parameters against null. It’s too much code and too many duplicate tests. Besides, I would be testing the wrong thing.
When a method receives null as a parameter, the invoker—and not the receiver—is missing a test

I think I've got to disagree with him on this one. Unless I'm misunderstanding 'general use'. It might be OK not to test for null if it's just your code calling your code (of course what happens when this gets pushed out, maybe serviceized and suddenly others are calling it?) Even if you do not guarantee the right results if the method is passed a null, I would think you should at least make sure that it doesn't blow up. If it fails, it should fail gracefully. It's true that it's not the responsibility of the receiver to make sure the invoker is calling it correctly, and I wouldn't expect a test for every conceivable possibility necessarily. But nulls happen all the time, its not unreasonable to expect a quick check. There wouldn't even be a lot of code. something like
if (someVar == null) {
    Log.error("it was null");
    return;
}
would do nicely. I do agree there is test duplication between the reciever and invoker, in that both are testing for the null condition on the same variable, but this doesn't really bother me. It seems to me there are lots of tests that have overlap or dependencies. Maybe I'm just crazy. I left a comment on the post.

Integration Tests Are a Scam?

I recently listened to a talk given by J. B. Rainsberger (author of JUnit Recipes) with the title Integration Tests Are a Scam (summary notes here). If the idea seems crazy, blame the fact that he's from Canada ;) These are some quick thoughts I had, I may expand on them later.

Here's some definitions he gives:

Basic Correctness

"Given the myth of perfect technology, do we compute the right answer?"

Myth of Perfect Technology

"Assuming we can use an arbitrary large amount of memory, for an arbitrary amount of time, on a Turing machine for spherical people[...]"

Integration Tests

"...any test whose result (pass or fail) depends on the correctness of the implementation of more than one piece of non-trivial behavior."

"You should never need to write an integration test to show basic correctness." He believes our largest problems lie in basic correctness. After we get this right, then we can worry about issues of performance, security, etc. The question of basic correctness is where he focuses his efforts. (He paraphrases a quote I believe based on the Pareto Principle).

Downsides of integration testing:

• Intergration tests are slow
• Integration tests don't tell you where the failure occurred (may be difficult to find even with debugger, assuming TDD hasn't caused you to forget how to use one)
  
• In order to have enough tests at the integration level to test thoroughly, the number of tests that need to be written increases combinatorially, based on code paths
• There is a lot of duplication in test setup

Now, it should be noted that he is not talking about acceptance tests. He says that acceptance tests tend to be end-to-end, and that is OK. But end-to-end tests should not be used for developer tests. He is also not altogether against integration tests for finding bugs, he just doesn't want them permanently added to the project. Bugs found through an integration test should create new object tests. "I don’t doubt the necessity of integration tests. I depend on them to solve difficult system-level problems. By contrast, I routinely see teams using them to detect unexpected consequences, and I don’t think we need them for that purpose. I prefer to use them to confirm an uneasy feeling that an unintended consequence lurks."

Instead, he recommends 'collaboration tests' (commonly called 'interaction tests') and 'contract tests'. By collaboration tests, he means to stubbing out or mocking the collaborators to isolate functionality and make sure all the ways it can interact with collaborators behave as expected. This is 1/2 of the work (and actually the easier 1/2). You've checked if you've asked the right questions and able provide an answer for all the responses.
The missing piece (that commonly causes people to rely on integration tests) is a misunderstanding between the interaction of piece in question and its collaborators.

The second 1/2 is 'contract tests'. The first of the two checks on the other side of the interface is whether the collaborator able to provide a response when "the star" (Class in Test CIT) asks for it (is it implemented? can it handle the request in the first place?). The second is whether the the collaborator responds in the way the CIT is expecting. "A contract test is a test that verifies whether the implementation respects the contract of the interface it implements." There should be a contract test for every case we send the collaborator and every case the collaborator might send back. Again this will using stubbing and mocking. The advantage of this approach is that you know when you have enough tests (two for each behavior). I've tried to diagram the idea thusly:

He claims that if you ask these questions between every two services and focus on basic correctness, we can be "arbitrarily confident" in the correctness. The number of tests increases additively instead of combinatorially and is easier to maintain, with less duplication, and faster to run. If something goes wrong, you are either missing a collaboration test or missing a contract test or the tests do not agree. This makes troubleshooting easier. As of yet, there is no automated way of testing that every collaboration test has a matching contract test.

When I saw the title of the talk, I initially reacted rather violently against the notion. I'm still not sure if I'm 100% behind it, but I think there are some good points raised about integration tests and their utility. However, as Dan Fabulich points out in a reply to a response Rainsberger gave to a comment about a Mars rover failure, figuring out that you are missing a test may not come easily.

"The ability to notice things" is high magic. If you have that, you can find/fix any bug without any tests... why don't we all just "notice" our mistakes when writing production code? In this case you're just using intuition to notice a missing test, but that's no easier than noticing bugs.

As you know, I share your view that integration tests are tricky, in the sense that writing one tempts you into writing two, where instead you should be writing more isolated unit tests. But unit tests have the opposite problem: once you have some unit tests, it's too easy to assume that no more testing is necessary, because your unit tests have covered everything. By exaggerating the power of unit tests and the weakness of integration tests, you may be doing more harm than good.

Imagine you're actually coding this. You just finished writing testDetachingWhileLanded and testDetachingWhileNotLanded. (It was at this point in your story that you first began to "notice" that a test was missing.) You go back over the code and find you have 100% branch coverage of the example application. Your unit tests LOOK good enough, to a superficial eye, to an ordinary mortal. But you're still missing a critical test. How are you supposed to just "notice" this?

More generally, how are you supposed to build a habit or process that notices missing tests *in general*?

I've got just the habit: write all the unit tests you can think of, and then, if you're not sure you've got enough unit tests, do an integration test. You don't even necessarily have to automate it; just try it out once, in real life, to see if it works. If your code doesn't work, that will help you find more unit tests to write. If it does work, don't integration-test every path; you were just double-checking the quality of your unit tests, after all."

<edit>
While I wouldn't go so far as calling it 'magic', finding all the edge cases can be difficult and may require a fair amount of knowledge about the collaborator. Rainsberger later commented that his method of ensuring every condition is tested is

Every time I stub a method, I say, "I have to write a test that expects the return value I've just stubbed." I use only basic logic there: if A depends on B returning x, then I have to know that B can return x, so I have to write a test for that.

Every time I mock a method, I say, "I have to write a test that tries to invoke that method with the parameters I just expected." Again, I use only basic logic there: if A causes B to invoke c(d, e, f) then I have to know that I've tested what happens when B invokes c(d, e, f), so I have to write a test for that.

Dan Fabulich suggests adding either "Every time I stub a method that can raise an exception, I have to stub it again with a test that expects the exception" or "Every time I stub a method to return X, I also have to write a test where the stub returns Y. And Z. For all possible return values of the method." Of course, it's impossible (or at least very difficult) to be sure you've gotten all edge cases.

My takeaway from all this is that integration tests are overused, often perhaps as a half-baked attempt to remedy poor unit tests (even though the two different tests try to solve different problems). While I'm not quite ready to do away with integration tests entirely (I think they provide a useful documentation of examples of use without going into the nitty gritty details of a unit test and make a nice supplement to unit tests), I think one should recognize their place: performance testing, and as general review. NOT for finding bugs or ensuring changes didn't break anything and certainly not for finding where they occurred. One should add them as a separate module that is only built when requested, or using something like the FailSafe plugin for Maven.


</edit>

One idea that he mentions early on in the talk is the idea of having only one assert per test. This is something I'm occasionally guilty of (especially if the method being tested does several things). This should be a testing smell that may indicate the need for some refactoring.

He also mentions what first got him interested in TDD, which I thought was one of the most compelling reasons I've heard so far to use TDD. When you don't use TDD you have a seemingly endlessly depressing cycle of writing tests, fixing bugs, writing more tests, and so on...how do you know when you're finished? When you do TDD, it has a bit more definitive ending point:

• Think about what you want to do
• Think about how to test it
• Write a small test. Think about the desired API
• Write just enough code to fail the test
• Run and watch the test fail. (The test-runner, if you're using something like JUnit, shows the "Red Bar"). Now you know that your test is going to be executed
• Write just enough code to pass the test (and pass all your previous tests)
• Run and watch all of the tests pass. (The test-runner, if you're using JUnit, etc., shows the "Green Bar"). If it doesn't pass, you did something wrong, fix it now since it's got to be something you just wrote
• If you have any duplicate logic, or inexpressive code, refactor to remove duplication and increase expressiveness -- this includes reducing coupling and increasing cohesion
• Run the tests again, you should still have the Green Bar. If you get the Red Bar, then you made a mistake in your refactoring. Fix it now and re-run
• Repeat the steps above until you can't find any more tests that drive writing new code
  

(from the C2 wiki)
I like that. This would help address my previously mentioned fear of knowing when you've tested everything. (Though I'm sure it's not foolproof).

Unit Testing Struggles

So, over the last 1.5 weeks or so I've been having a bit of an identity crisis with unit tests. Sure, I got the basic idea of writing tests for each method in college but like most never used it much while in school. Now that I'm at OCLC and unit testing is expected, I'm trying to develop my own philosophy on the matter and get a feel for it.

I felt like I should bring code coverage up on the project I was working on, but for purely religious reasons. Now, granted, this project is heavy in IO (it's the same project I mentioned previously). So perhaps I'd see less benefit from unit tests on this project than others, but despite the fact I now have >70% coverage (up from 0%), I haven't found a single bug by using unit tests, only with integration tests. The results might also have been different had I used TDD or my home-brewed syncretic TOD approach.

Despite these facts, I don't argue that Unit Testing used with CI can help as a tool for preventing software regressions and as up-to-date documentation for the code (though I still don't find it a very natural read except for BDD approaches like EasyB). It's also useful for finding the root cause of a problem. Whereas an integration test might only be able to say "something blew up" a unit test might be able to tell "here is what blew up". However, the value of the unit tests I've created I think remains to be seen. Meanwhile, it did deliver value as a learning platform.

Groovy has served as an excellent testing platform for me. This particular project was written in Groovy, but I think this would work well with Java projects as well (there is, however, the slight overhead of the additional dependency). I was able to do everything I wanted (still a few kinks to work out) with stubbing using the wonderful, magical, ExpandoMetaClass. There are a few tests I have yet to do where I may have to use their mocking framework.

A couple gotchas:

// getters cannot be overridden using just the property name, even though they can be called that way
class Foo {
  int bar
}

Foo.metaClass.getBar {->
  return 44
}
foo1 = new Foo()
assert foo1.bar == 44  // this passes

GroovySystem.metaClassRegistry.removeMetaClass Foo
Foo.metaClass.bar {->
  return 42
}
Foo foo1 = new Foo()
assert foo1.bar == 42  // this does not

I wanted to send some precanned input to a method that uses BufferedReader to get its input. The constructor from it eventually constructs a File to get the data. I can't extend File or create a new interface with all the File stuff for a test, because that would require modifying BufferedReader and Reader classes to match these changes. I've not found a way around this.

Another problem was a method that takes a String path of a file that contains filepaths of input files to process and adds the string and a BufferedReader to that file to a collection (not sure why that decision was made). So, I tried to mock out eachline(). But there is a problem...

// you cannot metaclass constructors, therefore, this code doesn't work. I've still got to figure out a way of faking a file, since File cannot use map coercion because it has no default constructor
String aPath = "a/path/to/file"
String fakeData = "some\nfake\nstuff\n"

File.metaClass.init {String filePath ->
  def mock = [eachLine: {return "${fakeData}"}, exists: {return true}] as File
  return mock
}

File f = new File(aPath)  // doesn't work

There are still some larger questions I have that maybe I'll never get THE ANSWER to. One question in particular I've been struggling with is "How simple is too simple to break?" The JUnit authors suggest this is a never-ending source of pain:

becomeTimidAndTestEverything
while writingTheSameThingOverAndOverAgain
    becomeMoreAggressive
    writeFewerTests
    writeTestsForMoreInterestingCases
    if getBurnedByStupidDefect
        feelStupid
        becomeTimidAndTestEverything
    end
end

And it's still very easy for me to lose sight of what I'm actually testing in the midst of all the mocking, stubbing, and so forth. More than a few times this last week I've looked down and realized that what I've written is so paranoid that it is really testing stuff that can only fail if the compiler or JVM fails or cosmic rays come down and change my data.

Just as important as making sure your tests pass is making sure they fail. I struggled with this the most when I started this process. I thought "wonderful, everything works." When it turns out that the code didn't work quite the way I thought it did and my tests were actually written in such a way that they would NEVER fail. All those green bars might not actually mean much. That't not to say they're worthless, just maybe not as valuable as you might initially think.

And this leads to my greatest fear: How do you know when something is thoroughly tested? And can some sort of confidence be associated with your tests? Clearly, code coverage doesn't cut it. I'm still new to all this, but I'm not taking much comfort from unit tests. I feel a bit better when integration tests return exactly the result I'm expecting and I test several possible scenarios. Still, even with this, you cannot test all possible scenarios and when do you know you've got enough? I guess when something blows up and you didn't find it. (>_<)

P.S. by 'Mein Kampf' I just meant the literal 'my struggle' it has nothing to do with Hitler or his work.