Over the last few days there’s been a firestorm about unit testing. Some folks dislike it (Will Shipley), and some who think the other way.

Here’s my take: you need both unit testing and a QA suite to get your application past the “QA” cycle of software development. There are some extremes (frameworks or component code on one extreme, and code that is mostly interface - or that just drives other apps - on the other), but most apps need both. Here are my views on the whole thing… First, unit testing. I use unit testing when I’m designing a complex system. A system that I might not be able to get my head around in one sitting. For example, I wrote a bunch of unit tests for a datetime class I wrote that used 10.3’s new Time APIs to implement timezone stuff. Found an bug in Apple’s code too (radr://problem/3770859). Even though my class is a relatively thin interface over these APIs, I wrote about 1,000 lines of unit tests. Date/time stuff is so complex, and testing different date/time settings so hard (imagine 20 trips to the date/time control panel…), I thought it would be good to know that my code was passing correct stuff to Apple’s APIs, and getting what I expected back. (Proving that my wrapper code is sane, Apple’s code is sane, and that my understanding of the documentation is sane.)

I also understand that you can’t unit test everything, ala GUI stuff. I once spent a summer writing VirtualUser tests - press this button, check to see that this dialog box came up, etc. Very hard to unit test that stuff - even if you wedge tests into your controller classes, you can’t be certain that the dialog box will actually appear. “Oh, I would have put up a dialog box here” isn’t the complete test - maybe you forgot to include a critical DLOG resource, or couldn’t find the proper nib to load.

So you also need user testing. A user (automated or not) needs to sit down in front of your software and try to break it. For Helix Maintenance Manager we have a set of tests that we run in addition to the unit testing performed. We have a slick system that moves test documents in and out of the system, asking the tester if specific things happened, keeping track of the failures. Hard to unit test View level stuff, you have to just run your program and see what happens.

There’s also beta testing, throwing your code our to a small community and seeing how well it flies. This often finds up major bugs, because users will use your software in New And Different (and, sometimes, Odd) ways.

For more, see bbum’s excellent post on this topic