Unit Testing and Code Coverage with Xcode

Introduction We all know unit tests suck, right? However, I think they suck in the same way that writing design documents, requirements, or test plans suck. They're all annoying to write. However, every one of them is valuable, no matter how you produce them (OmniOutliner, Excel, TextEdit, a napkin, your arm). It's not glamorous to think of every possible execution path — but if you fail to plan, you're planning for failure. Here I will give you a quick overview of how (and why) I write unit tests, and how to use gcov to make your unit testing more effective. Updated on 2005-11-03. See bottom of article for a list of changes. Motivation Many of us independent developers work in our spare time. We have to treat that spare time with respect, and fire on all cylinders during our coding sessions. I know that I, for one, do not want to waste my time monkey-testing my code after making changes to underlying frameworks. So, I decided to spend some time building a better monkey to do the job for me. Perhaps I should rephrase that last sentence. I decided to find as many pre-built monkey parts to make the most efficient monkey possible. This monkey would make the perfect companion to my development efforts. Also, this monkey would be commanded to do his (or her — I hear she-monkeys make great testers, too) dirty work every time I build my code. The pre-built monkey parts are OCUnit (now bundled with the Xcode developer tools), and gcov (bundled with Xcode 2.1 and later). Now, because we only have our spare time to build our robo-monkey, we have to accept that the monkey will never be perfect. We cannot expect that our monkey will find every flaw in our application, and we cannot expect that the monkey will guarantee 100% defect-free code. However, I guarantee you that the monkey will do a very good job of watching your back during the development process. Frameworks In FuzzMeasure, I decided to start working with frameworks for the next release. I relocated my math routines to SMUGFoundation.framework, so that I could run a plethora of sanity tests after every build. For FuzzMeasure 1.3, I took care of this using separate command-line applications, but that got very annoying to maintain over time. Also, when tests are hard to add, then we simply don't add them. I suggest heading over to Wolf's site, where you can watch his awesome tutorial to help you embed your own frameworks into your Xcode project. As a starting point, rip out any code from your application that you could imagine re-using in another application, should you ever finish this one… OCUnit We already know how to write unit tests with OCUnit. I put this section in so that you could catch up if you haven't already. Note that the process is streamlined in Xcode 2.x. All you have to do is add a new Target to your framework project, and select Unit Test Bundle which you'll find listed under the Cocoa targets. I'll let you tinker to get this part running first. It's pretty easy. I'd like to provide some advice about deciding what tests to write. Surely, it's important that you catch the edge cases (test your minimums, maximums, and outside your bounds). However, it's important that you also exercise your knowledge about what you're writing. For example, how does one test Fast Fourier Transforms? I know how the FFT works, so I should be sure to test some standard behavioral edge cases. So, a unit impulse results in a flat spectrum, and a flat spectrum results in a unit impulse. Note that you have to use STAssertEqualsWithAccuracy for these sorts of tests with floating-point values. Another suggestion is to try and write test cases that behave like your UI will. If your UI contains, say, a library with a selection of books, and you allow users to delete one, multiple consecutive, and multiple random entries, then you want to ensure your model classes can handle those deletions. This really isn't rocket science, so I don't need to speak too much about this. gcov Don't start here until you have working unit tests, or you really know what you're doing and feel like skipping ahead. Create a new build configuration called “Coverage” to sit alongside your “Debug” and “Release” configurations, if you have them. I suggest copying the “Debug” target, as it has optimizations turned off already. Now, double-click the framework target (mine's called SMUGFoundation) and select the Build tab. Select the “Coverage” configuration from the “Configuration” popup button to make sure we don't trash the other configurations. Do the following: <ul> <li>Check “Generate Test Coverage Files”</li> <li>Check “Instrument Program Flow”</li> <li>Add “-lgcov” to “Other Linker Flags”</li> </ul> At this point, your have instrumented your framework to support the auto-generation of code coverage data. When you run your unit test using the framework built in this manner, it will generate a dump of what lines of been hit (and how many times). We can now move on to modifying the test target to actually give you the coverage reports. I chose to have gcov dump its output to the build window, just after the unit test output. Also, the results of gcov will be written to sourcefile.m.gcov in a coverage subdirectory alongside your build subdirectory, both which lie in your project directory. If you expand the unit test target's disclosure triangle, you should see the build phases. Double-click the Run Script build phase, which currently runs your unit tests. It should contain the following text: <pre> # Run the unit tests in this test bundle. "${SYSTEM_DEVELOPER_DIR}/Tools/RunUnitTests" </pre> Modify it to include the following text, changing SMUGFoundation with your own framework's name. This'll look ugly, but it should copy/paste okay from Safari. <pre> # Run gcov on the framework getting tested if [ "${CONFIGURATION}" = 'Coverage' ]; then FRAMEWORK_NAME=SMUGFoundation FRAMEWORK_OBJ_DIR=${OBJROOT}/${FRAMEWORK_NAME}.build/${CONFIGURATION}/${FRAMEWORK_NAME}.build/Objects-normal/${NATIVE_ARCH} mkdir -p coverage pushd coverage find ${OBJROOT} -name *.gcda -exec gcov -o ${FRAMEWORK_OBJ_DIR} {} \; popd fi </pre> And that's it! Build your unit test target with the “Coverage” configuration, and it should now output the overview of the coverage in each individual file. Yes, it's likely to include a few shocks. I also have a few files getting 0% coverage right now, too. The monkey's only as effective as the instructions you give him, so it's up to you to increase testing. If you have problems (missing .gcno files, maybe), try doing a clean and build on the framework using the “Coverage” configuration. If you're reading this, you're a developer and can probably figure out what's going wrong. If you still have trouble, drop me a line. To figure out where to increase coverage, open the coverage subdirectory in your project directory. Open one of the files using your favourite text editor (or, feel free to transcribe them to your arm — I'll wait). The number listed in the left column indicates how many times that line of code got executed. If you see hashes, then that means the line has never been hit. By increasing code coverage, you can have increased confidence in your automated testing. More than anything else, you will have a method for measuring how well your unit tests cover your framework code. While increasing your code coverage numbers, you will become more proficient at writing good unit tests. As you'll learn quickly, you can never get to 100%. For example, the return nil; in an if ( ![super init] ) block may never execute. Conclusion I think that without code coverage, writing unit tests can be a little bit unrewarding. Without any sort of feedback about what you're doing, you may never know how good your tests are. Of course, code coverage isn't perfect. If you design things incorrectly to start with, you can't save yourself with code coverage and unit tests. However, upping your code coverage will save you from the nasty little bugs that will inevitably come back to haunt you later. For me, code coverage and unit testing is worth the extra effort. Update (2005-11-03) Removed the check for a specific “Library Search Paths”, as it doesn't take into account multiple architectures and/or compilers. Just keep in mind that you need gcc4 and later to get -lgcov. Update (2005-11-01) Changed ${SYMROOT} to ${OBJROOT} in the script, because although they're the same on my setup, they're not the same for everyone. “The SYMROOT and OBJROOT build settings identify the build locations for build products and intermediate build files, respectively,” according to this link. In the case of code coverage, all your .gcno files sit alongside the intermediate .ob build files.