Blogblog 


03

November '05

Unit Testing and Code Coverage with Xcode

Introduction

We all know unit tests suck, right? However, I think they suck in the same way that writing design documents, requirements, or test plans suck. They’re all annoying to write. However, every one of them is valuable, no matter how you produce them (OmniOutliner, Excel, TextEdit, a napkin, your arm). It’s not glamorous to think of every possible execution path — but if you fail to plan, you’re planning for failure. Here I will give you a quick overview of how (and why) I write unit tests, and how to use gcov to make your unit testing more effective. Updated on 2005-11-03. See bottom of article for a list of changes.

Motivation

Many of us independent developers work in our spare time. We have to treat that spare time with respect, and fire on all cylinders during our coding sessions. I know that I, for one, do not want to waste my time monkey-testing my code after making changes to underlying frameworks. So, I decided to spend some time building a better monkey to do the job for me.

Perhaps I should rephrase that last sentence. I decided to find as many pre-built monkey parts to make the most efficient monkey possible. This monkey would make the perfect companion to my development efforts. Also, this monkey would be commanded to do his (or her — I hear she-monkeys make great testers, too) dirty work every time I build my code. The pre-built monkey parts are OCUnit (now bundled with the Xcode developer tools), and gcov (bundled with Xcode 2.1 and later).

Now, because we only have our spare time to build our robo-monkey, we have to accept that the monkey will never be perfect. We cannot expect that our monkey will find every flaw in our application, and we cannot expect that the monkey will guarantee 100% defect-free code. However, I guarantee you that the monkey will do a very good job of watching your back during the development process.

Frameworks

In FuzzMeasure, I decided to start working with frameworks for the next release. I relocated my math routines to SMUGFoundation.framework, so that I could run a plethora of sanity tests after every build. For FuzzMeasure 1.3, I took care of this using separate command-line applications, but that got very annoying to maintain over time. Also, when tests are hard to add, then we simply don’t add them.

I suggest heading over to Wolf’s site, where you can watch his awesome tutorial to help you embed your own frameworks into your Xcode project. As a starting point, rip out any code from your application that you could imagine re-using in another application, should you ever finish this one…

OCUnit

We already know how to write unit tests with OCUnit. I put this section in so that you could catch up if you haven’t already.

Note that the process is streamlined in Xcode 2.x. All you have to do is add a new Target to your framework project, and select Unit Test Bundle which you’ll find listed under the Cocoa targets. I’ll let you tinker to get this part running first. It’s pretty easy.

I’d like to provide some advice about deciding what tests to write. Surely, it’s important that you catch the edge cases (test your minimums, maximums, and outside your bounds). However, it’s important that you also exercise your knowledge about what you’re writing.

For example, how does one test Fast Fourier Transforms? I know how the FFT works, so I should be sure to test some standard behavioral edge cases. So, a unit impulse results in a flat spectrum, and a flat spectrum results in a unit impulse. Note that you have to use STAssertEqualsWithAccuracy for these sorts of tests with floating-point values.

Another suggestion is to try and write test cases that behave like your UI will. If your UI contains, say, a library with a selection of books, and you allow users to delete one, multiple consecutive, and multiple random entries, then you want to ensure your model classes can handle those deletions. This really isn’t rocket science, so I don’t need to speak too much about this.

gcov

Don’t start here until you have working unit tests, or you really know what you’re doing and feel like skipping ahead.

Create a new build configuration called “Coverage” to sit alongside your “Debug” and “Release” configurations, if you have them. I suggest copying the “Debug” target, as it has optimizations turned off already.

Now, double-click the framework target (mine’s called SMUGFoundation) and select the Build tab. Select the “Coverage” configuration from the “Configuration” popup button to make sure we don’t trash the other configurations.

Do the following:

  • Check “Generate Test Coverage Files”
  • Check “Instrument Program Flow”
  • Add “-lgcov” to “Other Linker Flags”
At this point, your have instrumented your framework to support the auto-generation of code coverage data. When you run your unit test using the framework built in this manner, it will generate a dump of what lines of been hit (and how many times).

We can now move on to modifying the test target to actually give you the coverage reports. I chose to have gcov dump its output to the build window, just after the unit test output. Also, the results of gcov will be written to sourcefile.m.gcov in a coverage subdirectory alongside your build subdirectory, both which lie in your project directory.

If you expand the unit test target’s disclosure triangle, you should see the build phases. Double-click the Run Script build phase, which currently runs your unit tests. It should contain the following text:

 # Run the unit tests in this test bundle. "${SYSTEM_DEVELOPER_DIR}/Tools/RunUnitTests" 

Modify it to include the following text, changing SMUGFoundation with your own framework’s name. This’ll look ugly, but it should copy/paste okay from Safari.

 # Run gcov on the framework getting tested if [ "${CONFIGURATION}" = 'Coverage' ]; then     FRAMEWORK_NAME=SMUGFoundation     FRAMEWORK_OBJ_DIR=${OBJROOT}/${FRAMEWORK_NAME}.build/${CONFIGURATION}/${FRAMEWORK_NAME}.build/Objects-normal/${NATIVE_ARCH}     mkdir -p coverage     pushd coverage     find ${OBJROOT} -name *.gcda -exec gcov -o ${FRAMEWORK_OBJ_DIR} {} \;     popd fi 

And that’s it! Build your unit test target with the “Coverage” configuration, and it should now output the overview of the coverage in each individual file. Yes, it’s likely to include a few shocks. I also have a few files getting 0% coverage right now, too. The monkey’s only as effective as the instructions you give him, so it’s up to you to increase testing.

<

p> If you have problems (missing .gcno files, maybe), try doing a clean and build on the framework using the “Coverage” configuration. If you’re reading this, you’re a developer and can probably figure out what’s going wrong. If you still have trouble, drop me a line.

To figure out where to increase coverage, open the coverage subdirectory in your project directory. Open one of the files using your favourite text editor (or, feel free to transcribe them to your arm — I’ll wait). The number listed in the left column indicates how many times that line of code got executed. If you see hashes, then that means the line has never been hit.

By increasing code coverage, you can have increased confidence in your automated testing. More than anything else, you will have a method for measuring how well your unit tests cover your framework code. While increasing your code coverage numbers, you will become more proficient at writing good unit tests. As you’ll learn quickly, you can never get to 100%. For example, the return nil; in an if ( ![super init] ) block may never execute.

Conclusion

I think that without code coverage, writing unit tests can be a little bit unrewarding. Without any sort of feedback about what you’re doing, you may never know how good your tests are.

Of course, code coverage isn’t perfect. If you design things incorrectly to start with, you can’t save yourself with code coverage and unit tests. However, upping your code coverage will save you from the nasty little bugs that will inevitably come back to haunt you later. For me, code coverage and unit testing is worth the extra effort.

Update (2005-11-03)

Removed the check for a specific “Library Search Paths”, as it doesn’t take into account multiple architectures and/or compilers. Just keep in mind that you need gcc4 and later to get -lgcov.

Update (2005-11-01)

Changed ${SYMROOT} to ${OBJROOT} in the script, because although they’re the same on my setup, they’re not the same for everyone. “The SYMROOT and OBJROOT build settings identify the build locations for build products and intermediate build files, respectively,” according to this link. In the case of code coverage, all your .gcno files sit alongside the intermediate .ob build files.

12 Comments

Chad

Over a year ago

Good article thanks! :)


neal

Over a year ago

this is a great article, relevant today still. i would like to point out that the NATIVE_ARCH in the build script should be CURRENT_ARCH (NATIVE_ARCH returns i386 even when building an x86_64 binary!)


Michael

Over a year ago

Great tip, cheers.

I added a bit of text processing to make the output a little more readable:

find “${OBJROOT}” -name .gcda -exec gcov -o “${OBJ_DIR}” {} \; | egrep “^File|^Lines” | sed -E “s@File ‘$SRCROOT/@@;s@(.[a-zA-Z])’@\1: @;s@Lines executed:([0-9.%]+) of ([0-9]+)@\1 (\2)@” | paste -d” ” – - | sed -E “s@^([^:]+):([^(])(([^)]+))@\2:\1\3@” | sort -n | sed -E “s@^([^:]+):([^(]*)(([^)]+))@\2:\1\3@”

Rather complicated looking command line, but it produces output like the following, sorted by coverage percentage (lowest first):

Classes/TSClassFile.m: 0.82% (243) Classes/TSSomeOtherFile.m: 0.89% (676)


Michael

Over a year ago

Oh yes, the other thing: I originally got build errors after following these directions:

vproc_transaction_begin”, referenced from: __gcov_init in libgcov.a(_gcov.o) _vproc_transaction_begin$non_lazy_ptr in libgcov.a(_gcov.o) “_vproc_transaction_end”, referenced from: _gcov_exit in libgcov.a(_gcov.o) _vproc_transaction_end$non_lazy_ptr in libgcov.a(_gcov.o) ld: symbol(s) not found

Turns out (http://lists.apple.com/archives/xcode-users/2009/Sep/msg00066.html), the active SDK has to be the same as the current OS; setting it to 10.6 resolved the problem.


Blaine

Over a year ago

I have overcome all problems with setting up gcov on my iPhone project but now I am getting the following:

profiling:/Users:Cannot create directory

Does anyone have any suggestions as to what I might be doing wrong?

I originally setup gcov following Google’s instructions: http://code.google.com/p/coverstory/wiki/UsingCoverstory

I am not doing unit testing at this stage. I am planning on doing GUI-based testing using a tool similar to Sikuli (an awesome tool developed by MIT, check it out at sikuli.org)


Ryan

Over a year ago

Love the monkey references. Keep up the good work :)


Add New Comment

Post your comment

Anti-Spam Quiz: