Troubleshooting Verilog

In this article, I look at
unit tests using two unit test harnesses that work in C. Along
the way, I will also discuss some of the common
terminology of automated unit testing. Let me start by discussing the fundamental tool, the test harness.

What Is A Unit Test Harness?

A unit test harness is a software package that allows
a programmer to express how production code should
behave. A unit test harness’s job is to provide these
capabilities:

  • A common language to express test cases
  • A common language to express expected results
  • Access to the features of the production code programming language
  • A place to collect all the unit test cases for the
    project, system, or subsystem
  • A mechanism to run the test cases, either in full
    or in partial batches
  • A concise report of the test suite success or
    failure
  • A detailed report of any test failures

I’ll shortly look at two popular harnesses for testing embedded C.They are both easy
to use and are descendants of the
xUnit family of unit test
harnesses.

First, I’ll employ Unity, a
C-only test harness. Later, I will use
CppUTest, a unit test harness
written in C++, but not requiring C++ knowledge to use.
You’ll find that the
bulk of the material in this article can be applied using
any test harness.

Here are a few terms that will come in handy while
reading this explanation:

  • Code under test is just like it
    sounds; it is the code being tested.
  • Production code is code that is
    (or will be) part of the released product.
  • Test code is code that is used
    for testing the production code and is not part of
    the released product.
  • A test case is test code that
    describes the behavior of code under test. It
    establishes the preconditions and checks that
    significant post conditions are met.
  • A test fixture is code that
    provides the proper environment for a series of test
    cases that exercise the code under test. A test
    fixture will assist in establishing a common setup and
    environment for exercising the production code.

To take the mystery out of these terms, let’s look at
a few tests for something we’ve all used:
sprintf. For this first
example, sprintf is the code under test; it is production
code.

sprintf is good for a first
example because it is a standalone function, which is the most
straightforward kind of function to test. The output
of a standalone function is fully determined by the
parameters passed immediately to the function. There
are no visible external interactions and no stored
state to get in the way. Each call to the function is
independent of all previous calls.

Unity: A C-Only Test Harness

Unity is a straightforward, small unit test harness.
It comprises just a few files. Let’s get
familiar with Unity and unit tests by looking at a
couple example unit test cases.
If you are a long-time Unity user, you’ll notice some
additional macros that are helpful when you are not
using Unity’s scripts to generate a test runner.

A test should be short and focused. Think of it as
an experiment that silently does its work when it
passes, but makes some noise when it fails. This
test checks that sprintf
handles a format spec with no format operations.

TEST(sprintf, NoFormatOperations) 
{ 
    char output[5]; 
    TEST_ASSERT_EQUAL(3, sprintf(output, "hey")); 
    TEST_ASSERT_EQUAL_STRING("hey", output); 
}

The TEST macro defines a
function that is called when all tests are run. The
first parameter is the name of a group of tests. The
second parameter is the name of the test. We’ll look
at TEST in more detail
later.

The TEST_ASSERT_EQUAL macro
compares two integers. sprintf should report that
it formatted a string of length three, and if it
does, the TEST_ASSERT_EQUAL
check succeeds. As is the case with most unit test
harnesses, the first parameter is the expected
value.

TEST_ASSERT_EQUAL_STRING
compares two null-terminated
strings.
This statement declares that
output
should contain the string
“hey”: Following
convention, the first parameter is the expected
value.

If either of the checked conditions is not met, the
test will fail. The checks are performed in order,
and the TEST will terminate on the first failure.

Notice that
TEST_ASSERT_EQUAL_STRING
could pass by accident; if the output just happened
to hold the “hey” string,
the test would pass without sprintf doing a thing. Yes,
this is unlikely, but we better improve the test and
initialize the output to the empty string.

TEST(sprintf, NoFormatOperations) 
{ 
    char output[5] = ""; 
    TEST_ASSERT_EQUAL(3, sprintf(output, "hey")); 
    TEST_ASSERT_EQUAL_STRING("hey", output); 
} 

The next TEST challenges
sprintf to format a string
with %s.

TEST(sprintf, InsertString) 
{ 
    char output[20] = ""; 
    TEST_ASSERT_EQUAL(12, sprintf(output, "Hello %s\n", "World")); 
    TEST_ASSERT_EQUAL_STRING("Hello World\n", output); 
} 

A weakness in both the preceding tests is that they do not
guard against sprintf
writing past the string terminator. The following
tests watch for output buffer overruns by filling the
output with a known value and checking that the
character after the terminating null is not changed.

TEST(sprintf, NoFormatOperations) 
{ 
    char output[5]; 
    memset(output, 0xaa, sizeof output); 
    TEST_ASSERT_EQUAL(3, sprintf(output, "hey")); 
    TEST_ASSERT_EQUAL_STRING("hey", output); 
    TEST_ASSERT_BYTES_EQUAL(0xaa, output[4]); 
} 
TEST(sprintf, InsertString) 
{ 
    char output[20]; 
    memset(output, 0xaa, sizeof output); 
    TEST_ASSERT_EQUAL(12, sprintf(output, "Hello %s\n", "World")); 
    TEST_ASSERT_EQUAL_STRING("Hello World\n", output); 
    TEST_ASSERT_BYTES_EQUAL(0xaa, output[13]); 
}

If you’re worried about
sprintf corrupting memory
in front of output, we
could always make
output a character
bigger and pass
&output[1] to
sprintf. Checking that
output[0] is still
0xaa would be a good
sign that sprintf is
behaving itself.

In C, it is hard to make tests totally fool-proof.
Errant or malicious code can go way beyond the end
or way in front of the beginning of
output. It’s a
judgment call on how far to take the tests. You
will see when we get into TDD how to decide which
tests to write.

With those tests, you can see some subtle
duplication creeping into the tests. There are
duplicate output declarations, duplicate
initializations, and duplicate overrun
checks. With just two tests, this is no big deal,
but if you happen to be
sprintf’s maintainer, there
will be many more tests. With every test added, the
duplication will crowd out and obscure the code that
is essential to understand the test case. Let’s see
how a test fixture can help
TEST
cases.

Test Fixtures in Unity

Duplication reduction is the motivation for a test
fixture. A test fixture helps organize the common
facilities needed by all the tests in one place.
Notice how TEST_SETUP and
TEST_TEAR_DOWN keep
duplication out of the
sprintf tests.

TEST_GROUP(sprintf); 

static char output[100]; 
static const char * expected; 
TEST_SETUP(sprintf) 
{ 
    memset(output, 0xaa, sizeof output); 
    expected = ""; 
} 

TEST_TEAR_DOWN(sprintf) 
{ 
} 
static void expect(const char * s) 
{ 
    expected = s; 
} 
static void given(int charsWritten) 
{ 
    TEST_ASSERT_EQUAL(strlen(expected), charsWritten); 
    TEST_ASSERT_EQUAL_STRING(expected, output); 
    TEST_ASSERT_BYTES_EQUAL(0xaa, output[strlen(expected) + 1]); 
}

The shared data items defined after the
TEST_GROUP are initialized
by TEST_SETUP before the
opening curly brace of each
TEST. The data
items comprise file scope, accessible by each
TEST and all the helper
functions. For this
TEST_GROUP, there is no
cleanup work for
TEST_TEAR_DOWN.

The file scope helper functions, expect and
given, help keep the
sprintf tests clean and low
on duplication.

In the end, it’s just plain C, so you can do what
you want as far as shared data and helper functions.
I’m showing the typical way to structure a group of
tests with common data and condition checks.

Now these tests are focused, lean, mean, and to the
point.

TEST(sprintf, NoFormatOperations) 
{ 
    expect("hey"); 
    given(sprintf(output, "hey")); 
} 

TEST(sprintf, InsertString) 
{ 
    expect("Hello World\n"); 
    given(sprintf(output, "Hello %s\n", "World")); 
} 

Notice that once you understand a specific
TEST_GROUP and have seen a
couple examples, writing the next test case is much
less work. When there is a common pattern within a
TEST_GROUP, each test case
is easier to read, understand, and evolve, as change
becomes necessary.

Installing Unity Tests

It is not evident from the example how the test
cases get run with the necessary pre- and post-processing. It’s done with another macro: the
TEST_GROUP_RUNNER. The
TEST_GROUP_RUNNER can go in
the file with the tests or a separate file. To
avoid scrolling through the file, I use a separate
file. For the two sprintf
tests written, the
TEST_GROUP_RUNNER looks
like this:

#include "unity_fixture.h" 
TEST_GROUP_RUNNER(sprintf) 
{ 
    RUN_TEST_CASE(sprintf, NoFormatOperations); 
    RUN_TEST_CASE(sprintf, InsertString); 
} 

Each test case is called through the
RUN_TEST_CASE macro.
Essentially, this RUN_GROUP_RUNNER
calls the function bodies associated with each of these macros:

TEST_SETUP(sprintf); 
TEST(sprintf, NoFormatOperations); 
TEST_TEAR_DOWN(sprintf); 

TEST_SETUP(sprintf); 
TEST(sprintf, InsertString); 
TEST_TEAR_DOWN(sprintf); 

Invoking TEST_SETUP before each TEST
means that each test starts out fresh, with no accumulated state.
TEST_TEAR_DOWN is called to clean up after each test.

Now that the tests are wired into a
TEST_GROUP_RUNNER, let’s
see how the
TEST_GROUP_RUNNERs are
called. For this last step, we have to look at
main. You will have a
main for your production
code and one, or more, for your test code. The Unity
test main looks like this:

#include "unity_fixture.h" 

static void RunAllTests(void) 
{ 
    RUN_TEST_GROUP(sprintf); 
}
 
int main(int argc, char * argv[]) 
{ 
    return UnityMain(argc, argv, RunAllTests); 
}

RUN_TEST_GROUP(GroupName) calls the function defined by
TEST_GROUP_RUNNER. Each
TEST_GROUP_RUNNER you want
to run as part of your test
main has to be mentioned in
a RUN_TEST_GROUP.
Notice that RunAllTests is passed
to UnityMain.

One unfortunate side effect of using a C-only test
harness is that you have to remember to install
each TEST into a
TEST_GROUP_RUNNER, and the
runner is invoked by calling
UnityMain. If you
forget, tests will compile, but not run — potentially
giving a false positive.

Because of this opportunity for error, the designers
of Unity created a system of code generators that
read your test files and produce the needed test
runner code. To keep the dependencies low for
getting started with Unity, I’ve opted to not use the
code-generating scripts and manually wire all the
test code.

When I discuss CppUTest later in this article, you
will see another solution to that problem. But
before doing that, let’s look at Unity’s output.