Unit Testing FRC C++ with CppUTest in Eclipse

Introduction

Participating as a programming mentor in this year's FIRST Robotics Competition has been a lot of fun, but I found myself wishing, once again, for better ways to test our robot's code. The existing methods are the traditional direct tests on the actual robot hardware, and simulation using Gazebo. Directly testing the robot is often difficult to even get started with, since the robot is not constructed for much of the 6-week build season. Once it is constructed, though, deploying builds and manually working the controls to verify functionality is slow and error-prone. Working with the simulator also has its own shortcomings. A realistic SolidWorks model is required to drive in a virtual arena, and the simulator itself consumes a lot of processor power and memory.
  Fortunately, there is another way, and that way is unit-testing. Unit tests are small, simple tests written in code (often in the same language as your production code) that set up inputs, exercise some part of the production code, and verify the outputs that are produced. These tests can run directly on the programming computer, no robot hardware required, and execute very quickly. For Java and C++, there are several great frameworks available for free to make the process of writing unit tests easier. In this post, I will describe how the CppUTest framework can be used to make unit tests for a C++ robot program based on WPILib. CppUMock, which is a part of CppUTest, is also used to create a mock version of WPILib that can be precisely controlled in the unit tests. Example code based on the 'Arcade Drive' sample project is available at GitHub.
  Please note that this strategy does not require any modifications to your existing robot code. The only change to your project will be an additional build configuration that allows it to be tested this way. Switching between the default configuration and unit-testing configuration (described below) is easy and fast.

Requirements

The following utilities are required to develop FRC programs with CppUTest.

1. Eclipse with C/C++ development plugins (eclipse.org)
2. CppUTest (github.com/cpputest/cpputest) (release 3.7.2 was used for this example)
3. MockWPILib from the example code (unless you want to write your own mocks)
4. Unix development tools (if developing on Windows, use Cygwin (cygwin.com)
1. autoconf
2. autogen
3. automake
4. make
5. gcc-core
6. gcc-g++
7. libtool

Building CppUTest

The first step is to acquire the CppUTest sources either from a release package or the git repository. If you're using Cygwin, the sources should be placed somewhere in your Cygwin installation path (ex: C:\cygwin64\home\user). Then, using a bash shell (or Cygwin Terminal), navigate to the CppUTest root directory and run the following commands, in order:

1. ./autogen.sh
2. ./configure
3. make

If you have all of the packages listed in the requirements section, this should generate the files libCppUTest.a and libCppUTestExt.a in the lib/ subdirectory. If the process is interrupted with errors, then one or more packages are probably missing and should be installed before continuing.

Building WPILib Mocks

The next step is to use CppUMock (included in CppUTest) to begin writing the mock classes you need for your tests. These will be located in a separate project in the same workspace as your main robot project. To create your own mocks project, select 'File > New > C++ Project' and set the project type to 'Static Library > EmptyProject'. For Windows users, set the toolchain to 'Cygwin GCC'.

Click 'Next' to get to the 'Select Configurations' window. Then, click on 'Advanced Settings' to go to the Tool Chain Editor (the Tool Chain Editor for an existing project can be accessed in it's Properties window).

Make sure you're in the 'C/C++ Build > Settings' section (in the menu on the left). Under 'Compiler > Includes', add an include path that points to the 'include' subdirectory of your CppUTest installation (ex: C:\cygwin64\home\user\cpputest\cpputest-3.7.2\include). Click 'Apply' at the bottom of the window.

For Windows users, go to the 'Environment' section (select 'C/C++ Build > Environment' in the menu on the left). There should be an environment variable with the name 'CYGWIN_HOME' with an empty value. Set the variable to the path of your Cygwin installation (ex: C:\cygwin64). Leave the other environment variables unchanged.

Click 'OK' at the bottom to leave the properties window and click 'Finish' to create your mock library project. You may now begin writing the code for your mocks. To learn how to create mock classes, see the CppUMock documentation at the CppUTest site (cpputest.github.io/mocking_manual.html). For examples, see the MockWPILib project. In addition to mock classes, the START_ROBOT_CLASS macro should be defined as empty (#define START_ROBOT_CLASS ()).

  If users want to use and extend the MockWPILib project instead of creating their own mocks project, it can be acquired from GitHub and imported into the Eclipse workspace. To import a project, select 'File > Import > General > Existing Projects into Workspace'.

The build properties explained above should be checked after importing the project to make sure that the correct include path for CppUTest is set. If Cygwin is being used, the Cygwin installation environment variable should be set appropriately as well.

Robot Project Configuration

The next step is to add a build configuration to the main robot project that uses the mock library instead of the real WPILib library. First, go to the project's properties (right-clock on the project and select 'Properties'). In the 'C/C++ Build' section, click on 'Manage Configurations' on the right. In the 'Manage Configurations' window, click 'New' and give a name and description for the new configuration. Copy the settings from the 'Debug' configuration.

Click 'OK' to create the configuration, and then select it and click 'Set Active' in the 'Manage Configurations' window. Click 'OK' to go back to the properties window.

In the properties window, go to 'C/C++ Build > Tool Chain Editor' and set 'Current toolchain' to 'Linux GCC' for Linux users and 'Cygwin GCC' for Windows users. For Windows users, the Cygwin installation directory needs to be set as explained above in 'Building WPILib Mocks'.

After setting the toolchain, go to the section 'C/C++ Build > Settings'. Under 'C++ Compiler > Includes' add an include path for the mock library.

In the 'Project References' section, check the mock library project to indicate that the main robot project now references it.

Click 'OK' to exit the properties window. The main robot project can now be built with the new testing configuration. Be sure that all necessary mock classes and functions are defined in the mock library project to avoid errors. To switch the build configuration to link in the actual WPILib and deploy to the roboRIO, right click on the projects and select 'Build Configurations > Set Active > Debug'.

Writing Tests

Now that the mock library is written and the robot project is configured for unit testing. the unit tests themselves can be written. These tests will reside in a separate project. Create a new project by clicking 'File > New > C++ Project'. Set the project type to 'Executable > Empty Project'. Windows users should set the toolchain to 'Cygwin GCC'.

Click 'Next' and select 'Advanced Settings' to go the properties window. For Windows users, the Cygwin installation path should be set as explained in the section 'Building WPILib Mocks'.

  In the section 'C/C++ Build > Settings', include paths for CppUTest, the mock library, and the main robot project should be added under 'C++ Compiler > Includes'.

In the same section under 'C++ Linker > Libraries', library names and search paths should be added for CppUTest, the mock library, and the main robot project. The library for the main robot project will be called 'FRCUserProgram' and is located in the project's subdirectory for the testing configuration. For CppUTest, both the CppUTest and CppUTestExt libraries should be linked.

After the compiler and linker are configured, go to the 'Project References' section and check both the main robot project and mock library project. Then, click 'OK' to exit the properties window, and click 'Finish' to create the project.

The unit tests can now be written in the new tester project. To learn how to write unit tests with CppUTest, see the documentation on the CppUTest site (cpputest.github.io/manual.html). For examples, see the 'Arcade Drive Test' project on GitHub. Remember to define the 'main' function with a call to 'CommandLineTestRunner::RunAllTests'.

  Once the unit tests are written and built, the tester project can be executed to run the tests. Right-click on the tester project and select 'Run As > Local C/C++ Application'. The test results should show up in the console in Eclipse. If a test fails, the exact location of the failing assertion will be given.

  If an error concerning a missing library appears when running the tests on Windows, then the Cygwin libraries may be missing from the dynamic linker search path. To fix this, go to the 'Run/Debug Settings' section in the project properties. Select the executable in the launch configurations list and click 'Edit' on the right. In the 'Edit Configuration' window, select the 'Environment' tab and click 'New' to add an environment variable. Set the variable name to 'PATH' and the value to the 'bin' subdirectory of the Cygwin installation (ex: C:\cygwin64\bin). Click 'OK' and make sure that 'Append environment to native environment' is selected at the bottom.

Click 'OK' to exit the 'Edit Configuration' window. Click 'OK' to exit the properties window. The tester project should now be able to run.

Introducing a Learning and Prototyping Framework for FRC Software Development

For the past year, I have been volunteering as a mentor for a robotics team at a nearby high school. This team has focused on building robots for the FIRST Robotics Competition for several years. The competition runs every year from January to April. For the first six weeks, each participating team builds a robot, roughly the size of a dishwasher, to compete in a game against robots from other teams. The game changes every year and can range from fast-paced basketball-like matches to precision-intensive construction challenges.
   The latter game type was the one that I had the pleasure of working on with the team for the first time. For the six weeks of build time plus extra hours spent between our competitions, I helped the students and other mentors develop the electronics and software that made up the control system. Since this was a project targeted for high-school students, a lot of that involved taking pre-built modules provided by approved vendors and fitting them together into a system that allowed a pair of human operators to drive the robot.
   Though the process of constructing this control system was relatively easy to manage, my small control systems group and I struggled with a problem that I have encountered before in previous projects: a lack of prototyping and testing techniques. The control system is firmly centered around a large, clunky, expensive controller board called the roboRIO of which our team had only two: one for the robot and one for backup. Destruction or loss of either one of these boards would be a huge blow. In addition, the software utilities and development tool suite were limited to Windows, which in my opinion is a terrible platform for learning how to program. These are only a couple of the factors that I believe resulted in our group having to wait weeks until the robot's hardware was finished before the software could be properly tested. All through the build season I yearned to break free of this suffocating environment.
   After the robot was completed and the team returned from the competitions, I eventually found the time to ponder an alternative development system that would be suitable for prototyping robot code as well as teaching new students the basics of programming. I started with one basic requirement: that code written for the normal system should be able to run unchanged on the new system. That meant re-implementing the main control system C++ API, WPILib. An additional constraint that I adopted was that the robot's hardware should be relatively cheap and easy to modify and expand. After considering a number of educational robotic kits, I settled on the RedBot by SparkFun. The reasons for this choice include the platform's huge popularity, a plethora of plug-and-play sensors and actuators, Arduino compatibility, and affordability. The fact that it's design files have been open-sourced is another significant benefit.
   After nearly three months of sporadic work, I'm finally ready to present an initial status report. Most of the time has been spent on the new implementation of WPILib. It is currently developed only for Linux, though a Windows port is at the top of the to-do list. At this point, only a small fraction of WPILib's interfaces are available for use, including digital output and input and driving capabilities. A lot of work went into the underlying architecture that runs the user's program on a PC base-station while communicating with the robot via a minimal serial protocol. The robot itself runs a simple control loop written in the Arduino language that gets packets from the base-station, pokes the necessary hardware, and spits packets back. More technical details on this architecture as well as plans for future development will soon follow in another post.

Building an Instruction Set Simulator to Unit Test Assembly Code

In my last post, I described a basic technique for using the AVR simulator Simulavr to perform unit testing on assembly code. The code under test along with a minimal entry function were assembled by avr-gcc and loaded into Simulavr. The tests themselves were written in Tcl since Simulavr features an appropriate interface. I prefer to write unit tests in higher-level languages, anyways. Other cool features that Simulavr offers are a GDB interface to step through troublesome functions and cycle-level accuracy.

   However, I didn't feel like that method could be used for much besides the very basic use cases I demonstrated in my examples. Simulavr only simulates a subset of AVR devices and its Tcl interface is not documented very well. In addition, I found Tcl's built-in unit testing framework awkward to use. These reasons plus a bit of Not Invented Here Syndrome led me to try putting together my own simulator. Because this project's primary purpose is to facilitate basic assembly code testing, cycle-level accuracy is not necessary. Leaving that out greatly simplifies the implementation.

   I chose Python as the primary language because I am already familiar with it and it has a pretty good unit testing framework in the unittest package. Its object-oriented programming features also make it easy to build the simulator, which is currently made up of the following modules:

• System: Encapsulates the registers, program counter, and stack of a device.
• Instructions: Each instruction has its own class which derives from a base class that contains its location within the program. The classes each have their own unique execution methods that operate on a given instance of a system.
• Program: Contains a collection of instruction objects and map of labels to addresses. There are also functions here to build the instruction objects given an assembly code listing.

  An implementation of this basic simulator along with some example assembly code, unit tests, and helper scripts are present here. The test-driven development flow used can be described as follows:

1. Write some tests in a Python class that is derived from 'unittest.TestCase'.
2. Implement the desired functionality in a new assembly function.
3. Assemble the function into an unlinked object.
4. Dump the contents of the object into a simple ASCII-based format that contains only labels and instructions in hexadecimal (this is accomplished with avr-objdump and an Awk script).
5. Parse the listing and build a Program object.
6. Build a System object and initialize it so that the given function runs a certain way.
7. Use the System object to run the Program.
8. Analyze the System object to determine if the function ran as expected. If it did, the unit test passes.

  The example demonstrates the above flow with a function that adds the values in two registers and stores the result in a third register. The given unit test initializes the System's registers with two known values and compares the value of the third register to the expected sum to determine success.

   The framework was relatively quick and easy to code up, but only a handful of instructions have been implemented so far. In addition, there is no ability to perform branching or subroutine calls. After those are taken care of, I want to build some more advanced testing functionality, like a method for mocking out specified subroutines. And, of course, this framework should be ported to and tested with different architectures, like MSP430.

Unit Testing Assembly Code

So I've been continuing to play around with AVR assembly and struggling to get things working. With my first full-time job underway, I've had limited time to devote to this hobby, and I don't have all of the tools that I should have to properly test my prototype systems. However, I still have had the nagging suspicion that I was approaching the development process the wrong way. I finally realized what was missing while listening to an episode of Elicia White's Making Embedded Systems podcast that discusses test-driven development (TDD) for embedded systems. At first, I simply refused to believe that TDD and bare-metal programs were compatible, but the more I thought about it, the more I understood that the process of modularizing your code and running each module one at a time in a specially-configured environment was almost exactly the same.
   That podcast, though, only discussed the semantics of testing C programs, which are easily compiled for almost any platform out there (as long as there aren't any platform-specific elements in your program, which should be discouraged). Assembly programs, though, are the least portable of all. A stable hardware environment with debugging support or an architectural simulator is required. Fortunately, Simulavr is available for AVR programs (though it only supports a limited subset of all devices). In addition, its developers have implemented a nice Tcl interface for it, which makes the process of writing unit tests much easier.
   I have started a simple project to implement and test the above concepts. The files can be found on GitHub. Downloading the files (as well as installing Tcl, Simulavr, and all of their prerequisites) and running 'make test' launches the testing process. As of this writing, it is still not completely functional due in part to the lack of documentation for the simulator interface, but some more experimentation should produce a system that is able to test individual assembly routines by providing arguments and testing outputs via core registers. If it becomes robust enough, I can hopefully merge it into my other projects to help with their development.

Update: AVR-LCD Assembly Project

Well, my holiday break has been mostly uneventful, besides the fact that I received a Raspberry Pi (Model A) as a gift, so I'm looking forward to getting into that soon. In the meantime, though, I've finally had the chance to dig back into the AVR-LCD assembly project.
   After a couple weeks of on-and-off work, I'm pretty much at exactly the same place. Except that I've managed to switch the compiler to avr-gcc. Exciting! For a while, the conversion broke the code in ways that confounded me until I stopped being dumb and actually took a look at the listing produced by the compiler, which converts the register defines into plain hexadecimal. Apparently, avr-gcc doesn't do you the favor of subtracting the IO-memory-space-offset from special function register addresses used with IO-specific instructions while the Atmel compiler does. When using IN, OUT, SBI, CBI, etc., the _SFR_IO_ADDR() macro can be used to subtract the offset. Once that was inserted to all of the appropriate places, the program worked properly. Hello again, world!
   The online avr-libc manual became a valuable resource for me during the conversion by providing code samples and a Makefile template. I also took some time to play around with Simulavr. Since that simulator only supports a limited number of AVR devices (not including the ATMega328P), I needed to make my assembly compatible with the ATMega328. Fortunately, all this required was replacing the CALL instructions with RCALL. With the help of this page, I was then able to put together a command to run the simulator and output a VCD file to verify the functionality. The command is recorded in a script that is up on the project's GitHub page (avr-gcc branch) along with the updated assembly source and Makefile.

Making an Automatic Test Pattern Generator

Another semester is over, which means I have at least a  few weeks to work on my personal projects before being thrown back into the meat grinder. There is already a large heap of unfinished projects that deserve more progress (see previous posts), but a new problem has entered my mind that stubbornly refuses to be ignored. This is a bonus project that I never had time to work on from one of the classes that I just finished, Fault Detection in Digital Circuits. The objective is to write a program for automatic test pattern generation (ATPG). The high-level flow will look something like the diagram below.

High-level flow for ATPG, by Dr. Jia Wang

   The input to the program is to be a circuit netlist. For this development, circuits from the ISCAS89 benchmark are being used. Some of these designs are pretty huge, so it's important that this program be fast and efficient. Another thing to consider is that these designs contain sequential elements (flip-flops), which can really complicate the process since the state of the circuit has to be considered. For now, I'm assuming that all registers are scan-enabled, which means that inputs and outputs can be serially read and loaded directly. This enables ATPG and simulation techniques to be used for the combinational part of the circuit.

   The thing that's really giving me headaches at this point is extracting the functional model of a circuit from the structural model, which is the provided netlist. I can most likely achieve this by constructing a reduced ordered binary-decision diagram (ROBDD) using an algorithm explained in Algorithms for VLSI Design automation by Gerez. Once that's done though, it needs to be exclusive-OR'd with a copy of the circuit with errors inserted. Then it needs to be converted to conjunctive normal form (CNF) so it can be analyzed by a SAT solver (in this case, MiniSAT) for the actual test pattern generation. And I haven't even started thinking about fault simulation yet.

   Suddenly, that AVR LCD library is looking a lot more attractive.

AVR-LCD Assembly Project

So much for the two-week posting schedule. Let's see if we can fix that.
  I was actually holding off on another post until I had something of my own up and running. Well, it turns out that hardware is hard, so this seemingly simple project that I started at the beginning of the summer didn't reach a notable milestone until today. The milestone in question is one that is familiar to any programmer.

  The breadboard pictured above contains an Atmel AVR ATMega328P microcontroller (top) and a NewHaven Display NHD‐0216K1Z‐FL‐YBW LCD module (bottom). The goal of this project was to program the AVR with an assembly program to control the LCD module to print "Hello, World!" to the display. To the right of the AVR on the breadboard is a 7-segment LED display and BCD-converter chip that I started playing around with after being stymied by the LCD module for a few weeks.
   It's taken me a ridiculously long time to reach such a simple goal, but I've learned quite a bit about the AVR's internals, assembly syntax, and the Atmel Studio development environment in the process. One byproduct of this work is a Python script I wrote to convert output from Atmel Studio's built-in simulator to VCD format so data nearly any internal register can be collected and plotted as a digital waveform.
   The assembly code, simulation script, and schematic files for the test-bench circuit in the photo is available at this GitHub page. The included README contains some tips for setting up the development environment in Windows, such as using the simulator, converting the output, and programming the AVR with a USBTinyISP.
   My next steps will be to port this project to AVR-GCC, as Linux is my preferred development environment. SimulAVR seems like a promising replacement for the simulator in Atmel Studio, which was difficult to interface with external signals. Afterwards, I'll either work on reorganizing the code into a library that can be included with future assembly projects or add more code so that the functions can be exposed via a bus interface, so that a single AVR can be addressed by several other modules on a network to print out diagnostic data. I get excited about the possibilities just typing about it. I'm so easily amused.
   Oh, and another program to display a countdown on the 7-segment display is available here.