Section 3: Verilog Testing
In the previous discussion section, we manually ran Verilator and Icarus Verilog on the command line. This process can quickly become quite tedious and error prone. The key to being a productive hardware designer is to automate as much of the process as possible. In this discussion section, you will learn how to automate the process of linting your designs, building test simulators for your designs, and testing the functionality of your design with these test simulators. We will also learn about a variety of different kinds of testing you can use to help verify the functionality of your hardware designs.
1. Logging Into ecelinux with VS Code
Follow the same process as previous discussion sections. Find a free workstation and log into the workstation using your NetID and standard NetID password. Then complete the following steps (described in more detail in the last discussion section):
- Start VS Code
- Install the Remote-SSH, Verilog, and Surfer extensions
- Use View > Command Palette to execute Remote-SSH: Connect Current Window to Host...
- Enter
netid@ecelinux-XX.ece.cornell.eduwhere XX is anecelinuxserver number - Install the Verilog extension on the server
- Use View > Explorer to open your home directory on
ecelinux - Use View > Terminal to open a terminal on
ecelinux
There is no need to fork the repo for today's discussion section. Simple clone the repo as follows.
% source setup-ece2300.sh
% mkdir -p ${HOME}/ece2300
% cd ${HOME}/ece2300
% git clone git@github.com:cornell-ece2300/ece2300-sec03-verilog-testing sec03
% cd sec03
% tree
The repo includes the following files:
Makefile.in: Makefile for the build systemconfigure: Configure script for the build systemconfigure.ac: Used to generate the configure scriptscripts: Scripts used by the build systemece2300: ECE 2300 unit testing library and miscellaneous macrosptd/ptd.mk: Tells build system about files in this sectionptd/PairTripleDetector_GL.v: Verilog for pair/triple detectorptd/PairTripleDetector2_GL.v: Verilog for two pair/triple detectorsptd/test/PairTripleDetector_GL-test.v: test cases for pair/triple detectorptd/test/PairTripleDetector2_GL-test.v: test cases for two pair/triple detectorptd/sim/detector2-sim.v: interactive simulator for hardware module
Note that unlike last week, this week we have organized our files into
subdirectories. The ptd subdirectory has all of the code for the
pair/triple detector including hardware designs, test benches, and
interactive simulators.
To make it easier to cut-and-paste commands from this handout onto the
command line, you can tell Bash to ignore the % character using the
following command:
Now you can cut-and-paste a sequence of commands from this document and
Bash will not get confused by the % character which begins each line.
2. A Simple Makefile
We will start by writing a very simple Makefile to automate our work from the last discussion section. Let's remind ourselves how to lint, build, and run a test simulator for the pair/triple detector from the last discussion section.
% cd ${HOME}/ece2300/sec03
% verilator -Wall --lint-only ptd/PairTripleDetector_GL.v
% iverilog -Wall -g2012 -o PairTripleDetector_GL-test ptd/test/PairTripleDetector_GL-test.v
% ./PairTripleDetector_GL-test
Let's now remove the test simulator so we are back to a clean directory.
Manually performing these steps can be very tedious. We can use a new
tool called make which was specifically designed to help automate the
process of running a sequence of programs. The key to using make is
developing a Makefile. A Makefile is a plain text file which contains
a list of rules which together specify how to execute commands to
accomplish some task. Each rule has the following syntax:
A rule specifies how to generate the target file using the list of
prerequisite files and the given Linux command. make is smart enough to
know it should rerun the command if any of the prerequisites change, and
it also knows that if one of the prerequisites does not exist then it
needs to look for some other rule to generate that prerequisite first. It
is very important to note that make requires commands in a rule to start
with a real TAB character. So you should not type the letters <TAB>,
but you should instead press the TAB key and verify that it has
inserted a real TAB character (i.e., if you move the left/right arrows
the cursor should jump back and forth across the TAB). This is the only
time in the course where you should use a real TAB character as opposed
to spaces.
Let's create a simple Makefile to automate the process of linting an
building a test simulator. Use VS Code to create a new file named
Makefile.
Enter the following content into the Makefile:
PairTripleDetector_GL-test : ptd/PairTripleDetector_GL.v ptd/test/PairTripleDetector_GL-test.v
<TAB>verilator -Wall --lint-only ptd/PairTripleDetector_GL.v
<TAB>iverilog -Wall -g2012 -o PairTripleDetector_GL-test ptd/test/PairTripleDetector_GL-test.v
clean:
<TAB>rm -rf PairTripleDetector_GL-test
We can use the newly created Makefile like this:
make will by default use the Makefile in the current directory.
make takes a command line argument specifying what you want "make". In
this case, we want to make the PairTripleDetector_GL-test simulator.
make will look at all of the rules in the Makefile to find a rule
that specifies how to make the PairTripleDetector_GL-test simulator. It
will then check to make sure the prerequisites exist and that they are
up-to-date, and then it will run the command sequence specified in the
rule for PairTripleDetector_GL-test. In this case, those commands are
verilator and iverilog. make will output to the terminal every
command it runs, so you should see it output the command line which uses
verilator to lint the design and iverilog to build the
PairTripleDetector_GL-test simulator.
Try running make again:
make detects that the prerequisite (i.e., PairTripleDetector_GL.v and
PairTripleDetector_GL-test.v) have not changed and so it does not
rebuild the test simulator. Now let's try making a change in the
PairTripleDetector_GL.v source file; change the final OR gate to a NOR
gate.
Now rebuild and rerun the test simulator like this:
make will automatically detect that the prerequesite has changed and
rebuild the test simulator appropriately. This ability to automatically
track dependencies and rerun just what is necessary is a key benefit of
using a tool like make. Makefiles can also include targets which are
not actually files. Our example Makefile includes a clean target
which will delete any generated simulators. Let's clean up our directory
like this:
3. Automated Build System
So far we have been doing all of our work in the root of our repository. We will need a better way to organize our work as we start to develop larger and more complicated hardware modules. As you have already seen, each "subproject" will be in its own subdirectory and that subdirectory will organized as follows:
subproject: subdirectory for Verilog hardware design filessubproject/test: subdirectory for Verilog test simulator filessubproject/sim: subdirectory for Verilog interactive simulator files
For this section the subproject name is ptd which stands for
pair/triple detector. In Lab 1, the subproject name is lab1. From now
on, we will always be building all of our simulators in a separate
build directory. Keeping simulator executables, trace files, and
waveforms separate from the source code is critical to maintaining a
clean project. It enables us to easily do a "clean build". We can simply
trash the build directory and start from scratch. You should never
commit and push your build directory to GitHub!
We provide you a sophisticated Makefile you will be using in the
remaining lab assignments. The Makefile is generated through a
configuration step that checks to make sure your environment is setup
correctly. Let's go ahead and create a build directory, run configure,
and look at the provided Makefile.
You do not need to understand everything in the Makefile, but just
appreciate that it will be a powerful way to automate the process of
linting, building, and running our simulators.
3.1. Composing Two Pair/Triple Detectors
Let's implement a more complicated Verilog hardware module that composes
two instances of our pair/triple detector called PairTripleDetector2.
It has the following block diagram.
The PairTripleDetector2 has two three-bit input ports and one one-bit
output port. It should output a one if either of the input ports contain
two or three ones and output a zero otherwise. We need to use wire
slicing on each three-bit input port to slice out each bit and connect
it to each input port of the PairTripleDetector. Go ahead and use VS
Code to implement PairTripleDetector2.
3.2. Testing Child Modules
What strategy should we use to test our new composition? The wrong thing
to do is to immediately start testing the composition. Instead, we must
test and thoroughly verify every child module before doing any kind of
integration testing. Let's use the provided Makefile to lint, build,
and run the test simulator for the child pair/triple detector.
You can build and run the test in a single line like this:
The && bash operator enables running multiple commands on the same
command line. Remember you can "zoom-in" to run a single test case with
the +test-case command line option.
% cd ${HOME}/ece2300/sec03/build
% ./PairTripleDetector_GL-test +test-case=1
% ./PairTripleDetector_GL-test +test-case=2
When you zoom-in the test framework will displace a trace of the inputs
and outputs for the design-under-test. You can output waveforms using the
+dump-vcd command line option and then open them using the Surfer
extension for VS Code.
% cd ${HOME}/ece2300/sec03/build
% ./PairTripleDetector_GL-test +test-case=2 +dump-vcd=waves.vcd
% code waves.vcd
3.3. Basic Testing
Now that we know the child modules are correct, we can do integration testing of the composition. Run the basic tests that we provide for this new composition.
% cd ${HOME}/ece2300/sec03/build
% make PairTripleDetector2_GL-test && ./PairTripleDetector2_GL-test +test-case=1
3.4. Directed Testing
In the previous discussion section, we used exhaustive testing to verify
our pair/triple detector. Exhaustive testing is a powerful technique but
only applicable to hardware modules with just a few input ports each with
just a few bits. The number of exhaustive checks scales exponentially
with the number of bits across all input ports. For example, if our
hardware module has four eight-bit input ports, we would need 2^(8*4)
or four billion checks for exhaustive testing!
We can use directed testing in situations where exhaustive testing is
not possible. In directed testing, we focus on including checks for as
many interesting input values as possible. We usually organize directed
testing into many test cases, where each test case is focusing on a
different kinds of input values we are interested in testing. For
PairTripleDetector2, we might want a directed test case that focuses on
input values with a many ones and a different directed test case that
focuses on input values with few ones. Go ahead and add appropriate
checks to test_case_3_few_ones in PairTripleDetector2_GL-test.v.
Then rebuild and rerun the test simulator using make.
% cd ${HOME}/ece2300/sec03/build
% make PairTripleDetector2_GL-test && ./PairTripleDetector2_GL-test
Zoom-in on each of these two test cases to see the trace output.
% cd ${HOME}/ece2300/sec03/build
% ./PairTripleDetector2_GL-test +test-case=2
% ./PairTripleDetector2_GL-test +test-case=3
3.5. Random Testing
Directed testing is a good first step, but how do we know if we have covered all of the corner cases? We can use random testing to help increase our confidence that we have not missed any important corner cases. Random testing involves generating random input values, determing the correct output given these random input values, applying these random input values to the design-under-test, and then verifying that the design-under-test produces the correct output values.
Here is an example of a random test case for our PairTripleDetector2
module:
logic [2:0] random_a;
logic [2:0] random_b;
logic random_out;
int random_a_num_ones;
int random_b_num_ones;
task test_case_4_random();
t.test_case_begin( "test_case_4_random" );
// Generate 20 random input values
for ( int i = 0; i < 20; i = i+1 ) begin
// Generate a 3-bit random value for both a and b
random_a = 3'($urandom(t.seed));
random_b = 3'($urandom(t.seed));
// Calculate the number of ones in random value a
random_a_num_ones = 0;
for ( int j = 0; j < 3; j = j+1 ) begin
if ( random_a[j] )
random_a_num_ones = random_a_num_ones + 1;
end
// Calculate the number of ones in random value b
random_b_num_ones = 0;
for ( int j = 0; j < 3; j = j+1 ) begin
if ( random_b[j] )
random_b_num_ones = random_b_num_ones + 1;
end
// Calculate the correct output value
random_out = (random_a_num_ones > 1) || (random_b_num_ones > 1);
// Apply the random input values and check the output value
check( random_a, random_b, random_out );
end
t.test_case_end();
endtask
Recall that Verilog has two subsets: the synthesizable subset is used
to actual real hardware hardware, while the non-synthesizable subset is
used to test hardware. Very different constructs should be used when
designing vs. testing hardware. Here we are clearly using Verilog
constructs (e.g., $urandom, for loops, etc) that are not suitable for
modeling hardware.
The random test generates two random 3-bit values using the $urandom
system task. We pass in t.seed to ensure our random tests are actually
"pseudo-random", i.e., the test case will always produce the same
sequence of random values. This way if a random test case fails we can
rerun the test case to debug it without worrying that the failure will go
away due to different random input values. We then include Verilog
behavioral code to determine the correct answer; we use two for loops to
count the number of ones in each random 3-bit value. Once we know the
number of ones in each random 3-bit value, we can determine what the
correct output should be for the design-under-test.
Copy and paste this new test case into PairTripleDetector2_GL-test.v.
Then add this new test case to the list of test cases in the initial
block like this:
initial begin
t.test_bench_begin( `__FILE__ );
if ((t.n <= 0) || (t.n == 1)) test_case_1_basic();
if ((t.n <= 0) || (t.n == 2)) test_case_2_many_ones();
if ((t.n <= 0) || (t.n == 3)) test_case_3_few_ones();
if ((t.n <= 0) || (t.n == 4)) test_case_4_random();
t.test_bench_end();
end
Now rebuild and rerun the test simulator. Verify that your new random test case is running.
% cd ${HOME}/ece2300/sec03/build
% make PairTripleDetector2_GL-test && ./PairTripleDetector2_GL-test
% ./PairTripleDetector2_GL-test +test-case=4
You should see output like this:
../ptd/test/PairTripleDetector2_GL-test.v
test_case_4_random
0: 101 011 > 1
1: 010 110 > 1
2: 001 010 > 0
3: 000 101 > 1
4: 101 100 > 1
5: 010 001 > 0
6: 101 100 > 1
7: 111 010 > 1
8: 111 010 > 1
9: 000 001 > 0
10: 101 111 > 1
11: 011 000 > 1
12: 111 010 > 1
13: 101 111 > 1
14: 111 110 > 1
15: 100 000 > 0
16: 110 000 > 1
17: 100 101 > 1
18: 101 000 > 1
19: 100 110 > 1
It is always useful to try and break your code to make sure your test
cases really are testing what you think they are. Replace the OR gate in
PairTripleDetector2_GL with an AND gate. Do your directed tests catch
this bug? If your directed tests do not catch a bug but your random
tests do catch a bug, then you want to add a new directed test case
that tests for this behavior. In fact, adding a new directed test case to
reproduce a bug caught during randoming testing is the best way to start
debugging a random test case failure. If the bug was caught by your
random test, but not your directed test cases, go ahead and add a new
directed test case that will catch this bug.
3.6. Running All Tests
The provided Makefile provides a check target which will build and
run all test simulators in a single step.
Note that this should only be used to produce a high-level overview of
which test simulators are passing and which test simulators are failing.
You should not just always type make check. You should "zoom-in" and
build and run a single test simulator for which ever hardware module you
are currently working on. Periodically run make check to get a
high-level overview.
4. Interactive Simulators
Once we have finished testing our hardware design, we can use an
interactive simulator to experiment with the design. The source files for
our interactive simulators will be in the sim subdirectory. Take a look
at the provided ptd-sim.v interactive simulator.
You can build the simple interactive simulator by simply specifying it as a target:
Then you can specify the input values for a and b on the command
line.
% cd ${HOME}/ece2300/sec03/build
% ./ptd-sim +a=000 +b=000
% ./ptd-sim +a=001 +b=000
% ./ptd-sim +a=000 +b=001
% ./ptd-sim +a=111 +b=111
Note that an interactive simulator is not for testing. Using an interactive simulator is neither systematic nor automatic. One should only start experimenting with an interactive simulator once a hardware module has been thoroughly tested using a combination of basic, exhaustive, directed, and/or random testing!
5. To-Do On Your Own
We have pushed the automated build system into your Lab 1 repos, so let's go ahead and give it a try!
where as always, netid is your NetID. Now take a look at the files we
have added to your repo:
You should see the following new files:
Makefile.in: Makefile for the build systemconfigure: Configure script for the build systemconfigure.ac: Used to generate the configure scriptscripts: Scripts used by the build systemece2300: ECE 2300 unit testing library and miscellaneous macroslab1/lab1.mk: Tells build system about files in this lab
Let's start by creating a build directory and configuring the Makefile.
Now we can build and run the Lab 1 Part A tests using the automated build system.
% cd ${HOME}/ece2300/netid/build
% make BinaryToSevenSegUnopt_GL-test
% make BinaryToBinCodedDec_GL-test
% make DisplayUnopt_GL-test
% ./BinaryToSevenSegUnopt_GL-test
% ./BinaryToBinCodedDec_GL-test
% ./DisplayUnopt_GL-test
It can be convenient to build and run a test using a single command line like this so you can just use the up-arrow key to quickly rebuild and rerun a test.
% cd ${HOME}/ece2300/netid/build
% make BinaryToSevenSegUnopt_GL-test && ./BinaryToSevenSegUnopt_GL-test
You can build and run all of the test for Lab 1 Part A with the check target like this:
You can build and run the interactive simulator like this:
Hopefully can now appreciate the power of an automated build system, but if necessary, you can also run commands explicitly to debug any issues.