23 September 2008

VSTS How To: Code Coverage for Manual Tests

I was just reading through Michael Ruminer's recent post about code coverage for Manual Tests and it got me to thinking about manual testing and code coverage.  I've often discussed code coverage in conjunction with automated testing but never with manual tests.  I guess I'll walk through the process so that I can discuss and demo the topic the next time testing and code coverage comes up with a client.

Kudos go out to the VSTS Quality Tools team for a great blog post that was the basis for a large part of this post.

There are 3 basic cases:

  1. You have the source code for the app and tests.
  2. You only have the binaries for the app but have the source code for the tests.
  3. You only have the binaries for the app and tests. 

In each case, the same steps are implemented:

  1. Instrument the DLL to test
  2. Start Manual Test run
  3. Start app
  4. Run all manual tests
  5. Stop app
  6. End Manual Test run
  7. Review Code Coverage

In each of the Cases described below we will be using the attached "Hello World" app.  This app is a simple WinForms app written in VB.Net 2008.  The basics of the app are that you enter a value in the textbox and click the Click Me! button.  The app then displays a dialog specifying if the value is an integer or not and if it is an integer, then whether the value is odd or even.  This will give us a simple code path to test and view coverage.

Here's the layout that we will use:

SolutionExplorer

Here's the method that we will be testing and retrieving coverage data.

   1:  Public Class Form1
   2:  
   3:      Private Sub Button1_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button1.Click
   4:          Dim msg As String = "The entered value is not an integer."
   5:          Dim value As Integer
   6:   
   7:          If Integer.TryParse(TextBox1.Text, value) Then
   8:              If value Mod 2 = 0 Then
   9:                 msg = String.Format("The number {0} is even.", value)
  10:              Else
  11:                  msg = String.Format("The number {0} is odd.", value)
  12:              End If
  13:          End If
  14:          MessageBox.Show(msg, "Silly little test app", MessageBoxButtons.OK, MessageBoxIcon.Information, MessageBoxDefaultButton.Button1, MessageBoxOptions.DefaultDesktopOnly, False)
  15:      End Sub
  16:   
  17:   
  18:  End Class

Case 1 - You have app and test source code

The first case is the similar to the way we implement code coverage for Unit Tests.  Instead of selecting a group of Unit Tests to run, you run a set of Manual Tests.  In our case, we have the app project and the test project.  So let's run this manual test and grab code coverage data.

1. Instrument the DLL to test

Double-click on LocalTestRun.testrunconfig and navigate to the Code Coverage tab.  In here you will be shown all of the assemblies and web apps in your project.  Check the box next to assemblies that you want to get coverage on.  Leave the Instrument assemblies in place checkbox checked (default).  Click the Close button to finish

testrunconfig

2. Start Manual Test run

Open the Test View window (Test | Windows | Test View...), select the manual tests that will be part of this test run and click the Run Tests button.

testview

This will start the code coverage service and display the first manual test in the test run.

3. Start the application being tested

You could press the F5 key to run the app but if you did you would see this dialog.

errordebug 

You can't run code coverage and debugging at the same time because instrumentation of the assembly injects probes into the call stack thus throwing off debugger when it tries to match up the stack frames with the PDB information.

To get the app running you need to press CTRL + F5 to start the app without debugging.

4. Run all manual tests

Follow the steps in each test, select whether the test passed or failed and add any comments.  Once the test is done, don't forget to click the Apply button to submit that test result.

testrunning 

5. Stop the application

Once your manual tests are all finished but before you Apply the last test's results, close the application.  You close the application to flush the coverage data that has been recorded.

6. End Manual Test run

Finish marking the last test in the run, mark it passed or failed and then click the Apply button to save the test results and finish the test run.  This will let the test run results window load with the final test data.

testrunning

7. Review Code Coverage

To open the Code Coverage window you can either select the Test | Windows | Code Coverage menu item or right-click on your test results in the Test Results windows and select Code Coverage.

testresults

This will open up the Code Coverage windows where you can drill-down into your application and view the covered and non-covered code metrics.  These include the blocks of code or the lines of code.  You can right-click within the Code Coverage Results window and select Add/Remove Columns to display the coverage metrics by Lines covered.

coverageresults

To see the source code with coverage hilighting, right-click on the method you wish to view and select Go to source code. 

gotosourcecode

This will open your source code and bring up the selected method with hilighting applied.  The code in blue was covered by the tests in this test run and the code in red was not.

coverage

 

Case 2 - you have test source code but only app binaries

In this case you are doing effectively the same steps as in Case 1 except you will need to add the assemblies to the instrumentation list in Step 1.

If you do not have access to the app's source code you will not be able to see the source code coloring.

1. Instrument the DLL to test

Double-click on LocalTestRun.testrunconfig and navigate to the Code Coverage tab.  In here you not see any assemblies, unlike in Case 1.  Click on the Add Assembly... button to open an Explorer window.  Select the assembly or assemblies to instrument.  Leave the Instrument assemblies in place checkbox checked (default). 

If the assemblies have been strong-named, you will need access to their key to re-sign them after instrumentation.  If you can't get access to the key, you can't instrument the assemblies. 

Click the Close button to finish

testrunconfig2

2 through 7

Follow the steps in Case 1.

 

 

Case 3 - You have app and test binaries but no source code

Manual tests aren't able to be run from the mstest.exe command-line tool, so this post is actually applicable for any scenario where you have app and test binaries.  Unfortunately, this means that you will need to have the manual tests printed out and available to your testers.

As we discussed, we are going to follow the same set of steps as the earlier 2 cases but we are going to use the tools in a bit of a different way. 

1. Instrument the DLL to Test

a. To instrument the DLL we need first need to copy them from wherever they are stage to a directory on the local machine.

b. Next we have to open a Visual Studio 2008 Command Prompt  (Start | All Programs | Microsoft Visual Studio 2008 | Visual Studio Tools | Visual Studio 2008 Command Prompt)

c. Change directories to the directory on the local machine use in Step 1a.

d. Run the following command for each assembly you wish to instrument
    vsinstr –coverage [YourAssemblyNameHere]

2. Start Manual Test run

To start our test run we need to spin up the code coverage monitor.  We do this by running the following command from the Visual Studio 2008 Command Prompt we opened in Step 1b.

vsperfcmd –start:coverage –output:[AMeaningfulOutputName].coverage

3. Start app

You start your application by running it from the location specified in Step 1b.

4. Run all manual tests

Here's where those copies of the manual tests come in.  In Cases 1 & 2 we were able to run the manual tests inside of Visual Studio but we can't do that if we only have test binaries. 

5. Stop app

Shut down your app the way a user normally would.

6. End Manual Test run

We need to shutdown the code coverage monitor. This will cause the code coverage data to be written out to the file that we specified on the command-line in Step 2.   Do this by running the following command in the Visual Studio 2008 Command Prompt we opened in Step 1b.

vsperfcmd –shutdown

7. Review Code Coverage

To look at the code coverage results, open the .coverage file (specified on the command-line in Step 2) in Visual Studio 2008.  If someone looking at the .coverage file has access to the source code, they will be able to see the source code coloring.

 

Sources

Running Tests with Code Coverage 
 
http://blogs.msdn.com/vstsqualitytools/archive/2005/06/08/426979.aspx

MSDN: Manual Test Overview
 
http://msdn.microsoft.com/en-us/library/ms182496(VS.80).aspx

Perform Code Coverage Analysis with .NET to Ensure Thorough Application Testing(MSDN Magazine)
 
http://msdn.microsoft.com/en-us/magazine/cc163981.aspx

MSDN - VSInstr
  http://msdn.microsoft.com/en-us/library/ms182402.aspx

 MSDN - VSPerfCmd
  http://msdn.microsoft.com/en-us/library/ms182403.aspx

1 comments:

Anonymous said...

When i do the "Case 3 - You have app and test binaries but no source code". I am seeing the following error message.
"Strong name signature could not be verified. The assembly may have been tampered with, or it was delay signed but not fully signed with the correct private key. (Exception from HRESULT: 0x80131045) "
Let me know how to proceed.

Post a Comment