Announcing Visual Studio Team System 2010

Microsoft has 'officially' named the Visual Studio Team System Code-name 'Rosario' suite of tools to "Visual Studio Team System 2010" in an announcement on 29-Sep-2008.  They have also posted information on MSDN and Channel 9.

Here are 2 of the items that are important to me and my clients of the many announced:

• The VSTS Developer and Database Editions are being combined into a single product so you no longer have to buy Team Suite for all of your coders that also write stored procs and mantain databases.  The separation of these 2 tools has been one of the barriers to bringing the database into Version Control with some of my clients.  As Brian Harry puts it:

"Another key announcement is that we will be combining the Team Development product and the Team Database product into a single Team Development product (essentially giving you 2 for the price of 1).  That change will be in effect go into effect on 10/1/08 (the day after tomorrow)."  - Brian Harry

• VSTS 2010 allows the developer to see the state of the test system when the test failed.  This was internally called "TiVo for Debuggers" which pretty much gives a nice, clear picture of what it does.  Removing "Can't Reproduce" from a team's vocabulary is a big win in itself.

In Brian Harry's blog post he also stated that there would be another CTP this fall.  Yippee!

Here are some links to the announcements, feature sets, goodies and articles:

Microsoft PressPass - Microsoft Unveils Next Version of Visual Studio and .NET Framework

Brian Harry's Blog - Shining the Light on Rosario

MSDN - Visual Studio Team System 2010 Overview

MSDN - Visual Studio 2010 and the .Net Framework 4.0 Overview

Channel 9 - Norm Guadagno: Announcing Visual Studio Team System 2010 

Channel 9 - Visual Studio Team System 2010 Week on Channel 9!

Information Week - Microsoft Details Visual Studio 2010, .Net 4.0

CNet News - Visual Studio 2010 to come with 'black box'

VSTS How To: Code Coverage for Manual Tests

I was just reading through Michael Ruminer's recent post about code coverage for Manual Tests and it got me to thinking about manual testing and code coverage.  I've often discussed code coverage in conjunction with automated testing but never with manual tests.  I guess I'll walk through the process so that I can discuss and demo the topic the next time testing and code coverage comes up with a client.

Kudos go out to the VSTS Quality Tools team for a great blog post that was the basis for a large part of this post.

There are 3 basic cases:

1. You have the source code for the app and tests.
2. You only have the binaries for the app but have the source code for the tests.
3. You only have the binaries for the app and tests. 

In each case, the same steps are implemented:

1. Instrument the DLL to test
2. Start Manual Test run
3. Start app
4. Run all manual tests
5. Stop app
6. End Manual Test run
7. Review Code Coverage

In each of the Cases described below we will be using the attached "Hello World" app.  This app is a simple WinForms app written in VB.Net 2008.  The basics of the app are that you enter a value in the textbox and click the Click Me! button.  The app then displays a dialog specifying if the value is an integer or not and if it is an integer, then whether the value is odd or even.  This will give us a simple code path to test and view coverage.

Here's the layout that we will use:

Here's the method that we will be testing and retrieving coverage data.

   1:  Public Class Form1

   2:  

   3:      Private Sub Button1_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button1.Click

   4:          Dim msg As String = "The entered value is not an integer."

   5:          Dim value As Integer

   6:   

   7:          If Integer.TryParse(TextBox1.Text, value) Then

   8:              If value Mod 2 = 0 Then

   9:                 msg = String.Format("The number {0} is even.", value)

  10:              Else

  11:                  msg = String.Format("The number {0} is odd.", value)

  12:              End If

  13:          End If

  14:          MessageBox.Show(msg, "Silly little test app", MessageBoxButtons.OK, MessageBoxIcon.Information, MessageBoxDefaultButton.Button1, MessageBoxOptions.DefaultDesktopOnly, False)

  15:      End Sub

  16:   

  17:   

  18:  End Class

Case 1 - You have app and test source code

The first case is the similar to the way we implement code coverage for Unit Tests.  Instead of selecting a group of Unit Tests to run, you run a set of Manual Tests.  In our case, we have the app project and the test project.  So let's run this manual test and grab code coverage data.

1. Instrument the DLL to test

Double-click on LocalTestRun.testrunconfig and navigate to the Code Coverage tab.  In here you will be shown all of the assemblies and web apps in your project.  Check the box next to assemblies that you want to get coverage on.  Leave the Instrument assemblies in place checkbox checked (default).  Click the Close button to finish

2. Start Manual Test run

Open the Test View window (Test | Windows | Test View...), select the manual tests that will be part of this test run and click the Run Tests button.

This will start the code coverage service and display the first manual test in the test run.

3. Start the application being tested

You could press the F5 key to run the app but if you did you would see this dialog.

 

You can't run code coverage and debugging at the same time because instrumentation of the assembly injects probes into the call stack thus throwing off debugger when it tries to match up the stack frames with the PDB information.

To get the app running you need to press CTRL + F5 to start the app without debugging.

4. Run all manual tests

Follow the steps in each test, select whether the test passed or failed and add any comments.  Once the test is done, don't forget to click the Apply button to submit that test result.

 

5. Stop the application

Once your manual tests are all finished but before you Apply the last test's results, close the application.  You close the application to flush the coverage data that has been recorded.

6. End Manual Test run

Finish marking the last test in the run, mark it passed or failed and then click the Apply button to save the test results and finish the test run.  This will let the test run results window load with the final test data.

7. Review Code Coverage

To open the Code Coverage window you can either select the Test | Windows | Code Coverage menu item or right-click on your test results in the Test Results windows and select Code Coverage.

This will open up the Code Coverage windows where you can drill-down into your application and view the covered and non-covered code metrics.  These include the blocks of code or the lines of code.  You can right-click within the Code Coverage Results window and select Add/Remove Columns to display the coverage metrics by Lines covered.

To see the source code with coverage hilighting, right-click on the method you wish to view and select Go to source code. 

This will open your source code and bring up the selected method with hilighting applied.  The code in blue was covered by the tests in this test run and the code in red was not.

 

Case 2 - you have test source code but only app binaries

In this case you are doing effectively the same steps as in Case 1 except you will need to add the assemblies to the instrumentation list in Step 1.

If you do not have access to the app's source code you will not be able to see the source code coloring.

1. Instrument the DLL to test

Double-click on LocalTestRun.testrunconfig and navigate to the Code Coverage tab.  In here you not see any assemblies, unlike in Case 1.  Click on the Add Assembly... button to open an Explorer window.  Select the assembly or assemblies to instrument.  Leave the Instrument assemblies in place checkbox checked (default). 

If the assemblies have been strong-named, you will need access to their key to re-sign them after instrumentation.  If you can't get access to the key, you can't instrument the assemblies. 

Click the Close button to finish

2 through 7

Follow the steps in Case 1.

 

 

Case 3 - You have app and test binaries but no source code

Manual tests aren't able to be run from the mstest.exe command-line tool, so this post is actually applicable for any scenario where you have app and test binaries.  Unfortunately, this means that you will need to have the manual tests printed out and available to your testers.

As we discussed, we are going to follow the same set of steps as the earlier 2 cases but we are going to use the tools in a bit of a different way. 

1. Instrument the DLL to Test

a. To instrument the DLL we need first need to copy them from wherever they are stage to a directory on the local machine.

b. Next we have to open a Visual Studio 2008 Command Prompt  (Start | All Programs | Microsoft Visual Studio 2008 | Visual Studio Tools | Visual Studio 2008 Command Prompt)

c. Change directories to the directory on the local machine use in Step 1a.

d. Run the following command for each assembly you wish to instrument
    vsinstr –coverage [YourAssemblyNameHere]

2. Start Manual Test run

To start our test run we need to spin up the code coverage monitor.  We do this by running the following command from the Visual Studio 2008 Command Prompt we opened in Step 1b.

vsperfcmd –start:coverage –output:[AMeaningfulOutputName].coverage

3. Start app

You start your application by running it from the location specified in Step 1b.

4. Run all manual tests

Here's where those copies of the manual tests come in.  In Cases 1 & 2 we were able to run the manual tests inside of Visual Studio but we can't do that if we only have test binaries. 

5. Stop app

Shut down your app the way a user normally would.

6. End Manual Test run

We need to shutdown the code coverage monitor. This will cause the code coverage data to be written out to the file that we specified on the command-line in Step 2.   Do this by running the following command in the Visual Studio 2008 Command Prompt we opened in Step 1b.

vsperfcmd –shutdown

7. Review Code Coverage

To look at the code coverage results, open the .coverage file (specified on the command-line in Step 2) in Visual Studio 2008.  If someone looking at the .coverage file has access to the source code, they will be able to see the source code coloring.

 

Sources

Running Tests with Code Coverage 
  http://blogs.msdn.com/vstsqualitytools/archive/2005/06/08/426979.aspx

MSDN: Manual Test Overview
  http://msdn.microsoft.com/en-us/library/ms182496(VS.80).aspx

Perform Code Coverage Analysis with .NET to Ensure Thorough Application Testing(MSDN Magazine)
  http://msdn.microsoft.com/en-us/magazine/cc163981.aspx

MSDN - VSInstr
  http://msdn.microsoft.com/en-us/library/ms182402.aspx

 MSDN - VSPerfCmd
  http://msdn.microsoft.com/en-us/library/ms182403.aspx

Hyper-V: Must Have Windows Sidebar Gadget

I've been doing a lot of work recently that requires me to run virtualized environments.  My laptop is running Windows Server 2008 Standard with Hyper-V RTM installed.  It is working out great (now that I understand its networking).  One of the tools that I find myself using all the time is a Windows Sidebar gadget by Tore Lervik that monitors and manages Hyper-V machines.

What's cool about this is that it handles all of the basic management of your VMs in a gadget that's always available on the Sidebar. 

It gives you the current run status of your VMs.  It show the VM is Stopped (Shut down), Running,  Suspended (Saved), Starting and Saving.

Moving the mouse over the status will change the status display to one or more buttons.  From there you can do an appropriate task for that state. If the state is Stopped or Suspended you will see a single icon that will start the VM when clicked. 

If the VM is in a Running state, hovering over the status will give you 3 buttons.  You can Stop the VM (hard shutdown), Shut down the VM (tell the OS to shut down) or Save the VM (save the state).  All of these functions are accomplished through the use of WMI.

If you want to connect to the VM with a Virtual Machine Connection windows you only have to click on the VM's name in the Gadget and the window will open and attempt to connect. 

Here's the kicker for all those IT Pros out there managing multiple Hyper-V installations, a single gadget will work with multiple Hyper-V servers.  I don't have multiple Hyper-V servers running to get a screen shot, but Tore did so I'm going to point over to his graphic.

I am putting this on my "highly-recommended" list.  It is simple, elegant and most of all...useful.

Hyper-V: Give your Virtual Machines access to your Wireless Broadband card

Whenever I'm on the road I rely heavily on the broadband card built into my Dell D630 laptop for connectivity at client sites, airports, train stations and hotels.  There have been a number of occasions when I've had to update a VM or create a new one on-the-fly.  One of the biggest problems I've encountered is how to update the VM's OS to the latest Service Pack or KB since Hyper-V doesn't recognize wireless devices as network cards.  Therefore I can't simply add a Network Adapter bound to my broadband card and update the VM. 
I initially tried to create a Network Bridge between my internal broadband card and the Hyper-V virtual switch but Windows Server 2008 won't let me bridge those two connections.  I think it is because the virtual switch is classified as LAN or High-Speed Internet and the broadband card is classified as a Dial-up connection, but I haven't found any confirmation to that effect. 
My next thought was to follow a blog post by Ben Armstrong on connecting Wireless networks to your VMs using Internet Connection Sharing.  I've taken those steps and modified them to work with the Broadband card.  This post will be very close to his only with more pictures.  Here are the steps...

 

1. Prepare the network connections

You need to have a virtual network device available for the Broadband card to share with.  I've found that it is best to share with either a brand-new Internal Network created solely for this purpose or with an existing Virtual Network that isn't being used at the same time as sharing. 
In my case, I rarely have the Broadband card active when I'm using a physical network cable connection so I'm going to use the Broadcom NetXtreme 57xx Gigabit connection that was configured against my physical NIC when I installed Hyper-V.

Here's what that looks like on my machine.  The Internal Network that I created shows up in Network Connections as Local area Connection 4 and the Broadcom NetXtreme card is showing up as Virtual Switch. (Note:  I changed the name of the Broadcom connection to Virtual Switch after installing Hyper-V to differentiate it from the Physical NIC it is bound to.  Yours will not be called Virtual Switch or Physical NIC.)

 

2. Configure the Broadband card for Sharing

The next step is to configure your broadband card to share it's connection with the virtuals.  The steps I show here are for the Dell Wireless 5270 card for VZW Mobile.  If you use this card with some other carrier or a different card your screens may differ from the ones shown here.
First off we need to get to the Properties window for the broadband connection.  Right-click on the connection and select Properties.

This will bring up the VZAccess dialog letting you know that you really shouldn't be playing around with this stuff unless you know what you are doing.  Since we really know that this dialog is here to minimize the costs incurred by Verizon's Help Desk for screwed-up configurations, just go ahead and click the Advanced... button.
 
This will bring up the actual Properties dialog.  we need to click on the Sharing tab to get to the stuff we really want to play with.  Now you need to click on the first check-box entitled Allow other network users to connect through this computer's Internet connection. 

As soon as you check this, the dialog below will appear letting you know that they can't save the username and password for use by all users.  This dialog can be safely ignored since we are going to be logged-on whenever we use the VMs.  Click Ok.
The next thing to select a connection from the Home networking connection list.  You want to select the connection that you identified at the start of this process.  In my case, I'm selecting the Virtual Switch connection.  The remaining checkboxes on this tab should be unchecked.  Click Ok to continue.

 

3. Verify the settings

If all went as planned, you can verify the settings in the Network Connections window.  You should now see that your broadband connection is marked as Default and Shared. 

 

4. Connect to the Internet using the Broadband card

Fire up your broadband card and connect to your ISP. 

Once connected, bring up a Command Prompt and verify your connectivity by running an IPConfig command.  You should see the adapter that you selected has an IP Address of 192.168.0.1 which is the default for Internet connection Sharing.  If I was configured incorrectly and my laptop wasn't connected to a network cable I would see a 169.254.x.x address because the connection couldn't find a DHCP server to provide an address.

5. Configure the Virtual Machine

Open up the Settings window for the virtual machine and select the Add Hardware entry in the current settings list.  The right-hand pane will change to the Add Hardware settings.  Next, select Network Adapter and click the Add button.  This will change the right-hand pane to Network Adapter settings.

Notice that the current settings list now contains a new Network Adapter whose status is Not Connected.  In the Network Adapter settings pane select your shared virtual adapter from the Network drop-down.  Click the Ok button to finish.

6. Verify connection on the Virtual Machine

You can verify the connection by starting the virtual machine and logging into it.  Once logged-in, bring up a Command Prompt window and run an IPConfig command.  You should see that one of your connections should have an IP Address in the 192.168.0.x range and the DNS suffix should be mshome.net. 
The next step is to verify connectivity.  You can do this by running a Ping command against your favorite server on the Internet.  I use ftp.apple.com because it is one of the few server that I know will reply to a ping (Most don't for security reasons).  Feel free to use any server that you know is external to your network and will reply.

That's it.  You can now open up a web browser and browser the web or run Windows Update to keep your VM OS' secure and up-to-date.

7. Caveats

From PracticallyNetworked.com:

"Consider these points when deciding whether to enable ICS.
WARNING #1: When you enable ICS, the network adapter connected to the local area network is assigned a static IP address of 192.168.0.1. The client computers are assigned other IP addresses in the 192.168.0.x range. These addresses may not be compatible with an existing network
WARNING #2: Don't enable ICS if any computer in your network is configured as a domain controller, DHCP server, or DNS server. Don't enable it if another computer is running ICS or Network Address Translation (NAT).
WARNING #3: To enable ICS, you must be logged on as a user that is a member of the Administrators group.
WARNING #4: If you establish a Virtual Private Networking (VPN) connection while sharing a different connection, the client computers won't be able to access the Internet until the VPN connection is ended."