Team Build: How do you deal with dependencies in builds?

A question came up yesterday from Tim Mulholland on the MSDN forums around directory structure when you need to share assemblies between different teams.  As I was throwing in my two cents, I realized that this might make a good post, so here's Tim's original question and my comments.  I've also linked to the entire thread as there are other comments that are worth the read.

Tim's Question:

"I am trying to set up my build machine to be able to build some of our projects (we've moved to TFS for source/version control and work item tracking, but not yet for builds).

What are the "best practices" for having all of the required files on the machine to be able to build the sources. Not everything is included in source control for the project being built.

Examples of things i'm talking about might be:

• Output of other projects developed by our team.
• Output of other projects developed by other teams (or 3rd parties).
• Includes/Libs from SDKs - WM SDK is the one i'm dealing with right now.

I imagine that this can be done quick and dirty by just mimicking the directory structure on the dev machine, but i'd prefer to make it "cleaner" and have some type of best practices to follow."

My comments:

I can't say that these are "Best Practices", but only that they are "My Practices." 

I am configuring our build as we speak.  Our system is composed of (currently) 4 Team Projects with about a dozen solutions housing 100+ projects.  This is our initial build under Team Build, it will get much worse.

To deal with the question of "How do we access the output of other projects", we (the dev leads and I) decided on a "well-known" location for all references.  We created a local folder (C:\ReferenceLibrary) that would be mapped to the R: drive letter(to make it machine-agnostic).  We then added a post-build task to each project to copy the project's output to the R: drive in a folders structure like:

R:\[TeamProjectName]\[AssemblyName]\[AssemlyVersion]\assemblyname.dll

The post-build task was a simple xcopy like this:

xcopy "$(TargetDir)$(TargetName).*" "R:\Common\$(TargetName)\1.0\" /Y

This allowed us to setup a project's references to the assemblies on R: drive.  (To make this work, you have to set the reference's SpecificVersion properety to True).  I then added a bit of MSBuild magic code to each project file to let it search the R: drive for referenced assemblies from the Team Projects that it uses.  Because of this searching, it is imperative that the SpecificVersion property on the reference is True.  The "RetrieveCommonReferenceFolders" (for dependencies in the Common Team Project) target is hooked into the project by overriding the "BeforeResolveReferences" target like so:

<Target Name="BeforeResolveReferences"
        DependsOnTargets=
            "RetrieveCommonReferenceFolders" />

Then it was just a matter of creating the RetrieveCommonReferenceFolders target to recurse through the folder list on the R: drive and add all of the found folders to the standard "AssemblySearchPath" property.

<Target Name="RetrieveCommonReferenceFolders">
  <CreateItem Include="R:\Common\**\*.*">
     <Output ItemName="Common"
             TaskParameter="Include" />

  </CreateItem>
  <CreateItem Include=
         "@(Common->'%(RootDir)%(Directory)')">

    <Output ItemName="CommonReferencePath"

            TaskParameter="Include" />

  </CreateItem>


  <CreateProperty Value=
        "@(CommonReferencePath->'%(FullPath)');
         $(AssemblySearchPaths)">
    <Output TaskParameter="Value"
           PropertyName="AssemblySearchPaths"/>

  </CreateProperty>


  <!-- FOR DEBUGGING -->

  <CreateProperty Value=
        "@(CommonReferencePath->'%(FullPath)');
         $(ReferencePath)">
    <Output TaskParameter="Value"

          PropertyName="CommonReferencePath" />

  </CreateProperty>


  <Message Text="GetCommonReferences:
               RefPath=$(CommonReferencePath)" 
           Importance="High" />


  <Message Text="GetCommonReferences:
   AssySearchPathBefore=$(AssemblySearchPaths)"
           Importance="High" />

  <!-- END DEBUGGING -->

</Target>

Since we prepended our folders to the existing AssemblySearchPath property, MSBuild will look in the R: drive folders first.  When we have references in multiple Team Projects, we just add another target for the next Team Project and then add that target's name to the BeforeResolveReferences target's DependsOnTargets list.  Also to make adding this to all of these project file easier, I put the targets into their own References.targets file and then just used an "Import" in the project file.  This cuts down on the places that I have to change when a new Team Project is added.

Addidional Comments: To make this work you need to be able to run the SUBST command to map the drive letter "R" to your ReferenceLibrary folder (which must be shared).  Because we reflect over these "referenced" assemblies in our code generation, we had to run the SUBST command under the System account so that if could be seen during code generation.  Luckily I work with some very bright and ingenious people that solved this problem by creating a small Windows Service that runs at Startup and does the SUBST on the System account for us.

How To: Get MSBuild to run a complete Target for each Item in an ItemGroup

Scenario: I am trying to create a generic target that will create my MSI packages.  This process consists of a number of distinct steps including MESSAGE tasks for logging and a call to either InstallShield or DevEnv.exe to build the actual MSI.

Problem:  My Team Builds are configured to create anywhere from zero to six MSIs during a build run.  I needed to configure the project so that I could list a number setup projects to be run and the build would create the MSIs from that list.  I also needed to be able to add additional metadata related to the MSI to be built.  In pseudocode, my task was this:

For each project in InstallerList
  Remove read-only flag from Installer project file 
  Set an Environment variable pointing to BinariesRoot 
  If PackagerType = "IS" Then Run InstallShield 
  If PackagerType = "VS" Then Run DevEnv.exe
Next project

 [The color coding on the pseudocode lines above will be used throughout the rest of this article to correlate the output received to the tasks needed.] 

So I wrote a test project file to see if I could get this to work.  My initial file looked something (actually exactly) like this:

<?xml version="1.0" encoding="utf-8"?>
<Project DefaultTargets="Test" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">

  <ItemGroup>
    <Package Include="CommonWebSetup.ism">
      <PackagerType>IS</PackagerType>
      <SetupProjFolder>CommonWebSetup</SetupProjFolder>
      <ISProductConfig>Server</ISProductConfig>
      <ISReleaseConfig>Release</ISReleaseConfig>
    </Package>
    <Package Include="CommonClientSetup.vdproj">
      <PackagerType>VS</PackagerType>
      <SetupProjFolder>CommonClientSetup</SetupProjFolder>
      <ISProductConfig>Client</ISProductConfig>
      <ISReleaseConfig>Release</ISReleaseConfig>
    </Package>
  </ItemGroup>


<Target Name="Test" >
	<Message Text="Removing read-only flag for %(Package.Identity)" Importance="High" />  

	<Message Text="Setting Environment variable for %(Package.Identity)" Importance="High" />  

	<Message Condition=" '%(Package.PackagerType)' == 'IS' " 
	         Text="Running InstallShield for %(Package.Identity)" Importance="High" />  

	<Message Condition=" '%(Package.PackagerType)' == 'VS' " 
	         Text="Running DevEnv.exe for %(Package.Identity)" Importance="High" />  

</Target>

</Project>

 I thought that this would do exactly what I wanted, since I had grouped all of the metadata into a nice self-containedItems.  So I ran it through MSBuild and received the following results:

 
__________________________________________________
Project "C:\test7.proj" (default targets): 
Target Test:
Removing read-only flag for CommonWebSetup.ism
Removing read-only flag for CommonClientSetup.vdproj
Setting Environment variable for CommonWebSetup.ism
Setting Environment variable for CommonClientSetup.vdproj
Running InstallShield for CommonWebSetup.ism
Running DevEnv.exe for CommonClientSetup.vdproj 
Build succeeded.
0 Warning(s)
0 Error(s) 
Time Elapsed 00:00:00.03

 I've colored each MESSAGE task's output to show the results a little clearer. 

As you can see, the Test target was run only once with all of the items passed into it.  So Batching did not occur at the Target level, but rather at the Task level.  During this build, each Task had the opportunity to process  the batched metadata before passing control to the next Task in the Target.  This isn't what I expected, so I did a little research and (of course) this is by design.  I did a little more research and came across an MSBuild forum post that discussed a situation similar to mine.  In this post, Neil Enns (Lead Program Manager, Microsoft Visual Studio) went to great lengths to not only show how this could be done, but to explain the concepts behind the behavior was seen.  Kudos to Neil for making this whole "batching thing" just a little bit clearer (and giving me a quick answer so I can move on with my project).

Solution: To paraphrase the thread, you need to add an Outputs attribute to the Target.  This attribute must point to a metadata entry that is unique for each Item in the ItemGroup.  In my example above, the Identity (shown in the Include attribute of the Package element) is unique, so all I have to do is add it to the Outputs attribute of my Target and it will work.  Here' the updated project file (changes in bold) and MSBuild output:

<?xml version="1.0" encoding="utf-8"?>
<Project DefaultTargets="Test" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">

  <ItemGroup>
    <Package Include="CommonWebSetup.ism">
      <PackagerType>IS</PackagerType>
      <SetupProjFolder>CommonWebSetup</SetupProjFolder>
      <ISProductConfig>Server</ISProductConfig>
      <ISReleaseConfig>Release</ISReleaseConfig>
    </Package>
    <Package Include="CommonClientSetup.vdproj">
      <PackagerType>VS</PackagerType>
      <SetupProjFolder>CommonClientSetup</SetupProjFolder>
      <ISProductConfig>Client</ISProductConfig>
      <ISReleaseConfig>Release</ISReleaseConfig>
    </Package>
  </ItemGroup>


<Target Name="Test" Outputs="%(Package.Identity)" >
	<Message Text="Removing read-only flag for %(Package.Identity)" Importance="High" />  

	<Message Text="Setting Environment variable for %(Package.Identity)" Importance="High" />  

	<Message Condition=" '%(Package.PackagerType)' == 'IS' " 
	         Text="Running InstallShield for %(Package.Identity)" Importance="High" />  

	<Message Condition=" '%(Package.PackagerType)' == 'VS' " 
	         Text="Running DevEnv.exe for %(Package.Identity)" Importance="High" />  
</Target>

__________________________________________________
Project "C:\test7.proj" (default targets): 
Target Test:
Removing read-only flag for CommonWebSetup.ism
Setting Environment variable for CommonWebSetup.ism
Running InstallShield for CommonWebSetup.ism
Target Test:
Removing read-only flag for CommonClientSetup.vdproj
Setting Environment variable for CommonClientSetup.vdproj
Running DevEnv.exe for CommonClientSetup.vdproj 
Build succeeded.
0 Warning(s)
0 Error(s)
Time Elapsed 00:00:00.03

 

For additional information on Batching and Target Batching specifically, check out MSDN Help's MSBuild Concepts MSBuild Batching.