Agile Practice

2009-10-29T13:18:00.001-07:00

In my article XXX I talked about how to improve the speed of LINQ to SQL queries with multiple one to many associations. The post propsed to use the following statement.

var orderItems= db.OrderItems
.Where(oi => orders.Select(o => o.ID).Contains(oi.OrderID)).ToList();
foreach (var o in orders)
{
var oClosure = o;
oClosure.OrderItems.SetSource
(orderItems.Where(oi => oi.OrderItemID == cClosure.ID));
}

one more tips here is to store the result of orderItems in a Dictionary using .GroupBy() and .ToDictionary(). If you do not do that you will eventually get a performance problem because you do a linear search inside a loop -- basically you will get a O(n^2) performance for your database retrievals -- not very good. If you use a dictionary lookup you will get something close too O(n) for the foreach loop.

Do not forget to check if you have matches for the key before you use it inside the foreach. If you do not have a match you should use something like oClosure.OrderItems.SetSource(Enumerable.Empty());.

SQL to Linq Performance: Retrieving Multiple One-to-Many Associations without Multiple Round Trips

2009-10-16T11:45:00.000-07:00

I have been using SQL to Linq for a while. It is a fairly straight forward framework for retrieving data for my data access layer. The Linq syntax makes it very easy to read while allowing for complex expressions.

Unfortunately, the performance may be really bad in some cases. This is typically because of lazy loading where a round trip is required per row because associated data are loaded. The typical example from the Northwind database is a Customer with multiple Orders. All the Customers are retrieved in one go, however for each customer the associated orders are retrieved in one SQL request. This will generate one SQL request for the customers and then n for the orders. This is hopeless. Fortunately, Microsoft provided us with the DataLoadOptions.LoadWith method. In this case, the following statements solves the problem:

Northwnd db = new Northwnd(@"c:\northwnd.mdf");
DataLoadOptions dlo = new DataLoadOptions();
dlo.LoadWith(c => c.Orders);
db.LoadOptions = dlo;

But what happens if each order has a set of associations? Lets say each order has a collection of OrderItems. You will end up with a round trip to the database for each order. Each round trip will retrieve the associated OrderItems.

So what do you do? You may try to add a second LoadWith statement... but it turns out that this is not supported by Linq to SQL. So if you add the second LoadWith, the framework will do what you tell it. However, it will use multiple SQL statements. So what is the point then?

In order to fix this, use this code:

var orderItems= db.OrderItems.Where(oi => orders.Select(o => o.ID).Contains(oi.OrderID)).ToList();
foreach (var o in orders)
{
var oClosure = o;
oClosure.OrderItems.SetSource
(orderItems.Where(oi => oi.OrderItemID ==
cClosure.ID));
}

So basically, by pre-fetching orderItems and manually setting the source by using SetSource, the number of round trips are reduced to two. Great? ;-)

SQL Server 2008 SP1 brok my SQL Server

2009-10-10T10:07:00.000-07:00

After installing SQL Server 2008 SP1 my SQL Server stopped working. The SQL Server reported the following in the log file:

2009-10-10 19:01:09.15 spid7s Error: 15404, Severity: 16, State: 19.
2009-10-10 19:01:09.15 spid7s Could not obtain information about Windows NT group/user 'XXX\XXX', error code 0x5.
2009-10-10 19:01:09.15 spid7s Error: 912, Severity: 21, State: 2.
2009-10-10 19:01:09.15 spid7s Script level upgrade for database 'master' failed because upgrade step 'sqlagent100_msdb_upgrade.sql' encountered error 15404, state 19, severity 16. This is a serious error condition which might interfere with regular operation and the database will be taken offline. If the error happened during upgrade of the 'master' database, it will prevent the entire SQL Server instance from starting. Examine the previous errorlog entries for errors, take the appropriate corrective actions and re-start the database so that the script upgrade steps run to completion.
2009-10-10 19:01:09.16 spid7s Error: 3417, Severity: 21, State: 3.
2009-10-10 19:01:09.16 spid7s Cannot recover the master database. SQL Server is unable to run. Restore master from a full backup, repair it, or rebuild it. For more information about how to rebuild the master database, see SQL Server Books Online.
2009-10-10 19:01:09.16 spid7s SQL Trace was stopped due to server shutdown. Trace ID = '1'. This is an informational message only; no user action is required.
2009-10-10 19:01:09.84 Server The SQL Server Network Interface library successfully deregistered the Service Principal Name (SPN) [ MSSQLSvc/XXX ] for the SQL Server service.

No luck? Then I tried to run the SQL instance as a privileged user, and now it works again. It looks like the upgrade script required some extra permission.

ReSharper 5.0 is comming

2009-10-09T13:21:00.000-07:00

According to the official blog for JetBrains, ReShaper 5.0 is just around the corner. I am looking forward to this. The new release will provide improvements such as:

Web Development. Web developers have long been asking for top-class support, so we’re starting to deliver on the promise with a pack of features for HTML, ASP.NET, and ASP.NET MVC.
Project and Team. This is a valuable addition for developers having to explore much unfamiliar code and/or perform batch modifications in large-scale projects. With access to and navigation within external sources, structured patterns for searching and customizing code, and location/namespace synchronization tools working in batch mode, you can handle your colossal solutions easier.
Code Analysis. In addition to fresh code inspections, ReSharper 5.0 presents multiple functional style “enumeration to LINQ” transformations, accepts warnings and suggestions into the “Errors in Solution” tool window, and introduces two major features to track what’s going on with your calls and data throughout application execution.
Visual Studio 2010 and Tools. We’re on the run to support Visual Studio 2010 earlier than ever. More info on that when VS2010 Beta2 comes out. Of course, Visual Studio 2008 is supported as well.

ITransformableFilterValues and List View Web Part in SharePoint

2009-10-06T00:56:00.000-07:00

After preparing a new web part filtering that implements ITransformableFilterValues I though I was near a solution: I wanted to filter a standard Sharepoint list by a set of values that I provided in my filter web part. Now it looks like the standard list view web part only accepts the one value, not multiple. So if you want it to show all rows where the value is A or B, then you are out of luck.

So what can you do? You try to find a different list view web part? That should be easy. After looking for a few days I am still out of luck.

Any suggestions?

Team Build Tray

2009-08-19T13:48:00.001-07:00

I just tried Team Build Tray 1.0.4, but it was a big disappointment. It shows you the status of your builds, but only for one of your projects. I want them all!

I guess I have to stick to the one bundled with TFS Power Toys!

Team Foundation Build -- running build script from command line

2009-08-18T01:43:00.000-07:00

Often you want to verify that your build will work correctly on your build machine before you do a commit. In Visual Studio you can run all your tests etc, but your build script will propably do some things such as settings up databases. This can be archived this way:
Open the DOS command window
Go to the TFS Build directory where TFSBuild.proj is located
Enter "msbuild /p:SolutionRoot=..\..\.. /t:DesktopRebuild"

The solution root may have to be changed depending on how you mounted the workspace.

I usually keep a shortcut on my desktop that I can just double click on for a fast compile.

QueryBuildDetail updated

2009-08-18T01:23:00.000-07:00

I just made an update to my QueryBuildDetail MSBuild task. The task will not retrieve the dropfolder of the the most recent successfully build if a build number is not specified. This is really useful if you want to do some integration testing etc.

using System;
using System.Linq;
using Microsoft.TeamFoundation.Build.Client;
using Microsoft.TeamFoundation.Client;
using Microsoft.Build.Framework;
using Microsoft.Build.Utilities;

namespace QueryBuildDetail
{
public class QueryBuildDetail : Task
{
private IBuildDetail buildDetail;

[Required]
public string TeamFoundationServer { get; set; }

[Required]
public string TeamProject { get; set; }

[Required]
public string BuildDefinition { get; set; }

public string BuildNumber { get; set; }

public override bool Execute()
{
try
{
RetrieveBuildData();

return true;
}
catch (Exception ex)
{
Log.LogError(ex.Message);
return false;
}
}

[Output]
public string DropLocation { get { return buildDetail != null ? buildDetail.DropLocation : null; } }

private void RetrieveBuildData()
{
TeamFoundationServer teamFoundationServer = TeamFoundationServerFactory.GetServer(TeamFoundationServer);
IBuildServer buildServer = (IBuildServer)teamFoundationServer.GetService(typeof (IBuildServer));

IBuildDetailSpec spec = buildServer.CreateBuildDetailSpec(TeamProject, BuildDefinition);

if (String.IsNullOrEmpty(BuildNumber))
spec.Status = BuildStatus.Succeeded;
else
spec.BuildNumber = BuildNumber;

spec.MaxBuildsPerDefinition = 1;
spec.QueryOrder = BuildQueryOrder.StartTimeDescending;
spec.QueryOptions = QueryOptions.None;
IBuildQueryResult res = buildServer.QueryBuilds(spec);
buildDetail = res.Builds.Single();
}
}
}

Retrieving droplocation of a build from a msbuild script using a msbuild task

2009-08-16T07:48:00.000-07:00

I just wrote a msbuild custom task for retrieving the drop location of a build. I am using it for my test deployment script.

The task may be used like this:

TeamFoundationServer="http://myServer:8080/"
TeamProject="MyProject"
BuildNumber="MyProject - CI_20090813.1">

Here is the source code for the msbuild custom task

using System;
using System.Linq;
using Microsoft.TeamFoundation.Build.Client;
using Microsoft.TeamFoundation.Client;
using Microsoft.Build.Framework;
using Microsoft.Build.Utilities;

namespace QueryBuildDetail
{
public class QueryBuildDetail : Task
{
private IBuildDetail buildDetail;

[Required]
public string TeamFoundationServer { get; set; }

[Required]
public string TeamProject { get; set; }

[Required]
public string BuildNumber { get; set; }

public override bool Execute()
{
try
{
RetrieveBuildData();

return true;
}
catch (Exception ex)
{
Log.LogError(ex.Message);
return false;
}
}

[Output]
public string DropLocation { get { return buildDetail != null ? buildDetail.DropLocation : null; } }

private void RetrieveBuildData()
{
TeamFoundationServer teamFoundationServer = TeamFoundationServerFactory.GetServer(TeamFoundationServer);
IBuildServer buildServer = (IBuildServer)teamFoundationServer.GetService(typeof (IBuildServer));

IBuildDetailSpec spec = buildServer.CreateBuildDetailSpec(TeamProject);
spec.BuildNumber = BuildNumber;
spec.QueryOptions = QueryOptions.None;
IBuildQueryResult res = buildServer.QueryBuilds(spec);
buildDetail = res.Builds.Single();
}
}
}

Shared binaries and continuous integration

2009-08-14T07:03:00.000-07:00

Lets say you have two component A and B. Component A does not depend on anything, but component B uses component A for some reason.

How do you build such a thing using continuous integration. It is easy to build component A because you do not need anything else. However for component B you somehow have to include component A. You could solve it by building both components all the time, however that will make it hard to manage as the system grows?

In the paper Enterprise Continuous Integration Using Binary Dependencies, Roberts suggests that you store the binaries in the source control system. Each successful build should update the binary in the source control so that it will trigger a new build of any dependent component. The paper also suggests how to handle multiple versions/branches by dedicating separate names for latest version and for fixed versions.

Automatic deployment to staging area

2009-08-14T06:58:00.001-07:00

I just came across the tool TFS Deployer. It allows you to automatically deploy a new build to a machine. The tool is really simple. You just install it on your deployment machine. Then you configure it to subscribe to the correct Team Foundation Server and the correct build definition. TFS Deployer will then start a Power Shell script when you change the build quality of a build i Team Foundation. The power shell will be a script that installs all the components you need, configures the database and makes sure the web site is up-n-running. You can then easily change the deployed version of your application on your staging machine.

Refactoring in Matlab

2009-07-27T13:18:00.001-07:00

Popular programming languages like Java and C# has various refactoring tools available that automatize the refactoring process. ReSharper is such a tool for C#.

For the Matlab programming language I am missing similar features. Basic refactorings like Extract Method or Introduce Variable has to be done manually. This is very error prone and makes refactoring boring and time consuming. I guess many refactoirngs are never caried out in this language because of the missing feature.

So Mathworks -- why don't you provide we some basic refactoring features:

Extract Method/Function
Introduce Variable
Inline Method/Function
Inline Temp
Rename Method / Function
Rename Variable
Add Parameter
Remove Parameter

A video exist where a basic Introduce Variable refactoring is implemented, but it is currently very basic.

Refactoring: Replace Iteration with Linq Query

2009-07-27T13:07:00.000-07:00

string[] items = new[] { "item1", "item2", "item3" };
List itemsFiltered = new List();
foreach (string item in items)
if (item != "item2")
itemsFiltered.Add(item);

=>

string[] items = new[] { "item1", "item2", "item3" };
List itemsFiltered = items.Where(item => item != "item2").ToList();
Motivation
You often see methods in which the programmer iterates over a list, and then checks a predicate regarding the current item. If the predicate is satisfied the item is added to a newly constructed list. After the iteration the list contains all items satisfying the predicate. In C# this is not necessary because the Linq library has built in support for doing these queries. The issue is not performance, but that you end up with unnecessary lines of code that adds no value to your program—you have four lines of code that can be expressed with one line of code.
Mechanics

Identify a foreach block with a single if statement where the item is added when the predicate is satisfied.
Replace the foreach block with a Where() query on the list with the predicate as a parameter, apply a ToList() query to convert the result to a list, and assign the result to the list variable.
Remove the constructor of the filtered list.
Compile and rerun tests.

Example
Start with this code.
string[] items = new[] { "item1", "item2", "item3" };
List itemsFiltered = new List();
foreach (string item in items)
if (item != "item2")
itemsFiltered.Add(item);
Finally, we simplify the code:
string[] items = new[] { "item1", "item2", "item3" };
List itemsFiltered = items.Where(item => item != "item2").ToList();

mlUnit - a great unit test alternative for Matlab

2009-06-10T13:35:00.000-07:00

I am working on various expert applications that use Matlab for computation.

Matlab is great for mathematics, however it is lacking features such as strong typing, reasonable scoping (you do not even have to declare a variable). The result is a lot of error in the code due to mismatches in methods/function signatures after adding parameters, type mismatches, undeclared variables etc.

I have been using test-driven development and unit testing for a long time in C#, however I missed that feature in Matlab. Earlier this year I came across mlUnit, which is a full blown unit testing framework. Now I am developing test first, and my code is getting more and more stable. To further improve the situation, I have included the unit tests as part of my continuous integration of my Matlab code.

So should you!

Buddy build using Team Foundation Build 2008

2009-06-06T13:54:00.000-07:00

I am a big fan of continuous integration, and I waiting for the pre checkin validation features of Team Foundation Build 2010. But until 2010 is released, we will have to stick with 2008. What can we do?

I just found out that there is a CodePlex project allowing buddy builds for Team Foundation Build 2008. Want to try? Check out http://www.codeplex.com/BuddyBuild .

Visual Studio 2010 Beta is out

2009-06-05T13:30:00.000-07:00

Now you can download the new Microsoft Visual Studio 2010 from MSDN.

Visual Studio Team Foundation 2010 includes new interesting features such as gated checkins for the Team Foundation Build system. This makes sure that broken changes are not committed to the source control system.

MSBuild and attaching SQL Server database

2009-03-11T02:57:00.000-07:00

The msbuild script I am running on the continuous integration server updates the database in order to run some tests.

The database I am using is SQL Server 2008.

Unfortunately I have not found a task that does this job. However, I ended up with this script:
exec command="SSEUtil.exe -m -s $(DeployDatabaseInstance) -d name=$(DeployDatabase)"
...to detach the old database...and...
exec command="SSEUtil.exe -m -s $(DeployDatabaseInstance) -a "$(DeployFolder)TuneIT.mdf" $(DeployDatabase)"

I guess I could write a MSTest task, but I do not want to maintain it. Any ideas?

ReSharper -- the refactorings

2009-03-04T00:59:00.001-08:00

I am a big fan of the ReSharper tool. It lets you do a lot of code cleanup and refactorings directly from Visual Studio.

Here is a list of the refactorings it supports:

Change Signature

This refactoring allows you to modify a method signature in the following ways:

· Add, remove, rename, or reorder parameter(s)

· Change return type

· Change parameter type(s)

Rename method
Along with changing the signature, ReSharper searches for all usages of the method and modifies all calls, implementations, and overrides of the method to reflect the change. In case of newly added parameters, default values for them supplied by the user are substituted in all method calls. In overrides, the original parameter passed to the method is passed to the base call.

Convert Abstract Class to Interface

You can apply this refactoring to convert an abstract class to an interface. This is especially useful when you want a class to inherit from more than one abstract class, thus you need to convert one of them to an interface. The reverse functionality is also possible with the Convert Interface to Abstract Class refactoring.

Convert Anonymous to Named Type (C# only)

This refactoring converts anonymous types to named types in the scope of either the current method (locally) or the whole solution (globally). In the dialog box that this refactoring provides, you can specify whether ReSharper should generate auto-properties or properties with backing fields, and also opt to generate equalify and formatting method overloads.

Convert Extension Method to Plain Static (C# only)

This dialogless refactoring converts an extension method to a static method in the same class. The reverse functionality is provided by 'Convert Static to Extension Method'.

Convert Indexer (Default Property) to Method

This refactoring works a lot like 'Convert Property to Method' but it is applied to indexers (default properties in VB.NET). You can convert only getter, only setter, or both, to a method. The reverse functionality is provided by 'Convert Method to Indexer (Default Property)'.

Convert Interface to Abstract Class

If you want to add some logic to interface or feel it better be a class you should invoke the refactoring and all appropriate conversions will be made. More importantly, it checks for any conflicts, i.e. for types implementing the interface that already have base type (an error in C#, which has no multiple inheritance).

Convert Method to Indexer (Default Property)

This is the reverse of 'Convert Indexer (Default Property) to Method'. It works in dialogless mode, but you may have to apply it to two methods to generate both a getter and a setter in your indexer.

Convert Method to Property

This refactoring allows users to convert non-void methods without parameters to properties with read access, and void methods with exactly one parameter to properties with write access. Paired methods can be converted to a single read/write property. This refactoring is especially useful for converting more traditional code with setter- and getter-type methods (for example, J#) to C#.

Convert Property to Auto-Property (C# only)

You can now convert traditional properties with backing private fields to auto-properties that were introduced in C# 3.0.

Convert Property to Method(s)

Using this refactoring, properties with read access can be converted to getter-type methods (non-void methods without parameters), and properties with write access can be converted to setter-type methods (void methods with exactly one parameter). As to properties allowing both read and write access, you can convert them to pairs of getter- and setter-type methods.

Convert Static to Extension Method (C# only)

This new refactoring converts a static method to an extension method. For the conversion to be carried out successfully, the static method must (1) have at least one argument and (2) reside in a static class. The reverse functionality is available with 'Convert Extension Method to Plain Static'.

Copy Type

It will take you just a second to create a copy of a type with a different name or within another namespace. ReSharper will take care of creating a separate file for it. For partial types, all parts will be copied even if they are located in separate files.

Encapsulate Field

The Encapsulate Field refactoring allows you to quickly create an accessor property from an existing field. Usages of the field are automatically replaced with usages of the property. An apparent advantage of this refactoring is that you can disallow direct access to a field through the use of properties.

Extract Class from Parameters

This refactoring creates a new class or struct and converts parameters of the selected method into encapsulated fields of the newly created type (with constructor taking parameters, fields to store values and properties to retrieve values). Usages of parameters are converted to usages of properties of created type.

Extract Interface

This refactoring allows you to create an interface from a class and make that class implement the created interface. You can choose the members to extract to the interface and specify a name for the interface.

Extract Method

Select a block of code and invoke the Extract Method refactoring to transform it into a method (Sub or Function in VB.NET). ReSharper will automatically analyze the code to detect the return value and/or out/ref parameters.

Extract Superclass

With this refactoring, you can create a base class for a class and move some members to it. Just place the caret at a class declaration, select members to extract, and run the refactoring. It is very useful for moving some logic up to share it later.

Inline Method

Whenever possible, this refactoring is able to transfer a method's body into the body of its callers and remove the method altogether. The reverse functionality is provided by 'Extract Method'.

Inline Variable

Select an arbitrary variable or local constant and invoke the Inline Variable refactoring. All occurrences of the selected variable in your code will be substituted with its initializer. The reverse functionality is provided by 'Introduce Variable'.

Introduce Field

Select an expression or a local variable and apply this refactoring. It will create a new field or constant and initialize it with the expression or local variable initializer. The field can be assigned in its initializer, type constructors or current members.

Introduce Parameter

Select an expression or a local variable in a method and use this refactoring to create a new parameter from it. All call sites will be updated to reflect the changed signature, while maintaining the logic and the semantics.

If an expression uses local variables that are inaccessible at a call site, this refactoring allows to pass them as delegates.

Introduce Variable

Select an arbitrary expression inside the member code and invoke the Introduce Variable refactoring. A new implicitly or explicitly typed local variable will be declared and initialized with the selected expression. The original expression will be replaced with the name of the variable. Should there be multiple occurrences of the original expression in your code, you will be given an option to replace all of them with the newly created variable. In the same manner you may also introduce constants for constant expressions.

Make Method Non-Static/Non-Shared

This refactoring converts a static method (Shared in VB.NET) to an instance method in the selected parameter's type (i.e. moves the method to parameter's type and transforms the parameter into 'this') and transforms method calls accordingly.

Make Method Static/Shared

This refactoring converts a non-static method into a static one (Shared in VB.NET) by making 'this' a parameter of the method. After refactoring, the target method will be declared as static (Shared in VB.NET), and necessary parameters will be added to its call sites. The refactoring is also useful when you want to move non-static methods (non-Shared in VB.NET). In this case, use Make Method Static as a preparatory step for the Make Method Non-static refactoring.

Move Static Member

The refactoring moves static fields and methods to another type. If you want to move non-static methods, you can perform the following sequence of refactorings: Make Method Static -> Make Method Non-static, with the move happening in the latter.

Move Type to Another File or Namespace

You can easily move types between namespaces, with references automatically updated by ReSharper. You can also move a type to outer or nested scope or to another file.
Using this refactoring, you can swiftly move the code implementing a given type into a separate file. This refactoring can be accessed through the Move refactoring command.

When you use XAML and you move a .NET class to another namespace, XAML markup references to the class are updated, and XAML import directives are inserted/changed accordingly. This refactoring can also be invoked right from XAML markup on any type reference.

Move Type to Outer Scope

This refactoring moves an inner type to an upper level. In case that the inner type uses members of the enclosing type, a reference to the enclosing type is passed as an argument to the moved type's constructors. This refactoring can be accessed through the Move refactoring command.

Pull Members Up

This refactoring helps you move type members to a superclass or an interface. This operation is useful as a way to generalize behavior. ReSharper analyzes all members in the current class and makes a list of members that you can pull up. Before completing the refactoring, ReSharper also checks for possible conflicts, for example, whether the members that you are trying to move to a superclass will be accessible in the destination type. The reverse functionality is available with 'Push Members Down'.

Push Members Down

This refactoring helps you clean up your type hierarchy by moving type members to a subtype. This operation is useful as a way to specialize behavior. ReSharper analyzes all members of the selected type and makes a list of members that you can push down. Before completing the refactoring, ReSharper also checks for possible conflicts, for example, whether the members you are trying to push down will be accessible in the destination type. The reverse functionality is available with 'Push Members Up'.

Rename

The Rename refactoring allows you to rename any symbol, including: namespaces, types, methods, parameters, local variables, properties, fields, and events. It automatically finds and corrects all references to the symbol. The Rename refactoring can be invoked directly from the editor and sometimes from other views (Class View, Object Browser).

Rename works with all supported languages and technologies - namely, C#, VB.NET, ASP.NET, XML, XAML, and build scripts.

Automatic reference correction fully extends to XAML markup when you use the Rename refactoring. In addition, specific XAML symbols such as namespace aliases and resources can be renamed easily:

For build scripts, when you rename a property or target with ReSharper, all its references and even usages in comments and strings are automatically updated to reflect the change.

Replace Constructor with Factory Method

This refactoring encapsulates a constructor with a static method that returns a new instance of a class.

The Factory Method pattern is a way of creating objects without specifying the exact class of object that will be created. ReSharper generates a separate method for creating objects. Subclasses can override it to specify the derived type of object that will be created.

Safe Delete

If you are going to delete a type, type member, or any other symbol in C# or VB.NET code files, ASP.NET code and markup, build scripts, or XAML, use this refactoring to ensure that the delete operation is safe. If there are no symbol usages found, or if there are some that are safely collapsible, the symbol will be deleted. Otherwise, ReSharper will show you all the usages that are not safe to delete, allowing you to edit the corresponding code.

When you invoke Safe Delete on a target or a property in a build file, ReSharper does all necessary validations and operations to remove selected symbol. ReSharper can also remove any conflicting references for you.

Use Base Type where Possible

With this refactoring, your code is generalized by replacing particular type usages with references to its base type or interface where possible, that is, where no members of the derived type are used. It is especially useful after the Pull Members Up refactoring.

TFSBuild.exe is missing something

2009-03-02T06:27:00.001-08:00

I just wrote a post on how to rebuild an old change set using Team Foundation Build. By using TFSBuild.exe you may start a build on the build server. However, in order to be useful for private builds and gated builds (which will be a part of Team Foundation 2010), it should be possible to disabled the publishing of the results.

With the current strategy the build output is public so that it shares the build number with the other builds. This means that if the build fails the results will be shared with the whole team. So it makes it less useful. Furthermore, the build numbers are shares.

I guess the workaround is the create a new build definition only for private builds:

"MyProject - CI" for Continuous integration
"MyProject - private" for private builds

Want to recompile a build on the buildserver?

2009-03-02T06:21:00.000-08:00

When using Team Foundation Build it is possible to recompile an old version. Typically you will be using continuous integration, and the most recent change set is build. However, from time to time you want to rebuild some old version. This is not possible using the GUI, so after some digging I found this page: The strategy is also described in Inside the Microsoft® Build Engine: Using MSBuild and Team Foundation Build.

tfsbuild start http://hostname:port TeamProject BuildDefinition /getOption:Custom /customGetVersion:VersionName

Where VersionName is C1000 for changeset #1000, or LMyLabel for the label called MyLabel.

Inside the Microsoft® Build Engine: Using MSBuild and Team Foundation Build

2009-03-02T04:30:00.000-08:00

I just got my copy of the new book "Inside the Microsoft® Build Engine: Using MSBuild and Team Foundation Build". The book describes how MSBuild works. The book start with a getting started chapter, and then continues with a more in depth chapter on advanced topics.

The book even includes chapters on how Team Foundation Build works. Being one of the few books on MSBuild and Team Foundation Build, I guess it is worth having a copy. The table

Part I. Overview
1. MSBuild Quick Start
2. MSBuild Deep Dive, Part 1
3. MSBuild Deep Dive, Part 2
Part II. Customizing MSBuild
4. Custom Tasks
5. Custom Loggers
Part III. Advanced MSBuild Topics
6. Batching and Incremental Builds
7. External Tools
Part IV. MSBuild Cookbook
8. Practical Applications, Part 1
9. Practical Applications, Part 2
Part V. Team Foundation Build
10. Team Build Quick Start
11. Team Build Deep Dive
12. Team Build Cookbook
Appendix A: New Features in MSBuild 3.5
Appendix B: Common Properties and Items
Appendix C: New Features in Visual Studio Team System 2010 Team Build
Index

The Buildmaister's Guide

2009-02-27T03:35:00.000-08:00

I just received the new book "The Buildmeister's Guilde: Achieving Agile Software Delivery" from Amazon. The book is written by Kevin Lee, an expert in software build tools.

I am really looking forward to reading this book.

If you also are interested in continuous integration, this is the book for you -- besides the standard book on the topic "Continuous Integration: Improving Software Quality and Reducing Risk".

Continuous Integration -- the original article

2009-02-27T03:25:00.000-08:00

For those of you who has not read the original article on continuous integration, here is the link. Enjoy.

Code Contracts in CLR 4.0 (.NET 4.0)

2009-02-26T02:20:00.000-08:00

It looks like CRL 4.0 (Visual Studio 2010 / .NET 4.0) will include some nice features. It will allow your compiler to statically check for pre and post conditions of the input and output. Take a look at this code block:

public void BuyMoreStuff(Item[] cart, ref Decimal totalCost, Item i)
{      
   CodeContract.Requires(totalCost >=0);
   CodeContract.Requires(cart != null);
   CodeContract.Requires(CodeContract.ForAll(cart, s => s != i));

   CodeContract.Ensures(CodeContract.Exists(cart, s => s == i);
   CodeContract.Ensures(totalCost >= CodeContract.OldValue(totalCost));
   CodeContract.EnsuresOnThrow(totalCost == CodeContract.OldValue(totalCost));

   // Do some stuff
   …
}

The CodeContract class allows you to set these conditions, and they are properly converted so that the compiler may check that the caller do not violate the rules. Much better than a lot of assers all over?

Linq to SQL and predicates with foreach: PredicateExtensions

2009-02-23T06:44:00.000-08:00

Do you want to write code like this?

            NorthwindDataContext db = new NorthwindDataContext();
 
            string[] searchTerms = new string[] { "Maria", "Pedro" };
 
            var predicate = PredicateExtensions.False<Customer>();
 
            foreach (string searchTerm in searchTerms)
            {
                string temp = searchTerm;
                predicate = predicate.Or(c => c.ContactName.Contains(temp));
            }
 
            dataGridView1.DataSource = db.Customers.Where(predicate);

Using the PredicateExtensions class, you will be able to generate a complicated as a proper predicate that can be used with Linq to SQL. The problem with using .Union() is that not all operations are supported. But here you may construct a nice predicate that does what you want--iterate over a collection. Take a look at this post.

Usefull Unit Test feature in Bamboo

2009-02-19T14:23:00.000-08:00

I admit it. I have a few bad "unit test" that are run in each continuous integration build. Bamboo has this nice feature that shows statistics about the tests. By using Bamboo, I can easily see how often each test has failed a build on the build server!

Bamboo Continuous Integratoin --- it is not that easy

2009-02-19T14:16:00.000-08:00

As you understand from this blog, I am a big fan of continuous integration. Today I browsed the web site of Bamboo--a continuous integration system--to have a look at the features. The web page read like this:

Bamboo distributes builds across different machines. These distributed builds allow you to harness the computing power of your entire network, maximising your build productivity and decreasing wait times.

Simply install a new Bamboo Agent on any additional build computers and your main Bamboo server will manage it. When a build is detected, the Bamboo server will automatically and intelligently distribute the builds to the agents.

Really, who believes it is that easy? The story has been different for me. I always end up with a few applications or libraries that have to be installed on the build machine. And those applications and libraries are not necessarily the same for all the projects I am building. So it means that each build machine may need some tweaking before it is ready to run!

More Effective C#: 50 Specific Ways to Improve Your C#

2009-02-19T13:45:00.001-08:00

"More Effective C#: 50 Specific Ways to Improve Your C#" is a book by Bill Wagner on how to improve your C# code. The book present 50 specific improvement. Each item includes a description of the problem the improvement solves and how it works. The book includes the topics:

Use generics to express your design intent more effectively
Master advanced generics techniques, such as constraints, method constraints, and generic specialization
Use the multithreaded techniques you’ll need to work with the .NET framework every day
Express modern design idioms using the rich palette of C# language features
Successfully mix object oriented and functional programming constructs
Create composable interfaces and avoid confusion in public interfaces
Use extension methods to separate contracts from implementation
Program successfully with C# closures and anonymous types
Write more effective LINQ queries
Make the most of LINQ Lazy Evaluation Queries and Lambda Expressions
Distinguish and convert between delegates and expression trees
Efficiently utilize nullable types and partial classes
Use implicit properties for mutable, nonserializable data

The book is a followup of the book "Effective C#: 50 Specific Ways to Improve Your C#" by the same author. The Effective C# book was mainly on .NET 1.1, but the new book is on .NET 2.0 and .NET 3.5.

Run recently failed, new, and modified tests first!

2009-02-18T05:46:00.000-08:00

Lets say you just modified a unit test and committed the change. In that situation you want to get feedback as soon as possible from the continuous integration system if the modified test passes. TeamCity does just that. By enabling the two configuration settings shown in the picture above, the build will automatically run recently failed, new, and modified tests first. This allows for fast failing, making the response times much faster. Furthermore, TeamCity does not wait until all the tests are completed to report that the build broke.

Too bad I have to use Team Foundation Build!

Comparason of Continuous Integration solutions

2009-02-18T01:17:00.001-08:00

I just came across a report called "Continuous Integration: What companies expect and solutions provide" by Georg Fleischer. If you are considering buying a new continuous integration system you should read this first. It really gives a good comparison of what features are provided in the different solutions.

The report has a section where "Individual features and concepts are discussed. These features and concepts are specific to some of the solutions, but they may be important to you. So have a look.

Treating dependencies between projects

2009-02-18T00:09:00.001-08:00

I have previously been writing about dependency management between projects in Team Foundation Server when doing continuous integraiton. This is basically a missing feature.

However, Cruse 1.2 does support this in a nice way. As you can see from the picture, a successful build can trigger a build of another project. Furthermore, Cruse Control keeps track of which build triggered which build. This information will typically get lost when kicking off a build from the msbuild task in Team Foundation Build.

Team Foundation Build Server -- Dependency Management

2009-02-16T01:25:00.000-08:00

After reading about other continuous integration systems, I see that Team Foundation Build Server is missing many of the advanced features. I heave earlier written about pre-test commits, but this time I will take about dependency management.

With Cruse Control you can have dependencies between your Cruse Control projects. It means that if a build completes successfully, a new build of any project dependent on this first project is started. Furthermore, artifacts may be shared between the projects.

However, in Team Foundation Build you are basically forced to build everything from scratch, or check the binaries into the source control system. This can get very complicated if multiple projects shares a component such as a assembly.

Working with source control systems

2009-02-16T00:50:00.000-08:00

When working with source control systems, I see from time to time developers committing multiple changes in a short period of time after being silent for many days. This is an anti pattern, and I will describe it below.

What is really going on is that the developer is doing some major changes on his computer. After spending a week or two with the changes -- often on unrelated parts of the code -- they are ready to commit the changes. When they try to commit, they figure out that some of the files has changed, so they have do to a merge. After the merge, they commit the files they think are related at once. The combination they choose is often okay, however the develop often misses a point: they do not commit all the changes that are required, so the build will fail. This may be because they change an interface (which is okay), but they did not include all the related changes where the consumers of the interface are changed as well. After doing maybe five commits, the build server is able to compile the program again, but the build still fails due to a failing automated test.

At that time, the defect indicated by the automated test is not fixed because the develop is busy with "something else".

By doing commits more often, I think much of this can be avoided. Furthermore, I hope that the introduction of pre-test commits will mitigate most of this by forcing the developer to only commit working change sets.

Let me state it one more time: Every change set should build!

Continuous Integration != Having a Build Server

2009-02-16T00:44:00.000-08:00

A common misconception is that you are practicing continuous integration by just setting up a build server to continuously build the most recent source code.

The reason why is a misconception is that you are not really doing continuous integration if:
* You ignore broken builds -- a broken build should be fixes NOW, not tomorrow!
* You just compile the source code, but does not run any tests. The test coverage should be sufficient as well.
* Failed tests do not fail the build.
* Developers do not check in every day.
* Developers do not check in change set -- the smallest working changes. Changes should work together when you check in.

Team Foundation Build running test agains database

2009-02-15T05:54:00.000-08:00

I just read an old post where the author included the following code:

<TestDBServer>fredTestDBServer>
<TestDBName>$(BuildNumber)TestDBName>
<TestDBCreateConnectionString>
data source=$(TestDBServer);integrated security=SSPI;Pooling=true
TestDBCreateConnectionString>

I then just realized that I should change my current practice of having a fixed database name and instead attaching a new database every build. And the perfect database name could be $(BuildNumber), which should be unique -- each new build will get a new database and for each build definition.

Just do not forget to detach the data base, or your sql server will have a lot of databases after a while.

Some reasons why this is probably a good idea is that if a build for some reason fails to free the database all the subsequent build will not fail.

OLE for Process Control and Continuous Integration

2009-02-13T04:58:00.000-08:00

OLE for Process Control (OPC) is an interface for getting simple values such as booleans, floats, doubles from one machine to the other. It is used heavily in process control. Measurements such as pressures, temperatures, speeds, flow rates are transferred from one system to another periodically (e.g. every second).

Anyway, when doing development of such applications you want to make sure that you get everything right. It means that you want to see that the given OPC input gives the correct output. So basically you write unit or component tests that does that. You could mock the OPC interface class, or you could just operate with an OPC Simulator what allows other applications to write values.

Usually we do the last thing to get enough confidence in the application. Unfortuantly I have not been able to find a good OPC simulator. I have been using Matrikon OPC Simuator, but it is not really made for automation. In order to define tag items (the information that is shared), you will have to use the GUI. And even staring the server requires some manual actions.

So... I am looking for alternatives. If you know some, please give me some feedback in the comment area.

Team Foundation Build Load balancing

2009-02-13T04:51:00.000-08:00

When doing continuous integration, you may often run into a situation where you do a commit, and then 30 seconds later you do a new commit. Or maybe not you, but your colleague. In this situation the builds will typically queue up because you have only one build server for each project.

For Team Foundation Build I came across Team Founation Bulid Load Balancer. It monitors the build queues of your project, and if it finds a queue at one build server and an idle build server, it simply moves the build from the one server to the other. Nice?

Of cource, now you are really forced to make both build server equal. This means that the installed software should be the same on bother servers. If not, a build will pass at one server, and fail at the other.

Pre-test commit

2009-02-13T04:41:00.000-08:00

From reading Paul M. Duvall's "Continuous Integration: Improving Software Quality and Reducing Risk" book, I learned about the pre-test commit feature of continuous integration systems.

In the standard continuous integration system you do a private build and run your unit tests. If you like the result, you commit to the source control system. The continuous integration system gets the latest source code and starts a build. if the build succeeds, everything is fine. However, sometimes it does not because of other commits between the private build and checkin, differences between the private environment and the continuous integration environment.

However, with a pre-test commit system, your changes are not really commited before a build succeeds on the build machine. This means that broken code will not get into the source control, and the commiter is forced to fix and recommit the code.

What I like about this way of operating is that it makes it much easier to enforce the continuous integration system; it is not possible to just ignore it.

I know TeamCity includes support for this, but reading about Team Foundation Server 2010, I can see that this will soon become a part of my continuous integration solution as well.

Build Notifications

2009-02-13T04:13:00.000-08:00

What is your favorite build notification tool when doing continuous integration? Please use the comment of this post to share your view.
Personally I like to have a system try application, but I also like the idea of having a big screen in the coffee area showing the current status.
Here is a list of methods I know about:
* System try
* Email
* Sound
* Big screen
* Web page
* Java lamp
* RSS feed

I guess the most used is a complaining colleague. Telling that you broke the build. It must be a better way?

What is continuous integration -- really?

2009-02-13T03:45:00.001-08:00

Continuous integration is an interesting topic, however we mean different things by it.

Here is a list of what I think should be included in a minimum continuous integration:
* Every checking should trigger a build
* The build does a clean compile and linking of the software
* Unit and custumer tests should be ran on each build

But the result of the above is just a collection of .exe and .dll files. What you really want is a .msi file or similar that you can ship to your customer. This .msi file should be build! I will later talk about how you can do that using WiX -- Windows Installer XML.

Continuous (test) deployment is also something that should be considered. After each successful build, the software should be deployed to a test envirnoment so that you can always have a look at what is current. This typically requires some more thought.

Personally I have not worked so much with continuous deployment because the challenges has been to get all the basic pieces together, but I firmly beleave that having a test environment that is always up to date with the most recent successful build is something that will give very much.

For further information on Continuous Integration, have a look at Martin Fowler's "Continuous Integration: Improved Software Quality and Reduction Risk" book.

Team Foundation Build Hotfix improving build speed

2009-02-13T02:25:00.000-08:00

Team Foundation Build 2008 SP 1 includes a bug that causes a project to be build multiple times, one extra time it is referred to by another project. This makes it very slow. I just found the hot fix for this, and it cut down the build time by a 20 seconds for a small build I have.

Try it yourself at by downloading the hotfix.

ConfigurationManager.ConnectionStrings and Unit Testing in .NET

2009-02-12T00:51:00.000-08:00

We have a lot of legacy code where we use ConfigurationManager.ConnectionStrings["MyConnectionString"] to get the connection string to the database. Some of the tests I would like to make is to see what happens if the database fails to open (or something like that). Because of the (bad) design, I am not able to change this connection string programmatically.
Does anyone have a suguestion of how to do this in the best way?
ConfigurationManager.ConnectionStrings["MyConnectionString"].ConnectionString = "MyNewString";
reports:
System.Configuration.ConfigurationElement.SetPropertyValue(ConfigurationProperty prop, Object value, Boolean ignoreLocks)
System.Configuration.ConnectionStringSettings.set_ConnectionString(String value)

Test-Driven Development: By Example

2009-02-10T04:07:00.000-08:00

I just want to recommend a book this time. It is called Test-Driven Development: By Example. The book is written by the test-driven development guru Kent Beck. It is a must have for every test-driver.
If you spend a few hours reading this book you will definitely improve your code quality and reduce the number of defects significantly. Of course, I assume you follow the practice.
Give it a try!

Continuous Integration – What are your alternatives

2009-02-10T04:04:00.001-08:00

We currently using Team Foundation Build as our continuous integration system, but I know there are other alternatives. Here is a list I found at Wikipedia:

AnthillPro — a continuous integration server from Urbancode, very feature rich supports all major build related technologies.
Apache Continuum — a continuous integration server supporting Apache Maven and Apache Ant.
Apache Gump — Apache's continuous integration tool.
Atlassian Bamboo — commercial continuous integration server from Atlassian.
Automated Build Studio — commercial automated build, continuous integration and release management system by AutomatedQA.
BuildBot — a Python/Twisted-based continuous build system.
CABIE — Continuous Automated Build and Integration Environment. Open source, written in Perl, works with CVS, Subversion, AccuRev, Bazaar and Perforce.
Cascade — commercial continuous integration tool. Provides a "checkpointing" facility by which changes can be built and tested before they are committed.
Cruise — Commercial continuous integration and release management server from ThoughtWorks, the creators of CruiseControl and CruiseControl.NET.
CruiseControl — Java-based framework for a continuous build process.
CruiseControl.NET — .NET-based automated continuous integration server.
CruiseControl.rb - Ruby based CI server.
Draco.NET — .NET-based automated continuous integration server inspired by CruiseControl.
ElectricCommander — commercial build-test-deploy automation solution developed by Electric Cloud.
Hudson — MIT licensed, written in Java, runs in servlet container, supports CVS, Subversion, GIT, Ant, NAnt, Maven, and shell scripts.
TeamCity — commercial continuous integration server from JetBrains with free professional edition.
Team Foundation Build - Microsoft's commercial continuous integration server and source code repository.
Tinderbox — a Mozilla based product written in Perl.
SCLM — an SCM system part of z/OS to allow integrated source control, build and promotion of work packages as a work area control system. It is part of IBM Rational Software's adaptive framework.
Xinc — a continuous integration server written in PHP 5. It has built-in support for Subversion and Phing (and therefore PHPUnit), and can be easily extended to work with alternative version control or build tools. Google project.

If you known some other alternatives, please let me know? Later I will try to give a comparison of the alternatives.

Try Resharper – it really improves your code quality

2009-02-09T02:23:00.001-08:00

Resharper is an excellent plug-in to Visual Studio. It exist the basic refactoring such as Extract Method and Rename Method/Variable. There is a free 30 day trial, so go ahead and try it yourself. I ended up buying the software.

In addition to the refactoring part, it is really a tool for resharping your code (I guess that's why they gave it that name). For instance, using the StyeleCop plug-in you may easily check the compliance to coding standards as you.

BuildWatcher – continuously keeping an eye on your continuous integration builds

2009-02-09T00:37:00.001-08:00

If you are like me, you have multiple software projects or productions that you want to keep in good shape. For me, this means bringing attention to broken builds so that the team fixes it as soon as it breaks.

Each team member may install the Build Notification tool that comes with Team Foundation Server 2008 Power Tools. This tools pops up a message when a build is queued, started, and completed. Unfortunately, there are some bugs in the software so that every time the TCP/IP connection breaks the software stops giving updates. This reduces the value of the software significantly because we are all laptop users doing frequent docks and undocks. I really hope Microsoft addresses this in the next release of the Power Tools.

Another way to bring focus to the team is to show the build status on a big screen next to the coffee machine. Lately, I have been developing such software.

The software continuously pools the Team Foundation Server for updates on the latest builds and shows the results. When the build fails the status is observed by a coffee thirst team member.

Enjoy!

Matlab Compiler and Team Foundation Build

2009-02-08T14:09:00.001-08:00

At my company, we have many internal tools written in Matlab, which is an interpreted language. Until recently these programs were distributed as source code, and the user was required to have Matlab installed on the computer. This made it very easy for the user to do modifications to the program in order to make it fit his needs and fix bugs. Unfortunately, this also meant that many users changed the source code without using source control. Furthermore, bugs were often just fixed locally and never reported back to the maintainers of the software. Thus, the robustness of the software did not improve very much, and each of the changes would have to be manually merged each time they upgraded the software. The result was that the software was "never" upgraded once installed.

To mitigate some of these concerns, I decided to use the Matlab Compiler to make an executable of the program and shipping this executable instead of the source code. This meant that the users were forced to use an official version, and thus that all the bugs were fixed once and for all.

Unfortunately, the build process requires a few steps making it error prone. The build process has to figure out which files to compile in the executable, and in some cases the algorithm misses a file. In such cases the program will not work.

After successfully applying continuous integration to other C#-based projects I was wondering if it would be possible to use the same practice here as well. I figured out that I could start the .bat file I had used for generating the executable directly from the msbuild script. I was thinking about doing some unit testing as well, and it would be nice to run the tests as part of the build process. I managed to do this by compiling the unit test as a separate executable. The executable was then started from the msbuild script.

<Target Name="AfterTest">

<Exec Command="$(OutDir)\MyApp_test" />

</Target>

By ending the build code by an assert statement that checked if the unit test run was successful, the exit code of the application would be non-zero when a unit test failed. By hocking the test up to the AfterTest target, the build gets a partial successful build when a test fails. This is the very same way as MSTest works in Team Foundation Build.

To summarize, I had the continuous integration framework up-n-running in a few hours. I guess this is also relevant for other languages as well.

Welcome to my new blog on agile practices

2009-02-08T13:42:00.001-08:00

I am a big fan of agile practices. In this blog I will share some of my experiences made in my work. Two of the practices I have been working much on implemented are continuous integration and test driven development. Expect that this blog will provide you with valuable lessons learned.