Well, I wanted to provide a little update on my previous discussion on the my performance testing methodology; I've refined it a bit while getting ready for the Austin Code Camp.

Of course, GC.Collect() is still very important ... but I must correct myself in the previous post. It's called before each test method run. This ensures that the garbage collector is all cleaned up and collected before the test run even starts executing.

Now, on the calculations. I still do a normalized (or perhaps weighted, but we're getting into semantics here) average. But ... I've altered the equation a bit to subtract the overhead associated with the profiler probe. These were, surprisingly, pretty different across the board with the different test methods. It really is appropriate to discount these from the overall results as they do impact the overall numbers. And, considering the differences between them in the various methods (in one set of tests, it ranged from .1 msec to 2.54 msec), they really needed to be removed from the results.

The final tweak was to make a call to each of the test methods before I went into the actual test. This was done in a separate Initialize method. This ensures that all of the classes being used (as was mentioned in the previous post) are loaded into memory and initialized. It also ensures that the methods themselves are JIT'd before the test runs begin as well; again, this is something that we need to take out of the final equation.

Here's the latest on Brian Harry's blog: http://blogs.msdn.com/bharry/archive/2008/01/09/jan-08-devdiv-dogfood-statistics.aspx.  Keep in mind that this is just for DevDiv (Developer Division) ... so we're talking about the developer tools (i.e. Visual Studio, etc).  There are other groups using TFS inside of Microsoft.  Looking at this ... wow ... it gives you an appreciation of the amount of development that is done here. 

Speaking of TFS, I heard a customer story about TFS recently that really speaks to the value. So ... a remote dev center was coding away on an application, but Central IT noticed that they had a lot of bugs being re-opened and that overall, things didn't seem to be going too well.  Of course, re-opened bugs ... well ... that's definitely a problem. This was exposed to the Central IT team through the OLAP and reporting capabilities of TFS. But the data and processes ... well, it's easy and natural for devs to work with so they tend to actually do it. Having the working items completely integrated into VS and into your check-in process was a huge thing for me.  So ... Central IT was very concerned and they had the remote team come down to see them. They went over what was going on. As it turned out, the remote team didn't have a clear idea of the vision of the project and this was leading to a lot of issues. They got this resolved ... and the team got back to work. This time, though, the stats were better. It took the business another 3 weeks to notice this; by that time, the underlying issue had been corrected. Three weeks of development time that would have been ... ummm ... less than productive had already become very productive. It saved 3 weeks on a time-critical project. And, of course, the largest cost of a software project is the people ... the developers' time. Well, I thought it was cool.