Some fascinating graphs of World Bank Data

• CO2 emissions per capita: Carbon dioxide emissions in metric tons per capita.


• Energy use per capita: Primary energy use (before transformation to other end-use fuels) in kilograms of oil equivalent, per capita.


• Electricity consumption per capita: Electricity consumption in kilowatt-hours per capita.


• Exports as percentage of GDP: Value of exported goods and services as percentage of GDP.


• Fertility rate: The average number of births per woman.


• GNI per capita in PPP dollars: Gross National Income per capita, expressed in purchasing power parity dollars to adjust for price level differences across countries. Not adjusted for inflation.


• Gross Domestic Product: GDP in current U.S. dollars. Not adjusted for inflation.


• Gross National Income in PPP dollars: Gross National Income, expressed in purchasing power parity dollars to adjust for price level differences across countries. Not adjusted for inflation.


• GDP deflator change: Annual change of the implicit GDP deflator shows the rate of price change in the economy as a whole.


• GDP growth rate: Percentage change of real GDP compared to previous year. Real GDP is adjusted for inflation.


• Imports as percentage of GDP: Value of imported goods and services as percentage of GDP.


• Internet users as percent of population: People with access to the Internet per 100 inhabitants.


• Life expectancy: The average number of years a newborn is expected to live with current mortality patterns remaing the same.


• Military expenditure as percentage of GDP: Military expenditure as percentage of Gross Domestic Product.


• Mortality rate, under 5: The probability per 1,000 that a newborn baby will die before reaching age 5.


• Population: Estimates of the resident population.


• Population growth rate: Percentage change of resident population compared to previous year.

Source: Official Google Blog: World Bank public data, now in search

Deep cloning without serialization.

A client asked me to write an object cloning implementation today that did not use serialization. Sometimes you cannot mark the types you need to clone as serializable.

I ended up with the code below. In the future, I'll modify it to use IL or something to lose the reflection performance hit.

It's implemented as an extension method so that it should work for any object. Enjoy!

using System;
using System.Collections;
using System.Reflection;

public static class Extensions
{
   /// 

   /// Clones the specified object.
   /// 
   /// 
   /// The object.
   /// 
   public static T Clone(this T obj)
   {
       var type = obj.GetType();
       var clone = Activator.CreateInstance(type);
       var fields = clone.GetType().GetFields();
       int fieldIndex = 0;
       foreach (FieldInfo fieldInfo in type.GetFields())
       {
           fields[fieldIndex].SetValue(clone, fieldInfo.FieldType.Implements("ICloneable") ? ((ICloneable)fieldInfo.GetValue(obj)).Clone() : fieldInfo.GetValue(obj).Clone());
           if (fieldInfo.FieldType.Implements("IEnumerable"))
           {
               var enumerable = (IEnumerable)fieldInfo.GetValue(obj);
               if (fields[fieldIndex].FieldType.Implements("IList"))
               {
                   int i = 0;
                   IList list = (IList)fields[fieldIndex].GetValue(clone);
                   foreach (var item in enumerable)
                       list[i++] = item.Clone();
               }
               else if (fields[fieldIndex].FieldType.Implements("IDictionary"))
               {
                   IDictionary dictionary = (IDictionary)fields[fieldIndex].GetValue(clone);
                   foreach (DictionaryEntry entry in enumerable)
                       dictionary[entry.Key] = entry.Value.Clone();
               }
           }
           fieldIndex++;
       }
       return (T)clone;
   }

   /// 

   /// Returns true if the type implements the specified interface, otherwise false.
   /// 
   /// The type.
   /// Name of the interface.
   /// 
   public static bool Implements(this Type type, string interfaceName)
   {
       return type.GetInterface(interfaceName, true) != null;
   }
}

Hosting multiple WCF Services under a single Windows Service

I am often asked how to host multiple services under a single windows service and through trial and error have been streamlining this process for clients.

I thought I'd blog an example here for reference:

class Program {
    static void Main() {
        if (Environment.UserInteractive) {
            ServiceManager serviceManager = new ServiceManager();
            serviceManager.OpenAll();
            Console.ReadKey();
            serviceManager.CloseAll();
        }
        else
            ServiceBase.Run(new WindowsService());
    }
}

public class WindowsService : ServiceBase
{
    public static string WindowsServiceName = "Windows Service Name";
    public static string WindowsServiceDescription = "Windows Service Description";
    public static string WindowsServiceUsername = @".\username";
    public static string WindowsServicePassword = "password";

    private readonly ServiceManager serviceManager = new ServiceManager();

    private readonly IContainer components = new Container();

    protected override void Dispose(bool disposing) {
        if (serviceManager != null) serviceManager.CloseAll();
        if (disposing && (components != null)) components.Dispose();
        base.Dispose(disposing);
    }

    public WindowsService() {
        ServiceName = WindowsServiceName;
        CanStop = true;
    }

    protected override void OnStart(string[] args) {
        base.OnStart(args);
        serviceManager.OpenAll();
    }

    protected override void OnStop() {
        serviceManager.CloseAll();
        base.OnStop();
    }
}

public class ServiceManager {
    readonly List serviceHosts = new List();

    public void OpenAll() {
        OpenHost();
        OpenHost();
        ...
    }

    public void CloseAll() {
        foreach (ServiceHost serviceHost in serviceHosts)
            serviceHost.Close();
    }

    private void OpenHost() {
        Type type = typeof(T);
        ServiceHost serviceHost = new ServiceHost(type);
        serviceHost.Open();
        serviceHosts.Add(serviceHost);
    }
}

/// 
/// Enables application to be installed as a Windows Service by running InstallUtil
/// 
[RunInstaller(true)]
public class WcfServiceHostInstaller : Installer {
    public WcfServiceHostInstaller() {
        Installers.Add(new ServiceInstaller
                           {
                               StartType = ServiceStartMode.Automatic,
                               ServiceName = WindowsService.WindowsServiceName,
                               Description = WindowsService.WindowsServiceDescription
                           });
        Installers.Add(new ServiceProcessInstaller { Account = ServiceAccount.User, Username = WindowsService.WindowsServiceUsername, Password = WindowsService.WindowsServicePassword });
    }
}

And some configuration:

- Here, the binding & behaviour configuration is shared across services but you may need different configurations for different types of services.
- I use different ports for different services, but you don't have to.

<system.serviceModel>
  <services>
    <service behaviorConfiguration="DefaultBehavior" name="Namespace.Service1">
      <endpoint address="" binding="netTcpBinding" bindingConfiguration="TCPBindingConfig" name="TCPEndpoint" contract="Namespace.IService1" />
      <endpoint address="mex" binding="mexTcpBinding" bindingConfiguration="" name="TcpMetaData" contract="IMetadataExchange" />
      <host>
        <baseAddresses>
          <add baseAddress="net.tcp://localhost:8001/Namespace/Service1" />
        </baseAddresses>
      </host>
    </service>
    <service behaviorConfiguration="DefaultBehavior" name="Namespace.Service2">
      <endpoint address="" binding="netTcpBinding" bindingConfiguration="TCPBindingConfig" name="TCPEndpoint" contract="Namespace.IService2" />
      <endpoint address="mex" binding="mexTcpBinding" bindingConfiguration="" name="TcpMetaData" contract="IMetadataExchange" />
      <host>
        <baseAddresses>
          <add baseAddress="net.tcp://localhost:8002/Namespace/Service2" />
        </baseAddresses>
      </host>
    </service>
    ...
  </services>
  <bindings>
    <netTcpBinding>
      <binding name="TCPBindingConfig" maxBufferSize="5242880" maxReceivedMessageSize="5242880">
        <readerQuotas maxStringContentLength="5242880" />
        <security mode="None" />
      </binding>
    </netTcpBinding>
  </bindings>
  <behaviors>
    <serviceBehaviors>
      <behavior name="DefaultBehavior">
        <serviceMetadata httpGetEnabled="false" />
        <serviceDebug includeExceptionDetailInFaults="true" />
        <serviceThrottling maxConcurrentCalls="21" maxConcurrentSessions="50" />
      </behavior>
    </serviceBehaviors>
  </behaviors>
</system.serviceModel>

Shuffle: an extension method of random uselessfulness

UPDATE: I have updated this method here: http://thegrenade.blogspot.com/2010/02/when-random-is-too-consistent.html

Have you ever needed to randomly sort a list in C#? Add this to your nifty extensions library:

/// <summary>
/// Shuffles the specified list (Extension Method for any IList<T>).
/// </summary>
/// <remarks>
/// Algorithm described at: http://en.wikipedia.org/wiki/Fisher-Yates_shuffle
/// </remarks>
/// <example>list.Shuffle();</example>
public static void Shuffle<T>(this IList<T> list)
{
    Random rng = new Random();
    int n = list.Count;
    while (n > 1) {
        n--;
        int k = rng.Next(n + 1);
        T value = list[k];
        list[k] = list[n];
        list[n] = value;
    }
}

Source control shootout

I like to think that I keep up with changes in the continuous integration space but somehow I missed an important shift in source control thats been taking place over the last year and a half. While I was busy promoting subversion and TFS as SourceSafe replacements, the community has abandoned centralised source control alltogether in favour of distibuted variants. Mercurial, like Git and Bazaar is a distributed version control system and needs no central server control. If a server wants revisions of a project it simply clones or pulls revisions from clients it is interested in.

Large open source projects have used the model for many years with notable successes like the linux kernel but a recent shift by Mozilla to Mercurial as well as Google Code's adoption of Mercurial was what caught my attention. Having now caught up on my research of the subject, I feel like I've arrived late to the party. I'm not going to say too much about what I've learned but the long and short of it is this:

If you're still using centralised source control, stop it. At the time of writing, you have two options: Git or Mercurial. I leave it as an exercise for the reader to understand why if you don't already know...

Using expression trees for more elegant code

Caching WCF responses without making a mess in the code base is often problematic due to connection pooling limitations and the requirement to close connections after use.

Many implementations end up something like: attempt to retrieve entity from cache > if absent, instantiate client proxy > retrieve entity from service > close connection > add entity to cache.

Problems arise because it is possible that entities for a single solution may need to be supplied by numerous services and the service method signatures may differ for various entities or collections of entities. This makes it difficult to create a generic wrapper capable of populating any number of varying entity types let alone the associated CRUD calls for entity persistence, etc. Entity identifiers may differ in type between services or even within a service. When a client side caching mechanism is required, the code becomes cluttered with the numerous caching calls and connection instantiations and tear-downs.

Any one who has worked on large WCF driven apps knows that this kind of clutter is butt-ugly and a pain to maintain or add features to.

Expression Trees to the rescue:

public static class Services
{
  /// <summary>Gets the item.</summary>
  /// <typeparam name="TProxy">The type of the proxy.</typeparam>
  /// <typeparam name="TEntity">The type of the entity.</typeparam>
  /// <typeparam name="TIdentity">The type of the identity.</typeparam>
  /// <param name="expression">The expression.</param>
  /// <param name="proxy">The proxy.</param>
  /// <param name="id">The id.</param>
  /// <returns></returns>
  /// <remarks>Method first checks the cache for an entity with a matching id, if found, it is returned.
  /// If the cache does not contain the entity, the wcf service method in the supplied expression tree
  /// is used to obtain the entity which is then added to the cache.</remarks>
  /// <example>Product p = GetItem((client, identifier) => client.GetProduct(identifier), new CatalogServiceClient(), 123);</example>
  private static TEntity GetItem<TProxy, TEntity, TIdentity>(Expression<Func<TProxy, TIdentity, TEntity>> expression, TProxy proxy, TIdentity id)
    where TEntity : class
    where TProxy : ICommunicationObject
  {
    TEntity item = Cache.GetItem<TEntity, TIdentity>(id);
    if (item == null)
    {
      try
      {
        var originalDelegate = expression.Compile();
        item = originalDelegate.Invoke(proxy, id);
      }
      finally { proxy.Close(); }
      Cache.AddItem<TEntity, TIdentity>(item);
    }
    return item;
  }
}

Running .Net 2 web applications alongside .Net 4

I recently installed the Visual Studio 2010 RC on an XP dev machine and found that it reconfigured IIS to default to ASP.Net v4. That's not terrible, but it did stop me being able to debug my .Net 2 & 3 applications under IIS and I started getting the old "Unable to start debugging on the web server" error message.

It's simple enough to fix however. Just run the following command for each virtual directory that needs to run under .Net 2:
C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\aspnet_regiis /s W3SVC/1/ROOT/<YourVirtualDirectoryName>

Quirky psexec behaviour when run from MSBuild.

Part of a deployment process I have been working on requires that our build server deploy some WCF services to a remote host. This is accomplished by hosting the WCF services within a Windows service that listens for TCP connections.

We ran into some quirky behaviour with psexec hanging during the execution of InstallUTil on the remote host. I presume that installutil was prompting for some sort of confirmation and not getting it. I added the -d switch to the psexec command and then noticed that psexec started returning inconsistent exit codes. It turns out that -d causes the PID of the remote process to be returned as the exit code as well as suppressing any interactivity. I can see this being useful in many psexec scenarios but it makes msbuild think something has gone wrong and this means that I also needed to add an IgnoreExitCode attribute to my exec tag like so:

<Exec Command="psexec -d /accepteula \\RemoteHostName C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\InstallUtil.exe /i /LogFile=D:\Logs\ServiceInstallation.log D:\Services\WcfServiceGroupHost.exe" IgnoreExitCode="true" />

Running Visio SaveAsWeb in a web application or service

There are many reasons why you should not run Microsoft Office applications from inside a web application or service. But if you really want to do it anyway, here's what you need to know to use Visio SaveAsWeb on Windows Server 2003.

Create a new windows user. The Application Pool that your web application runs under will be using the Network Service account in its identity settings by default. Change this so that your Application Pool uses a custom identity account that you have created.

Add your new user to the local IIS_WPG group. Verify that Local Security Policy > Local Policies > User Rights Assignment > Log on as a service grants this role to IIS_WPG.

Run DCOMCNFG and navigate to Component Services > Computers > My Computer > DCOM Config > Microsoft Office Visio Drawing > Properties > Security > Launch and Activation Permissions > Customize > Edit. Add the IIS_WPG group and allow Local Launch & Local Activation rights.

Add the following code to somewhere appropriate in your application:

using System;
using Microsoft.Office.Interop.Visio;
using Microsoft.Office.Interop.Visio.SaveAsWeb;

public static class Visio
{
  public static bool Convert(string pathToVsd, string pathToHtml, string pageTitle)
  {
    bool result;
    try
    {
      InvisibleAppClass visio = new InvisibleAppClass();
      VisSaveAsWeb saveAsWeb = (VisSaveAsWeb)visio.Application.SaveAsWebObject;
      saveAsWeb.AttachToVisioDoc(visio.Documents.Open(pathToVsd));
      VisWebPageSettings webPageSettings = (VisWebPageSettings)saveAsWeb.WebPageSettings;
      webPageSettings.TargetPath = pathToHtml;
      webPageSettings.PageTitle = pageTitle;
      webPageSettings.DispScreenRes = VISWEB_DISP_RES.res768x1024;
      webPageSettings.QuietMode = 1;
      webPageSettings.SilentMode = 1;
      webPageSettings.NavBar = 0;
      webPageSettings.PanAndZoom = 0;
      webPageSettings.Search = 0;
      webPageSettings.OpenBrowser = 0;
      webPageSettings.PropControl = 0;
      saveAsWeb.CreatePages();
      visio.Quit();
      result = true;
    }
    catch (Exception ex)
    {
      //Put your error logging code here:
      //Log.WriteEntry(string.Format("Failed to convert {0} to {1}.", pathToVsd, pathToHtml), ex);
      result = false;
    }
    return result;
  }
}

Following is a usage example:

protected void UploadButton_Click(object sender, EventArgs e)
{
  if (FileUpload1.HasFile && FileUpload1.FileName.ToLower().EndsWith(".vsd"))
  {
    string documentName = FileUpload1.FileName.Substring(0, FileUpload1.FileName.Length - 4);
    string newFilename = string.Format("{0}-{1:yyyyMMddHHmmss}", documentName, DateTime.Now);
    string uploadFolder = Server.MapPath("Diagrams");
    string pathToVsd = uploadFolder + "\\" + newFilename + ".vsd";
    string pathToHtml = uploadFolder + "\\" + newFilename + ".htm";
    FileUpload1.SaveAs(pathToVsd);
    if(Visio.Convert(pathToVsd, pathToHtml, documentName))
      Response.Redirect("Diagrams/" + newFilename + ".htm", true);
  }
}

LINQ vs Lambda vs loop - A performance test

I was wondering about the benefits of using Linq over Lambda or vice versa for data filtering. I noticed that some of Scott Gu's recent MVC posts contain examples like:

var products = from p in northwind.Products where p.Category.CategoryName == "Beverages" select p;

Personally, I prefer the Lambda construct:

var products = northwind.Products.Where(p => p.Category.CategoryName == "Beverages");

This is largely a personal preference issue, but being regularly involved in architectural discussion you often get asked to justify your personal preferences. So I checked the interweb for some metrics and came across an older post about LINQ over loops.

Back in 2007, Håvard Stranden posted some performance metrics for LINQ vs Loop using a Visual Studio 2008 beta. His test concluded that there was a performance hit in using LINQ over loops but that it was within acceptable ranges. Anders Hejlsberg and others pointed out that the test would benefit from removing some unnecessary casting.

I have modified the tests to remove the unnecessary casting, include metrics for using Lambda, and take advantage of the improvements to the framework in Visual Studio 2008 SP1. I experimented with several different approaches because the various methods have dependencies and return constraints. LINQ and looping depend on IEnumerable collections (IEnumerable) and return the same, Lamda depends on Genercic Lists (IList) and returns the same. To account for this, I decided that the methods could have their test data provided in the format they need it in before the stopwatch starts. This decision favours Lambda but real world scenarios where Lambda expressions are used also rely on this constraint. For the same reason, I removed the List instantiation in Stranden's loop test because this handicaps the loop implementation since a real world looping example doesn't need to instantiate a List in order to filter on some property data.

The test results show that the framework is now seriously biased in favour of Lambda implementations. Presumably, because of changes to Generic internals.

The test code as well as the results on my PC are shown below.

Test:

using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.IO;
using System.Linq;

namespace IterationPerformanceTest
{
    class Program
    {
        public static void Main(string[] args)
        {
            Trace.Listeners.Add(new TextWriterTraceListener(File.CreateText("Output.txt")));
            RunTests();
            Trace.Flush();
        }

        static void RunTests()
        {
            const int SIZE = 10000;
            const int RUNS = 1000;
            Stopwatch stopwatch = new Stopwatch();
            Random r = new Random();
            int[] ints = new int[SIZE];
            for (int i = 0; i < SIZE; i++)
                ints[i] = r.Next(int.MinValue, int.MaxValue);
            List intList = new List(ints);

            //LINQ
            stopwatch.Start();
            for (int t = 0; t < RUNS; ++t)
            {
                IEnumerable result = (from int i in ints where i < 10 select i);
            }
            stopwatch.Stop();
            Trace.WriteLine(string.Format("LINQ:   {0}, avg. {1}", stopwatch.Elapsed, new TimeSpan(stopwatch.ElapsedTicks / RUNS)));
            stopwatch.Reset();

            //LAMBDA
            stopwatch.Start();
            for (int t = 0; t < RUNS; ++t)
            {
                IEnumerable result = (intList.Where(i => i < 10));
            }
            stopwatch.Stop();
            Trace.WriteLine(string.Format("LAMBDA: {0}, avg. {1}", stopwatch.Elapsed, new TimeSpan(stopwatch.ElapsedTicks / RUNS)));
            stopwatch.Reset();

            //LOOP
            stopwatch.Start();
            for (int t = 0; t < RUNS; ++t)
            {
                foreach (var i in ints)
                {
                    bool match = (i < 10);
                }
            }
            stopwatch.Stop();
            Trace.WriteLine(string.Format("Loop:   {0}, avg. {1}", stopwatch.Elapsed, new TimeSpan(stopwatch.ElapsedTicks / RUNS)));

        }
    }
}

Results:

LINQ:   00:00:00.0019557, avg. 00:00:00.0004552
LAMBDA: 00:00:00.0000574, avg. 00:00:00.0000133
Loop:   00:00:00.0601094, avg. 00:00:00.0139906

Running your .Net Web Application under a custom user account on Windows Server 2003

Some web applications will benefit from running under a different user account to the default 'Network Service' account. I had to set this up today to support some server side Visio COM execution.

The user account is specified under the Identity tab for the Application Pool properties and whatever user you use will need access to the following resources & groups:

• Add to group: IIS_WPG


• Run %windir%\Microsoft.NET\Framework\v2.0.50727\aspnet_regiis -ga domain\username


• Local Security Policy > Local Policies > User Rights Assignment > Log on as a service

Static methods and classes are always evil

So, I'm looking for a new gig and I take a call from an agency that represents a prestigious .Net development team that's paying well and using all the latest toys. Fantastic! The agent reckons I'm suited to the role and sends me a copy of the "perfume test". It's a technical test with an interesting twist. The constraints are rather few and far between and the test description is really just a description of a business problem. "Write your own solution to the problem and make sure you use good design practice." I'm told that I have 48 hours to write my solution but I finish in 4 and send it back. It's a complete TDD solution with all the required unit tests and even goes the extra mile with a few extra bells and whistles to spruce it up. I was feeling pretty confident and gave them a couple days to respond.

No news... I call them and am told "Your solution was great but you made use of a static class, we don't like that. It's hard to test. You're out of the running."

Uh oh. Well, if ever anyone wanted a good reason why you should never use static classes, the definitive answer is: It makes you unemployable. Q.E.D.