Microsoft Live - Why I don't use it

Microsoft has been busy doing all kinds of neat stuff with their Live services. Mostly Live has always been an all-around disappointment. But recently, they've gotten competitive with Google Docs. I suppose that's because Google Docs is a very serious threat to the MS Office empire.

The new SkyDrive and Online Office 2010 stuff is fantastic. In many respects it kicks Google Docs in the teeth. Google may be the king of the cloud, but Microsoft does pretty UI's second only to Apple, while Google's UIs tend to suck-balls. Normally Google's minimalist approach works well, but not with stuff like this.

So it is possible that Microsoft could eat Google's face with Live if they wanted to.

But they aren't eating Google's face. Why? Well, because no one wants to use it... and probably for the same reasons I don't use Live myself. It basically boils down to just two problems:

1. The Advertising. Look, I know Microsoft wants to make ad revenue. That's what Google's doing after all, and it's working. But come on! Live uses an absurd amount of screen space for their ads. The ads are so in-your-face that you can't help but want to vomit every time you open the site.   
          
2. The advertising. Did I mention how distracting and annoying the advertising was? The entire Live setup is designed to make you switch from one page to another every time you try to do anything (unlike Google, which lets you do a lot from oneas intuitive page). This is transparently a gimmick to let them show you a new ad on each page... not because it makes any sense from a user interface perspective.
     

It's sad really... Microsoft has some really awesome cloud services, and they always have. But between their tarnished reputation, failure to market their products, and then finding some super annoying way to shoot themselves in the face, it just seems that no one really cares about Live. Unless someone over at MS gets a clue soon, I doubt they ever will either.

Pitty...

I expect that with slow adoption, Microsoft may do what they've been done with everything else that they've had trouble selling recently... find some way to tie it to X-Box Live, which is about the only Live related service anyone has ever cared about. Maybe when they start giving out achievements and unlocks when you level up your spreadsheet, then someone might be entertained enough to look past the rest of the crap.

MEF and MVC - Limitations and workarounds for partial trust environments

A while back I wrote about using MEF in MVC environments with extensions provided by Hammet for the Nerd Dinner MEF sample application. Those extensions deal with dynamic discovery of parts based on MVC conventions (instead of attributes), as well as per-request composition containers. The extensions work great, after a few modifications that I talked about in the last post... but in partial trust environments it blows up in your face!

BOOM!

I spent hours and hours digging through the code, reading about CAS, trust policies, transparent code and whole mess of other junk that I'd really rather not have rattling around my skull. Long story short -- MEF isn't friendly with partially trusted asp.net environments.

Now, you could write your custom MEF code in a class library, flag the assembly with the APTCA attribute, sign-it, and install it to the GAC if you want. That will get around these limitations neatly, but if you are running in partial trust you probably don't have the luxury of installing things to the GAC either.

The first major limitation is that you cannot access the parts collection within catalogs or containers. If you try it, your get an exception like this:

Attempt by method 'DynamicClass.lambda_method(System.Runtime.CompilerServices.Closure)' 
to access type 'System.ComponentModel.Composition.Hosting.ComposablePartCatalogCollection' failed.
  
  

The easiest way to reproduce the problem is to simply add the trust element to web.config like this:

<trust level="High"/>
  
  

Then add this to application startup in global.asax:

 var cat = new DirectoryCatalog(Path.Combine(AppDomain.CurrentDomain.BaseDirectory, "bin"));
 var parts = cat.Parts.ToArray();
  
   

In the Nerd Dinner MEF sample, this limitation effectively kills the mechanisms that slit up parts into per-request containers vs. application wide containers.

If you've done any reading about MEF online, you've likely run across code for a FilteredCatalog class. This thing is so commonly cited on the net that it seems beyond retarded that it wasn't built-in with MEF. But these limitations from partial trust kills FilteredCatalog; which the Nerd Dinner MEF sample uses heavily.

The other major area of limitation is that you cannot use ReflectionModelServices, which is needed in order to dynamically create export/import definitions programmatically. This kills the Nerd Dinner MEF sample's auto-discovery of controllers.

Despite these limitations, you can still use MEF in medium trust, but only if you are careful to keep it simple and straight forward.

Honestly, I recommend that you just use Ninject or a similar IoC/DI framework until the next version of MEF or MVC (hopefully) fixes these issues.

In my case though, I really wanted to be able to support medium trust environments and I'm too damned stubborn to give up on MEF that easy.

I'm OK with having to use the MEF attributes to decorate my controllers, so losing auto-discovery isn't much of a problem. Hammet's extensions are brilliant, but the auto-discovery mechanism is a lot of VERY complicated experimental code.

Now, the simplest thing you can do is just use a custom MVC ControllerFactory that instantiates a new MEF container on each request. That works well, and is trivially easy to implement:

public class MefControllerFactory : IControllerFactory
{
    public IController CreateController(RequestContext requestContext, string controllerName)
    {
        var catalog = new DirectoryCatalog(Path.Combine(AppDomain.CurrentDomain.BaseDirectory, "bin"));
        var requestContainer = new CompositionContainer(catalog);
        var controller = requestContainer.GetExportedValue(controllerName);

        if (controller == null){ throw new HttpException(404, "Not found");}
        
        return controller;
    }
}

Sure, this is fine, but it sort of undermines a lot of the power of MEF. MEF's default behavior uses a singleton pattern to reuse parts that have already been instantiated, but this mechanism eliminates ALL reuse, by recombobulating the entire container on each request. It also has an appreciable performance impact since reflection has to go over and build up the entire catalog each time too.

Another solution is to just create an application wide container, and just keep the controllers from being reused by setting the PartCreationPolicy attribute to NonShared. That's a better solution, and simple to achieve too. It looks something like this:

public static class ContainerManager
{
    private static CompositionContainer _container;
    public static CompositionContainer ApplicationContainer
    {
        get
        {
            if (_container == null)
            {
                var catalog = new DirectoryCatalog(Path.Combine(AppDomain.CurrentDomain.BaseDirectory, "bin"));
                _container = new CompositionContainer(catalog);
            }
            return _container;
        }
    }
}

Then your controller just uses the application container from this static class. Very simple, and allows you to control reuse using MEF's standard attributes.

I actually recommend the above approach, but it bothered me to mark controllers as NonShared. It isn't that controller instances cannot be reused, it's just that in MVC they can't be reused across multiple requests.
So I came up with a more ghetto solution that can sort-of mimic a FilteredCatalog even in medium trust. This allows for a pattern more similar to the Nerd Dinner MEF sample; you can have application scoped containers, and smaller per-request containers for just for the controllers too.

It looks a little something like this:

First create a class derived from HttpApplication so you can boot-strap creating the MEF containers and catalogs on application startup:

public class MefHttpApplication : HttpApplication
{
    public static ComposablePartCatalog RootCatalog { get; private set; }
    public static CompositionContainer ApplicationContainer { get; private set; }
    public static ComposablePartCatalog ControllerCatalog { get; private set; }

    protected virtual void Application_Start()
    {
        if (RootCatalog == null){ RootCatalog = CreateRootCatalog(); }
        if (ApplicationContainer == null)
        {
            ApplicationContainer = new CompositionContainer(RootCatalog, false);
        }
        if (ControllerCatalog == null)
        {
            var controllerTypes = Assembly.GetExecutingAssembly().GetTypes().Where(t => t.GetInterfaces().Any(i => i == typeof(IController)));
            ControllerCatalog = new TypeCatalog(controllerTypes);
        }
        ControllerBuilder.Current.SetControllerFactory(new MefControllerFactory());
    }

    protected virtual void Application_End()
    {
        if (ApplicationContainer != null){ApplicationContainer.Dispose();}
    }

    protected virtual ComposablePartCatalog CreateRootCatalog()
    {
        return new DirectoryCatalog(Path.Combine(AppDomain.CurrentDomain.BaseDirectory, "bin"));
    }
}

On startup we create a master catalog of every part definition, and an application scoped container from that master catalog. But we also create a catalog containing just the controller parts by using a bit of reflection to pull out just controllers and shoving them into a TypeCatalog (which is built-in with MEF)... the poor man's filtered catalog!

Now just doctor up Global.asax to inherit the MefHttpApplication class:

public class MvcApplication : MefHttpApplication
{
    public static void RegisterRoutes(RouteCollection routes)
    {
       //normal route stuff
    }

    protected override void Application_Start()
    {
        base.Application_Start();
        AreaRegistration.RegisterAllAreas();
        RegisterRoutes(RouteTable.Routes);
    }
}

And finally, we need our ControllerFactory:

public class MefControllerFactory : IControllerFactory
{
    public IController CreateController(RequestContext requestContext, string controllerName)
    {
        var requestContainer = GetRequestControllerContainer(requestContext.HttpContext.Items);
        var controller = requestContainer.GetExportedValue(controllerName);

        if (controller == null){throw new HttpException(404, "Not found");}

        return controller;
    }

    public void ReleaseController(IController controller){/*nothing to do*/}

    public static CompositionContainer GetRequestControllerContainer(IDictionary contextItemsCollection)
    {
        var app = (MefHttpApplication)HttpContext.Current.ApplicationInstance;

        if (contextItemsCollection == null) throw new ArgumentNullException("dictionary");

        var container = (CompositionContainer)contextItemsCollection["MefRequestControllerContainer"];

        if (container == null)
        {
            container = new CompositionContainer(MefHttpApplication.ControllerCatalog, false, MefHttpApplication.ApplicationContainer);
            contextItemsCollection["MefRequestControllerContainer"] = container;
        }
        return container;
    }
}

As you can see, the overall technique here is similar to that used in the Nerd Dinner MEF sample. We have a static method that we can call to build a per-request container. It stuffs the entire container into context in case its needed later. The key to the container itself is that it is built from our catalog of just controller types, and uses the application scoped MEF container as an export provider for any other parts the controllers might need to import.

In the long-run, this is probably no better than just marking our controllers as NonShared and using an application wide container, but the general concept of this technique can be applied to other situations besides just dependency injection with controllers. While you can't truly filter catalogs and manipulate part in partial trust, you can still use reflection to create specialized catalogs and achieve similar results... for the simpler cases anyway.

ASP.NET MVC 2: validation and binding issues with rich-text input, DataAnnotations, view-models, and partial model updates

Here I'm going to tackle a series of validation related annoyances with MVC 2 that tend to come up rather frequently.

This will include:

• Input Validation: Potentially Dangerous Request even when using [ValidateInput(false)] attribute.
    
• DataAnnotations validation warnings with partial model updates.
    
• Dealing with View-Model binding AND DataAnnotations validation warnings with partial model updates together.
    

OK... so, you are writing a page on the asp.net MVC 2 framework that creates a record in the DB for you.

You have a view. Nice!

You have a controller. Also nice!

The view needs select lists and stuff too, so you have a view-model. Sure thing!
Your controller relies on a model class to handle business logic like auto-populating values on the entity and similar. You bet!

And the entity itself is an EF 4 generated class. Nothing special there!

And you created a meta-data buddy class for the entity so you can flag fields with attributes and get MVC to automate your validation. Absolutely!

So you type in your values into the page, hit submit, and the page blows up!

Fuck!

Now, since you were using a view-model, your controller looks something like this:

[Authorize]
public virtual ActionResult Create()
{
    Ticket ticket = new Ticket();
    var model = new TicketCreateViewModel(ticket);
    return View(model);
}

[Authorize]
[HttpPost]
[ValidateInput(false)]
public virtual ActionResult Create(FormCollection collection)
{
    try
    {
        Ticket ticket = new Ticket();
        UpdateModel(ticket, collection);
        if(TicketService.CreateTicket(ticket))
        {
            RedirectToAction("Success");
        }
        else
        {
            return View(new TicketCreateViewModel(ticket));
        }
    }
    catch { return View(new TicketCreateViewModel(ticket)); }
}

The first error you are likely to come across will be a potentially dangerous request error. This will happen if your view tries to submit a field that contains HTML characters, like with a rich text editor.
Now, you used to be able to work around this by just flagging the action method with [ValidateInput(false)] and all was well; but not with MVC 2.0 you don't!

In .NET 4.0, the MS dev team decided to overhaul (badly) the input validation mechanism. They wanted to have the same mechanism work for asp.net, web services, and just about any other kind of request too. So they now have a new input validation system that operates so high-up in the request pipeline that it dies long before it actually gets to your controller to see if you've overridden the default behavior. More info in this whitepaper.

Basically, to work around this you need to tell ASP.NET to use the old .NET 2.0 validation mechanism instead of the fancy new one. You do this in web.config by adding this to the system.web section:

<httpRuntime requestValidationMode="2.0"/>
  
  

Basically, this change is so retarded, that any application with a rich text editor ANYWHERE has to turn off the new security feature for the entire application. The good news is that the old mechanism still protects asp.net pages, but you are on your own again for web services and other request types.

Good job guys!

Now, once you've fixed that up the next problem you'll likely run across is that the UpdateModel method has trouble populating the data into the right properties. This is because you are using a view-model class, but when the form is posted back you are trying to use UpdateModel on just an entity.

Generally this is pretty easy. You just tell the UpdateModel method the "prefix" for the values in the form that should match the entity you are updating. In my example here, the property in the view-model that had the ticket fields was called "NewTicket"... we we just alter the controller to supply the prefix "NewTicket" and the UpdateModel method can figure out what properties in the form data should go get shoved into our entity.

[Authorize]
[HttpPost]
[ValidateInput(false)]
public virtual ActionResult Create(FormCollection collection)
{
    try
    {
        Ticket ticket = new Ticket();
        UpdateModel(ticket, "NewTicket", collection);
        if(TicketService.CreateTicket(ticket))
        {
            RedirectToAction("Success");
        }
        else
        {
            return View(new TicketCreateViewModel(ticket));
        }
    }
    catch { return View(new TicketCreateViewModel(ticket)); }
}

This is also how the standard example apps for MVC typically do things. And it works fantastic as long as you are posting back ALL of the properties from the view, or at least all the ones that have validations via DataAnnotations attached to them.

But when you have a [Required] attribute from DataAnnotations on entity properties that are NOT being passed back from the view, then you run into the third major problem... the UpdateModel will blow up with validation errors for the fields that your view didn't post to the controller.

This is also due to a late-breaking, and very unwise, change in behavior that the asp.net team made right before MVC 2.0 was released.

The MVC validation stuff in 1.0 only validated properties that were posted up; a mechanism called input based validation. The problem with it is that a hacker could, in theory, get around triggering validation errors by hacking out required fields from the HTTP post. So, they decided to switch to "model based validation" where the validation would check the validity of ALL properties in the model even if they weren't included in the post.

Again... this is an ok idea, but the asp.net team should not have implemented such a breaking-change without giving you some way to easily override the behavior too... after-all, doing a partial model update from a form is NOT exactly an edge case scenario. It is damned common!

Now... Steve Sanderson (smart guy!) posted about a really slick action filter attribute way to handle partial model updates. In short, his mechanism allows you to flag an action method with an attribute, and an action filter will automatically loop in after the model binding is done and remove any model state validation errors for fields that weren't included in the post.

Now... suppose you were doing this instead of what we were doing above:

[Authorize]
[HttpPost]
[ValidateInput(false)]
public virtual ActionResult Create(Ticket newTicket)
{
 // stuff
}

With Steven's simple attribute, you could get the desired partial model update to work, without barfing on the data annotations by just flagging the action with one more attribute; like this:

[Authorize]
[HttpPost]
[ValidateInput(false)]
[ValidateOnlyIncomingValuesAttribute]
public virtual ActionResult Create(Ticket newTicket)
{
 // stuff
}

Fantastic!

Except that it only works if you are using a model binder, but we weren't. Remember, in our case we're using a view-model. We don't want to bind to the view-model on the postback... we just want to bind to an instance of the entity we are trying to create.

The UpdateModel method is NOT a model binder though largely it does the same thing as one. So the action filter attribute mechanism Steven describes doesn't work when using the UpdateModel method.

sigh...

OK, now I could just put similar code to what Steven's attribute was using in the controller itself. All his attribute does is loop through the model state and remove errors for fields that weren't in the posted form collection. But honestly, that's an ugly beast and I HATE having my controllers marshaling values around like that.

Now... this example is actually simplistic, and the easiest way to work around this is to use the default binder and bind to the model automatically. This can be done simply with the Bind attribute, which allows you to supply a "prefix" to use for the binding. It looks something like this:

[Authorize]
[HttpPost]
[ValidateInput(false)]
[ValidateOnlyIncomingValuesAttribute]
public virtual ActionResult Create
(
 [Bind(Prefix = "NewTicket")] Ticket ticket
)
{
 if(TicketService.CreateTicket(ticket))
 {
  RedirectToAction("Success");
 }
 else
 {
     return View(new TicketCreateViewModel(ticket));
 }
}

This works fantastic for simpler cases like this one. We just supply the prefix to use and the binder takes care of it.

But, for reasons I don't want to delve into here, my code was a tad more complex and this technique didn't quite work out for me...

So... what I did to work around this when the Bind attribute wasn't enough, was to create a custom model binder that could do the same thing we're doing with UpdateModel. Anyway, once we have a custom binder we can then modify the controller action to use the custom binder AND Steven's attribute both.

So... here was my custom model binder:

public class NewTicketModelBinder : DefaultModelBinder
{
    public override object BindModel(ControllerContext controllerContext, ModelBindingContext bindingContext)
    {
        bindingContext.ModelName = "NewTicket";
        return base.BindModel(controllerContext, bindingContext);
    }
}

Simple enough. All it does is supply the "prefix" to the binding context to achieve pretty much the same results as our UpdateModel method used to. Otherwise it uses the DefaultModelBinder's behavior as-is. Again though, this example is omitting some details that just aren't necessary to describe here, but my custom binder has a few other overloads besides what's shown here.

Now the controller looks like this:

[Authorize]
[HttpPost]
[ValidateInput(false)]
[ValidateOnlyIncomingValuesAttribute]
public virtual ActionResult Create
(
 [ModelBinder(typeof(NewTicketModelBinder))] Ticket ticket
)
{
 if(TicketService.CreateTicket(ticket))
 {
  RedirectToAction("Success");
 }
 else
 {
     return View(new TicketCreateViewModel(ticket));
 }
}

All I had to do here was declare what binder to use and specify Steven's [ValidateOnlyIncomingValuesAttribute]. I was also able to clean up the controller a bit, because now I don't need the try/catch for binding failures either.

Now... one thing I still didn't like was that the custom model binder here is specific to each view model. There isn't a clean way to supply the "prefix" to the binder without some ugly reflection or such. But I didn't like the idea that every time I needed to bind this way I'd have to make a new custom model binder.

So I settled on a "convention" for this and created a single custom model binder that could be used with any view-model, as long as it followed the convention:

public class ViewModelBinder : DefaultModelBinder
{
    public override object BindModel(ControllerContext controllerContext, ModelBindingContext bindingContext)
    {
        bindingContext.ModelName = bindingContext.ModelType.Name;
        return base.BindModel(controllerContext, bindingContext);
    }
} 

Here, the binder requires that the property on the view-model be named the same as the entity's type. So if we want to expose a Ticket entity, we need to name the property in the view-model "Ticket" too.  Previously in my view-model I was using the property named "NewTicket", so I just change that to use property name "Ticket" instead. This way the ViewModelBinder can always find the right prefix name to supply for the default binder.

This still seems like an awful lot of work for a common scenario. I sure hope the MVC team gets all this worked out in the next release of the MVC framework to support this kind of situation more elegantly.

Hooked on MEF - Using MEF in ASP.NET MVC, and the Nerd Dinner MEF sample fix

I'm about to embark on a major project in Silverlight 4. In advance of that project, I've been exploring some of the newer technologies such as WCF RIA Services and the Microsoft Managed Extensibility Framework (MEF); both of which shipped in .NET 4.0. While my usage for MEF in Silverlight is more about plug-in and modular architectures, I also have a good bit of interest in MEF as a basic IoC mechanism too.

To get a handle on MEF, I decided to plug it into my TicketDesk 2.0 project, which is being written on the ASP.NET MVC platform. TicketDesk 2.0 uses a class library for all the business and entity framework bits. I generally avoid the IoC design pattern though, preferring instead to just use overloaded constructors; one that takes dependencies for unit testing, and the other that supplies default dependencies to be used by the application at runtime. This technique is common, and is sometimes called a poor-man's IoC. Insults aside, it is simple, easy to code, and works. Traditional IoC implementations on the other-hand tend to add complexity that doesn't do much to advance the application's core functionality.

But MEF is interesting because it offers functionality that can improve how the application actually works, and it has the by-product of being a decent, and simple, IoC container too.

Anyway, once I started using MEF for IoC in TicketDesk, I ran into a slight problem. See, MEF was designed with persistent applications like Silverlight in mind. But in an MVC web environment you have multiple user requests coming in, and many of the objects cannot easily be shared across multiple request threads. So using MEF in this environment requires that you deal with the fact that some objects have to be instantiated on a per-request basis, while others might be scoped to the entire application.

Fortunately, there is this genius person named Hamilton Verissimo de Oliveira (Hammett). Hammet is apparently one of the core devs on the MEF project, and he's done a good bit of writing about MEF in MVC environments. According to his blog, he's even working with the MVC team to get MEF officially supported in the ASP.NET MVC 3 platform.

His most recent sample code is a modified MEF enabled version of the Nerd-Dinner MVC sample application. Scott Hanselman blogged about the MEF version of Nerd Dinner and hosts the downloadable version of the code.

The code in the MEF version of Nerd Dinner is basically a beta of an extended version of MEF for use in MVC scenarios. It includes two class libraries that extend MEF and MVC. These extensions do two things:

1. Provide lazy MEF compositions on a per-request or per-application basis as appropriate. The per-request composition allows your application to handle compositions only for objects you need to service a request at runtime, while the per-application composition can be used for shared application scoped needs.
     
2. Provide convention driven MEF design pattern. MEF is normally attribute driven, where you explicitly declare exports and imports by decorating your code with MEF attributes. But MVC applications are convention driven, so this feature set allows MEF to auto-discover composable parts based on similar conventions... for example, the nerd dinner application treats controllers as composable exports without you having to decorate them with the MEF attributes.

Overall, the code is a mostly complete MVC compatible set of MEF extensions. But the nerd dinner sample only really uses MEF for an IoC design pattern... the models are declared as MEF exports and the extensions handle supplying the appropriate dependencies to the controllers at runtime. But nothing about the way this same is setup should, in theory, prevent you from using MEF in other ways such as a plug-in extensibility mechanism (which is MEF's strong suit anyway).

But... once I got to using the extensions in TicketDesk, I ran into a couple of unusual problems.

First, code in my referenced class library wasn't being passed to my controllers. If I moved the code into the MVC app itself though, it worked like a charm. Second, code in my MVC application that was marked as exports using attributes weren't being passed into constructors for objects in my class library. The nerd dinner example contains the models and controllers both directly in the MVC application assembly itself, so these issues didn't occur there. But once you split your code into two assemblies, things didn't work too smooth. Well... this is just sample software.

So... to track down and fix these problems.

One of the more confusing bits about the nerd-dinner code sample is how the MEF catalogs are built when the app starts up. In global.asax.cs the sample code looks like this:

protected override ComposablePartCatalog CreateRootCatalog()
{
    var agg = new AggregateCatalog();

    foreach (Assembly assembly in BuildManager.GetReferencedAssemblies())
    {
        agg.Catalogs.Add(new AssemblyDiscoveryCatalog(assembly));
        agg.Catalogs.Add(new AssemblyCatalog(assembly));
    }

    return agg;
}

Basically this just loops through all the assemblies in the application and builds an Aggregate catalog of all the MEF composable parts from each assembly. What's odd though is that, for each assembly, it builds two separate catalogs and adds both to the aggregate catalog. One is a standard AssemblyCatalog, and the other is a custom type of catalog called an AssemblyDiscoveryCatalog which is part of the extended MEF code shipped with the Nerd Dinner example.

Now, the aggregate catalog is supposed to merge multiple catalogs, eliminating duplicates and all that... but why build two separate catalogs from each assembly in the first place? Shouldn't AssemblyDiscoveryCatalog alone contain all the parts from any one assembly?

Now... I have been unable to understand exactly what it is that causes the problems I was seeing. When I examine the catalogs in the debugger, and the way they are used, the aggregate catalog appears to contain all the composable parts it should. And when the controller factory is invoked, it finds the right controller, and uses a container that has all the right parts in it.

What I did find that was that by removing the duplicate catalog being created in global.asax.cs did fix the problem where my controllers weren't being given the imports when those parts were coming from the referenced class library.

I chose to remove the AssemblyCatalog from the aggregate, leaving just the AssemblyDiscoveryCatalog being added for each reference assembly.

protected override ComposablePartCatalog CreateRootCatalog()
{
    var agg = new AggregateCatalog();

    foreach (Assembly assembly in BuildManager.GetReferencedAssemblies())
    {
        agg.Catalogs.Add(new AssemblyDiscoveryCatalog(assembly));
        //agg.Catalogs.Add(new AssemblyCatalog(assembly)); 
    }

    return agg;
}

This worked fine, but then I ran into the second problem... if my class lib needed imports from the MVC application, they weren't getting them... basically the opposite problem.
Digging deeper into the second problem, I was able to find a cause. The AssemblyDiscoveryCatalog type has a minor bug:

private IEnumerable InspectAssembliesAndBuildPartDefinitions()
{
    var parts = new List();

    foreach(var assembly in this.Assemblies)
    {
        var attributes = assembly.GetCustomAttributes(typeof(DiscoveryAttribute), true);

        if (attributes.Length == 0)
        {
             parts.AddRange(new AssemblyCatalog(assembly).Parts);
        }
        else
        {
            foreach (DiscoveryAttribute discoveryAtt in attributes)
            {
                var discovery = discoveryAtt.DiscoveryMethod.New();
                var discoveredParts = discovery.BuildPartDefinitions(assembly.GetTypes());

                parts.AddRange(discoveredParts);
            }
        }
    }

    return parts;
}

First this code checks the assembly for a "DiscoveryAttribute". This attribute marks the entire assembly as one that contains classes for which the modified MEF system should "infer" composable parts based on conventions rather than by looking for the explicit MEF attributes. In the nerd dinner example, this attribute is declared at the top of the Conventions.cs class.

Notice how the code immediately afterwards works though. If the assembly isn't marked with the DiscoveryAttribute, it uses the standard MEF mechanisms to add all the parts to the catalog; those mechanisms do so by looking for the explicit MEF attributes in the code.

But when the assembly is marked with the DiscoveryAttribute, then the code does something quite different; it goes through the assembly looking for dynamically discoverable parts based on any defined conventions. It then adds those dynamically discovered parts to the catalog.

And that's the bug! The branch handling DiscoveryAttribute assemblies doesn't have any code that looks for traditional MEF parts declared by attributes!

So... all I had to do was modify this code so it adds both discoverable and inferred attributes both:

private IEnumerable InspectAssembliesAndBuildPartDefinitions()
{
    var parts = new List();

    foreach (var assembly in this.Assemblies)
    {
        var attributes = assembly.GetCustomAttributes(typeof(DiscoveryAttribute), true);

        parts.AddRange(new AssemblyCatalog(assembly).Parts); // add the standard MEF locatable parts
        if (attributes.Length > 0)
        {
            //add any convention inferred parts 
            foreach (DiscoveryAttribute discoveryAtt in attributes)
            {
                var discovery = discoveryAtt.DiscoveryMethod.New();
                var discoveredParts = discovery.BuildPartDefinitions(assembly.GetTypes());

                parts.AddRange(discoveredParts);
            }
        }
    }

    return parts;
}

Now we have a single kind of catalog that can locate both inferred and explicitly declared MEF parts in any assembly. And after this change, my MVC controllers are getting imports from the class lib, and the class lib is getting imports from the MVC app.

Everyone is happy...

Except that it still bugs me why having the two kinds of catalog added to the aggregate catalog keeps the controllers from getting imports from my class library. I suspect strongly though that this may be some kind of bug in the core MEF implementation, perhaps with how Lazy composition actually works.

But either way... fixing the bug in AssemblyDiscoveryCatalog allows you to handle everything in one kind of catalog and works around both problems.

A WMD markdown editor variant that works

I'm a big fan of markdown. If you've used Stack Overflow, then you are probably very familiar with markdown via their excellent variation of the WMD markdown editor. Markdown is popular in its way. A number of different web based content systems out there have adopted it, and there are plug-ins for a wide range of desktop utilities and development tools too.

But if you are just wanting a javascript based WYSIWYG style editor for use in your own web applications, you are kinda screwed; there just aren't a lot of markdown editors out there. I know of only two working markdown editors of any merit that aren't tied to some specific application; MarkItUp! by Jay Salvat and WMD by John Fraser.

WMD is fantastic, but John Fraser vanished from the net shortly after making WMD's initial code open source. His initial release was an obfuscated form, which makes it hard to deal with. He had plans to go ahead with a more developer friendly and advanced version of his editor, but it never happened. I have no idea what might have happened to John, but I suspect it was not good (active programmers don't typically disappear off the net entirely for years at a time without a trace). I do hope John is well, but the internet is a much poorer place without him I can tell you that!

MarkItUp! is also a fine editor in its own way, but it was not designed with Markdown as the primary target and it isn't exactly a WYSIWYG editor. Technically, MarkItUp! is just a markup editor with some macros on the toolbar. It does have support for the markdown syntax, and it does a decent job as a markup editor even with markdown's syntax. But using it to author content in markdown isn't very approachable for a public facing application. The editor is just a tad too "bare-metal".

WMD on the other hand is smoother and more viable. It is also more of a markup editor than a real WYSIWYG, but it does has some really subtle, but important features that help make authoring markdown content enjoyable. But John's baseline version has several pretty major problems, and since the source is obfuscated, fixing it up and customizing it is next to impossible.

Fortunately Dana Robinson and some others at Stack Overflow managed to de-obfuscate the original WMD source code, and published their their own Stack Overflow specific version over at github. Now! That was some fine work, and I really do appreciate Dana's contribution. But the Stack Overflow version stripped out a lot of the original WMD's configuration features, and has several Stack Overflow specific tweaks besides. The end result of the SO version is that you can't really control it well. For example, I've found it nearly impossible to instantiate the SO version of WMD via JavaScript in response to a user action (like showing the editor in a pop-up for example). The SO version also doesn't deal well with multiple instances on the same page.

There are quite a few branches from the SO version floating around on github. Fortunately there IS one branch that seems to have worked out most of the major problems and has actually advanced WMD quite a ways beyond the initial SO version. Anand Chitipothu maintains the openlibrary branch of WMD.
Basically, this version is a WMD in a JQuery wrapper. Anand has added back much of the configuration stuff that the SO branch broke, tweaked a few of the behaviors (in pleasant ways), fixed up the multi-editor support, and most importantly made the editor JavaScript/JQuery instantiable.

Overall, this is an excellent variant that can be used simply in just about any web application. It even has decent documentation too!

I'm using openlibrary / wmd Master branch tagged as 2.0, but there are a couple of minor catches I've found in this variant (as it was on July 1st 2010 anyway):

First, there is a random stray undeclared variable that blows up in chrome (I didn't check other browsers for this one).

The line with the problem is:

WMDEditor.Checks = Checks;

Commenting out this line fixes the problem simply enough.

The other problem is much more complex. Basically, when putting this together, Anand made some fundamental changes to how the code is arranged, and some of the stuff in an IE specific branch hasn't been property updated with the new object model yet (the author probably isn't testing in IE 8 yet). Fixing this is rather annoying to describe, but the simplest way to do it is:

• do a search/replace for "wmd.ieCachedRange" and replace with "WMDEditor.ieCachedRange"
    
• do a search/replace for "wmd.ieRetardedClick" and replace with "WMDEditor.ieRetardedClick"
    

I have to say that I'm not fully up-to-speed on the innards of WMD, so my fix may not be optimal here... but it seems to work OK in my limited usages so far.

For your convenience I've made my own variation of jquery.wmd.js and available for download. All of the lines I altered end with "//SMR". I am not including the minified version, but you should make your own minified for use in production environments.

openlibrary-wmd-2.0-modified.zip (43.40 kb)

  

Using T4 templates to generate DataAnnotations buddy classes for EF 4 entity models

One of the more frustrating things for me lately has been leveraging the ASP.NET MVC 2 validation support using DataAnnotations with a generated Entity Framework 4 model.

Wow! That's was a mouthful!

Anyway, it's great that the MVC Framework 2.0 can leverage DataAnnotations for both validation, as well as stuff like generating label text and such. Fantastic! But, for reasons that remain retarded, Microsoft chose not to include DataAnnotations attributes with generated EF 4 entity models. You can generate the models, and the models have attributes that mirror many of the ones DataAnnotations uses, but the attributes EF 4 generates are useless for use with MVC 2.0's validation features.

DataAnnotations does have a rather interesting mechanism that you can use to add the annotations to your generated model though. You can't add the attributes directly to the generated code of course, they'd just get blown away next time the code gen ran. And you can't add the annotations to a custom partial class that extends the entities because the partial classes cannot each define two properties with the same name.

Instead, what you are supposed to do is this:

• Create a custom "meta" class (sometimes called a buddy class) that mimics the entity's public properties
    
• Annotate the public properties in the custom meta class using attributes from Data Annotations
    
• Create a partial class that extends the generated entity you are annotating
    
• Flag the custom partial entity class with an attribute called MetadataTypeAttribute to link the entity class to the meta class where the annotations live  
    

Assuming you have a generated EF 4 entity named "Ticket", this will look something like this:

[MetadataType(typeof(TicketMetadata))]
public partial class Ticket
{
    //whatever extensions I might want to the entity

    /// 
    /// Class that mimics the entity so you have a place to put data annotations
    /// 
    internal sealed class TicketMetadata
    {
        [DisplayName("Ticket Id")]
        [Required]
        public int? TicketID { get; set; }
    }
}

So, we have a way to add the annotations to our generated model, but you have to manually code this up... which is a BIG pain in the ass honestly, especially for larger entity models.

This is a job for T4, but I have no interest in modifying the core T4 templates that generate the actual EF 4 entity model. Fortunately Raj Kaimal was kind enough to blog his custom T4 template for generating metadata buddy classes.

This is a pretty slick template that auto generates meta classes by just looking at an EF 4 generated edmx file. The template is super simplistic. It doesn't account for complex types, and there isn't a built-in mechanism where you can custom override or control how the attributes are generated. For example, the T4 generates the display name by just word-splitting at the capital letters in a Pascal cased property name. So if you want a different display name, there isn't a way to override the generated display name. The template does generate the meta classes as partials, so you can implement your own extension to catch any of the complex types that the template cant automatically generate attributes for.

But because this template is so super-simple, it is a great place to start if you wanted to create your own template that can account for more complex needs specific to your own application.

In my case, I found the template useful for generating the buddy classes initially, but then I just removed the T4 template and customized the code it had generated directly. That way I could make my customizations without worrying about the T4 over-writing my changes later. The T4 saved me a boat-load of time though by just doing that initial code generation, and the code it produced was about 95% sufficient for the final product's data annotation needs.

The annoyance of using data annotations with EF models comes up so often for me that I'm seriously considering writing a Visual Studio Extension, or my own complex T4 template to deal with the problem. Sadly, I doubt I'll ever find the time to actually write the code though.

Diaspora Follow-Up: How I'd Do Social Networking...

So... after having spent so much time bashing the Diaspora project in my last post, I'd promised to provide some constructive input.

Here it is!

Rather than address Diaspora's design directly, what I'll do instead is outline how I'd go about starting a project like Diaspora myself, with the same general goals in mind.

Name:

Any good project has to have a name, and preferably one that regular Americans, as the lowest common denominator, might pronounce correctly. It also helps not to include random special characters for no apparent reason (other than to confuse search engines perhaps).

Since I'm not feeling inspired today, I'll just steal the fictional name "Gink" (from the college humor parody video on social networking --very funny stuff!). Gink would have been a pretty good name for a real social-network; easy to remember, catchy, fun, and unlikely to be mispronounced... it also makes a good verb (which is one of Google's marketing strengths too). As a word, Gink does have some definitions already, but none seem so well entrenched yet that we couldn't usurp it for ourselves.  

Defining the Mission:

There are two ways to go about Gink.

Creating fantastic social-networking software can be the mission. Here the software is your focus, and the network a by-product of success. In the wild, it would (hopefully) evolve organically and displace entrenched social-networks through just sheer awesomeness. This seems to be the angle the Diaspora is taking, and it is a viable one --Linux started out like this.

Me though, I'd set a more explicit mission. My primary goal would be to create the social-network everyone wants to use; the one that empowers users. The software would exist only as a means to achieve that mission.

Security, Privacy and Control - Organization and Politics:

Giving the users a sense of ownership of their data and control over their privacy is the driving factor for Diaspora's public interest, and is central to its design. Even with Gink's slightly modified mission, this still remains the primary requirement.

So let's start by solving this problem with Gink...

As I mentioned in my last post, these requirements have been misunderstood by the Diaspora team. Users want control over how their data is used, and a sense of ownership. They want a binding promise that the system will not abuse their privacy. The part Diaspora misunderstands is that users have no interest in taking responsibility for the technical details themselves. They don't want to manage infrastructure, nor keep up with the data themselves.

At heart, this is a just a trust issue. The requirement is political, and political requirements are best solved through political architectures, not technical ones.

Since our team has the mission of providing a social-network, not just to develop software; the easiest way to handle this whole trust thing is by establishing a formal non-profit organization to oversee the network itself.

It is this organization to which I'd give the name "Gink".

The org can make enough promises in its charter, sign-up policies, etc. to solve the trust issues. The promises would take lawyer time to word up, but the highlights would be:

• The system would not be a vehicle for advertisers; an ad-free network.


• Users would have full control of their own personal data and how it used


• Users would have the ability to examine all data the system collected about them (down to the logs) at any time.


• The network would not censor user generated content.


• The network would not be able to sell or disclose personal data (individually, or in aggregate) to 3rd parties.


• Any non-personal data that may be shared would be public to all.


• A prohibition against the transfer of ownership of the data --the network cannot be bought out by someone else who then changes the rules.


• These basic promises would be unalterable after the fact; none of that "we reserve the right to screw you with 30 days written notice" trickery.

There are probably several other promises to make too, but being a non-profit organization and having made our policy plain, unalterable and user-centric, we've pretty much solved the critical trust issues; and without any code at all.

All the code has to do is avoid leaking data all over the place.

The Gink organization would be chartered to oversee the network itself and steward the direction of the "official" branch of the software. It would also completely control the infrastructure of the network, though it need not necessarily own that infrastructure directly.

System Architecture:

In my previous post, I also went into detail about my misgivings on Diaspora's technical design. Having solved Gink's trust issues, a fully distributed system of user operated peer-server nodes is unnecessary. Such a design would not advance Gink's mission; which is just to provide a network of pure amazing!

Still though, a system like this will be very large, and a distributed architecture of some type is absolutely essential. We control the infrastructure ourselves though, so we can manage a lot more architectural diversity on the back-end.

I would not design a "do-everything" kind of distributed server, such as what Diaspora proposes. Instead, I'd break the system down into separate service roles; each having its own server design using whatever architecture best supports that particular role.

Why should my aggregation services be limited by constraints imposed by the data storage architecture?

I'm not going to go into specifics about the design here. We have not fleshed out the requirements enough to even begin thinking about more specific details yet.

Software Project:

While the Gink network is operated as a non-profit organization, there is no reason to develop the software through the same organization. There several good reasons not too though. As non-profit, employees of the org would be limited in any income potential... Not that the org can't pay them well, but there may be opportunities related to the software beyond just the Gink network itself. Separating the development from the non-profit would allow the development team to capitalize on other opportunities without unnecessary restrictions.

So I'd develop the software though a different organization, which we'll call "Ginkworks" for lack of a better name. Ginkworks would produce open, royalty-free specs for the protocols, APIs, etc. of the Gink network. It would then develop an open source reference implementation of those specs.

The version of the software in use on the Gink network itself doesn't have to be the same internally as the reference implementation. There are cases where the network, due to practical realities, might need different or extended internal implementations... as long as it conforms to the open specs, this should be allowed where needed.

This is how Google runs projects like Wave, and Microsoft did much the same with the .NET CLR and C# languages. This strategy seems to work quite well, allowing each organization to use the best political architecture to meet their different requirements in much the same way as our system architecture does.

Driving User Adoption:

In order to be successful, we need to get users to switch to using our network as their primary social networking tool. This is the hardest part of the entire plan by far. It isn't enough to just enable users to do what they can do on other networks. You have to give them reasons beyond that, or they will just stay where they are comfortable.

This is the make or break requirement, and will decide the priorities and focus for the rest of the project's design. The focus would be on user features, and all back-end considerations must align to these.

I can identify four major areas of user feature enhancement that could generate the needed attractiveness for user switch-over. There are probably others, but these are the ones that stand out to me.

Interoperability:

Here Diaspora has the right idea. To get users on the system, you have will have let them bring their established networks with them. They aren't going to give up existing communities that they've spent years building on other sites. So the system must aggregate the user's other online identities, and allow them to participate across multiple networks simultaneously.

You can see the general shape of this already, especially in the smart-phone market, with social aggregator apps, but there is a lot more that needs to be done in this area.

The interoperation needs to be dumb-ass simple; so intuitive, seamless and effortless that cross participation appears to just happen by pure-fucking-magic (for you non-programmer types, this is a real technical term).

This interoperation needs to be bi-directional and highly cooperative, not just a read-only aggregation. For example, if users like flickr better than gink's own photo systems, let them use flickr through gink as if it were a native part of our system (as much as it is possible to do so, of course).

This will probably be the hardest of the technical challenges to meet since interoperation depends on the APIs of other networks, which are beyond our control and highly variable in their capabilities.

Mobile:

So far, the other social networks are providing mobile support as a secondary add-on service. Their mobile experience is limited to a sub-set of the network's features. But with social networks in particular, mobile devices are fast becoming the primary means by which people interact.

I'd develop the entire system as if the primary clients were mobile devices, treating full desktop browsers as a secondary design consideration. There will still be a small set of features that just can't be adapted to mobile (yet) of course, but these should be limited to admin and management functions where possible.

There are also some opportunities to provide features that leverage the unique strengths of mobile devices too. The most obvious of these are location services, and this is an area of opportunity that the competition seems to largely ignore (except Google).

Neglected Social Networks:

One of the bigger failing of the major social networks is that they all fail to recognize older social network technologies, many of which are quite well established.

Email is the most obvious one. Email is very much a form of social networking; and is the oldest such mainstream service still in existence. The existing social nets all treat email as a just an out-of-band notification channel for their own convenience. At best, they provide a poor-man's "message" feature, but otherwise treat actual email like it were a plague (which it probably is).

But why duplicate, badly, email-like functionality in Gink when we can try to aggregate real email into our system? So I'd try to treat this as if it were just another social network users participate in.

Online forums are another neglected social network that needs to be pulled in, along with users' blogs, if they have one.

I might not put much effort into real-time chat networks though. I'd have to investigate this area more myself to be sure how to handle it.

Non-Social Information Streams:

Social Networking, from the user's point of view, is largely just a personalized information stream. But on most networks, one of the most popular sets of 3rd party add-on features are ones that bring in read-only subscriptions and news. Since we're aggregating so much already, I'd want to subscription service aggregation to be a core, and first-class, feature set.

These four broad swaths of user-centric feature sets are things that might generate enough user acceptances to get them to switch. Of course, you also have to do a bang-up job on the expected feature sets as well.

So before I started on the deep technical nitty-gritty of the Gink back-end architecture, we'd flesh out as much about these user features as we can. Identify how these features might work, so we can ensure that we build a framework that directly addresses the technical needs of our feature set.

And that's how I'd go about starting a social networking project.

Diaspora - Good Idea, But...

I've been watching the Diaspora project carefully for little while now. If you aren't familiar with it, the simplest explanation is that it is an open-source version of facebook where users owns their own server.


Diaspora's own explanation is not so concise, but here it is:

Diaspora aims to be a distributed network, where totally separate computers connect to each other directly, will let us connect without surrendering our privacy. We call these computers ‘seeds’. A seed is owned by you, hosted by you, or on a rented server. Once it has been set up, the seed will aggregate all of your information: your facebook profile, tweets, anything. We are designing an easily extendable plugin framework for Diaspora, so that whenever newfangled content gets invented, it will be automagically integrated into every seed.

[read the rest here]: 

I've been looking for something like this to come along for a while. There have been a few feeble attempts so far, but none have gained any traction in the wild yet... and I'm not expecting this one to either.

Here's my take on it...

The Diaspora project has gotten a lot of media attention recently including a feature story in the New York Times. The reason it has gotten so much coverage is mostly just lucky timing. They happened to launch a $10k request for donations on kickstarter.com right as privacy and security concerns with Facebook became a mainstream press issue. Being lazy, the media just latched onto Diaspora as the underdog counter-story to Facebook, rather than do something rash like actual journalism.

Diaspora, thanks to the coverage spike, has accumulated over $175,000; much more than the mere $10k they'd originally asked for. If the donations keep rolling in, they can just skip code and go straight to being famous, rich, and retired.

I'm always pretty excited by these kinds of projects though, and I love the general idea of an open source social network. And, really, who doesn't like a good under-dog?

But, despite my enthusiasm, I am not all that impressed by what I've seen and read about Diaspora so far. I don't have a good feeling about the technical approach, nor the fundamental concepts.

Fundamental Concepts:

The project's fundamentals are subtly flawed in at least two very important ways.

The first is related to the design architecture; that Diaspora will be a distributed multi-node system.

The idea of distributed systems itself is quite solid of course. Spread storage and processing out to as many machines as possible, each fully responsible for their own information domains. This is the principle upon which the web itself operates, so it is well proven.

Diaspora's flaw is the assumption that every user will "own their own server".

This isn't 1994!

Social networking users are regular people, and regular people do NOT want to manage a server node. It doesn't matter how easy you make node management!

Users can barely manage copy and paste, and you want to put them in charge of a distributed information server?

I think the team has completely misunderstood the user's requirements here. Users "want control of their data". They never said that they "want to control their data". This isn't just semantics; it is the critical distinction between what they want, and what Diaspora is proposing to build.

For several years, the trend among users, even the technical ones, has been away from user owned software. Today's users don't want to install or configure anything themselves. They use pure internet apps almost exclusively. The same is true with user data. They don't want to have to deal with keeping track of it, or backing it up, or moving it around. In short, they don't want the responsibility for all that technical crap. They just want to use the software, have the data, and have someone else to blame when things break or data gets lost.

You can see this user trend clearly in the rise of gmail, youtube, flickr, google apps, etc. Wherever possible, people are steadily switching to web supplied software, and are moving data off of their own vulnerable machines into the cloud. All the users have been asking for is a sense of control over what the cloud does with the data once it gets there.

The second conceptual flaw is more subtle. It is the simple lack of focus on end-user features. Their explanations of the user features are, at best, vague; when they bother to mention them at all.

Their focus seems confined to the technical back-end details. Even worse, they use the term "feature" to label the various plumbing aspects of the system. These are not features, these are mechanics. Mechanics are just the means to the ends; it is in the ends that the features hide.

My concern here is really that the team just doesn't seem to understand their true goal well enough to build towards it reliably. They seem focused too narrowly on "how", but not enough on "what".

The plan appears to be to design and build a framework first, and deal with user features later. They say so on their blog in some detail actually, but frameworks are not applications.

I like the iterative and pragmatic approach they describe, but they are already building a framework and still they don't provide coherent explanations of actual user facing features. They only describe them in abstract vagaries; and this is what concerns me.

This kind of thinking is tragically common in application design, and it rarely turns out well. If you don't clearly define the user features you are aiming for, in quite some detail, then you cannot know if you are building the *right* framework; one that will usefully support the user features you'll layer on top later. The framework is likely to be too generic to be very useful during the real feature development stages. When developing front-end code, nothing is harder than having to code against an abstract framework's limitations and constraints, but not getting much help with the very concrete requirements you are faced with.

There is nothing wrong with building a great framework of course, but the team is getting donations based on the promise of a better social-networking application; not just a social networking framework.

Perhaps the fault just lies in the public's misconception about what Diaspora is proposing to build.

The Technical Approach:

Distributed systems are difficult, but distributed systems with a variable number of member nodes, coming and going at will, are many orders of magnitude more so.

Then there are the problems of distributed document and NoSQL database systems, which are still rather new and immature technologies. Even the best of these still have serious limitations.

Performance is a significant concern in such a system. Nodes have to be able to locate and query other nodes for data very quickly, and a social network will need to hit a large number of peer-nodes for even the simplest user features.

And then there is the problem of fault-tolerance in such a highly distributed system. When a member node is down, slow, or unreachable, the system has to gracefully degrade, without error, and without significant disruption in the user's experience.

Any of these areas present the kind of technical challenge that even companies the size of Google struggle with, but few systems combine so many such elements as Diaspora would. Their back-end architecture is up against some of the nastiest technical challenges the field has to offer, plus competing with established players whose centralized systems do not suffer the same.

To me, the worst part is that, after having overcome the design problems, the system itself will have gained almost nothing that would add any value for the users. I can't think of anything this design allows Diaspora's users to do, that a centralized social network doesn't also provide. Even the promise of giving users control over their data is not in any way unique to this design. Facebook can give users the same control, and with little technical effort (though, it would have a severe impact to their business model).

Despite my misgivings about the architecture, I still think Diaspora should try this approach. Just because I shat all over it, doesn't mean it's wrong to give it a try. If nothing else, they might advance the applied sciences of distributed system design. Even if Diaspora doesn't succeed with a distributed social-network, someone eventually will.

Whether real users will care or not is an entirely different matter. With the team putting so much effort into the complex back-end architecture, they will likely not have the resources left to do a bang-up job on the front-end features; at least not quickly enough to still be relevant.

The front-end features have to be amazing, or the system will not have much user interest. They are up against well-established competition that has had a lot of time to evolve and design their UIs. Creating a compelling experience to compete with them will be quite a daunting, and non-trivial, technical challenge all by itself.

Next...

It isn't fair to spend this much text bashing someone else's project without at least doing them the courtesy of providing constructive suggestions or alternative ideas. But, I'm going to save the constructive ideas for another post.

How Apple Will Lose the Mobile Computing Race

I've heard several people say that "Apple is the new Microsoft". These people are wrong; dead wrong.

Apple is the new Apple. They've not changed in the last 20 years, not even a little bit.

Google is the new Microsoft, and Apple is going to lose the mobile device market to Google the exact same way it lost the desktop market to Microsoft 20 years ago.

How is this going to happen? First you have to know the history.

Let me tell you a little story...

Back in the 80's there were two companies; Apple and Microsoft. Suddenly Apple comes out with a magic new product way ahead of the competition. Enter the Macintosh. The Mac was guaranteed to change personal computing forever, and everyone knew it before the Mac even hit the shelves. There was media hype and gobs of nerd excitement well in advance of the Mac's debut. And the Mac delivered spectacularly on all of its promises.

But Apple insisted on near complete control of the entire platform; hardware and soft. They were terrified that someone else might come along and spoil the pure awesome.

Apple used deliberately unusual hardware, and controlled it with an iron fist. Only Apple and a few licensees were allowed to make Mac hardware. The 3rd party accessory market was as restricted as Apple could make it too.

Programmers could make applications for the Mac of course, but Apple maintained tight control over how applications used the hardware and interacted with the OS. Developers played in the sandbox, but it was Apple and select partners that created the majority of useful Mac software.

Apple was also territorial about who could sell their precious. Macs were distributed mostly through select retail partners. Anyone else that wanted to sell a Mac paid full retail price --which made it very hard to compete with Apple's preferred retailers.

Because Apple had tight control end-to-end, the Mac was a very stable and reliable computer. Most of the software was also high quality, which made the Mac a joy to use.

On the other side, you had Microsoft. They weren't really a hardware company though. They concentrated on making a decent operating system and left the hardware to other companies that specialized in that sort of thing.

There was a lot of compromise and cooperation between Microsoft and the hardware makers though, of which there were many. They created a large market with a wide range of different systems, each with different capabilities, quality, and prices. Microsoft just did what they could to make sure that Windows worked reasonably well with whatever crazy-ass hardware the other guys came up with.

Microsoft made a decent enough OS, but what they did really well was make programming languages, compilers, and development tools --all that stuff you need in order to make software for the regular folks. Microsoft made it easy to develop applications for Windows, and they stayed out of the way as much as possible. They didn't put up a fight when developers released software to compete with Microsoft's own, and they didn't stop developers from extending the operating system itself in new and unusual directions.

Microsoft was later than Apple in coming out with a good GUI based OS. Windows wasn't too pretty nor much fun compared to the Mac; especially at first. But PCs were cheaper, and came in a variety that fit different people's needs and budgets. The most important thing was that a Windows PC could do everything that a Mac could, though not always with the elegance of the Mac. People often preferred a Mac, but they could (and did) settle for PCs instead.

In the end, what killed Apple the first time around was Apple's own paranoia. All by itself, Apple couldn't evolve the Mac hardware nor the OS fast enough to keep up with a young and rapidly changing personal computer market.

All Microsoft had to do was try and keep up while other companies raced forward with new ideas.

Now jump forward 20 years.

There are two companies; Apple and Google. Suddenly Apple comes out with a magic new product way ahead of the competition. Enter the iPhone. The iPhone was guaranteed to change mobile computing forever, and everyone knew it before the iPhone even hit the shelves. There was media hype and gobs of nerd excitement well in advance of the iPhone's debut; and the iPhone delivered spectacularly on all of its promises.

But Apple insists on near complete control of the entire platform; hardware and soft. They are terrified that someone else might come along and spoil the pure awesome.

Apple uses deliberately unusual hardware, and they control it with an iron fist. Only apple is allowed to make iPhones, and the 3rd party accessory market is as restricted as Apple can make it.

Programmers can (and do) make applications for the iPhone of course, but Apple maintains near complete control over how those applications use the hardware and interact with the OS. Developers are allowed to play in the sandbox only if they use the specific tools that apple permits. Apple and a few select partners make the majority of the useful iPhone apps. Most developers that make apps to competes with Apple's own are denied access to the app store; which is the only way users can obtain apps for the iPhone. Apple will even deny an app if they just don't like what it does.

Apple is territorial about who sells their precious. The iPhone is (currently) only distributed through one network in the U.S., and apple exerts enormous control over how that network does business. AT&T had to wait to roll out 3G services until there was an iPhone that also supported it. There were other mobile phones sold by AT&T that had  3G support as much as a full year before, and AT&T had deployed the network hardware long before Apple's 3G iPhone came out.

Because Apple is in such tight control, the iPhone is a very stable and reliable smartphone. Most of the software is also of high quality, which makes the iPhone a joy to use.

On the other side, you have Google. They aren't really a hardware company though. They make a good mobile operating system, but have left most of the hardware in the hands of companies that specialized in that sort of thing.

There is a lot of compromise and cooperation between Google and the many hardware makers. They have created large market with a wide range of different Android phones with different capabilities, quality, and price. Google just does what it can to make sure the Android OS works reasonably well with whatever crazy-ass hardware the other guys come up with.

Google makes a good mobile OS, but what they do really well is provide cloud services --services developers can use to make useful mobile software for the regular folks. Google makes it easy to build applications for Android, and supply much of the cloud services free of charge. They stay out of the way as much as possible, but give developers a way to distribute software through a centralized app store, with very few restrictions. Apps can also be obtained through independent channels too. Google doesn't put up a fight when developers release software to compete with Google's own products, and they don't stop developers from extending the Android OS in new and unusual directions.

Google was later than Apple with a smartphone. Android isn't quite as pretty compared to the iPhone; though it is rapidly getting there. Android phones are cheaper, available for nearly any network, and come in a wide variety of forms to fit different people's needs and budgets. And the important thing is that an Android Phone can do anything that an iPhone can; sometimes even with the same elegance as the iPhone. People often prefer the iPhone, but they can (and do) settle for Android phones instead.

In the end, what will kill Apple the second time around is Apple's own paranoia. All by itself, Apple cannot evolve the iPhone fast enough to keep up a young and rapidly changing mobile computing market. Their treatment of developers is pushing them to other platforms fast.

All Google has to do is try and keep up while other hardware and software companies race forward with new ideas.

This same exact pattern of behavior is repeated with the new iPad.

The situations are so similar it is almost insane that Apple can't see their own doom coming.

There are some major differences between now and 20 years ago, but none of them bode well for Apple.

Google is in a much more competitive position than Microsoft was 20 years ago, and Android is a much better product to pit against the iPhone than Windows was against the Mac.

Perhaps the biggest difference between now and then though is that Apple wasn't crapping all over their software developers back then. Apple did limit developers in some ways, but never in the arrogant and insulting way they do now... and especially not through any legal or contract trickery like the iPhone developer agreement.

Apple is also much more blatantly motivated by pure monetary greed. Twenty years ago, it felt like the Mac more about a vision of excellence, making a better tomorrow, being different, and enabling new and wonderful possibilities. The iPhone today though is clearly just a cash cow. Apple's every legal and marketing move is so transparently profit motivated that it doesn't even fool kindergarteners.

There is no greater vision behind the iPhone like there was the Mac; behind the iPhone it's just dollar signs, abused developers, and shackled customers.

Peter Watts - about birthdays....

I'm one of those people that just has a knack for seeing the world a bit 'differently" than most people. What some people see as absolutely normal social behavior, I often see as utterly absurd.

This is not usually an advantage, if you were wondering.

But I've got nothing on Peter Watts, the now famous Sci-Fi author, border-guard crushing bad-ass, and sadly... convicted felon. His writing is filled with this kind of totally brain breaking logic.

Like birthdays...

Below is an excerpt from his excellent novel "Blindsight" that exemplifies exactly the kind of thinking that breaks most people's brains. The novel is available online for free under a Creative Commons license (CC), or you can buy it in paper or audio if you prefer.

In prelude, know that the character Sascha has multiple personalities (the Gang of four as they are known  in the book).

"Yeah? Name one."

"Birthdays," I said, and immediately wished I hadn't.

Sascha stopped chewing. Something behind her eyes flickered, almost strobed, as if her other selves were pricking up their ears.

"Go on," she said, and I could feel the whole Gang listening in.

"It's nothing, really. Just an example."

"So. Tell us." Sascha cocked James' head at me.

I shrugged. No point making a big thing out of it. "Well, according to game theory, you should never tell anyone when your birthday is."

"I don't follow."

"It's a lose-lose proposition. There's no winning strategy."

"What do you mean, strategy? It's a birthday."

Chelsea had said exactly the same thing when I'd tried to explain it to her. Look, I'd said, say you tell everyone when it is and nothing happens. It's kind of a slap in the face.

Or suppose they throw you a party, Chelsea had replied.

Then you don't know whether they're doing it sincerely, or if your earlier interaction just guilted them into observing an occasion they'd rather have ignored. But if you don't tell anyone, and nobody commemorates the event, there's no reason to feel badly because after all, nobody knew. And if someone does buy you a drink then you know it's sincere because nobody would go to all the trouble of finding out when your birthday is — and then celebrating it — if they didn't honestly like you.

Of course, the Gang was more up to speed on such things. I didn't have to explain it verbally: I could just grab a piece of ConSensus and plot out the payoff matrix, Tell/Don't Tell along the columns, Celebrated/Not Celebrated along the rows, the unassailable black-and-white logic of cost and benefit in the squares themselves. The math was irrefutable: the one winning strategy was concealment. Only fools revealed their birthdays.

Sascha looked at me. "You ever show this to anyone else?"

"Sure. My girlfriend."

Her eyebrows lifted. "You had a girlfriend? A real one?"

I nodded. "Once."

Don't let unit testing push you around!

Test Driven Development (TDD) and the ever increasing trend towards Unit Testing as a religion among developers is probably a good thing for software quality in general. I've just  spent a couple of weeks trying to employ the recommended unit testing techniques with Entity Framework 4, and this has reinforced my misgivings about the wisdom of some of the  basic concepts behind how unit testing is supposed to be done.

I've always been somewhat lukewarm towards unit testing. I certainly don't do as much testing as I probably should, but I do agree that some unit testing is essential to developing a quality application. So, I always do some unit testing, but only to the degree that I feel the testing is appropriate or necessary.

But unit testing has grown in popularity over the last 10 years, to a point where it appears that common sense is not a part of the process anymore.

The source of my misgivings about this blind commitment to unit testing deals with one of the fundamentals; that a unit test should only test logic within the specific block of code that is under test.

I don't disagree with this principal, but the sticking point is that you have to fake your code's dependencies, especially the external ones like real databases and external services. In most modern applications, external dependencies are pervasive and sometimes very hard to avoid.

To support testing without hitting external dependencies, you generally design your code in a way that allows for dependency injection. There are several ways to do dependency injection, but the different techniques can range in complexity a lot. On the simpler side you can just use overloaded constructors or write methods so they take their dependencies as parameters. On the other end, there are horribly complex programming acrobatics such as the Inversion of Control (IoC) design pattern.

No matter which techniques you use, the overall point is that your unit tests are able to supply their own versions of any dependencies instead of using real ones. These mock (fake) dependencies will mimic the behavior external systems or other complex code routines.

I'm not opposed to making some concessions in my code in the name of enhanced testability. Indeed, many of the simpler dependency injection techniques lead to better code in general. But there are cases where simple techniques just don't work. This is where I start to have serious misgivings about the wisdom of unit testing.

I've always had the opinion that the use of fancy OOP tricks, complex inheritances in particular, should only be used if there are direct benefits to the application. Likewise, employing advanced design patterns is only worth the effort if the patterns solve issues within your application's actual problem domain.

To some degree, facilitating better unit testing can be considered a good reason to employ some of these more advanced techniques. Indeed, the TDD guys will usually tell you that unit testing is a part of the application's problem domain, and that a full suite of unit tests must be part of the delivered product. I do see their point, but I do believe that there are cases where it is misguided --and unwise.

In my opinion, you should never compromise your application's primary requirements just for the sake of unit testing. Testing is at best a secondary concern; an optional requirement when it doesn't interfere with other priorities.

But that doesn't stop developer after developer from doing insanely stupid shit to their code for the sheer sake of testability. I've seen so many otherwise simple applications become bloated and convoluted beyond reason, and often to the detriment to both developers and the application's functionality.

The most recent case of this that I've run into was with the next version of TicketDesk. I have a class library to encompass the application's core domain logic. Here I had planned to auto-generate my entity model from my database, add some data annotations and custom logic of my own, then just create a simple service API to expose the model to my web application; and in the future potentially other clients such as Silverlight.

All-in-all this is a pretty standard design. My entire reason for choosing the Entity Framework was to leverage the more advanced features, both within the library as well as within the client code consuming it.

After spending a solid two weeks exploring how to make Entity Framework 4 testable the correct way, I'd ended up with an insanely complicated hand-hacked variation of a standard generated EF model. But to support the necessary dependency injections, especially with the problematic EF ObjectContext, I had produced gobs of abstract interfaces and classes. Return types at the service API had become so abstract that the consuming clients had lost access to all of the special functionality of EF models; the very features that had made EF an appealing choice in the first place.

I got far enough into this to prove that achieving a useful level of testability could be done, even with the dreaded ObjectContext. But well before I reached that point, it was apparent that doing so would make my code into a complex and time consuming maintenance nightmare.

I still needed to be able to unit tests my custom logic and service layer though, but I was done with doing it "the right way". Instead, for my tests I pointed the ObjectContext at a real database server, used the built-in CreateDatabase method to generate a temporary fake database schema from the EF model, and wrote my tests against it.

My tests are able to add any fake data needed directly to the temporary database via the strongly typed and auto-generated ObjectContext. With this approach, I can still test the relevant code blocks, but without jumping through hoops to dodge a dependency on a database server.

Sure, it isn't technically the correct way to do unit testing... and yes, many of my tests do spend a bit of time talking to the external database, or managing the test data within it; but all of the extra complexity is encapsulated directly within the unit tests or helpers classes within the test project; not junking up my actual application.

I'm sure the more fanatical unit testing advocates, especially the TDD guys, would foam at the mouth at this approach, but I have nearly 100% code coverage in my tests (excluding generated code). And for me, the biggest advantage is that my application code is simple, elegant, effective, and highly maintainable...

Isn't that the whole point of unit testing?

Visual Studio 2010 - Revisiting Themes and Fonts

[update: 4/2010 - there is a new color schemes site for visual studio that also has several alternate themes, including the humane studio theme discussed here. ]

With the pending release of VS 2010 and it's shiny new UI, I thought it time to revisit the topic of color schemes and fonts. I have typically use a hacked up "dark theme" in VS 2008 with a black or near-black background, but I wasn't too thrilled with the results after porting my previous color scheme over to VS 2010.

I've always been decently happy with Visual Studio's text editor once I change a few of the more retarded defaults (really? Line numbers aren't enabled by default?). But that default white background with dark text is a bit harsh on the eyes.

I personally prefer darker color schemes, but Visual Studio has never been a "theme" friendly app. You can easily import and export settings, which serves as a poor-man's kind of theme though, but overall personalizing the theme is Visual Studio is always a pain in the ass.

So the first thing I did was try to find a new base theme online that I could start from, then customize it to my own preferences.

Every time I try a black background based theme, I keep running into the same problems:

1. There too few colors that are bright enough to contrast readably against a black background. This is exactly why so few web pages use black-grounds despite their popularity among web users.  


2. A good editor color-codes text to help the developer quickly interpret the meaning and structure of their code. For Visual Studio you need about a dozen distinct colors, which is not easy against a black-ground.


3. Most colors that are usable on black are the really bright ones like hot pink, lime green, and electric blue. These tend to clash easily with each other though, and I often end up with a nauseatingly horrible color scheme as a result.

With VS 2010, these problems are just harder to get around. The new version adds yet more color-coding, all of which is useful. So this time, I tried a few of the compromise themes.... Ones that aren't based on black/near-black backgrounds. The most interesting of these turned out to be a theme called HumaneStudio:




The colors remind me of an old editor (whose name I cannot remember) from back around 1995 or so. It uses a series of browns for backgrounds, and for text a mix of various earth tones from across the color spectrum.

I didn't expect to actually like this theme when I downloaded it. Earth toned stuff isn't my thing typically, but after only a few minutes working with it I discovered this was quite possibly the most fantastic theme ever! The colors are dark enough not to be too harsh on the eyes, and the wide array of available text colors works effectively with Visual Studio's color coding. I did have to make some minor changes though: First I replaced the font...

A mono-space font is non-negotiable for code editors (so says me!), but there are several good mono-space fonts to choose from. There are six that I personally recommend, all of which achieve a decent balance between long-term readability and efficient use of screen space:

Serif fonts:

A serif is the little ticks at the edges of characters. These come from the old days where chisels were used to carve letters in stone or wood. Serifs tend do improve readability of text, especially with larger blocks of smaller text. Most books, newspapers, and other text-heavy documents use serif based fonts for this reason. These fonts are less visually appealing at larger sizes though, so serifs are less common for titles, signs, and short text scenarios. Serif fonts also tend to be more readable on older low-resolution electronic displays. A major drawback to serifs is that they tend to bleed into darker backgrounds. The serifs can even disappear completely when used on black at small font sizes.

• Courier New (Serif) - This is an old font, and been the default for windows text editors for a very long time. The problem is that it wasn't designed for modern high-resolution LCD displays. Quite the opposite actually, as this variation was specially designed for readability on older low resolution CRTs. Courier New wastes a lot of horizontal space, and doesn't scale smoothly. Still though, this font and has been around a very long time for very good reasons, and remains a quite serviceable font today.
  
  
       
• AnonymousPro (Serif) - This is a modern serif font. It is more stylistic than courier and has been designed with today's higher-resolution LCD displays in mind. It remains highly readable even on dark backgrounds, though the serifs are subtle enough to bleed out almost completely on black backgrounds. This is my favorite of the serif fonts.
  
  
       

Sans-Serif fonts:

These fonts lack the ticks at the edges and most people find them much more visually appealing. The drawback is that sans-serif characters can be less recognizable at smaller sizes. Sans-serif fonts do allow more flexibility in character shapes, which leads to more artistic styles. With today's higher resolution displays, sans serif fonts are increasingly common even for dense and small text cases where serif fonts used to dominate. Sans-serif fonts don't suffer as much from use on black backgrounds so long as the characters aren't over-stylized or too compressed.

• Consolas (Sans-Serif) - This is Microsoft's new favorite mono-space font, and is the default for VS 2010. It was designed specifically for LCDs and is further optimized for Microsoft's Cleartype font rendering technology. On dark backgrounds at smaller sizes though this font tends to flatten out awkwardly (in my opinion). On light backgrounds though, this is one of the best fonts ever invented. It scales very smoothly and is exceptionally pleasant at medium to large sizes. Though not a serif font, the character styles are very similar to serif fonts like courier new.
  
  
       
• DejaVu (Sans-Serif) - This is a variation of the Vera font family, and is very popular on linux systems. The general shape is similar to Consolas, but the characters are taller, a tad wider, and the style is a little more rounded. It doesn't flatten out like consolas at smaller sizes, and for this reason it has become my personal favorite on black or near-black backgrounds.
  
  
       
• Envy Code R (Sans-Serif) - This is a somewhat bizarre font with a wilder style. It uses thin lines and is somewhat compressed, but overall I'm not personally fond of it. It is a popular developer font though, and has decent readable even on dark backgrounds, but long-term readability does suffer at smaller sizes.
  
  
       
• Monaco (Sans-Serif) - This is a windows version of the popular font used in TextMate from the Apple universe. This is the largest of the fonts listed here. It also has a fancy stylistic look with unusual character shapes. While pleasant to look at, I find that it wastes too much space all around. It does render very well on dark backgrounds, mostly because of the exaggerated size of the characters. It is a solidly readable font once you get used to it.
  
  
       

The default font for the HumaneStudio theme is "Envy Code R". While this is a neat font, it is just too "stickish" for my tastes. I switched the default font to either AnonymousPro or DejaVu. I also removed all of the "bold" text effects from the theme's settings; the bold effect work well with Envy Code R, but not so much with my own preferred fonts.

I'm torn between AnonymousPro at 11pt and DejaVu at 10pt with this theme.

In VS 2008 AnonmousPro is my preference, but VS 2010 renders text differently. I'm finding that I am leaning more towards DejaVu in VS 2010. Another area that required some customization was the output and find windows. HumaneStudio's default settings uses a custom font called Palm OS. This is a terrible font! The color scheme for these windows is also a green-on-green "lime explosion" type thing. I like the stark contrast of these windows compared to the warm tones of the main editor window, but I had to change the background to black to make the output and find text comfortably readable at such extremely small sizes. For the font in these windows I stuck with AnonymousPro at 8pt.

Overall, HumaneStudio is probably the best theme I've ever used in a code editor. It was very easy to get used to, and it holds up amazingly well during long coding sessions without causing me any eye fatigue. This theme also has a lot more flexibility for customizations than most black based themes.

On VS 2010, the warm earth-tones in the editor window tends to contrast poorly with the default VS 2010 scheme for menus, borders, and the other non-text editor features. By default, these are based on a stark blue color theme, and these settings cannot be changed directly in the options menu. Fortunately there is an extension for VS 2010 that allows you to customize these non-editor colors. The extension's built-in "autumn" scheme seems to fits in with the Humane Studio editor theme perfectly.

I've put my own variants of the Humane Studio theme if you want to try them out yourself:

Redd's Variant HumaneStudio Theme for VS 2008

Redd's Variant HumaneStudio Theme for VS 2010

You'll want to get the AnonymousPro and DejaVu fonts and install them before you load the themes.