Rinat Abdullin on Efficient Software Development

Mercurial Is Getting Better for .NET Projects

Rinat Abdullin — Fri, 05 Mar 2010 04:33:43 +0000

Recently I've started using Mercurial (distributed version control system, an analogue of Git) more and more for working on various projects. This applies even if the project itself lives in the Subversion for collaboration.

Icon overlays have got more stable in the latest versions of Windows 7, Jungle Disk, TortoiseHg (shell extension for the Mercurial) and TortoiseSvn. The last two, share same overlays, as a matter of fact, along with showing much smarter overlay management. There is no need to mess with the registry (works on my machine).

So it is a real pleasure to work with Mercurial these days. This also allows to perform solution-wide refactorings and cleanups featured by the ReSharper 5.0.

Here's the snippet of the .hgignore file that I use:

# Mercurial .hgignore file template by Abdullin.com
# For syntax see: http://linux.die.net/man/5/hgignore
# Source: http://bitbucket.org/abdullin/snippets/

# Visual Studio 
glob:*.user
glob:*.suo
glob:*.cache
glob:_ReSharper.*/
relre:/obj/
relre:/bin/

# Subversion
glob:.svn/
glob:_svn/

# Build structure
relre:^Build/

# Misc
glob:Thumbs.db
glob:*.bak
glob:*.log

The snippet itself is saved (link to the latest version) on the bitbucket, although Google Code already supports Mercuarial as well.

In order to use the snippet, simply paste it into the ".hgignore" file in the root of the working copy. Chances are, the file has already been created for you by Mercurial.

And here's a bit of PowerShell code for wiping clean a SVN working copy, in order to merge latest branch with it:

ls -r -force | 
where {!$_.Mode.StartsWith("d") }  | 
where {!$_.FullName.Contains(".svn")} | 
del -force

Basically, the command wipes clean all files (you can put it in a single line), leaving an empty folder tree with the Subversion files. Then you can copy another solution export over it (i.e. export of a Mercurial branch) and simply accept the changes. That's how we've been syncing some folders in SVN repositories in one of the projects a few years ago.

Fortunately copying snapshot from the SVN to Mercurial is more simple than that: just wipe the all files, except for the single hg folder in the root; export contents of a Subversion solution over to this folder and commit changes.

Scalability and Concurrency Challenge

Rinat Abdullin — Tue, 05 Jan 2010 15:20:41 +0000

It is interesting, how concepts of separated features and messaging become more important as the distributed applications evolve. I keep encountering these again and again in the solutions at Lokad and in new reads showing up:

Articles about cloud computing;
Effectus codebase by Ayende;
Command-Query Responsibility Segregation by Udi;
MindTouch Dream principles of building lightweight decoupled services;
Composite Application theory.

So for the last few weeks I’ve been discovering things like Advanced Message Queue Protocol (reading its specs is a pleasure of its own) and its implementations (Apache Qpid and RabbitMQ). There had also been a different approach to the REST services and concurrency handling as in the inspiring MindTouch Dream framework.

Obviously, the toolset is not your native Microsoft .NET toolset, but:

MindTouch Dream works natively on Mono and .NET;
Apache QPID and RabbitMQ have .NET clients;
It takes just a few lines of code to write .NET Dream Service to act as a gateway between AMQP and REST;
For some reason I like AMQP more than MSMQ;
In a current world it takes less than an hour to procure a cheap running server out in the cloud.

It is also interesting to see how REST services and AMQP middleware services have their parallels with the classical composite applications:

AMQP == EventBroker
Services == IoC Container environment (roughly)

So basically, there are quite a lot of concepts and technological approaches that suddenly start making a lot of sense to me.

At the moment I’m interested in giving a try to build some learning prototype application using these technologies, just for the sake of practicing and learning pros and cons better.

Does anybody know of a common or specific problems out in the field, that require potential scalability and async processing? Something, where scalability, concurrency and sync processing create a lot of challenges.

Update: Jonathan has really nice article about Getting Started with DDDD and CQS (there are a lot of references there as well).

Windows Azure in Lokad

Rinat Abdullin — Tue, 15 Dec 2009 20:44:13 +0000

If you are interested in Cloud Computing, Windows Azure and some practical applications, then this short movie by Microsoft might interest you. It had been featured at PDC’09 and is partially based on the use case around Lokad (company I’m working in):

If YouTube player does play properly, then there is a separate link or downloadable movie files.

There’s also the original Cloud Computing Case Study on Windows Azure in Lokad made by Microsoft after a few interviews.

Given all the latest changes in the technology, and the possible trends, that start showing up, we can say that the cloud computing is going to be extremely important in the mid-term future.

Here’s another curious trend in the picture. It seems that stock markets might eventually add yet another commodity to their list – cloud computing instances. Amazon had recently announced EC2 Spot Instances as an ability to obtain computing capacity at a price matching proposed bit. Once there is some cost-effective and efficient platform for redistributing cloud computing resources, all these datacenters with idle server rack might follow in. Forces of Economics and expense optimization might bring forth interesting development solutions afterwards.

Microsoft Feedback 2010

Rinat Abdullin — Thu, 10 Dec 2009 07:56:34 +0000

There is usually a lot to learn from large companies like Microsoft, or less large ones.

However, it takes a bit of learning on your own before you could understand and appreciate details and partially hidden interactions behind what is revealed.

Fore example, that’s how Jetbrains ReSharper feedback looks like:

Here’s the one currently employed by us at Lokad.

And that’s how Microsoft Office 2010 feedback looks like:

Each of these three has it’s own benefits, including but not limited to:

UI simplicity;
Level of complexity of cliend-side and server-side infrastructure behind the feature;
Support for multiple software applications within the same product line (all the way through various versions, deployment options);
Ability to offer user to provide additional data, essential for the problem resolution;
Level and quality of sensible automation behind the interaction.

There is a lot to learn from large companies for Lokad.

Fixing Icon Overlays for Dropbox + TortoiseSVN

Rinat Abdullin — Mon, 26 Oct 2009 06:42:36 +0000

If you are using Windows 7 (x64), Dropbox and TortoiseSVN, then you might eventually encounter that half of your icon overlays went missing.

Since TortoiseSVN seems to be sharing same overlays with TortoiseCVS, TortoiseHG and TortoiseGit, these programs could experience same glitches.

TortoiseXXX - windows shell integration for a given version control system.

Dropbox - SaaS backup solution that works like version control, but performs all the sharing automatically in the background.

Loosing overlay icons hurts badly, since both Tortoise and Dropbox are extremely good in what they are doing.

Problem

There is only a limited number of overlay slots available in Windows. That's the byte order limitation (probably around 15, with 4 reserved by the OS). Whenever the limit is breached, we've got problems.

Solution

So we've got to reduce the number of overlays. Since there are no settings in Dropbox or TSVN to disable certain overlays, we'll have to do this manually.

Step1: Start the registry editor (regedit) and navigate to the:

HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\
  Explorer\ShellIconOverlayIdentifiers

Step2: Backup the entire branch. JIC.

Step3: Delete overlays that are rarely used and not essential for everyday work. I personally deleted these:

Offline Files
TortoiseLocked
TortoiseIgnored
TortoiseUnversionsed

Step4: Reboot.

As the result, Tortoise icons finally show up. Behavior of the Dropbox is more erratic, but this is not critical.

Caveat

There could be better solutions to the problem.

However, I didn't investigate the case a lot, for the lack of time - just checked that there are no proper solutions in the google, fired up the regedit, deleted a few entries and found out that it works for me.

NB: Copy of DropboxPath utility for x64.

Infrastructure Shards and Optional Components

Rinat Abdullin — Wed, 21 Oct 2009 05:56:05 +0000

Component-driven development really helps to deliver multiple applications and systems efficiently. It happens because:

important functionality is abstracted in components;
components are coded and unit-tested only once;
every component has multiple use cases that provide additional feedback and help to find problems or limitations, if there are any;
whenever a component is fixed or improved, all its consumers would benefit from that automatically.

As we go up in the complexity of the development, it becomes too cumbersome to manage components on the individual basis. That's where infrastructure shards come into the play.

Infrastructure Shards

Infrastructure shard is a group of components that work together to carry out some specific job in the application or system. They are joined to simplify reuse in multiple applications and contexts.

Normally all these components are developed to support Inversion of Control, but if needed they could be wired together manually. In the IoC scenario, components are fit together using registration modules (or xml configuration blocks for the legacy containers). In the rare cases, when IoC context is not available (i.e.: we need to offer minimal feedback reporting to deal with possible unhandled exceptions on startup), a simple factory method would do the job.

A single infrastructure shard could join components coming from the different libraries and application blocks.

Feedback infrastructure shard (sources), for example, would include:

Model-View-Controller elements:
- User control view to be shown in the currently available IViewspace;
- Immutable models used to pass feedback reports and information around;
- Controller responsible for managing the view and user interactions;
- Validation rules injected into the view in order to check and enforce user input.
Feedback helper components, aggregating system information to be attached to the report.

However, if we attempt to start our feedback reporting window, just with these few components, we would fail. This would happen because usually in the IoC environment an infrastructure shard is not self-sufficient. It has to interact with the other components and services as well. The view, for instance, does not even have a window to present itself in; it is merely a user control:

So in order for the shard to work, we need to provide some external services. These are really easy to spot in the code if we consistently stick to the guidelines of:

only constructor injection is used (no property injection);
every component has only a single constructor;
components can inherit only from the interfaces (except for the views that have to derive from some user control).

Given that we could find out that core feedback service components use these components:

IViewspace - responsible for displaying view properly in the UI environment available;
IReportService - responsible for establishing contact with the communication service shard (which will handle all the exception handling and reliability on its own) and sending the feedback report;
IClientInteractions - provides UI-specific methods that allow to interact with user;
Optional{ServiceConnection}.

The last item is the most interesting one here. Here's why and how it works.

Optional Components

When we show feedback report window - we want to suggest user to use his existing email address and connection information. This way the feedback could be routed to the appropriate server and associated with the user's account.

And we want to have code working in situations, when user context is not available (i.e.: user logged off or context is not even known).

That's what the Optional{ServiceConnection} does. It allows to pass some information via IoC container, while indicating that the reference could be empty at the moment of retrieval. It's declaration is extremely simple:

/// <summary> Simple delegate that returns optional result </summary>
public delegate Maybe<TResult> Optional<TResult>();

That's it. Yes, that resembles Lazy of MEF, but is a bit simpler, does not require MEF libraries and can benefit from the synergy with Maybe Monad.

For example, if we want to get current user name in one of the views (if the user name is available), the code might look like:

// somewhere in the class
readonly Optional<ServiceConnection> _contextProvider;

// somewhere in the code
var sender = _contextProvider()
  .Convert(sc => sc.Username)
  .GetValue(AnonymousUserName);

While the registration of the optional might look like:

// extension method for Autofac ContainerBuilder
public static void RegisterExport<TComponent, TExported>(
  this ContainerBuilder builder,
  Func<TComponent, Maybe<TExported>> func)
{
  builder.Register<Optional<TExported>>(c => 
    () => func(c.Resolve<TComponent>()));
}

// actual registration
builder.RegisterExport<IApplicationController, ServiceConnection>(c => 
  c.Context.Convert(ec => ec.Connection));

With this registration at hand, all trusted components that need to have access to the current connection context, can do that by requesting an instance of Optional{ServiceContext}.

This technique is really useful when defining boundaries of infrastructure shards, required and optional dependencies that they might have.

Unit Testing

We can even easily isolate the shard in unit test in order to allow manual testing of feedback reporting:

[Test, Explicit]
public void Show_synthetic_feedback()
{
  var data = RandFeedbackModels.NextTextDatas(0, 3);
  CreateService().ReportFeedback(data);
}

Where CreateService explicitly establishes all dependencies in the MockContainer and grabs the service:

IFeedbackService CreateService()
{
  Build(cb =>
  {
    // factories
    cb.Register<Func<ServiceConnection, ILokadService>>(ServiceFactory.GetConnector);
    cb.Register<Optional<ServiceConnection>>(() => Maybe<ServiceConnection>.Empty);

    // services
    cb.Register<FeedbackService>().As<IFeedbackService>();
    cb.Register<ReportService>().As<IReportService>();

    cb.Register<FeedbackSenderController>().As<IFeedbackSenderState>();
  });

  WhenViewIsShown.Do(f => f.ShowDialog());

  return Resolve<IFeedbackService>();
}

As you can see for the service connection provider we are passing a lambda expression that returns an empty connection context.

Note that we are not establishing any view-related components, since ViewFixture (this test fixture derives from it) already does this:

cb.Register<TView>().As<TView, TInterface>();
cb.Register<TestViewspace>().As<IViewspace, TestViewspace>();
cb.Register<TestInteractions>().As<IClientInteractions, TestInteractions>();
cb.Register(Specifics.Empty);
OnBuild(cb);

Result looks like this:

Filling in the information and clicking Send would create a ticket in Lokad support system. By the way, you can use this opportunity to say what you think about the company.

Shard Synergies

Here are some additional synergy effects that come from using infrastructure shards to reuse similar components between multiple applications:

Business functionality could be used to extend multiple applications. Since UI is shared, we could use shared documentation, tutorials and screenshots as well.
Sometimes context requires infrastructure shard to be evolved. This brings potential improvements in all existing and new applications.
When a group of components is developed to be used in multiple scenarios from the start, this polishes the overall design even further than TDD or CDD would alone.
Localization or UI changes across the software product line become easier, since the effort does not have to be duplicated and tested in multiple codebases.

What do you think about all that?

Notes

I understand that this looks really similar to what Microsoft Extensibility Framework does with Lazy and Export . However, my belief is that:

extensibility discovery and management framework should stay completely detached from the components;
inversion of control container should stay completely detached from the framework as well;
attribute-based declarations on the components are evil;
all component dependencies should be explicitly declared in the orchestrating code in strongly-typed manner.

This means that the core components should not use extensibility framework. Infrastructure shard boundaries might do that, while acting as a simple gateways. But that's another story.

You can subscribe to this journal to get all the updates. Voting for the next article to be written is also possible.

Efficient Development, Deployment and Customer Feedback at Lokad

Rinat Abdullin — Mon, 19 Oct 2009 07:25:27 +0000

A while ago I wrote about deployment and updates of the desktop applications and outlined the technology we use at Lokad for rapid and efficient delivery of various updates to the customers.

Let's expand on the topic a little bit more, showing how a small company could create quite a bit of business value via some efficient development practices.

There will be a small questionnaire at the end of this article as well.

Development and Deployment Experience

Current application deployment and publication loop for Lokad Client Applications works like this: when developers commit new features or fixes to version control repository and hit a special build button on integration server, following happens automatically:

All unit tests, integration and code quality tests are executed against the latest codebase (build breaks if these fail).
New version of install package (and optional downloads) is deployed to publicly available location.
Download page is updated with the links to the new version.
Existing users of an application get a nice "Update Available!" notification in the corner of their toolbar. Clicking on it launches semi-automatic download and upgrade process.

Note, that we are not using ClickOnce technology here, since it has important limitations we couldn't live with. Everything took a bit of time to setup once and then it is just started flowing.

Customer Feedback Experience

Recently, in order to enhance customer experience, yet another simple feature had been added to the client applications: integrated feedback reporting. Simply put, customer could click a button and send a message to the company. It will immediately show up in the customer support system. Additionally, all unhandled exceptions (should they happen) could be reported, too.

Feedback reports (if customers decide to) could include following anonymous information:

system descriptor of the software (version, OS version);
versions of the assemblies loaded into the AppDomain;
latest portion of application log;
statistics of some performance counters showing client-side information about interactions with our servers (number of communication failures, retries, amount of data transferred);
exception counter statistics (information about unique exceptions handled by the application along with their counts).

Note, that sources for the SystemDescriptor, ExecutionCounter and ExceptionCounter are available in Lokad Shared Libraries. Actual feedback reporting shard is shared with the community in the Lokad SDK

Feedback system had been implemented as an infrastructure shard within the shared codebase. This implied IoC-friendly component-driven design and allowed to efficiently reuse all components in multiple applications (even in these that didn't have any IoC infrastructure around). All improvements and fixes to the shared components, in such scenario, automatically benefit all applications that use them.

Additionally, since the feedback system is implemented as an independent infrastructure shard, it can be used to trap and report possible exceptions that take place on the application startup (even before the Inversion of Control Container is setup). Later, when the application and composite UI elements are ready, we switch stand-alone feedback handler for integrated handler (that uses, application's Viewspace and action policies, for instance)

This reduced feedback friction for the customers and made issue reports a lot more useful and helpful for the development team. Since all feedback reports get routed to the internal customer support system, we could feel the change really fast and resolve problems and provide solutions more efficiently.

New Challenge - Update Flood

This kind of automatic and tight integration between development, deployment and customer support spawns new possibilities and brings forth new challenges. First of all, it definitely affects the way we think about the development and plan future changes.

Second, since it is too easy to deploy an update, we've got to start thinking about protecting customers from being flooded with a number of updates (software should solve the problems, not create new ones), since theoretically we could fire the deployment update after every single verified fix that is landed in the trunk.

Here's how one of the implementation scenarios looks like right now.

All software updates (packaged as a complete version, as usually) could be described with a:

version;
download url;
tags (Critical, RC, Test, Major etc)
optional description.

Primary download page is automatically updated with the links only when "Critical" and "Major" versions are released.

Customers are unformed about the Major updates to the installed software with a usual manner (i.e.: toolbar notification). Less-important updates are not shown to them explicitly but could be pulled (i.e.: from the menu), if customer needs that.

This scenario should work if we are facing scenario, when there is a minor fix affecting only a single customer. We could deploy an update via the normal procedure, but it does not make a lot of sense to push updates to every single customer.

Implementation should be rather straightforward:

Instead of updating simple txt on the server per every update, we could update an XML file in the RSS/Atom format with the build version, url, description and tags.
By default every application filters out all non-Primary updates, displaying the rest on the toolbar. This way we deliver important functionality to customers, while saving them from the barrage of less important builds.
Customers might explicitly look into the update stream, picking and firing the specific update they need (i.e. when informed about the fix by the customer support).
Beta testers and partners might update application settings to be notified about RC and Minor deployments
Developers and Testers might switch their filtering settings to include Beta and Minor updates.

Potential synergy effects:

Updates could be subscribed to via a mere RSS reader and mashed with a company news (automagically).
Since we are already tracking Delivery-Build labels for issues, this information could be pasted to the description dialog, providing customers and testers with more context (could be available from the application and from the web/RSS).
It is possible to push the idea even further and let customers set simple filter in their application ("inform me when ticket #X is fixed"). Then, whenever the next deployment is detected, a simple regex could scan the description for the ticket number, raising friendly alert, whenever it is mentioned there.
If we have a critical update at hand (i.e.: breaking API change or an important security vulnerability) we can block the entire app and insist on updating.

Everything might get even more interesting as extensible and composable application architectures (with extensions being developed and delivered concurrently) get into the picture, allowing fine-grained and yet automated customer-company interactions.

Summary

In this short article we've walked over some development practices adopted in Lokad. These practices help the company to resolve customer problems and provide new solutions to their business challenges.

In short the outlined practices include:

efficient software deployment and updates;
efficient customer support and feedback collection;
considering synergy effects and automation, whenever possible and feasible.

Of course, there's a lot more in efficient development, than these few practices. I'll address them eventually. You can subscribe to this journal to get all the updates. Voting for the next article to be written is also possible.

As always, all feedback to the article is welcome and appreciated. Here are also a few questions, if you are willing to share your experience with the community around this Journal:

How much time does it takes for the customer to report a problem and get an update in your development environment?
Does your company share the code with the community?
Are you a customer of Lokad? What do you think about this development approach? How could it be improved?
Are you a competitor of Lokad? What do you think? ))

Let's Make Blogging More Efficient, Just like the Development

Rinat Abdullin — Sat, 10 Oct 2009 11:59:08 +0000

Over the course of the last few months I've gathered quite a few ideas and drafts about things that I wanted to share thoughts about. However, since there are too many of these, sometimes it is hard to pick one topic that deserves being turned into an article before the others.

To clear up the whole blogging process and make it more efficient (just like in the software development), I've created a feedback forum for this Journal on uservoice and filled it with a list of some planned articles.

It works like this:

You can check out these article ideas and post some feedback on them (adding comments, questions).
Every uservoice user is granted with 10 votes. You can distribute your votes between existing article ideas or even create a new one for a topic on efficient software development, that's dear to your heart.
Whenever I'm stuck picking the next article to work on, uservoice should provide the clear picture of what interests the community around this Journal (eventually also providing an additional motivation to move forward).

Let's see if this works out, shall we?

Here's the link: What articles are you interested to read next?

Update on Maybe Monad: Q&As and Caveats

Rinat Abdullin — Thu, 08 Oct 2009 12:24:43 +0000

My last Zen dev article on Maybe Monad for C# had caused quite a few arguments and comments coming from different directions.

To make the information exchange slightly more efficient, I've composed a few additions for the article:

Questions and Answers
Caveats

You can find this new content either below or in the original article that had been updated. All new major updates on the topic will be delivered in the same way.

Questions and Answers

Question: Why add another needless layer of complexity for such a simple task? Just learn how to use the null coalescing operator (??) and/or inline if-thens.

Console.WriteLine(number != null ? "True" : "False");
Console.WriteLine(number ?? "Null Value");

Imagine a large real-world project that spawns multiple components and tiers, while facing problems slightly more challenging, than writing output to the console. If developers know that this project uses Maybe to declare potentially empty variables, then they are completely freed from checking for nulls and even thinking about that. That's one less thing to bother, making quite a difference in complex scenarios (and human being can simultaneously bother only about 7 things max, anyway).

This actual usage of Maybe{T} might happen in just one or two methods of the project. However, when developers use them, they will know immediately (and not from the documentation) and will be forced to handle them because of the syntax.

Things like that reduce development friction and allow complex project to be delivered efficiently and with really small teams.

Question: How is this different from Nullable{T} in C#?

Only value types are supported by Nullable generic (or its shorthand syntax). So you can't use a class variable in a nullable type (which makes sense, since the reference itself can be null).

Question: Why do Lokad Shared Libraries feature null checks, although Maybe{T} is declared in the code?

Because Lokad.Shared.dll, as a really lightweight library, might be used in various projects. Some of these projects might not have "no nulls returned and expected" policy. So it is logical to protect shared methods from the nulls explicitly. in order to avoid the unpredictable behavior.

Question: How do I use Maybe{T} if method can fail because of the multiple reasons and I want to know them?

Maybe{T} on its own is not designed to handle such situations. These are the primary logically-complete solutions for this situation:

You can return Maybe{T} while printing multiple failure details to the IScope passed as an argument. That's how Lokad Rules are working. It also works well in long-running complex (especially asynchronous) operations that require logging and detailed failure control.
You can use Result{TValue,TError} monad returning strongly-typed error result (i.e.: enumeration value) or it's generic counterpart Result{TValue} (if the strong-typing is not possible for some reason).
You can always use good old exceptions, if the above options do not suite you for some reason and you are OK with try-cath and breaking the functional code flow.

Caveats

This approach, as any other piece of knowledge from the xLim body of knowledge, might be controversial and not applicable for you projects, development practices and beliefs.

All that I can say - it works for me, allowing slightly more efficient delivery, evolution and maintenance of .NET development projects ranging from business analytics servers, RESTful NoSql automation engines and up to Smart Client applications with reusable MVC infrastructure shards.

This limited improvement in development efficiency sometimes requires using approaches and principles that may be somewhat incompatible with the traditional ones (MS .NET or Alt.NET), while continuously enforcing them across the codebases. But in the end, I think, it's worth it, just like Continuous Integration or Inversion of Control.

But again, that's the way of Zen - everybody has his own way to satori.

As this is an update article, all comments are disabled here. Please use original article on Maybe Monad for sharing your feedback (which is welcome and appreciated).

Zen Development Practices: C# Maybe Monad

Rinat Abdullin — Mon, 05 Oct 2009 19:25:10 +0000

One of the things I like about software development is a simple fact, that efficiently developed code, block or software, takes much more time to document, than to deliver.

This efficiency could be based on certain practices, constraints and principles, followed by discipline. These items allow to take away small bits of complexity and development friction, allowing us to concentrate on the things that really matter and generate business value.

Let's talk about one of this small things - a really simple construct in C# called Maybe (of the monad fame) that had completely freed me from all cases of NullReferenceException in my code, while making it cleaner.

Sounds interesting? Let's have a look at the C# code and .NET usage patterns.

Introduction

Maybe<T>, as the simplest monad implementation in .NET, is a generic class that can either hold value of type T or be empty. There is a little bit of helper syntax and some usage guidelines and patterns.

In this article we'll cover all of these; links for the sources in C# and binaries would be provided, too.

Here's how the declaration of the Maybe class starts in C#:

[Serializable, Immutable]
public sealed class Maybe<T> : IEquatable<Maybe<T>>
{
  readonly T _value;
  readonly bool _hasValue;

  Maybe(T item, bool hasValue)
  {
    _value = item;
    _hasValue = hasValue;
  }

  /// <summary> Gets the underlying value, if it is available </summary>
  /// <value>The value.</value>
  public T Value
  {
    get
    {
      if (!_hasValue)
        throw Errors.InvalidOperation(
          ResultResources.Dont_access_value_when_maybe_is_empty);

      return _value;
    }
  }

  internal Maybe(T value)
    : this(value, true)
  {
    // ReSharper disable CompareNonConstrainedGenericWithNull
    if (value == null) throw new ArgumentNullException("value");
    // ReSharper restore CompareNonConstrainedGenericWithNull
  }

  /// <summary> Gets a value indicating whether this instance 
  /// has value. </summary>
  /// <value><c>true</c> if this instance has value; otherwise, 
  /// <c>false</c>.</value>
  public bool HasValue
  {
    get { return _hasValue; }
  }

  /// <summary>
  /// Default empty instance.
  /// </summary>
  public static readonly Maybe<T> Empty = new Maybe<T>(default(T), false);

There are a few implicit operators and helpers methods that improve the usage experience (we'll talk about them later), but the simple code above is the core of the approach. Primary monad operations could be expressed in this simple snippet:

// function that MIGHT return a number
Maybe<int> DetectNumber(string number)
{
  switch (number)
  {
    case "Zero":
      // we are allowed an implicit conversion
      // so here we return a number directly
      return 0;
    case "One":
      return 1;
    case "Two":
      return 2;
    default:
      // empty singleton for missing value
      return Maybe<int>.Empty;
  }
}


// Actually getting a Monad in C#
var number = DetectNumber("Zero");

// prints "True"
Console.WriteLine(number.HasValue);
// prints "0"
Console.WriteLine(number.Value);


var number2 = DetectNumber("Hm..");
// prints "False"
Console.WriteLine(number2.HasValue);
// throws InvalidOperationException
Console.WriteLine(number2.Value);

So basically this Maybe monad is extremely simple class that can either be empty or have value. If it is empty, and we attempt to access value, then the exception is thrown. In other words, that's a strongly-typed reference to an object that acts like null with some syntactic flavor.

How is it even better than using nullable references and getting NullReferenceException when you forget to check for null?

Here are the reasons justifying the usage of Maybe{T} for me:

Reason 1: Good Citizenship

By using Maybe{T} in my code I promise that all the code will be following good citizenship principles and will keep all object references non-null (initialized to some value from the default). This promise is to be uphold by a self-discipline and unit tests.

This helps to avoid all null references completely.

Reason 2: Explicit behavior and clean code

If there is a potentially nullable reference, parameter or a function return result, then it should be explicitly marked with the Maybe{T}, exposing its behaviour to the reader and forcing him to explicitly handle the states.

Additionally, using Maybe{T} allows us to avoid throwing weird exceptions or returning nulls in situations, where result could be undefined.

Consider declaration of the method like this:

/// <summary> Parses an encoded string into
/// authentication information. </summary>
/// <param name="encodedString">Encoded string.</param>
/// <returns>authentication information</returns>
/// <exception cref="ArgumentNullException">when string is null</exception>
/// <exception cref="InvalidOperationException">when string is invalid</exception>
AuthInfo ParseStringToAuthInfo(string encodedString);

This method declaration is really similar to a lot of existing methods from the .NET BCL (i.e.: int.Parse as the simplest one) and it shares a lot of excessive noise and inherent problems. Most importantly we are throwing some exceptions, when the input is not expected and could not be handled gracefully. This:

blows the code execution and must be handled by a separate try-catch;
requires code consumer to read the documentation instead of reading the method signature;
does not prevent the code from potentially breaking in every single place, should we refactor and replace InvalidOperationException by InvalidAuthDataException.

Let's rewrite the method signature using the Maybe monad style:

/// <summary> Attempts to parse an encoded string into 
/// authentication information. </summary>
/// <param name="encodedString">Encoded string.</param>
/// <returns>authentication information</returns>
static Maybe<AuthInfo> ParseStringToAuthInfo(string encodedString)

With this refactoring we:

decoupled ourselves from the ArgumentNullException, since we believe that input will never be null. Method body would live without the appropriate null check as well;
decoupled ourselves from the InvalidOperationException and use optional value, that would be valid only if the parsing succeeded;
no longer require the developer using the method to look in the code documentation in order to understand what exactly the method works;
do not need defensive and noisy try-catch programming around the calls to this method (or anywhere in the call stack).

Of course, there are other refactoring alternatives to the signature:

// Option1 : use bool
static bool TryParseString(string encodedString, out AuthInfo);   
// Option 2: return value that could be null
static AuthInfo ParseString(string encodedString);

First option uses the TryGetValue approach you can see on the IDictionary interface. This approach resorts to the out keyword (which I always avoid), breaks the functional style of the programming (method should always have a single value being returned) and adds some noise to the using code.

Also AuthInfo is returned as undefined, if the parsing fails, violating the good citizenship.

Second option returns value that would be null in case the parsing fails. Developer, in order to discover this, has to read the documentation. And he has to protect himself from the possible problems with null checks against the result.

In my opinion, method using the Maybe monad is the most clean approach of them all.

Reason 3: Chaining and pipeline syntax

Ok, so we have our Maybe-powered method signature:

static Maybe<AuthInfo> ParseStringToAuthInfo(string encodedString);

Let's actually see the full power of the concept in C# on a sample taken fresh from the production code. Consider the method:

// simple pipeline sequence
Maybe<AuthPrincipal> LoadPrincipal(string key)
{
  return ParseStringToAuthInfo(key)
    .Combine(u => _service.BuildAccountSession(u))
    .Convert(sa => new AuthPrincipal(sa));
}

In this LINQ-style and purely functional sequence of 3 lines, we:

Attempt to parse the key into AuthInfo.
If the attempt was successful, we use some service to try to build an account session out of the parsed AuthInfo. BuildAccountSession operation might also fail, by the way. This is indicated in the method signature (by using Maybe) and would happen if the provided AuthInfo didn't have a match in the repository.
If we were successful in parsing the original string and in building the account session, then we create a new AuthPrincipal class using this session; otherwise an empty Maybe{AuthPrincipal} is returned to indicate the composite failure.

As you can see, detailed method description takes a bit more space, than the actual implementation. And the implementation itself is less fragile and smaller compared to traditional approach with nulls, throwing and handling various exceptions.

After getting used to this benefit of Maybe, dealing with the code that has exceptions and nulls feels like working in Visual Studio without ReSharper installed.

All the benefits listed above come from two classes (that contain less code, than this article) and some discipline. That's the way of Zen.

If you want to try Maybe{T}, the code is available in Lokad.Shared.dll from the Lokad Shared Libraries. You can either copy this class to your solution (while keeping an eye on the project for any improvements and changes in the guidelines) or grab the latest download.

Let's continue our article by outlining common operations available for Maybe{T}.

Creating Maybe

// implicit conversion is the most common way to create Maybe<T>
// mostly it happens when we return actual value from within a
// method that returns maybe.
// Or, when we pass value into the optional function parameter
Maybe<int> maybeInt = 10;

// creates a maybe object when implicit casting is not available 
// for some reason (i.e.: anonymous types or failing type inferrence)
var nonEmptyMaybe = Maybe.From(value);

// each type has only a single empty maybe instance that is
// cached via the type-caching
var emptyAuthInfo = Maybe<AuthInfo>.Empty;

Getting Value

// tests if the Maybe has value
var hasValue = maybeInt.HasValue;

// returns value behind the Maybe, throwing exception if it is empty
// this approach is rarely used (because of the pipeline operators)
var value = maybeInt.Value

// get's value from maybe, using some default, if it is empty
var value = maybeUsername.GetValue("guest@abdullin.com");

// get's value from maybe, executing default generation if it is empty
// handy for scenarios, when generating default values is CPU expensive
var value = maybeUsername.GetValue(() => ComposeName());

Pipeline Operations

// get Maybe that either has value derived from the parent Maybe
// or is empty, if the parent is empty
var maybeUsername = maybeIdentity.Convert(i => i.Username);

// gets actual value that is derived from the parent Maybe, if it is not
// empty, or uses a predefined default 
// equivalent of Convert.GetValue
var username = maybeIdentity.Convert(i => i.Username,"guest@abdullin.com");
var username = maybeIdentity.Convert(i => i.Username,() => ComposeName());

// combines result of the two Maybe's, either of which could be empty
var maybeFullName = maybeUsername.Combine(k => LookupFullName(k));

// executes the action against the value, if Maybe is not empty,
// returns the original Maybe to continue pipelining 
maybeFullName.Apply(fn => Console.WriteLine(fn))

// executes the action, if maybe is Empty
// returns the original Maybe to continue pipelining
maybeFullName.Handle(() => Application.Terminate());

In addition to the operators above, Maybe{T} class also defines proper equality and hashing operations. It is debugger-friendly, too.

However, this does not end the list of reasons of using Maybe for the efficient development.

Reason 4: Simplified unit testing

Using Maybe monad in .NET also has the benefit of allowing to introduce simplified DSL syntax for unit-testing code leveraging the approach.

Here are the core extensions you need (they are pipelined together for the brevity)

// extensions checking different aspects of valid Maybe<T> monad
maybeValue
  .ShouldPass()
  .ShouldPassCheck(i => i == 10)
  .ShouldPassWith(10)
  .ShouldBe(10)
  .ShouldBe(true);

// checking empty Maybe Moned
MaybeNot
  .ShouldFail()
  .ShouldBe(false);

Each extension, basically throws a descriptive assertion exception, if the condition is not met.

Here's how the unit test might look like with one of these extensions:

[Test]
public void System_info_is_disabled()
{
  WhenViewIsShown.SetResult(DialogResult.OK);
  Subject.BindModel(RandClientModels.NextFeedbackView());
  Implementation.CheckSystemInfo(false);
  Subject
    .GetModel()
    .ShouldPassCheck(m => m.SystemInformation == "");
}

If the check fails, because the maybe has a value but this value is invalid, then the failure would look like:

FailedAssertException: Expression should be true: '(m.SystemInformation = "")'.

These test extensions are located within Lokad.Testing.dll, which is also a part of Lokad Shared Libraries within a Lokad.Testing namespace.

This separation is done in order to prevent IntelliSense from being polluted with all the Should statements, while we are working in the production code. Testing code, on the contrary can have access to these routines (which is done by referencing the testing assembly and using the namespace).

Reason 5: Initialization of Immutable classes

All we know that in F# it is extremely easy to create an immutable object from another record by providing a few override values in the cloning process:

let conn2 = { conn1 with Service = MyService }

If we are using only C#-style immutable classes as models in our application (which is beneficial for a number of reasons, but is a large topic on its own), then we are limited to the constructor initialization only, which could be rather messy:

// model definition
[Immutable, Serializable]
public sealed class ServiceConnection
{
  public readonly string Username;
  public readonly string Password;
  public readonly string AuthMode;
  public readonly Uri Service;

  public ServiceConnection(string username, string password, 
    string authMode, Uri service)
  {
    Username = username;
    Password = password;
    AuthMode = authMode;
    Service = service;
  }
}

// cloning with a custom service reference
var conn2 = new ServiceConnection(conn1.Username, conn2.Password, 
  conn2.AuthMode, MyService);

More fields the model has, more messy cloning the process becomes. Fortunately, in the upcoming C# 4.0 we will be able to do everything more efficiently and with different combinations:

var conn2 = conn1.With(service: MyService);
var conn3 = conn1.With(username: Anonymous, AuthType: Plain);

This could be achieved with a single method by using C# 4.0 Optional Parameters and Maybe{T}:

  public ServiceConnection With(
    string username = Maybe.String,
    string password = Maybe.String,
    string authType = Maybe.String,
    Uri service = Maybe.Uri)
  {
    return new ServiceConnection(
      username.GetValue(Username), 
      password.GetValue(Password),
      authType.GetValue(AuthType),
      service.GetValue(Service);
  }

where Maybe.String (and its Uri analogue) is just a short-cut for Maybe{string}.Empty that is already declared on the Maybe non-generic helper class in Lokad.Shared.dll.

Caveats

This approach, as any other piece of knowledge from the xLim body of knowledge, might be controversial and not applicable for you projects, development practices and beliefs.

But again, that's the way of Zen - everybody has his own way to satori.

Questions and Answers

Question: Why add another needless layer of complexity for such a simple task? Just learn how to use the null coalescing operator (??) and/or inline if-thens.

Console.WriteLine(number != null ? "True" : "False");
Console.WriteLine(number ?? "Null Value");

Things like that reduce development friction and allow complex project to be delivered efficiently and with really small teams.

Question: How is this different from Nullable{T} in C#?

Only value types are supported by Nullable generic (or its shorthand syntax). So you can't use a class variable in a nullable type (which makes sense, since the reference itself can be null).

Question: Why do Lokad Shared Libraries feature null checks, although Maybe{T} is declared in the code?

Question: How do I use Maybe{T} if method can fail because of the multiple reasons and I want to know them?

Maybe{T} on its own is not designed to handle such situations. These are the primary logically-complete solutions for this situation:

You can return Maybe{T} while printing multiple failure details to the IScope passed as an argument. That's how Lokad Rules are working. It also works well in long-running complex (especially asynchronous) operations that require logging and detailed failure control.
You can use Result{TValue,TError} monad returning strongly-typed error result (i.e.: enumeration value) or it's generic counterpart Result{TValue} (if the strong-typing is not possible for some reason).
You can always use good old exceptions, if the above options do not suite you for some reason and you are OK with try-cath and breaking the functional code flow.

Question: Does Maybe work with validation rules?

Yes, Maybe{T} works with Lokad validation and business rules. You can also use MaybeIs helper class to compose strongly-typed rules on top of the Maybe{T} monad. MaybeIs has two static methods:

MaybeIs.EmptyOr{T}(params Rule{T}[] rules) - composes validation rule that succeeds only if the Maybe{T} is empty or the underlying value passes additional rules.
MaybeIs.ValidAnd{T}(params Rule{T}[] rules) - composes validation rule that succeeds if the Maybe{T} is not empty and the underlying value passes additional rules.

Sample usage in composing more complex rule:

public static void ModelIsValid(FeedbackModel model, IScope scope)
{
  scope.Validate(() => model.Sender, 
    MaybeIs.EmptyOr<string>(StringIs.ValidEmail));

  scope.Validate(() => model.Model, ModelIsValid);
}

Question: So essentially you are replacing "optional != null" with "optional.HasValue".

Not exactly. HasValue is rarely checked against directly. Take a look at the piece of authentication code below. There are two possibly failing functions in it, that return Maybe{T}:

Maybe<AuthPrincipal> LoadPrincipal(string key)
{
  return ParseStringToAuthInfo(key)
    .Combine(u => _service.BuildAccountSession(u))
    .Convert(sa => new AuthPrincipal(sa));
}

Signatures are like:

static Maybe<AuthInfo> ParseStringToAuthInfo(string encodedString);
Maybe<AccountSession> BuildAccountSession(AuthInfo authInformation);

HasValue and Value aren't mentioned in this code. That's how it usually works.

Question: What about operator overloading:

public static implicit operator T(Maybe<T> mayBe)
{
  return mayBe.Value; //or mayBe._value;
}

public static implicit operator bool(Maybe<T> mayBe)
{
  return mayBe.HasValue;
}

This kind of implicit conversion from Maybe directly to value/state is not advised for two reasons:

It looses the intent of using Maybe{T} to force explicit handling of nullable situations.
Implicit conversion from Maybe to value is logically incorrect as it might return default(T) value or result in exception (depending on implementation).

Basically such an implicit operation would feel somewhat similar to the old C++ joke from bash.org:

#define TRUE false
#define FALSE true
// and good luck with debugging))

Side note: Implicit conversion from value to Maybe{T} or from error/value to Result{TValue,TError}, as opposed to the case above, does not violate this logic. This is because we are not forcing collapse of possibility of multiple states into one known state (while have side effects if the state is not what we expect).

This is why conversion from value to maybe is implicit, while the opposite has to be done either via the Value property or via the explicit conversion.

Summary

Let's wrap the things up.

In this article we've talked about Maybe{T} generic class that, as a simple monad concept, allows in .NET to:

promote Good citizenship in the code avoiding nulls and exceptions;
write clean code with explicit behavior;
use expressive LINQ-style syntax with chaining and pipeline operators;
benefit from the simplified unit testing syntax;
initialize immutable classes efficiently in C# 4.0.

C# source files and compiled binaries featuring Maybe monad and testing extensions could be downloaded from the Lokad Shared Libraries Open Source project.

It is important to note, that the current design of Maybe{T} and development practices around it are based on the ongoing R&D process and production usage cases. As such, they might gradually evolve and improve further (this being reflected in Lokad Shared Libraries).

This article is also considered to be an essential part of xLim 3 development body of knowledge.

If you are interested in more articles on Zen Development practices, you can stay tuned by subscribing to this blog via RSS.

Next article I'm considering for these series is about Result variations of the monads in C#, that allow handling more complex failure scenarios, than a simple Maybe (BTW, the previous Zen article was about Method-level IoC).

All comments, thoughts and any other feedback are welcome and appreciated.