Rinat Abdullin on CouchDB

10 Steps To Become Better .NET Developer

2010-11-19T10:20:17Z

Here's a list of things you might want to learn about in order to become a better .NET developer. Better developers are eligible to higher paychecks, exciting projects and more freedom in their lifestyles.

Learn Unit Testing
Get familiar with distributed Version Control Systems: Git or Mercurial.
Learn about IoC Containers: Autofac, Windsor Container and StructureMap.
Contribute to Open Source Projects of your choice.
Learn about Continuous Integration and try setting up TeamCity, Hudson or CruiseControl.NET.
Learn about NHibernate Object Relational Mapper, but be prepared to outgrow it for NoSQL persistence like CouchDB, MongoDB or RavenDB.
Check out the buzz about cloud computing, try Windows Azure Platform and also give a shot to running your .NET App with Mono on Linux (some free offers to try).
Familiarize yourself with messaging architectures and try NServiceBus, MassTransit or RhinoQueues; then learn about alternative to MSMQ - AMQP and RabbitMQ.
Read Domain-Driven Design and apply it in your projects.
Get Started with CQRS Architectures and practice it in a learning project.

As you can see, the list is not endless. There are just ten distinct topics structured in the recommended order of learning and practicing. Diligently walking through them is certain to get you way above the level of Senior Developer, opening a whole set of opportunities all around the world.

And if you happened to learn, understand and master all these steps and still don't have interesting job or project offer - drop me an email; I might get you in touch with folks desperately looking for developers with this set of skills.

CouchDB - Document Persistence Powered by CQRS

2010-04-08T10:09:46Z

I've been asked about my opinion on the CouchDb (MongoDb) multiple times. Here's my current analysis and opinion.

As any .NET developer I've tried working with the document-oriented databases. In my case the introduction started with the CouchDB. There was some research and development including the NCouch project.

CouchDB is a database famous for its distributed nature, great read performance and schema-less nature of the documents (CouchDb overview and introduction for .NET).

These nice features come right of the box, when you deploy CouchDB. And they are actually available out there for you because CouchDB partially implements Command-Query Responsibility Segregation principles. These benefits are inherent to the CQRS:

Free schema
High read performance
Replication

Indeed:

Free schema in CouchDb - persistence ignorance in the CQRS.
Multiple view engines in CouchDb - multiple subscribed event denormalizers in the CQRS.
High read performance of CouchDb views - high read performance against denormalized query tables.
Auto replication in CouchDb - Publish/Subscribe in CQRS.

From this standpoint CouchDb is an extremely nice base for building lightweight and high-performance document-oriented web applications, since you get all the performance features packed nicely with the REST API, JSON and ability to write view engines in multiple languages.

CouchDb might work for you even if you are not working on the web. If your solution is heavily into the domain of documents and unstructured data, then CQRS has chances of fitting nicely there.

However, if your domain deals with the business processes, where changes could be more important than just the data, then it is better to implement a CQRS-based solution yourself. You'd get similar performance and scalability benefits as CouchDb.

Let's take a moment to think about the logical differences between the data (state) and changes.

State is a version of a document, static snapshot or immutable representation of something. Change is a step in the process, that describes "how did we get to the version 2 from version 1" or "what happened between the lines". As such, change captures much more contextual information about the process, than the mere difference between the two versions.

In business contextual information is the king. It is so important, that changes are often treated like full-class citizens. Sometimes you can meet them under the names of events, commands or workflows. We even got persistence pattern that is based on events, rather than on the state - Event Sourcing.

Given all that, my impression about CouchDB is: CouchDB is a young document-oriented database that leans towards the web applications and is based on the CQRS principles. The latter gives it nice performance, distribution and scalability.

Unfortunately, as a .NET developer focused on the distributed enterprise applications targeting business scenarios, I don't have a lot of use cases for such an interesting tool.

This article concludes my study of the CouchDB, for now. It also represents implementation study for the CQRS research, which is a part of xLim 4 body of knowledge.

CouchDB in the Cloud - Persisting From .NET Code

2009-07-24T10:43:40Z

This article about CouchDB is a research that continues topics of xLim and Cloud Computing in .NET series.

In the previous article we've talked about deploying CouchDB servers in the cloud and locally. Now it is time to outline a simple .NET adapter for communicating with these servers.

In order to talk to CouchDB we need a few pieces:

JSON serializer - ISerializer;
Service for getting and posting REST commands - IRestClient;
Actual implementation of CouchDB client - CouchServer and CouchDatabase.

These pieces might fall together in an extremely simple (yet extensible) fashion:

Let's go with the most simple implementation that will let us execute code like this:

[Test]
public void RoundTrip_Class()
{
    // connect to the database
    var database = CouchFactory.ConnectToDatabase(DBUrl);

    // create new document objecy
    var user = new SampleUser
    {
        Name = "user@unknown.com",
        Tags = new[] { "User", "Can have attachments"}
    };
    // save document, specifying the identifier
    var revision = database.SaveDocument(user, u => u.Name);
    Assert.AreEqual(user.Name, revision.Id);

    // retrieve the document
    var retrievedUser = database.GetDocument<SampleUser>(user.Name);
    Assert.AreEqual(user.Name, retrievedUser.Name);

    // double check nested objects
    CollectionAssert.AreEqual(user.Tags, retrievedUser.Tags);

    // check out the attachment round-trip
    var info = database.AddAttachment(revision, "my.txt", WriteFile("my.txt"));
    Assert.AreEqual(info.Ok, true);
    database.GetAttachment(info, "my.txt", ContentsShouldBeEqual("my.txt"));
}

JSON Serializer for CouchDB

That's how JSON (JavaScript Object Notation) looks like. As you can see, it is yet another human-readable and platform-independent alternative for exchanging serialized data over the wire.

{
     "firstName": "John",
     "lastName": "Smith",
     "address": {
         "streetAddress": "21 2nd Street",
         "city": "New York",
         "state": "NY",
         "postalCode": 10021
     },
     "phoneNumbers": [ "212 555-1234", "646 555-4567" ]
 }

There are quite a few JSON serializers for .NET available out there. Some of these include:

Json.NET;
JayRock;
DataContractJsonSerializer from WCF;
JSONSharp;
LitJSON
JavaScriptSerializer from System.Web.Extensions.dll as mentioned by Oleg.

Each JSON serializer has its own advantages (primary one being - they already are implemented) and disadvantages for the purpose of integrating with CouchDB.

If I were to implement a proper .NET wrapper for CouchDB form the start, I would definitely go with my own serializer tailored for CouchDB needs (using JayRock architecture as a guideline but making it more strongly-typed and simple). However in this research article we'll stick with the DataContractJsonSerializer from Windows Communication Foundation.

This serializer has following major disadvantages that should be kept in mind:

This serializer does not handle anonymous types. This takes away all fun of working with schema-less databases.
Serializer requires classes to be declared with a DataContractAttribute that has members decorated with DataMemberAttribute. This does not allow custom mapping routines.

Let me explain why Attribute-based type declaration creates too much development friction for such a flexible database as CouchDB.

Just compare these two snippets below. First one is about declaring objects for CouchDB API in WCF-compatible (and C#-styled) way:

public sealed class DatabaseInfo
{
    [DataMember(Name="name")]
    public string Name { get; set; }
    [DataMember(Name="doc_count")]
    public int DocCount { get; set; }
    [DataMember(Name = "doc_del_count")]
    public int DocDelCount { get; set; }
    [DataMember(Name="update_seq")]
    public int UpdateSeq { get; set; }
    [DataMember(Name = "compact_running")]
    public bool IsCompacting { get; set; }
    [DataMember(Name = "disk_size")]
    public int DiskSize { get; set; }
}

Second snippet is about declaring objects using F# projects (which are first-class citizens in Visual Studio now) and style:

type db_information = 
  {
    db_name : string;
    doc_count : int;
    doc_del_count : int;
    update_seq : int;
    compact_running : bool;
    disk_size : int
  }

In my opinion, second type of declaration feels somewhat closer to the ideas behind CouchDB. More than that, F# compiler would automatically implement CompareTo, Equals and GetHashCode for these .NET objects. However using this kind of object declaration would require either extending an existing Json serializer or writing a new one.

Given all that, DataContractJsonSerializer from WCF is still perfect for this specific research article.

RestClient

The most simple implementation of a RestClient class is a wrapper around WebRequest class that could look like this:

public sealed class RestClient
{
    public readonly Uri Url;

    public RestClient(Uri url)
    {
        Url = url;
    }

    public Restponse Do(string query, string method, Action<Stream> writer);    
    public Restponse Do(string query, string method);
}

Two Do overloads are almost the same. The second one just does not create a stream for writing contents. Second overload looks like:

public Restponse Do(string query, string method, Action<Stream> content)
{
    try
    {
        var request = (HttpWebRequest) WebRequest.Create(new Uri(Url, query));
        request.Method = method;
        request.ContentType = "application/json";

        using (var writer = request.GetRequestStream())
        {
            content(writer);
        }

        return new Restponse((HttpWebResponse) request.GetResponse());
    }
    catch (WebException e)
    {
        // till we get to rewrite web requests to live without exceptions
        return new Restponse((HttpWebResponse) e.Response);
    }
}

where the Restponse is a simple wrapper class used to ensure that code stays decoupled from HttpWebResponse class and that developer does not forget to dispose the request:

public sealed class Restponse : IDisposable
{
    public readonly int StatusCode;
    readonly HttpWebResponse _response;

    public Restponse(HttpWebResponse response)
    {
        _response = response;
        StatusCode = (int)_response.StatusCode;
    }    

    public void RunReader(Action<Stream> read)
    {
        using (var s = _response.GetResponseStream())
        {
            read(s);
        }
    }
    public T RunReader<T>(Func<Stream,T> read)
    {
        using (var s = _response.GetResponseStream())
        {
            return read(s);
        }
    }    
    public void Dispose()
    {
        ((IDisposable)_response).Dispose();
    }
}

By the way, as we've mentioned in the schematics before, RestClient is a good place to hide behind the IRestClient interface and squeeze in a wrapper in between.

Thise way you'll be able to add exception handling and reliability layer to all requests. This layer could be implemented using IoC-injectable Action Policies intercepting all communication failures (i.e.: network break-downs) and retrying the last request a few times before giving up.

CouchServer

Now we have two simple pieces needed to talk to CouchDB. Let's actually try doing this. Our knowledge about the server-level CouchDB API will be encapsulated in CouchServer and CouchDatabase classes. CouchServer starts like this:

public sealed class CouchServer
{
    readonly ISerializer _serializer;
    readonly IRestClient _client;    

    public CouchServer(IRestClient client, ISerializer serializer)
    {
        _serializer = serializer;
        _client = client;
    }

As we remember, ISerializer is a simple interface abstracting away DataContractJsonSerializer.

In the environment without Inversion of Control CouchServer could be created with a factory method:

public static CouchServer ConnectToServer(string url)
{
  var rest = new RestClient(new Uri(url));
  return new CouchServer(rest, new WCFJsonSerializerWrapper());
}

By the way, that's a good approach for creating developer-friendly libraries. We implement all functionality as separated and decoupled components (as in component-driven development) ready to be consumed by whatever IoC Container the developer likes. Extensibility and flexibility come in there natively.

Should the project be too simple for wring up the proper IoC infrastructure, developer-friendly library will have some static factory class to use. This class would serve as a logical entry-point into the library and would host hard-coded creation of components (using recommended configurations and settings).

Let's get back to our code. That's how the CouchServer class goes on in C#:

public string[] GetDatabaseNames()
{
    using (var restponse = _server.Do("_all_dbs", "GET"))
    {
        return restponse.RunReader(r => _serializer.Deserialize<string[]>(r));
    }
}

public void CreateDatabase(string name)
{
    using (var restponse = _server.Do(name + "/", "PUT"))
    {
        if (restponse.StatusCode != 201)                
            throw CouchException.From(restponse);
    }
}

public ServerInfo GetInformation()
{
    using (var restponse = _server.Do("", "GET"))
    {
        if (restponse.StatusCode != 200)
            throw CouchException.From(restponse);

        return restponse.RunReader(r => _serializer.Deserialize<ServerInfo>(r));
    }
}

public void DeleteDatabase(string name)
{
    using (var restponse = _server.Do(name + "/", "DELETE"))
    {
        if (restponse.StatusCode != 200)
            throw CouchException.From(restponse);            
    }
}

As you can see, we are merely executing commands according to the CouchDB RestAPI documentation here.

Software architect, that had to deal with the user input before, would definitely notice one really important thing that is missing here - input validation (CouchDB has quite a few naming rules).

If needed, such validation could be easily added with the help of Lokad Rules. Any other framework that allows to define concise, readable and composable rules in .NET would, obviously, work as well.

When we need to start working with the specific database we just use this method to create another friendly CouchDB wrapper class CouchDatabase, by passing to it existing serializer and a RestServer instance with a database url:

public CouchDatabase ConnectToDatabase(string databaseName)
{
    var dbUrl = new Uri(_server.Url, databaseName + "/");
    var server = new RestServer(dbUrl);
    return new CouchDatabase(server, _serializer);
}

CouchDatabase class is similar to CouchServer but encapsulates database-level routines. Here's how it might look like in C#:

public sealed class CouchDatabase
{
  readonly IRestClient _client;
  readonly ISerializer _serializer;

  public CouchDatabase(IRestClient restClient, ISerializer serializer)
  {
    _client = restClient;
    _serializer = serializer;
  }


  public DocumentInfo SaveDocument<T>(T item, Func<T, string> id)
  {
    using (var r = _server.Do(id(item), "PUT", 
      tw => _serializer.Serialize(tw, item)))
    {
      return r.RunReader(tw => _serializer.Deserialize<DocumentInfo>(tw));
    }
  }

  public DocumentInfo AddAttachment(DocumentInfo info, string name, 
    Action<Stream> data)
  {
    var query = string.Format("{0}/{1}?rev={2}", 
      info.Id, name, info.Revision);
    using (var r = _server.Do(query, "PUT", data))
    {

      return r.RunReader(tw => _serializer.Deserialize<DocumentInfo>(tw));
    }
  }

  public void GetAttachment(DocumentInfo info, string name, 
    Action<Stream> reader)
  {
    var query = string.Format("{0}/{1}?rev={2}", info.Id, name, info.Revision);
    using (var r = _server.Do(query, "GET"))
    {
      r.RunReader(reader);
    }
  }

  public T GetDocument<T>(string id)
  {
    using (var r = _server.Do(id, "GET"))
    {
      return r.RunReader(tw => _serializer.Deserialize<T>(tw));
    }
  }

  public DatabaseInfo GetInfo()
  {
    using (var r = _server.Do("", "GET"))
    {
      return r.RunReader(tw => _serializer.Deserialize<DatabaseInfo>(tw));
    }
  }
}

OK, this should be enough to give you an idea, how we can interact with CouchDB server instances from .NET.

By the way, interesting part of this simple approach is that it could be reused in other REST adapters as well. For example, one could keep all the infrastructure intact but add different wrapper implementations in order to talk to some other RESTful Cloud APIs (i.e. Rackspace Cloud or Windows Azure) directly.

Source code behind the idea at the moment is based on a single usage scenario prototype. Thus, it is not ready for any kind of publicly shared library or framework, since any functionality in the shared library should be backed up by real usage scenarios coming from, at least, two different projects. Otherwise we might get inefficient code and bad development practices affecting everybody using the framework.

There might be another research article on the topic about using simple .NET layer (hosted within Mono runtime and implemented as IHttpHandler) to provide cost-effective and scalable server-side functionality in the cloud with the same RESTful API (suitable for a specific set of scenarios). Stay tuned for the updates on the topic.

Related links:

CouchDB in the Cloud - Cheap and Flexible Persistence For .NET

2009-07-20T23:44:10Z

This post about CouchDB is a research that continues topics of xLim and Cloud Computing in .NET series.

We’ll walk over one of the possible approaches for using CouchDB from .NET applications in a cloud scenario.

What is CouchDB?

CouchDB is a database engine designed for persisting documents without a predefined schema. It uses HTTP REST API for communicating with JSON-serialized messages. CouchDB claims to be designed for scalability (this includes multi-master replication model).

Simply put, you talk to CouchDB server with documents like this:

{
  "Subject":"I like Plankton",
  "Author":"Rusty",
  "PostedDate":"2006-08-15T17:30:12-04:00",
  "Tags":["plankton", "baseball", "decisions"],
  "Body":"I decided today that I don't like baseball. I like plankton."
}

Documents do not have predefined schema, so this is a valid one as well:

{
  "Author":"Rusty",
  "About": "Former baseball fan."
}

NB: although documents are unstructured, they always have a unique identifier associated with them and a revision number (CouchDB features Multi-Version-Concurrency-Control model).

In order to run queries against all this unstructured data, CouchDB features uses of View Servers that process and update structured views expressed in MapReduce queries (yes, that's the scalability opportunity here). These queries are like SQL in the relational world, yet they could be written in various languages (Javascript, Python, Ryby and Python are supported at the moment).

CouchDB as a project is a top-level open-source project within the Apache Foundation.

CouchDB is written in Erlang and is capable of running on POSIX systems. Theoretically it could even run on Windows, but the experience is far from being smooth. Yet, it is extremely easy to get yourself a cheap DB instance running in the cloud (as we'll see later in the article).

Primary disadvantages of the project are:

CouchDB is a young project and it has not gone through heavy stress usage;
.NET adapters are not present.

As always, using this technology in scenario where it does not fit well, will turn it into one big disadvantage as well.

How to run CouchDB in the Cloud?

At the moment there is no public CouchDB hosting available, yet. One future option worth of mentioning is a Couch.IO that will offer 10GB size databases for 30 USD per month.

However, let's see how we can get us a virtual CouchDB server at a fraction of this cost.

We'll use Rackspace cloud Virtual Machine slice in this scenario. You'll find the sequence for setting up a temporary development machine below.

Important: this configuration is not designed for production scenarios. You would at least need to adjust the firewall settings, change the ports, assign proper users and configure couchdb for auto-start.

Create new virtual cloud server:
- Properties: 256 MB - $0.015 per hour - 10 GB
- OS: Ubuntu 9.04 (jauntu)

Wait till confirmation email comes (a couple of minutes) and launch Putty (or any other SSH client) using the IP address from your confirmation email. Security warning will show up - Just hit "No" to accept the key temporarily. Username and password are provided in the email.

Tip: right-click with mouse acts as "paste" in Putty. This works even when system expects password input.

After a successful logon you should see something like:

Update everything to the latest by typing following command and hitting enter:

apt-get update
Install everything that is required for running latest CouchDB:

aptitude -y install build-essential
apt-get -y install libmozjs-dev libicu-dev libcurl4-openssl-dev erlang
Get the latest release and compile it:

wget http://mirrors.24-7-solutions.net/pub/apache/couchdb/0.9.0/apache-couchdb-0.9.0.tar.gz
tar zxvf apache-couchdb-0.9.0.tar.gz && cd apache-couchdb-0.9.0
./configure && make && sudo make install

NB: You can install CouchDB 0.8 by simply executing apt-get couchdb. But it will miss some of the 0.9 features.
Allow the instance to listen to public IP:

nano /usr/local/etc/couchdb/local.ini

In this window you will need to change httpd section to look like the snippet below and save:
```
[httpd]
port = 5984
bind_address = 0.0.0.0
```
Start CouchDB by typing:

couchdb -b
Verify in the command prompt that the instance is running locally (it should return "hi" from CouchDB):

curl http://localhost:5984
Verify from your local machine that CouchDB is accessible and ready by opening this Url in your browser :

http://[IP of CouchDB Server here]:5984/_utils/

How to install CouchDB on Windows?

Working with a real and cheap server in the cloud might be interesting. Yet it could be impractical in certain scenarios (i.e.: when working in offline mode or testing complex deployment scenarios). For that we need to learn how to install CouchDB on a local machine.

Although it is possible to run CouchDB directly on Windows, the solution is too bulky and unreliable. We can simply use Ubuntu virtual machine instead, sticking to the routine above.

Unfortunately Windows 7 Virtual PC is extremely limited (its primary goal is to support legacy applications by virtualizing Windows XP). So in order to create a new VM we'll use Sun VirtualBox

The process looks like this:

Download latest Ubuntu server distribution.
Create new VM instance for Ubuntu, named "Ubuntu".
Set up NAT networking for this instance and create sufficiently large HD (2 GBs should be enough).

Mount downloaded Ubuntu ISO into the virtual DVD and go through the bare server installation.
Logon into the instance and install SSH server:

sudo apt-get install openssh-server
Shut down the instance and configure virtual SSH forwarding so that we could use Putty to talk to the OS locally (we are assuming that the VM instance name is 'Ubuntu'). These commands have to be executed from the VirtualBox directory. Note, that there are no line breaks between "Ubuntu" and "VBoxInternal..." (I've introduced them for readability).
```
VBoxManage setextradata "Ubuntu" 
"VBoxInternal/Devices/pcnet/0/LUN#0/Config/guestssh/Protocol" TCP
VBoxManage setextradata "Ubuntu"
"VBoxInternal/Devices/pcnet/0/LUN#0/Config/guestssh/GuestPort" 22
VBoxManage setextradata "Ubuntu" 
"VBoxInternal/Devices/pcnet/0/LUN#0/Config/guestssh/HostPort" 2222
```
This binding will redirect all calls from localhost:2222 to the virtual Ubuntu:22.

Configure CouchDB forwarding, so that we could talk to the database, as if it were installed on the localhost:

VBoxManage setextradata "Ubuntu"
"VBoxInternal/Devices/pcnet/0/LUN#0/Config/guestcdb/Protocol" TCP
VBoxManage setextradata "Ubuntu"
"VBoxInternal/Devices/pcnet/0/LUN#0/Config/guestcdb/GuestPort" 5984 
VBoxManage setextradata "Ubuntu"
"VBoxInternal/Devices/pcnet/0/LUN#0/Config/guestcdb/HostPort" 5984

Create short-cut to the GUI-less interface of VirtualBox. Launching it will start our server-in-a-box:

VBoxHeadless.exe --startvm Ubuntu --vrdp=off
Connect Putty to localhost:2222 (port number we've configured for SSH forwarding) and perform the install routine from the previous section, starting from:

apt-get update
Now you should be able to talk to your CouchDB (don't forget to start it) using the following address:

http://localhost:5984/

You can open Futon interface in the browser with:

http://localhost:5984/_utils/

That should be it. Now we have a local CouchDB server that could be used as a development sandbox. We could also replicate this deployment in any cloud environment that supports Infrastructure as a Service scenario with VMs.

This database server does not have a predefined schema, is easy and cheap to deploy (once you know the drill) and has been designed with the scalability in mind. All this makes Couch DB an interesting technology that might fit well in xLim set of design principles.

In the next article we'll talk about using .NET to communicate with CouchDB in a strongly-typed manner. You can subscribe to this journal to stay tuned for any updates.

Related links: