Rinat Abdullin on Efficient Software Development

Make Code Explicit and Stupid

Rinat Abdullin — Mon, 06 Feb 2012 11:16:18 +0000

There is one thing that surprises me in CQRS-related development.

We, as developers, go at great lengths, explaining how event-centric architectures are so better than CRUD-based ones: you don't need to reverse engineer behaviors from state, since you express them explicitly (as events) and can project to any structural form.

At Lokad we leverage this heavily for web UIs, that tend to be slightly more intuitive because of that.

However, when things get to the code, we suddenly change our minds to the opposite and vastly favor implicit conventions and smart heuristics to discover and wire components in an application.

Maybe this is because, we are so good with solving complex problems, that we see them in every single problem?

(this is only a part of the picture, component instantiation is in another class).

We say, that such approach is smart, generic and reduces friction (when such approach is pushed to extreme, it is called Inversion of Control Container). That it generally is a better alternative than this:

Seriously, is it?

An important remark: Both pieces of code were written by me. At the moment of writing of each one, I was extremely proud of the approach being used :)

People Don't Think in Tables

Rinat Abdullin — Sun, 05 Feb 2012 12:14:26 +0000

Within the last few years I've seen major change in UI approaches in the systems I build. The change is caused by a shift from relational databases as persistence to something that does not require an SQL server (or DB server at all).

Take a look at this old UI of mine. It's pretty powerful (huge amount of buttons proves that), is intuitive for geeks and scary for normal people.

What's more curios, this UI is shaped by the relational persistence model that was created in mainframe days, when storage was expensive, memory even more so, and data had to be heavily normalized. However, technology has advanced slightly since then, and we no longer need to fit our UIs to something that is so spreadsheet-like. We can prevent user from overexposure to underlying data complexity, especially when data itself is simple, but just happens to be stored in a computer-optimized way.

Think of it again. We turn UI experience of users in sudoku game just because we know how to store data in a way that was good 20 years ago. No wonder Apple is making so much money - at least they don't try to turn every UI into spaceship control panel:

People don't think with tables. They would prefer something that is more simple, even if it does not benefit from a fancy styled UI framework. Something that does not involve solving sudoku puzzle every time you need to find a simple answer to your question or do your job.

or where we have the most simple way to search complex structured info, as patented and proven to be useful by google:

It is surprising, how underlying technologies affect what we build with them, is not it?

As you can guess, first 2 screenshots depict UI that runs on some sort of SQL, while the other UIs were influenced by a simple NoSQL storage model (to be precise: CQRS/DDD+ES model that uses for UI persistence blobs for cloud persistence and plain files for on-premises deployments).

BTW, I know that quite a lot of people are using Lucene to provide full text search capabilities for their read models. However in the simplest case (esp. with cloud deployments), it's easier to write a few lines of code as opposed to dealing with one more dependency.

Another side effect is that with the change of underlying tech from SQL (and more importantly - thinking model), there is less need to resort to data-aware UI frameworks (mostly used on conjunction with desktop apps) in order to provide rich user experience. This allows to have web-based applications that don't force framework installations (good luck with getting .NET or Java on all machines in some large organization with a lot of momentum) or complex upgrade routines (i.e. ClickOnce auto-update or automatic MSI updates).

I'm not talking about development simplicity and cloud-scalability - these come for free, and we don't really value things unless we struggle for them.

Obviously, as you can see from screenshots above, my UI skills are still below that of a kid with a box of crayons but lately Bootstrup and jquery have provided a really nice experience, especially, when you can reuse some great ideas and frameworks.

Wait, till we start getting into mobile experience with HTML5. It actually comes almost for free as well, since latest web layout frameworks provide a lot of mobile experience out-of-box.

NB: In no way I'm implying that this second approach is anywhere near close to being reasonably simple. And it's obviously bad looking (especially UIs that I hack together), but I think this is going in the right direction. At least, less time is wasted on writing UI docs and providing support.

Architectural Principle - Focus on Simplicity

Rinat Abdullin — Thu, 02 Feb 2012 08:10:50 +0000

A few hours ago I've posted a tweet that gathered unexpected response:

Popular things that I consciously avoid: IoC Containers, Mocking, ORM, SQL, NuGet, WCF, WWF, WPF, NServiceBus, DTC and TFS.

I did expect any kind of response. And I'd understand that perfectly, since I myself used to adore ORM, play with mocking, abuse IoC badly and do all sorts of other bad things.

However the only question people got was: "What's the problem with NuGet"? Well, answering it requires going a few steps back.

Keep in mind, that I'll be pushing my point of view slightly to the extreme, just to make the point more clear.

Every new junior developer (growing team, or a product) starts with the simple approach: we start writing software and adding features till we deliver (sometimes in waterfall approach quite a lot of time could be invested before this happens). Then we grow from there. As software and developers mature, they face new challenges and problems. Luckily we get new tools, skills and ideas to help us growing. New team members and frameworks join in to help everybody move forward and deliver more. Things that hurt or are uncool are thrown away (even though sometimes it might take some effort).

Sounds like fun.

However, somewhere in my career path, I've been lucky to get on a rocky path. We never had real ability to just add more tools, people or features. I never worked on a project that had more than 3 people involved in it full-time at once. All these projects were stressed in time (no man-years till the next release) and functionality (rapidly growing companies need to evolve and scale fast).

Sounds like a minefield and completely unrewarding environment, does not it? Every wrong decision shows itself really fast. Every step that is just good (and not outstanding) also shows (since you waste precious time that could've been invested more efficiently). So my development career was governed by pain: wrong decisions and poor choices lead to long hours, late nights and unsustainable work efforts. Not, that I don't make frequent mistakes any more, but these ones were really bad.

And that's actually good: harder life hits you, faster you learn. Bad ideas and tools don't get a lot of chances to stick around in such poor environment. Semi-useful things don't get to stick either. The core principle that still remains and always works is disciplined simplicity and focus. Among other things this means:

If I get a chance to remove something or break things down with a small effort, I will take that effort.
If I get a chance to add something with a small effort, I will fight it.
I prefer to refactor something first and then maybe add complexity as opposed to adding complexity and then maybe refactoring.
If I have a chance to defer solving the problem (or even avoid it altogether), I will try that.

Obviously this is the idealistic scenario, that does not factor business values and the rest of the real-world. However, these are just strengthened even more, when you start consciously measuring costs and try to avoid problems.

This often comes to things that other would see as extremes: dropping project references, avoiding to use new technology or framework, removing files, taking preventable risks and even turning down features. Some call this pain-driven development - you keep only things that are too painful to live without, because you can't afford keeping the rest; you focus only on problems that are the most painful for the business (and are measured in real money), because you don't have enough people and hours to deal with the rest.

This is like constant tending your garden or curating an exhibition: you continuously look look for bad combinations or things that are just good (and not great). You think twice before adding something new (Dependencies also count). Because each thing means:

One more thing for a new developer to learn.
One more thing that have a chance of going wrong in production.
One more thing to keep in mind while developing.
One more logical dependency that has a chance of causing regression.

All this zealotic obsession with simplicity would not be needed if I had environment that is more rich: with more resources, time and people to help us. Would it?

If we push the idea to the extreme, consider two imaginary products. One could be designed by a rich team with a strong vision, smart architects and man-years dedicated for their goal. The other product could be developed mostly by a semi-organized group individualistic hackers, that scavenge a few hours, accept risks, while working on diverse bits, that are kept stupid and isolated.

How do you think, will these projects perform in real life? How close is your project to either of these extremes?

An example of rich project environment would be something like: TFS.
An example of poor project would be a collection of hacks commonly known as Git.

I'm not trying to compare these projects or judge them. This is up to you :)

BTW: the answer to the question of "What's wrong with NuGet, NSB etc" is simple: these projects might be good, but I can live without them in my projects (or learned how to live without them). They didn't prove essential (just like almost all utilities and open source libraries I've created and thrown out myself).

Constraints of such projects are:

A few developers (distributed around the world) and rapid releases.
Necessity to compete with multi-billion transnational companies.
Necessity to have complex business intelligence (think business analytics) OR extreme scaling (think big data processing with 200x scaling in matter of hours) OR flexible and rich UIs OR cross-cloud low-friction development OR low operating costs OR tolerance to process failures and human errors.

More often than not, such projects require ALL of that at once (that's how we roll at Lokad). And guess who gets to babysit these projects and spend weekends fixing things that go wrong?

Such tight constraints led to the architectural principles and focus on simplicity that I've outlined in this article.

Tools and frameworks that we still stick to - are getting all the passion, gratitude and even free time. They are worth it, since they helped to work through the problems that we were unable to solve otherwise. However, among all these precious things, the most valuable tool and architectural principle is keep things simple, focus on that.

PS: You can continue reading on this topic in a fine article by Rob Ashton: Your container is not wanted here.

Handling Big Data in Cloud

Rinat Abdullin — Sun, 29 Jan 2012 08:19:29 +0000

Big Data is one of the new hype terms that is gradually gaining popularity. No big surprise - amount of data around us is gradually growing and by properly mining it we can get competitive advantage and eventually make more money. Money attracts money.

Usually by term big data we mean a collection of datasets that are so large, that they become too hard to be processed on a single machine. This data can even be so big, that it takes hours or days to process it in a data warehouse.

Fortunately, cloud computing comes to the rescue. It provides following resources:

nearly unlimited scalable storage capacity for data (i.e. Azure Blob storage or Amazon S3)
elastic compute capacity to process this data (i.e. Azure Worker Roles or Amazon EC)
network capacities and services to transfer data and coordinate processing (queues and actual networks).

These resources are elastic (you can get as much as you need) and paid-on-demand. Latter is really important, since you pay only for what you use.

I'm assuming here, that in your business model, more data processed means more money made.

There are three distinct approaches in handling big data, based on the specific challenges.

Batch Processing

First one is about batch processing, where you do not need extremely fast computation results but have terabytes and petabytes of data. This is essentially about implementing MapReduce functionality in your system, often resorting to hacky but extremely practical tricks. Below I'll talk about some of the lessons learned in this direction in Lokad (we provide forecasts as a service without any hard limit on amount of data to process).

Obviously, storage of this data becomes a primary concern with this approach. Fortunately starting companies can leverage already existing cloud computing resources, starting from Google's GFS and up to Azure Store. Or if you are big enough (and this is worth it), you can roll a data center of your own. Idem for computing and network capacities.

Stream Processing

Second approach deals with cases, where you need high-throughput and consistently low latency on gigabytes of data. High-frequency trading is one of the most known domains here (various real-world telemetry systems being the second one). Abusing event streams and ring buffers (like in LMAX) along with partitioning - helps to deal with the challenge.

Due to latency requirements, cloud computing might be not the best choice for latency-sensitive solutions. It is more efficient to roll out fine-tuned infrastructure of your own including specialized hardware and soft (things like InfiniBand and ASIC or even Mixed-signal chips).

However if you are OK with a bit of latency and are more interested in high-throughput continuous computation, then the flexible network and CPU resources provided by cloud are a good match.

Realtime Batch Processing

Third approach to Big Data involves dealing with more complex requirement - providing real-time capabilities over vast amounts of data (so we have to be both fast and capable of handling petabytes of data). Twitter is a vivid example here - it needs to provide (almost) real-time analysis over billions of tweets around the world (although fortunately it does not to go at microsecond level).

This challenge can be solved by actually mixing first two approaches: we would use slower batch-processing for dealing with actual bulk of data (tricks like preprocessing, append-only storage and MapReduce work here), while latest data will be handled through the real-time streams (stream processing and continuous computation). Results are merged as they come out of these two data pipes. Over the course of time (i.e. daily) latest data is pushed from "hot" zone into the bulk data.

More often than not, real-time data processing is done by simplified approximating algorithms that deal with subset of data. Batch processing is more through and precise (at the cost of being slower). By pushing real-time data back to the bulk data, results of the computations are actually corrected, as they are recomputed by batch algorithms. Twitter's Storm project features nice overview of this approach.

New Videos on CQRS/DDD and Practical CQRS

Rinat Abdullin — Fri, 20 Jan 2012 13:04:53 +0000

Thanks to awesome guys from Austrian Alt.NET (and particularly Jörg Egretzberger) there has been an opportunity for me to speak with Greg Young at Professional.NET in Vienna on September 2011. And here are the videos that were made there.

It's recommended to watch them in the provided order: first Greg's "CQRS and DDD" and then my "Practical CQRS"

Please, keep in mind, that this event took place more than 4 months from now; it also was the first talk I've done together with Greg (meaning that I became slightly less stupid and naive since then, especially during the CQRS RoadTrip 2011; at least that's what I hope).

CQRS and Domain Driven Design

by Greg Young. Direct link: http://youtu.be/KXqrBySgX-s

Practical CQRS in Cloud

by Rinat Abdullin. Direct Link: http://youtu.be/CnvO_nlvrps | PDF

I hope you enjoy them. Once again, kudos to terrific guys from Austrian Alt.NET community for making these recordings possible.

Run Lokad.CQRS code natively on Windows Azure

Rinat Abdullin — Sat, 14 Jan 2012 14:09:08 +0000

As it seems, the most common use of Lokad.CQRS Sample these days is for building CQRS-based applications for Intranet (and migrating legacy projects). I didn't see that coming, but glad that the latest "lean" version makes some sense to people (more about it).

However, one of the initial purposes of this Sample is to enable and demonstrate cross-cloud portability of projects developed using abstractions from Lokad.Cqrs.Portable. That's how we use it at Lokad.

So to highlight this (and to help folks from Microsoft CQRS Advisory Board that showed interest in trying Lokad.CQRS with Azure), I've added two new projects: Sample.Azure.Wires and Sample.Azure.Deploy. There is nothing really special, first project just wires SampleProject to run on Windows Azure in a Worker Role, without touching existing domain code. Second project is the actual deployment.

As expected, trace output is similar to the one we get from running sample in on-premises mode (via Sample.Engine console).

Obviously, this worker is wired to use native components of Windows Azure, instead of file system. These components (developed in Lokad and production-tested over the last few years in various scenarios) include:

BlockBlobTapeStorage (used for append-only event streams on Azure)
AzureAtomicStorage (used for storing documents, saga state and persistent read models)
BlobStreamingStorage (used for streaming large BLOBs for Big DataProcessing)
Azure Queues (for your message sending needs)

If you have latest Azure SDK installed, just open Lokad.CQRS.Sample.withAzure.sln to see these additional Azure projects and get a taste of event sourcing on the cloud (actually tastes exactly like any other ES, but feels cooler).

Maintenance Tip: By the way, you can download domain log file (sample-tapes/domain.tmd) from the Azure Storage to your disk and open it with Audit tool. This helps to visualize and investigate processes that happen in your CQRS system on the cloud (actually saved my day more than once, when production systems hit unexpected problems). Likewise you can also download and investigate event streams of any single aggregate.

Actually, this Audit tool was also designed to open event streams directly from Azure (and allow sending selected messages back or rebuilding all projections), however this functionality is not ported to open source yet. If somebody finds it useful, I'll try to find time and export it (there is nothing really fancy there, though - just using append-only nature of event streams for efficient data replication from remote cloud to local machine).

Maintenance Tip: Those, that plan to use this in a production, might want to plug the quarantine into the message handlers. For instance, here is a sample that records every handler failure and send a detailed report (with stack traces and message information) to email as soon as message is quarantined. The latter helps to deal with cloud problems really fast.

The next thing I want to include in this Sample Project is a lean Web UI, using ASP.NET MVC 3 Razor Views with jQuery and Bootstrap.css. Just to demonstrate how easy it is to add browser-friendly and fast Web UI to existing CQRS systems (and have it hosted on-premises or in the cloud, of course). What do you think?

Meanwhile, have fun, join the community of people abusing the project and be sure to fill in CQRS Survey put out by P&P Team

Announcing Lokad.CQRS vLean - SampleProject and Tools

Rinat Abdullin — Thu, 05 Jan 2012 17:49:25 +0000

I've just finished assembling pre-release of Lokad.CQRS vLean (lean branch in github repository). From of this version Lokad.CQRS is no longer a framework, it is a full sample.

As such, Lokad.CQRS could be used for learning about event centric development (CQRS/DDD with event sourcing and ability to deploy in cloud and on-premises). Gradually growing series of articles in my bliki could provide some theoretical background.

You can also use Lokad.CQRS to jumpstart new projects by copying the entire SampleProject to your system and then modifying it to your needs. That's essentially how we start new projects at Lokad these days.

Because of this transition from "Framework" to "Sample with Sources" (aka "Reference Implementation"), I was finally able to include a lot of new material into this branch.

Sample Project

I've ripped part of domain code (simple account+users setup) from one of our projects, along with all the accompanying tooling, tests and engine. Current approach to event sourcing is also included (as explained in the bliki)

The project currently does not include:

Deployment runners for Azure.Cloud (planned later, however Lokad.CQRS users should have pretty good sense of reconfiguring the domain to run in Azure).
Web UI (but I might add it later, if time permits, since Razor + Bootstrap make really pleasant and fast development with this architecture).

I'll try to add these later (community contributions are also awesome).

Absence of Web UI makes it harder to visualize what's going inside. However you can still run tests and also start Sample.Engine (configured with file-based persistence and queues), that will fire a few commands, which will produce events that will be wired to sagas and projections. Just like in the real world.

Plus, this project includes some sample code and tools that are wired to it. These are used exactly how we currently use them (although not necessary how we will use them a few months later). Read below for more info.

Contracts DSL

Lokad Contracts DSL is an optional console utility that you can run in the background. It tracks changes to files with special compact syntax (as mentioned in commands and events on bliki) and updates CS file. Changes are immediate upon saving file (and ReSharper immediately picks them). This is an improved version of Lokad Code DSL, it supports identities and can auto-generate interfaces for aggregates and aggregate state classes.

You can try this out by starting SampleProject\Tools\Dsl project and then doing some changes to SampleProject\Domain\Sample.Contracts\Messages.tt.

Current DSL code generates contracts classes that are compatible with DataContracts, ServiceStack.JSON and ProtoBuf. Here are how they look like:

[DataContract(Namespace = "Sample")]
public partial class AddSecurityPassword : ICommand<SecurityId>
{
    [DataMember(Order = 1)] public SecurityId Id { get; private set; }
    [DataMember(Order = 2)] public string DisplayName { get; private set; }
    [DataMember(Order = 3)] public string Login { get; private set; }
    [DataMember(Order = 4)] public string Password { get; private set; }

    AddSecurityPassword () {}
    public AddSecurityPassword (SecurityId id, string displayName, string login, string password)
    {
        Id = id;
        DisplayName = displayName;
        Login = login;
        Password = password;
    }
}

Audit Tool

After you have run Sample.Engine, try starting Audit and opening with it this file temp\sample-tapes\domain.tmd (located in folder where the engine run). You should see something like this:

You'll probably figure out the purpose of color coding (or you could look out in the sources).

BTW, here's how the other tab looks like - it is used for automatically detecting and wiring all projections and then running the selected event stream through them:

Please keep in mind, that this is a fast rip from internal code, so a few buttons and functionality might not make full sense in the file-based context (i.e. domain log synchronization).

Simple Testing

I've been mentioning heavy abuse of Greg's SimpleTesting for dealing with specifications to test Aggregates. Source code for that is included (and for printing out them in readable format). Just fire your NUnit (either directly or via ReSharper):

This setup currently supports usual AR+ES testing, plus testing exceptions thrown by the domain code (domain errors), plus dealing with services and time.

Core Changes

There were a few core changes in the actual Lokad.CQRS framework. They were caused by the need to move forward and support wider range of scenarios (and better scalability). Actually, these features were mostly related to throwing things out - all builder syntax and containers. In the current version you wire everything without using overcomplicated builders. This leads to fewer dependencies and simpler code.

If you don't know, how old builder code translates to the new one - just check Wires and Engine projects in SampleProject (in addition to Snippets and unit tests for core building blocks).

In the spirit, we are gradually drifting away from style of architectures that is similar to NServiceBus, MassTransit and RhinoQueues (where you have handler classes plugged into the container for you). Instead, we are going directly for the type of architectures described by the outstanding ZMQ guide.

What's Next

As always, your feedback is always welcome (it helped to shape the current release and let us move forward). Please share it (along with any questions you might have) in Lokad community

A lot more is planned in this world along the following directions:

Simplifying the core even further, along with solidifying the backing theory, based on real-world experience.
Better support of multiple clouds (just Azure is not enough)
Realt-time performance scenarios.

Stay tuned and participate in the community, if you are interested!

Using Git Revision in Windows Azure Cloud Deployments

Rinat Abdullin — Fri, 30 Dec 2011 11:19:14 +0000

Here's a quick approach for including git versions of your codebase into Windows Azure Deployment labels (as visible in the Portal). This applies to Azure SDK 1.6 and probably higher. Versions are appended automatically whenever you trigger a publish from Visual Studio.

git versions usually are reported by git describe and could look like r40 (if there is r40 tag on the last commit) or r39-31-g48ac07c (if last tag r39 was 31 commits ago).

It works like this.

Step 1: Create new Azure Deployment profile for your deployment project. It will be saved somewhere to Profiles\ProjectName.azurePubxml.

Step 2: Save project and then edit deployment project file to include this profile file only if it exists

<ItemGroup>
  <PublishProfile Include="Profiles\ProjectName.azurePubxml"  
     Condition=" Exists('Profiles\ProjectName.azurePubxml') " />
</ItemGroup>

Step 3: Drop a .gitignore file into the Profiles folder so that changes will not be tracked. Contents will be straightforward:

ProjectName.azurePubxml

Step 4: Add PreBuildEvent to the Azure deployment project by editing it further. This script should:

Fill <AzureDeploymentLabel> XML node in azurePubxml with a content of your choice (optionally grabbing output of git describe).
Fail gracefully if there is no git available.
Fail gracefully if there is no azurePubxml file.

In other words, we just take output of git describe command and put it inside the match of:

new Regex("<AzureDeploymentLabel>.*</AzureDeploymentLabel>", RegexOptions.Multiline);

MSBuild PreBuildEvent script can be wired to our process via editing of .ccproj file (or you can just fill that via Visual Studio UI - Deployment | Properties | Build Events):

<PropertyGroup>
  <PreBuildEvent>"PathToReplacementScript" "$(ProjectDir)\Profiles\ProjectName.azurePubxml"</PreBuildEvent>
</PropertyGroup>

For the actual replacement script you can use scripting environment of your choice or even C#.

By the way, there are two interesting twists upon this concept.

Twist 1: You can leverage git's super powers to print all changes in the code between any two deployment labels. They will be nicely formatted by committer, too.

git log r39..r40 --pretty=short | git shortlog

That's how we publish releases at Lokad.

Twist 2: You can set your system to publish InstanceStartedEvent with the following schema:

InstanceStarted!(string codeVersion, string role, string instance)

The latter can be used to link your code version trees directly with the event streams and life-cycle of a distributed system deployed to the cloud. This immensely simplifies post-mortem debugging of such systems along with adding a few maintenance tricks to your sleeve.

Example of Self-documenting Unit Test with Event Sourcing

Rinat Abdullin — Tue, 27 Dec 2011 15:13:37 +0000

One of the biggest advantages of event sourcing approach is it's inherent capability to turn unit tests into a living documentation. Below is an example of specification that I've worked my way through today (that's how NUnit prints it out).

registrations: duplicate email fails - Passed

Environment: 
  index includes email("contact@lokad.com")

When: Register 'd6e64e':
        Customer Name: Lokad
        Contact Email: contact@lokad.com
        Date:          2011-12-27

Expectations:
  [Passed] Registration 'd6e64e':
             Customer Name: Lokad
             Contact Email: contact@lokad.com
             Date:          2011-12-27
  [Passed] Registration 'd6e64e' failed:
             Email 'contact@lokad.com' is already taken.

And here's how actual NUnit code looks like:

public Specification duplicate_email_fails()
{
    var info = new RegistrationInfoBuilder("contact@lokad.com", "Lokad").Build();
    var index = new MockUniquenessService();

    return new RegistrationSpec(index)
        {
            Before = {() => index.includes_email("contact@lokad.com")},
            When = new CreateRegistration(reg, info),
            Expect =
                {
                    new RegistrationCreated(reg, info),
                    new RegistrationFailed(reg, new[]
                        {
                            "Email 'contact@lokad.com' is already taken."
                        })
                },
            Finally = index.Clear
        };
}

All was achieved without any special magic or even fancy tools. I've just pulled over sources of SimpleTesting and CompareObjects for additional readability.

For those who are interested in RegistrationSpec class, it is just a simple snippet wiring together dependencies of aggregate root to a strongly-typed specification deriving from TypedSpecification in SimpleTesting:

public sealed class RegistrationSpec : AggregateSpecification<RegistrationId>
{
    public RegistrationSpec(IRegistrationUniquenessService service)
    {
        Factory = (events, observer) =>
            {
                var state = new RegistrationAggregateState(events);
                return new RegistrationAggregate(state, observer, service, 
                    new TestPassword(), 
                    new TestIdentity());
            };
    }
}

Explicit strong-typing of aggregates (as described in bliki) works all the way back in unit test specification by allowing to benefit from compiler-time checking and IntelliSense support. In other words: you don't need to navigate through hundreds of messages to figure out which ones are actually applicable in test.

Migrating Legacy Systems to Event Sourcing

Rinat Abdullin — Sat, 24 Dec 2011 10:59:23 +0000

These days I'm working on migrating really legacy system towards the simplified CQRS/DDD design with event sourcing for the cloud.

As part of the migration process, I'm reverse engineering legacy SQL database into a stream of events. These events are not precise representation of what has happened in the past (this exact information is irreversibly lost, as in almost any data-driven system), but rather a pretty good estimate that could be used to prepopulate the new version.

Essentially, reverse engineering events is about writing a throw-away utility that will scan database tables (MS Access files or punch-cards) and spit out events that could be used to reproduce that state.

For instance, consider this customer record in DB table:

Customer {
  Name : "GoDaddy",
  Id : SomeGuid,
  Created : 2008-13-12,
  Status : Deleted,
  Phone : "111-22-22",
  Reason : "Supporting SOPA was poor PR move"
}

This record could be reversed into the following events

CustomerCreated!(
  Id: SomeGuid, 
  Name: "GoDaddy", 
  Created: 2008-13-12
)
CustomerPhoneSpecifid!(
  Id: SomeGuid, 
  Phone: "111-22-22"
)
CustomerDeleted!(
  Id: SomeGuid, 
  Reason: "Supporting SOPA was poor PR move", 
  Deleted: 2011-12-24
)

Note, that we actually had to improvise while coming up with this event stream: date of deletion was not stored in the original database (we were losing this information). So we are just substituting some predefined date here (i.e. date of upgrade to CQRS/DDD+ES).

When you have a system with a few years of history, quite a few events are generated. The system that I'm currently migrating has data that dates back to the early dates of Lokad, hence 300-400 thousand events is something expected.

As part of development process, these events are run through the aggregate state objects and also through the projections. The goal here is to pass all possible sanity checks and get read models that match exactly to the UI currently visible in the old system. If new system looks and behaves exactly like the old one (even if the guts are completely simplified), then we are moving in the right direction.

Obviously, during this process, a lot of problems show up, especially with logically inconsistent or corrupt data (i.e. accounting inconsistencies caused by race conditions and dead locks in the legacy database). These things are generally to be resolved manually - there is no magical silver bullet.