HKMP Technologies came to us with a tall order– upgrading its market-leading paperless accounting software on a tight deadline in order to secure a strategic customer deal. To make matters more complicated, a new release of Microsoft Silverlight – the technology platform for the product – was scheduled right in the middle of the development cycle.

Our team here at Nagarro approached the problem creatively, allowing HKMP to successfully seal a critical new deal and improve its eco-footprint. For more details, please check out our new HKMP Technologies case study and customer testimonial video.

Though this is not a very common scenario in many of the modern day applications out there but this definitely is a hallmark of databases handling loads from specific industries like banking. Applications intended for these industries normally have huge volume of small database transactions. In this article, we present one of the approaches you can take to handle a scenario such as this.

Introducing Broker Hub

To demonstrate the problem which has these type of database requirements, let’s use the example of a Broker Hub – a stock broking hub. Stock broking applications have very high volume of small database transactions and also there are spurts in database activity depending on market conditions. For simplicity sake, let’s assume that we were at a point when database design and usage pattern for Broker hub database was in the most optimal state.

First choice – Scale up

So to optimize Broker Hub further, we had a number of ideas and first choice was obviously to increase the hardware capacity. Increasing the hardware capacity did help the case. We could handle upwards of 50000 transactions a second by moving to a better system with 16 processors and a SAN array of high speed disks. But above 50000 in our load environments, we could still see the database to be the bottleneck.

Next obvious idea was to try SQL Server 2008 which has support for performance optimization features like advanced compression (reducing the overall disk IO) and support for virtually unlimited number of objects (2,147,483,647) and database size (524,272 TB). Again we could see the difference. Without enabling features like compression we could achieve a bigger number of around 70000 transactions per second.

Would it work? Probably not long term.

Probably enabling compression and using other features to optimize performance would have resulted in a higher figure but the problem here was that there was a limit to this. Adding hardware or moving to newer version of a database (or even to a different database) wouldn’t have given us the virtually unlimited (100 times 30000 transactions per minute) capacity we were looking at. Obviously we needed some way to deploy more than one server to split the load and thus increase the capacity to handle very high transaction loads. In simpler terms we needed a scale out solution instead of scale up for our database. Looking around for out of the box solutions in the market didn’t help, simply because there aren’t many scale-out solutions available in the market to load balance SQL server. Oracle has launched such a solution but even that requires syncing between different servers in cluster which takes up a lot of network bandwidth thus reducing the overall effectiveness of the solution.

Scale out – Approach

After a lot of brainstorming sessions, it was decided that it was time to create our own scale out solution. The idea was to create a design which could help us scale out as our user base grew but at the same time being able to handle sudden increases in transaction volumes. But as it happens all the time, we didn’t have too much money to be spent on this. We looked at various existing products such as SharePoint to see how they stored their data and came up with a very simple first draft of the scale out solution. It looked very similar to way SharePoint does load balancing for its data stores. We had a cluster of database servers connected to the Broker Hub. Each database was configured to handle a set of users with specific user Ids and thus held data only for those users. Merge replication was used to replicate data from all databases to a central database which was used for all reporting. Here’s how it looked like:

Broker Hub front end had to be modified a bit in terms of process flow. In this case, each user logging into Broker Hub had to be connected to a specific database based on his user id. The task of identifying which database has the information related to the user trying to log in was handled by a new module added to the application called DB Load balancer. The process flow in this approach looked like this:

• User foo with User ID (n) logs in.
• Application calls Custom DB load balancer to find out that all data specific to user foo is on database server DB 2.
• Application creates a connection to DB 2 on behalf of this user.
• All user transactions from User 1 are directed to DB 2 thereafter.
• Periodically user data is synchronized to the master database.

Design Decisions for this approach

This approach was pretty simple to implement and could achieve the results we were looking for. But there were a few practical issues:

• We have to introduce concept of ID buckets. So every transaction table on each server was assigned unique bucket from which it could allocate Ids. With this we overcame the problem of how to maintain unique IDs.
• All the masters would be replicated using transaction with update option. So a change on one gets replicated all over.
• Adding new users was not as simple as before. Every database was configured for a specific set of users and this set increased sequentially. This meant that we always had to add the new user in the last server available and if last server was full then we had to deploy a new database even for a single user (and probably even before the last server was at its full capacity). This was accepted as an acceptable fact as this would happen once a while.
• Any admin report had to combine the data from all the servers to be useful which meant an additional job to aggregate all the data. This was accepted as a design reality.

Conclusion

So we could see very clearly that we were achieving the desired performance figures for this approach. Our aim was to find out how best we could achieve load balancing capabilities for the specific database load our application was creating and that was achieved. It actually became a life saver for us in more than one situation.

The problem of making such general rules about programming practices is that you tend to miss out the benefits by following these rules too literally in every context. Think it on the lines – Reflection or any feature for that matter, if included must have a valid reason for its presence. We should be able to see and know how and when to use it and when specifically to avoid it.

Reflection is slow; but the question is how slow is slow? One should worry about performance but this worry should not keep you away from making best use of a technology. If you look around in .NET world, you will realize that .NET framework itself uses a lot of reflection. Ability to store meta-data about your code and to be able to use it at runtime opens the door to a whole lot of possibilities limited only by your imagination.

If you rethink it it’s just one another way of storing and working with information; difference being that this information is stored in code itself. So instead of retrieving information from some XML file or database, you are pulling it in from the code itself. Therefore your concerns towards writing optimized code while reading data from XML (e.g., load once read many times) or from database (e.g., making fewer trips to database), etc. will not be much different when using reflection. You might want to fetch type info once and hold it for future calls as well instead of fetching it every time you need it.

As in case of database, we know where to use joins, on which columns to create indexes and which where clauses to use to get optimized execution time; in case of reflection too there is a well defined grammar to make optimized reflective calls. For example, using BindingFlags enumeration correctly, not trying to use IgnoreCase flag, and limiting your searches to public or static or instance members as required, will definitely give you best possible results. Joel Pobar’s post talks at length about how reflection works internally and is a good reference to understand how to avoid performance pitfalls.

Don’t just assume that reflection will hit the performance, but use it, test it, tweak it for performance optimization and enjoy the flexibility it provides. If you think your problem statement has a solution in reflection; don’t hesitate in going ahead just because someone says that it will be slow. If this turns out to be slow; there are ways to optimize it too. For instance Jon Skeet’s post talks about using Delegate.CreateDelegate to turn a MethodInfo into a strongly-typed delegate. This can improve performance massively in scenarios where you need to set the same properties multiple times. Essentially this means that a lot of the type checking is done once when you created the delegate, rather than on every invocation.

Let’s explore a scenario where reflection is no harm to use and probably the only way you can achieve the end result with.

You want dynamic extensibility to your desktop application; your revenue model includes selling various groups of features separately. You do not want to build multiple deployment packages and you want to be able to provide even more features to your customers after they have started using your application. How would you do it?

You might do this by developing a plug-in model in your application.

You may define an IMyAppPlugin interface which can be implemented in each of your application’s plug-in assembly implementing a specific feature. At minimum you may have following definition:

public interface IMyAppPlugin
{
    Guid Identifier { get; }
    string Name { get; }
    string Copyright { get; }
    bool Start(IHostApp hostApp);
    void Stop();
    bool Run();
}

You will not have any reference of these assemblies in your project at design time. Its only at runtime when you will try to discover these plug-in assemblies (using some mechanism such as looking in specific folder or registry, an XML file or anything else) and load it using reflection.

You will write code to find and create instance of the class that implements this IMyAppPlugin interface. Once you have this object you can initialize it by calling its Start() method passing your application’s reference as required. You may also write code to add access points (such as menu items) for this dynamic functionality to appear in your application’s user interface when this plug-in is successfully loaded.

A call to Run() method anytime after this will do what is desired.

So essentially what we are doing here is that we have loaded the assembly using reflection and we are calling methods dynamically again using reflection. If we just go by numbers, yes dynamically invoking a method is slow. Referring to Rick Strahl’s post we can say that invoking a method dynamically is about 3.5 to 4 times slower than directly calling of a method. But again the question needs to be asked – how slow is slow? Here we are talking about few milliseconds extra. Unless you are making calls in a loop for a large number of times, does it really matter in the larger perspective of a typical application? Most likely – No!

Remember .Net Framework and especially ASP.Net uses a lot of reflection internally to provide dynamic execution of code and controls.

Talking again about the above scenario, if not reflection, do we have some other relevant approach of dynamically extending the application? You may talk about using COM interfaces to achieve it – but reflection or no reflection – the essence of every approach is to dynamically resolve the address of the method to be called, which is the only major difference wrt static calls where this address is readily available.

This is just one scenario where reflection proves to be the most effective and simplified approach to work with. There are a lot of other scenarios where reflective access to properties, fields and methods provides enormous flexibility that simply wouldn’t be there otherwise.

With Dynamic Language Runtime (DLR) in .NET 4.0 we are heading towards a cleaner reflection. Eventually this may replace reflection how we see it today, due to its simplified code and caching advantages. In-fact as mentioned in this Jim hugunin’s post, it appears that DLR too internally makes good use of reflection.

In short, do not shy away from using reflection, just because you hear warnings about it. Always verify if these warnings apply to your project or scenario. Often they will not. As they say, “trust but verify!

There are many limitations to this approach:

• one has to deal with dummy messages arriving in the inbox, when all that you want to test is your email sending code.
• automating the testing of the email sending code will result in a large number of emails being generated and delivered to your inbox.
• code that is supposed to deliver large amounts of email will generate a lot of load on the Internet bandwidth (and yes that can be a problem).
• if you are testing on a copy of a production database, you might inadvertently end up sending test emails to real users.

There are other issues present as well. One solution to solving these problems is to use your own SMTP server to handle all these emails in a manner that it receives an email and doesn’t sent it forward. So, it is like a dummy SMTP server. But setting up an SMTP server is usually a hassle.

There are plenty of options out there which make your job much easier than you think it is. In this post, I am going to list some solutions which can be used (though there are many more). The basic premise behind this approach is to run a local SMTP server (either on the developer box, or the test server, or both, or somewhere on the LAN). This local SMTP server address is what is configured as the server address in your application which is being tested. Any mail sent to such a server is kept with the server and not forwarded, thus preventing all the above listed issues.

Following are some of the more popular out-of-the-box programs that can be used (click on the names to go to their websites):

• Papercut - this is a simple windows application which will run in your System Tray and simulate an SMTP server; it receives and stores any messages it receives. It allows you to view the messages. It sits in your system tray and alerts you whenever a new message arrives.
• Neptune – this is another application which will do a similar task. It is however a little more advanced. It has an admin API interface which allows you to run custom queries to analyze the messages that it processes. It doesn’t have a UI through which you can view the received message though – but sometimes all you need to check is if the messages are being delivered. Very useful in a bulk messaging scenario.
• smtp4dev – this is another option similar to Papercut, which again sits in the system tray and will let you know whenever any message arrives and will let you look at the message details.

There are many more out there which can be used to do similar things with some features here and there (for example, Neptune doesn’t generate all SMTP responses). Here’s a post by Phil Haack from long ago which is also interesting: A testing mail server for unit testing email functionality.

So, when you feel the need to test your email sending code the next time, it might make sense to have one of the tools (or something similar) available.

Recently, we had the opportunity to give our opinion on the latest avatar of the Ado.Net Entity Framework to Jeffrey Schwartz from Application Development Trends magazine. Jeff was kind enough to publish our opinions in an article on the Visual Studio Magazine (thank you Jeff). The article is titled “Will Entity Framework 2 appease early adopters of LINQ to SQL?“. We, as can be deduced, were early adopters of LINQ (to SQL and others).

If you would like to see that question answered, I urge you to go and read that article. In this article, I wanted to discuss a little bit more about how we look at Entity Framework v2.

When Microsoft first announced the Ado.Net Entity Framework, we were unsure about what to do with it. We had already started training our developers on LINQ to SQL, and some of our projects were already using the technology. We found it (L2S) to be a very good way to implement the ORM layer of an application (and that worked very well in most cases). Additionally, we weren’t sure how mature the framework was. Sure, it looked perfect for some of our larger projects, but somehow it wasn’t fitting the bill (and we were not alone). And compared to LINQ to SQL, it felt quite lackluster in features.

The new version of the Entity Framework looks much better. It almost feels as if Microsoft went out of their way to ensure that Ado.Net EFv2 can stand on its own when compared to LINQ to SQL. However, by this time, we are pretty invested in LINQ to SQL, so there is no question of moving any of our existing projects over. But, we are very keen to try out the EFv2 in newer projects and internal applications. Specifically, we will looking at integrating it the next version of our Nagarro .Net Application Framework (which is a base framework that we use for most of our projects).

“Are we there yet?” – I would say, almost. The latest version of the Entity Framework looks a much more complete product as compared to the earlier version. Some of the features from an implementation-as-a-tool-in-a-software-development-services-shop perspective are really great:

• POCO support - the support for persistence ignorant objects make a great deal of difference because we can continue to leverage the features of the Entity Framework while using your own custom data layer logic alongside it. This is beneficial because this means we can at least consider using the EF in existing applications where the data model and the layers are already defined.
• First class code generation – this will come as a relief to most developers. The code generation engine, T4, is much more versatile that what we used to have earlier. This means that each application development team can customize the code generation a little bit according to their specific needs.

And of course there are hosts of improvements in general and there are tons of articles covering those.

From Nagarro’s perspective, I think we are almost there.

A quick  review of event processing in Windows Application tells us that there are 3 different types:

1. Quick Processing (less than a second) – Action performed is executed almost instantaneously without any significant wait period. In this case, you would generally not display any indication to the user about the background task execution. For instance, Open a dialog on the Click of a button.
2. Short Processing (1 second – 5 seconds) – User action consumed a few seconds in execution. In such cases, recommended practice is to show a wait cursor. For instance, Pick records from database to list them in the application screen requested by the user.
3. Long Processing (more than 5 seconds) - Actions which involve a significantly longer processing time, in such cases it is recommended to show progress bar indication to users.

In order to display a wait cursor (category 2 above) the following is the most common block of code seen in most applications:

public object GetData(int id)
{
    Cursor.Current = Cursors.WaitCursor;
    try
    {
        // Processing code here    
    }
    finally
    {
        Cursor.Current = Cursors.Default;
    }
}

The code piece above will have to be duplicated at all the places where we want to use wait cursor during processing, this would be a pretty tedious task. Now, let’s have a look at another approach to implement the same feature:

using System;
using System.Windows.Forms;

public class WaitCursor : IDisposable
{
    private Cursor cursor;

    public WaitCursor()
    {
        cursor = Cursor.Current;
        Cursor.Current = Cursors.WaitCursor;
    }

    public void Dispose()
    {
        Cursor.Current = cursor;
    }
}

The WaitCursor class above implements IDisposable interface, hence you may use this is the using block. Now see how simple it is to add a wait cursor to your application:

public object GetData(int id)
{
    using (new WaitCursor())
    {
        // Processing code here
    }
}

There are some very obvious benefits by doing it this way:

• It simplifies the code at several places into a single line statement.
• More importantly, we just ensured that a bug where some developer forgot to restore the cursor to normal can never happen (as long as we have trained our team to simply call the above code).
• Finally, it is always a best practice to encapsulate repeated code in one place.

One should always be on the lookout for such small coding patterns and implement them.

The first and most obvious change will be that programmers will start using many more threads and use current lock based approaches to synchronize access to shared data. Now detecting bugs in multithreaded programs is extremely difficult and as we write more concurrent programs, it will become increasingly important to have tools which can check our programs for correctness. Two such tools are CHESS from Microsoft research and jPredictor from UIUC. Both are still research projects but are definitely worth taking a look at and playing around with.

The transactional memory research community advocates getting rid of lock based approach for managing access to shared data. The idea comes from the database world where transactions have been around for a long time. Hardware transactional memory, hybrid transactional memory and software transactional memory, the three broad categories in which implementations are often split, are all under active research but there already are quite a few implementations out there for you to play around with. One of the core ideas behind using transactional memory for accessing shared data is to allow users to specify “what” needs to be synchronized and then let the machine take care of the “how” part (compare this to how you use locks today). This is supposed to greatly simplify the concurrent programming model but it is too early to say if it really does make the programming model simpler.

While  I cannot cover or summarize the work being done on transactional memory without making it into a small project, I can provide a few starting points that you may find interesting. Microsoft is doing very interesting work in this area and while a lot of it uses Haskel, there is a C-sharp API for software transactional memory that you should look at. If you are a Java programmer, you should take a look at DSTM2. And irrespective of what technology you use for building systems, I will recommend reading publications on transactional memory by Maurice Herlihy, Simon Peyton-Jones, Victor Luchangco and Mark Moir (there are others, I just picked up the four I see often in the literature).

Finally, we should not ignore the world of message passing. OpenMP and Erlang have been around for a long time and are designed for concurrent programming. OpenMP is an API you can use from C/C++ and FORTRAN to write parallel programs to take advantage of shared-memory architectures.  Erlang is a programming language in which you write processes that communicate exclusively using messages and if you write your Erlang programs correctly, you get the ability to scale to hundreds of processors for free.

Recently, I evaluated SharedCache, which according to their website is a high performance distributed and replication cache system built for .Net applications running in server farms. The system is made available to .Net applications through a Windows service.

 Some of the key features are listed of SharedCache are listed below:

• Works for both web and non-web solutions.
• Is extremely fast for fetch.
• It supports multiple topologies: distributed partitioned caching, replicated caching, and single instance caching. You can choose which you need based on your requirements (for example, you may want to choose replicated caching over single caching to ensure that you don’t have any single points of failure)
• In case of Caching, one of the most important aspect is purging. With SharedCache, you have a choice to make amongst the several that are supported:
  • Cache Item Priority
  • Least Recent Used Item (LRU)
  • Least Frequently Used Item (LFU)
  • Time based
  • Biggest Latency First (BLF)
  • Lowest Latency First (LLF)
  • Hybrid – Creating an index based on various item attributes such as: time in cache, Amount of requests, Size, Cache Life Time, etc.
• Configurable maximum cache size and the load factor when the system starts to purge cached items.

You can install the service by downloading it from CodePlex. And several samples and discussions are available at the author’s blog. The best thing about this service is that there is plenty of documentation to guide you.

Just try it out. Its pretty cool.

One method to log exceptions is to log them in the cloud. And there exists a useful service which makes your job easier.

A service in the cloud represents a service that is hosted on the Internet, available to you through an API, so that you have no idea how it is physically implemented. This makes things simple for you since you don’t have to worry about writing implementation code for the exception logging and reporting framework. You only need to call it.

The service that I am discussing is called HopToad. The service offers a simple to use API which allow you to log exceptions from your application. The exceptions are stored in the cloud and are available to you from a central web-based console. All the logged exceptions are presented in easy to understand and navigable way. For example, you can see which errors are occurring most often, or which one is the latest error that came up and from where. You can look at error details and stack traces.

The service developed to be used from within RubyOnRails applications. However, they have an open API, and can be accessed through any technology stack. K. Robertson has made a .Net wrapper available for logging to HopToad. It’s called HopSharp. It is fairly straight forward to use (just drop the DLL in your application and you are good to go). I can imagine that it would be straight forward to write a Log4Net extension for this service as well.

When to use this?

So, when would you use something like this. While the final decision to use this would vary from project to project and client to client, I can think of the following scenarios:

• The application is a work in progress and you are expected to release a number of updates. Typically, in such scenarios, a lot of testing happens while the application is in deployed state. A logging framework such as this will allow you to monitor the application as it is being used, and priorities on bugs can be assigned based on frequency and severity.
• You have an application deployed at multiple locations and you would like to see a consolidated view of the exceptions being reported.
• As a way to save time and effort. This is a ready made framework where you don’t need to impart any effort for implementing an exception logging and notification system.

I am pretty sure that there are other such services out there.