A little appreciation goes a long way. Global Services magazine and NeoAdvisory recently showed their appreciation for Nagarro by naming us to the Global Services 100 list for the second time. We appreciate being showcased among the world’s top IT outsourcing firms, and appreciate the customers that enabled us to showcase our successes. But most of all, we appreciate the Nagarro employees that allowed us to achieve success with our customers.

One of the findings that came out of the Global Services 100 survey is that in 2010, outsourcing firms should focus on getting and keeping the right talent. I would add to that: Keeping the right talent happy. Of course, this is nothing new at Nagarro, and while we can’t promise that every single Nagarro employee will be overjoyed every minute of the day, we can promise that employee satisfaction is a top priority, that we pride ourselves on running a meritocracy that rewards performance- and that we have never once had an employee evacuate the company via the emergency slide!

You can check out the video here-

HKMP Technologies came to us with a tall order– upgrading its market-leading paperless accounting software on a tight deadline in order to secure a strategic customer deal. To make matters more complicated, a new release of Microsoft Silverlight – the technology platform for the product – was scheduled right in the middle of the development cycle.

Our team here at Nagarro approached the problem creatively, allowing HKMP to successfully seal a critical new deal and improve its eco-footprint. For more details, please check out our new HKMP Technologies case study and customer testimonial video.

Though this is not a very common scenario in many of the modern day applications out there but this definitely is a hallmark of databases handling loads from specific industries like banking. Applications intended for these industries normally have huge volume of small database transactions. In this article, we present one of the approaches you can take to handle a scenario such as this.

Introducing Broker Hub

To demonstrate the problem which has these type of database requirements, let’s use the example of a Broker Hub – a stock broking hub. Stock broking applications have very high volume of small database transactions and also there are spurts in database activity depending on market conditions. For simplicity sake, let’s assume that we were at a point when database design and usage pattern for Broker hub database was in the most optimal state.

First choice – Scale up

So to optimize Broker Hub further, we had a number of ideas and first choice was obviously to increase the hardware capacity. Increasing the hardware capacity did help the case. We could handle upwards of 50000 transactions a second by moving to a better system with 16 processors and a SAN array of high speed disks. But above 50000 in our load environments, we could still see the database to be the bottleneck.

Next obvious idea was to try SQL Server 2008 which has support for performance optimization features like advanced compression (reducing the overall disk IO) and support for virtually unlimited number of objects (2,147,483,647) and database size (524,272 TB). Again we could see the difference. Without enabling features like compression we could achieve a bigger number of around 70000 transactions per second.

Would it work? Probably not long term.

Probably enabling compression and using other features to optimize performance would have resulted in a higher figure but the problem here was that there was a limit to this. Adding hardware or moving to newer version of a database (or even to a different database) wouldn’t have given us the virtually unlimited (100 times 30000 transactions per minute) capacity we were looking at. Obviously we needed some way to deploy more than one server to split the load and thus increase the capacity to handle very high transaction loads. In simpler terms we needed a scale out solution instead of scale up for our database. Looking around for out of the box solutions in the market didn’t help, simply because there aren’t many scale-out solutions available in the market to load balance SQL server. Oracle has launched such a solution but even that requires syncing between different servers in cluster which takes up a lot of network bandwidth thus reducing the overall effectiveness of the solution.

Scale out – Approach

After a lot of brainstorming sessions, it was decided that it was time to create our own scale out solution. The idea was to create a design which could help us scale out as our user base grew but at the same time being able to handle sudden increases in transaction volumes. But as it happens all the time, we didn’t have too much money to be spent on this. We looked at various existing products such as SharePoint to see how they stored their data and came up with a very simple first draft of the scale out solution. It looked very similar to way SharePoint does load balancing for its data stores. We had a cluster of database servers connected to the Broker Hub. Each database was configured to handle a set of users with specific user Ids and thus held data only for those users. Merge replication was used to replicate data from all databases to a central database which was used for all reporting. Here’s how it looked like:

Broker Hub front end had to be modified a bit in terms of process flow. In this case, each user logging into Broker Hub had to be connected to a specific database based on his user id. The task of identifying which database has the information related to the user trying to log in was handled by a new module added to the application called DB Load balancer. The process flow in this approach looked like this:

• User foo with User ID (n) logs in.
• Application calls Custom DB load balancer to find out that all data specific to user foo is on database server DB 2.
• Application creates a connection to DB 2 on behalf of this user.
• All user transactions from User 1 are directed to DB 2 thereafter.
• Periodically user data is synchronized to the master database.

Design Decisions for this approach

This approach was pretty simple to implement and could achieve the results we were looking for. But there were a few practical issues:

• We have to introduce concept of ID buckets. So every transaction table on each server was assigned unique bucket from which it could allocate Ids. With this we overcame the problem of how to maintain unique IDs.
• All the masters would be replicated using transaction with update option. So a change on one gets replicated all over.
• Adding new users was not as simple as before. Every database was configured for a specific set of users and this set increased sequentially. This meant that we always had to add the new user in the last server available and if last server was full then we had to deploy a new database even for a single user (and probably even before the last server was at its full capacity). This was accepted as an acceptable fact as this would happen once a while.
• Any admin report had to combine the data from all the servers to be useful which meant an additional job to aggregate all the data. This was accepted as a design reality.

Conclusion

So we could see very clearly that we were achieving the desired performance figures for this approach. Our aim was to find out how best we could achieve load balancing capabilities for the specific database load our application was creating and that was achieved. It actually became a life saver for us in more than one situation.

Nagarro is proud to partner with SIAM to develop paperless systems as an eco-friendly alternative to manual processes. To learn more about how Nagarro’s custom software development services can help your company’s business goals, please contact us at info@nagarro.com.

Read Nagarro’s new blog!
Become a fan of Nagarro on Facebook
Follow Nagarro on Twitter

If you are going to be at SAPPHIRE, please drop by Patrimonio’s session for a viewing of the video, an interactive discussion and more information on SAP BPO best practices:

Real-Life Integration and Process Management

Wednesday, May 19

3:00 – 3:20 PM presentation

3:30-3:50 PM discussion

Theater 3 Lines of Business

Problem: One of the challenges that you face, when you are selling web services is this: "how do you showcase your product to a potential customer". The answer can be simple, make a demo web page which allows prospects to play with the web service and try it out before subscribing to it. This is indeed simple, if you are selling one or two web services. However, if you are providing anything more than a handful of services, then creating, maintaining, updating, and updating a web page for every service and its various versions that you sell can be a nightmare.

Solution: A solution for such a scenario can be the use of an engine which can dynamically render a user interface for any given web service on demand. Let’s call this as the ‘web service demo engine’.

The advantages for such an engine are obvious:

• no additional effort when publishing new services
• smaller code-base to maintain

There are various challenges to as well, if such an engine is to be implemented. These include:

• the engine needs to be extensible to and flexible to absorb any changes to existing web services.
• web service parameters can vary from being simple data types (such as integers) to collections, custom objects, etc. The demo engine needs to be able to interpret any data type and render its UI efficiently.
• similarly, the web service response may contain a variety of data types (though XML) – tables, charts, images, etc. The engine should be able to interpret this data an render the output correctly.
• the engine should render the UI to allow users to enter the input data for the web service, and should perform basic business rule validation to ensure input data makes sense.
• the basic goal is that the engine should be completely configurable, and should not require re-coding whenever a new service is released.

As can be seen, some of this information is not even present in the WSDL of the web service, so any engine that hopes to achieve all of the above needs some help other than the WSDL. A possible solution is to have a configuration file published with every new web service that carries this information. Such a configuration file would carry a number of items:

• business data definition
• input controls information
• output controls information
• information about grouping of input and output controls
• business rules for input validation
• output data rendering information
• information about the web service itself
• mapping for input control to web service parameters

This configuration file can be generated by hand, or auto-generated partially through the use of custom code attributes or custom tools when the web-service itself is being developed.  Through the use of this information, an engine which renders demo pages for any given web service is conceivable.

How is creating a configuration file per web service better than creating individual web pages per service? There are several ways in which this is better:

• the configuration files generation can be largely automated.
• the central demo engine makes it easy to have consistent UI and style when rendering.
• you only manage one code base. Imagine if tomorrow, you wanted to add specific authentication mechanisms to provide access to the demo pages. If you had a demo engine, you only will have to make that change in one place.
• it decouples the rendering of the UI from the web service itself – a web page per service is very tightly coupled with the service it is demonstrating.

Has anyone implemented a similar solution or encountered a similar problem and provided a different solution? We would love to discuss this through the comments.

However, the most important achievements in winning this award are Nagarro’s near-perfect scores in customer satisfaction and employee management. As our customers know very well, Nagarro is only as good as the people who work here. As Nagarro’s CEO, I find it tremendously rewarding to know that all the hard work that goes into employee development behind the scenes has resulted in the desired effect: Satisfied customers.

A special thanks goes out to the hundreds of Nagarro employees working hard to help our customers achieve success. For me, this is what it’s all about.

What is SAP ESR

• SAP ESR is the evolution of Integration Repository (IR) in SAP XI. IR was already representing the end points as message interfaces. Message Interfaces were already exposed as WSDL. SAP has added the Modelling capability for Process Component Models (which is an extension on top of the earlier Integration Scenarios), and streamlined the usage of end points (now called Service Interfaces). It has also streamlined the usage of data types which are now called global data types.
• The service organization concept is similar to that of IR. All Service Interfaces are within a SCV and a namespace.
• SAP ESR provides the advantage of providing a consistent naming convention of Service Interface.
• Enterprise Service Builder (ES Builder) offers a modeling environment to create various models in the Enterprise. It was earlier called the Integration Builder in the SAP XI environment.

Salient features of SAP ESR:

• ES Builder allows to create Process Component Model (ProComp Model)
• In SAP NW 7.0 (2004s) and SAP NW ‘04, service interfaces have one operation per service. In SAP NW PI 7.1 the Enterprise Services Repository offers multiple operations per service.
• Global Data Types: Approved SAP-wide using the Governance process. Reusable semantic building blocks for service interfaces and message types.

Why to use SAP ESR:

• SAP ESR is the central repository for modeling the service interface. We model the service operations, and their request/response structures. This allows us to have a CENTRAL enterprise wide repository for managing these definitions.
• SAP ESR provides the capability for organizing the service definitions based on software components, their versions and namespaces. It also allows us to maintain service versions (since services invariably would get revised over time).
• Enable SOA design governance across the organization. We can model the Process Component architecture models in ES Repository. From the ProComp models, we can drill down to standards based service interface design (Service Metadata). The service interfaces are based on GDTs that provide resusable uniform representation of business structures.

SAP ESR is positioned to be the central enterprise wide repository for managing all service definitions for SAP and nonSAP services.

Our recommendation for SAP customers is to use SAP ESR for modeling and managing all SOA service definitions for SAP and non-SAP services across the enterprise.

One of our customers had a requirement which was slightly different from the standard update requirement. This customer offers a suite of applications (much like MS Office). There is a further twist, the business case for this customer is such that even within the same application, the end user might be able to run multiple versions of these applications side-by-side (like you can run Visual Studio 2005 and 2008 side by side). Each of these applications and the various active versions of a given application could have updates released independently of each other. One would think that all of these could be implemented as separate ClickOnce packages, but that would have been a maintenance nightmare on the update server, and also would have presented a very poor experience for the end user.

Problem statement – Create an updater solution which uses a single application to provide automatic update notifications for multiple applications, and manages download and installation for all of these updates.

Our solution was to create a utility application on the client end which acts more like an Application Manager. This application keeps track of all the applications installed on the client (and those that are not installed), and manages the updates and notifications for all of them. The solution mirrors the concept presented in the Microsoft UAB. Here is what it looks like:

The application manager can be implemented through a variety of paradigms. It can be a service which is always running in the background. It can be implemented as an application launcher – this means that to launch any of the applications in the suite, you first have to launch the application manager. There can be other ideas on how it can be implemented.

The basic solution for how it works is similar to the UAB. It keeps track of the currently installed application on the client machine (perhaps through reading registry settings). It keeps track if new releases by requesting a the server for a manifest file which contains information about the various versions of the applications present in the application suite – specifically, it contains information about the latest versions of the applications present on the update server, and information about how they can be accessed. The application manager then compares this information from the manifest to the local manifest that it maintains of the installed applications. The way this differs from the UAB is that in UAB, the manifest on the server contains metadata about the releases of a single application. In our scenario, it contains the information for multiple applications.

The application manager can provide various services such as provide notifications when updates are available, maintain user preferences for update notifications, download updates, install selective updates, or install updates automatically (or based on a schedule). All of these, and many other rich update features can be implemented through the use of the application manager.

On the server side, the process is simplified because whenever any of the applications are changed, a single change has to be made to a single manifest file/data source. For all updates, only one manifest needs to be managed, which makes update management easier and less error prone. The benefit of using an Updater Service (instead of simply requesting the file directly from the server) is that the manifest information can then be kept in a database, from where the service can generate the manifest file. Or another implementation could be that the manifest file is generated from a well-defined directory structure on the server. The point is that using an interface such as a service allows you to have more flexibility on how you implement your server. Another benefit could be to build-in licensing or role-based manifest generation (to prevent access to certain applications or updates).

Finally, the application manager itself needs to be updated. We chose to use ClickOnce for the Application Manager itself. The reason is the ease of deployment and update for a single application using ClickOnce is still the best. So, the use case for setting up new client machines becomes very simple. First install application manager to using ClickOnce. Then the application manager downloads and installs everything else.

Design Principles for service design and identification:

1. Granularity: The service should have a broad level functionality to be sufficiently useful to a large range of consumers. The service should not be too fine-grained i.e. not have a narrow functionality to be of use to a limited set of consumers.
2. True reuse at the Enterprise level: The service should have functionality to be of use across the enterprise by providing generic functionality in its functional domain.
3. Durable (standing the test of time/change in the landscape): A good service would stand the test of time and the test of constant change in the enterprise landscape.

Best Practices for Technical Characteristics

1. Idempotent: The Service should provide the same result when called more than once. In web service operations, due to technical failures, the caller may not get a valid reponse to its service invocation. The caller would invoke the business service again, which would lead to a repeated business transaction (an obvious blunder). Hence, for transactions, we have to ensure that the business transaction happens only once. A design approach is to use a unique GUID in the web service call, which would be used in the service response to correlate the request-response. In case of a repeated service invocation, the duplicate call would get detected due to the repetition of the unique GUID parameter.
2. Security Considerations: Considering the broad appeal of the service across the enterprise, many consumers are expected to invoke the service. Hence, it is important to control the security aspects of calling this business service such as roles, authentication and authorization. Even if the service is not a transaction (not committing any business data) and if it is a read-only service, the security has to be controlled to restrict the data subset being requested by the caller.
3. Error Control: Due to the distributed nature of services and participation in distributed business processes, the service should fail elegantly and provide a human-readable trace to provide an easy method to troubleshoot error causes. Additionally, the service should reverse-compensate any business data transactions on encountering errors and preserve data integrity.

The above are the essential aspects for designing good services that endure the test of time. These learnings are based on our direct experience in identifying enterprise services and establish SOA design and governance principles across customer projects.

SAP runs most of the core business processes across its customers. The SAP Business Suite provides comprehensive transactional support for core business processes such as financials, sales, procurement, manufacturing, operations, HR etc. SAP has built a rock solid business process platform for managing these core processes on the SAP application server to manage processes within the SAP ERP product.

With mergers, acquisitions and extended supply chain, the business processes are spanning multiple systems and business partners. SAP has recognized this industry need; SAP has a clear roadmap towards business process execution and management for extended processes spanning multiple systems, applications, users and partners.

SAP BPM suite is the business process platform for managing these extended processes. However, the SAP BPM product set is a relatively new entrant in the well entrenched BPM market. Hence, customers would tend to evaluate SAP BPM with the mainstream BPM vendors before taking a decision on the BPM adoption.

A few important BPM product evaluation criteria are:
1. Support for long running processes, process versioning, compensating transactions
2. BPM infrastructure related to performance, load balancing and scalability
3. Integration of business rules
4. Security, role based access
5. Reporting capabilities
6. Business activity monitoring
7. Simulation capability
8. Design and Development environment, productivity
9. Operational support, monitoring and management
10. Pricing, vendor stability, installed base, customer references, consultants availability

SAP BPM may not hit the top place today in comparison to the best-of-breed BPM products. However, SAP customers need to look at the holistic view of adopting and standardizing on the SAP platform for all their business and technology needs.

Advantages to standardizing on the SAP BPM platform:
1. Native integration with SAP Business Suite
2. Rapid prototyping capability for integration to SAP suite
3. Reuse of your team’s skills in SAP development using WebDynpro and ABAP
4. Integration with SAP security and consistent role management
5. Familiar SAP Portal look-and-feel to users for SAP BPM interactions
6. Unified monitoring through SAP Solution Manager
7. Pricing in line with your contract for the SAP product bundle

Our recommendation to customers having a SAP-centric strategy is to adopt SAP BPM for business process management. SAP BPM is central to SAP’s strategy on business process management, and the product is getting better with every release. Native integration to SAP products (SAP ECC, SAP SCM, SAP CRM etc.), usage of skills of the SAP development team and faster development would outweigh the adoption of a new BPM product in the long run.

MES stands for Manufacturing Execution System which is used to manage the operations; for example, production order, process data, work instruction storage etc. It is related to the manufacturing and operations. The main purpose of any MES system is to gain visibility, flexibility in operations and manage the supply chain effectively along with consistent documentation. In the market, there are numerous MES systems specific to industry type, product nature and manufacturing complexity.

In general, MES is vendor specific and as per explicit need of the customer. MES systems are independent of ERP systems and are “add-on” software. Usually, it is difficult to find all our requirements in any ERP suite that can manage the complex manufacturing flow of products; in such situations, specialized software products such as MES are suited, where ERP functionalities are not sufficient to run the operations. Indeed, MES assists in achieving the multifaceted requirements of the organization.

At the highest level, MES needs integration between the ERP and shop floor automation. SAP maintains the record of materials, BOM, routings and orders; the MES maintains the records/transactions such as WIP (work in progress). For implementing the MES, the consultant has to perform detailed analysis of MES requirements such that they can be mapped with the manufacturing environment. The consultant has to perform the analysis of the integration with SAP PP since there is overlap of functionality offered by SAP and MES software.

MES plays a major role when there are many shared resources or work centers, complex routings (master recipes) and substantial effort to manage the production of material into finished product. MES increases operation excellence and manages asset and its utilization, which in turn reduces overall operational cost of the business.

MES provides capabilities such as plant data collection, production process quality management, order tracking; additionally, the MES system provides detailed scheduling of operations, production process management, performance monitoring, resource allocation, manpower allocation and management control of engineering documents. Resource allocation can be manual or automated in MES. There is real time exchange of messages between the SAP ERP and MES system regarding various facets of the manufacturing operations. Let’s say in case of production management, one can easily download production schedule from SAP to MES system, upload production confirmation/consumption to SAP.  In case of asset management in MES, we can download equipment list from SAP and upload equipment runtime data to MES.

ISA (Instrumentation, Systems, and Automation Society) S95 standard describes the connectivity between ERP and the MES system; this includes MES functionality and database schema, message formats between application systems etc.

Integration – There are two major integration options for MES integration with SAP
1. SAP xMII (Manufacturing Integration and Intelligence) – SAP xMII facilitates real-time transactional integration between plant shop floor and enterprise (SAP ERP) systems out-of-the box. By using the schemas available in the xMII repository, we can perform faster integration between xMII and the MES systems conforming to the IS95 industry standard interfaces.
2. SAP PI : This is the integration middleware platform within the SAP NetWeaver suite. We can use SAP PI to exchange information between the MES and SAP ERP products.

SAP CAF provides discrete steps or activities which can be assigned to a role. The activities in the SAP CAF process can exchange data through process context parameters which is similar to a “global memory” where the common process data can be placed and can be accessed from any step in the SAP CAF process.

Though SAP CAF provides a rapid development facility of providing a code-free activity for common integrations such as web service, RFC …, in real world business processes, we have to use custom developed WebDynpros to implement each business process step.

Our design recommendation for SAP CAF processes is as follows:

1. Do not use the CAF process context for exchanging business data. Keep the process context light. Instead, use a unique process identifier across the process steps to identify the currently running process.
2. Design each process step as a standalone WebDynpro application. While establishing the functionality of this standalone WebDynpro application, think of it as a reusable component which can be reused later in any future business process step. The only input to this WebDynpro is the unique process instance ID. The standalone WebDynpro application would retrieve the business data from the persistent storage based on this process ID, and present the User Interface to the end user. The user would provide business data to this WebDynpro application; the WebDynpro would conduct its business processing; it would persist the business data into the persistent storage and mark this step as complete.

SAP BPM is similar to SAP CAF in the business modelling and process steps:

1. There are business activities in the business process model
2. SAP BPM provides process context for storing process specific business data.

In our experience, if we conduct the design based on our recommendations for SAP CAF process implementation, the migration effort to SAP BPM is significantly reduced. The WebDynpro applications need minor changes to conform to the SAP BPM framework vs the SAP CAF framework for calling WebDynpro methods.

In conclusion, our recommended design approach for standalone WebDynpro design towards reusability, and exchanging minimum data in the process context has stood the test of rapid migration from SAP CAF to SAP BPM with minimum effort or rework.

• SAP ABAP workflow is meant to streamline a business process that is completely contained within the SAP server. Typical usages are procurement approvals, leave or travel approvals etc where all the business data and business objects are within the SAP instance.

• SAP PI BPM is used for cross-system integration and managing a higher level business process between disparate systems. A typical scenario is a Sales Order acceptance process where several systems would need to be consulted before accepting the order such as a inventory check, global ATP check etc.

• SAP CE provides workflow capability at the User Interaction level where the user can see a sequence of screens to complete a business process. SAP CAF was the workflow product to provide the BPM capability for implementing a UI centric business process.

• SAP BPM is now replacing the SAP CAF product set to provide a standards based BPM capability. SAP BPM provides modeling and implementation for a UI centric business process – we can develop the business process based on the BPMN notation supported by most mainstream BPM modeling tools.

• SAP MDM workflow is a lightweight workflow capability within the SAP MDM product to allow the various processing steps for master data management such as cleansing, deduplication, referential integrity and broadcast to other consumers. It is a workflow engine for exclusive use within the SAP MDM product set for finer grained processing.

• SAP BW Process chains are a workflow capability for sequencing data processing steps within SAP BW such as index management, PSA/ODS load, cube load, statistics and aggregate creation.

• SAP Portal provides a capability for creating a workflow on the fly – it is SAP Portal Ad Hoc workflow capability. This feature allows the portal user to have better collaboration experience such as document proof reading and approval from other team members.

Considering the richness of the SAP product suite, there will always be workflow capabilities that are best suited for the specific product set.

Based on our experience, we recommend the following products for your workflow requirements:

• SAP BPM for UI centric business processes
• SAP PI BPM for cross-system business processes
• SAP Workflow for workflows within the SAP instance

As always, use SAP SOA based interfaces for all interactions between SAP systems.

What does this tell us? It is difficult to say with certainty that the recession is a thing of the past. However, we can conclude that regardless of size, industry or geography, companies recognize the value of using custom software products to make the most of their unique differentiators. Nagarro is proud to partner with the more than three dozen new customers who invested in long-term competitive strength. We are looking forward to an exciting 2010!

Since a programming language is just a tool for instructing a general purpose machine to perform certain activities (i.e. program the machine), the complexity rarely lies in the programming language. The complexity lies in the world and the abstractions that we agree upon for simulating certain aspects of the real world in our systems. We often try to bridge gaps in our understanding of the world by making assumptions. In an ideal (simpler, non-real) world, we wouldn’t have to make any assumptions. In the real world, the best we can do is to make every assumption as explicit as possible and get all stakeholders to buy into those assumptions.

An assumption can be made explicit through documentation but its better if the assumption can be formally described and automatically enforced. Good programs force developers to think through and be conscious of the assumptions even in absence of any documentation.

The idea of making assumptions explicit is not restricted to software interfaces alone. It is just as important to keep this in mind when you gather requirements, write specifications, create design documents, and design and document your processes. It is so important that I, at times, feel that  programmers should have lawyers working with them. Which (sort of) leads me to my other complaint: configuration heavy frameworks. We have spent a lot of time and effort making our programming languages (and APIs to certain extent) explicit and self enforcing, but with every file  based configuration (it’s actually not just file based but files will suffice for the time), we tend to hide some of the assumptions in an isolated, implicit and non-enforceable piece. Google’s Guice tries to avoid configuration files and Spring has been supporting annotation based dependency injection for a while now but the overwhelming trend even today seems to be: when in doubt, control by configuration.  The specification designers of the Java Enterprise Editions realized this several years ago but there still are configuration heavy frameworks out there. As a programmer, when I start looking into a new framework or a development environment, it’s not the core technology or a new programming language that takes time to learn. It’s always the configuration that takes the longest to figure out.

At a high level, the various SAP integration approaches are: File, Database, SAP RFC, SAP IDOC and SAP web services. The file and database are not specific to the SAP platform, and are a very common mechanism for data exchange. However, they suffer from latency issues, and are suitable for large data transfers or where the connecting application has rudimentary integration capabilities.

The SAP RFC and SAP IDOC have been the traditional means of integration. SAP RFC is suitable for request-response style of communication. SAP IDOC is suitable for bulk data transfers (10MB and larger message sizes typically). The SAP RFC and SAP IDOC are compatible with the oldest SAP versions, and they are a safe approach towards SAP integration for guaranteeing backward compatibility.

The SAP recommended integration mechanism is to use web services. These web services can be implemented and exposed by the SAP platform which can be easily invoked by the non-SAP application for request-response or data exchange. The web service support is fully supported in SAP ECC6. The web service support is weaker in the earlier SAP releases. However, most SAP customers are in the process of upgrading to SAP ECC6 and we should adopt the web service approach towards integration with the SAP platform. It can be safely assumed that most customers would be at SAP ECC6 release to make use of the web service based integration. This approach complies with the best practices and integration approach recommended by SAP.

Understandably, customers have hesitation and lack of clarity on how to work with their service provider for remote SAP consulting services. Customers would tend to pay higher to get local SAP consultants rather than “try” this new method of getting SAP development and support services.

There are challenges of timezone, language and the absence of physical face-to-face meetings which are the immediate concerns of any customer. Many service providers are located in India, and they are several hours ahead of the customers in Europe and the Americas. The consultants are fluent in spoken and written English, and have exposure to major European languages. We are able to address these challenges through hiring of local consultants that interact with the customer locally. Our teams work on flexible hours to match the working hours of the customer; we use collaboration tools like instant messenger, screen sharing and video conferencing.

Many large customers have been operating internally across their distributed global offices, and they are comfortable operating across languages and timezones. SAP AG, for instance, has the development teams distributed across Walldorf, Germany and the development centers in India and other locations.

Many mid-size SAP customers in the USA and Europe are now seriously considering the option of remote consulting and remote support development due to the access to expert SAP skills and compelling cost advantages. These SAP customers are the mid-size organizations that have traditionally used the local SAP consultants; they are now reaching out to SAP service providers to start a pilot for remote consulting and remote support services.

Conclusion

SAP remote consulting has challenges but it is a practical and cost effective option for getting effective and dependable SAP services. Many SAP customers are able to develop a deep partner relationship with their service providers. SAP customers should attempt to make use of this remote consulting service to complement their existing SAP service processes.

Nagarro is proud to be listed alongside some of the industry’s most highly regarded outsourcing firms. We fared well in evaluations of our HR practices, management team strength and corporate social responsibility programs. But ultimately, we took the greatest pride in being able to provide objective third-party validation and glowing customer references. As we add this award to Nagarro’s accolades, we are once again grateful to all the customers, partners and employees who have helped Nagarro continue to achieve success.

As part of a company-wide upgrade to the SAP platform, our SAP Consulting team is using the SAP Netweaver ™ platform, SAP IS-Banking and numerous other SAP technologies to help catapult Patrimonio Hipotecaria to the next level of success. Our team will create a 12-fold increase in overall system capacity for Patrimonio Hipotecaria –while improving the consumer experience as well as streamlining internal efficiencies.

You can read more about this new customer engagement in our recent press release and new customer case study.

Since a very compelling business reason for using cloud computing is the ability to scale on demand and when needed, using the best practices and guidelines for designing large scale  distributed systems is a good starting point. While academia has a preference for   distributed transactions, ACID properties, and all kinds of consensus protocols, you are better off with loosely coupled systems where nodes will eventually reach a consistent state (as opposed to waiting for all nodes in a very large system to  reach a consistent state before proceeding). Considering the state of existing systems, loosely coupled self healing self regulating systems should be preferred over system with exotic distributed transactional properties and consensus protocols.

Be prepared to lose computational appliances. As you already know, stateful systems are bad for scalability. Avoid storing the state in server sessions. If you are designing a client-server system from scratch, do not store client sessions on the server. For typical web applications, this means that your HTTP calls should be stateless. GWT and other rich AJAX or Flex clients let you do this relatively easily.

You should also try to avoid solutions that try to replicate state across multiple nodes. This almost always leads to overly complicated solutions. You are better off being prepared for  loss of data and designing with the assumption  that client application state, if at all available on the server side, may not  be synchronized across all nodes. UDP, which is often used for discovery, and negotiating other information may not even be available on many cloud platforms.

These are important design decisions that you must make right up front.  Retrofitting these into existing solution might be impossible.

Software engineering

In addition to strictly technical design and runtime issues, you also need to take into account other software engineering issues which affect you development.

As already discussed, you need to have a pool of engineers who understand the technologies you will be using. You may find it hard to find many engineers who have had experience with large scale, self healing, self managing systems. While one of the objectives of most cloud platforms is to take these complexities away from the developers, you will need engineers who understand these concerns.

You also need to find out if the platform you have selected provides good tools for development, deployment and debugging. You should ask the same questions you ask when selecting any technology.

• Find out if the platform integrates with the IDE you are planning to use (or if the platform comes with an IDE that will serve your development needs).  I would recommend staying away from custom IDEs. Amazon, Google and Microsoft support industry standard IDEs for their respective platforms.
• You need to design for testability right from day one and your design should be testable on the platform.
• If your organization has an enterprise architecture, you need to be sure that your design and tools meet the standards and approaches laid down by it. Different network topologies, moving data in and out of the cloud and integrating systems running in the cloud with the systems inside the enterprise will come with its own set of challenges. While most enterprises have already started looking into cloud as a viable means for extending their computational needs, they may not be ready for a complete adoption.
• Security is an important aspect that you may not have full visibility into when you are running on someone else’s platform. Work with your security experts, quality assurance and compliance team to make sure everyone understands the security risks. A perceived security risk and lack of visibility could be a major problem while adopting any new technology (anyone remember trying to use Java RMI across enterprise firewalls?) and clouds are no exception. In fact the shared nature of the services makes security a bigger concern.
• Finally, you need to design for monitoring. It is best if the platform supports automated monitoring. Otherwise you may need to design for it right from day one. Even when the platform provides some tools, you may still have to monitors aspects not supported by the platform tools.

This discussing should provide you with a good starting point for creating your own best practices and guidelines. These are important design decisions that you should try to get right  up front.  Retrofitting these into existing solution might be impossible.

What is SOA

Service Oriented Architecture (SOA) is a software design strategy in which there is a service provider and many service consumers. SOA is similar to the distributed computing principles of loosely coupled architecture. SOA differs from the traditional client server architecture in terms of distributing the functionality over separate business objects exposed through service interfaces, loose coupling through independent business objects, durable interface contracts and promoting reuse of functionality through a loosely coupled architecture.

What is a Web Service

A web service is an interface implementing the industry standards of SOAP and WSDL (and usually HTTP) to expose functionality to a web service consumer. It is a technical implementation to expose functionality through a standards based protocol.

The use of web services has enabled a tremendous leap in enterprise integration. Till a few years back, most applications provided proprietary or rudimentary methods of integration, such as SAP provided SAP RFC, BAPI, IDOC based integrations. Such integrations warranted the use of middleware adapters to provide integration with SAP.

The use of web services now allows a direct integration between applications with the SAP business suite.

Hence, we recommend the usage of web services as a central integration strategy for all integration initiatives.

With the proliferation of web services, it is a challenge to catalog and manage the usage of services in the enterprise. Hence, SAP has provided a robust set of SOA toolsets to manage the usage of services in the enterprise.

What is an SAP Enterprise Service

An SAP Enterprise service is technically a web service, and it has the following attributes:

• Based on web services standards of WSDL and SOAP
• Based on SAP global data types
• The service has been modeled within SAP ESR using business objects, process components and the SAP enterprise model
• Published in the SAP SR (service registry)
• Guarantee on the availability and functional correctness

What if my Web Service does not qualify to be an Enterprise Service

There are scenarios where the web service interface exposed by an enterprise application is not globally relevant, and is really a technical interface to a localized business functionality. The usage of such a web service may be limited to the department in the organization and its particular IT applications having its own semantics.

Such web services are not Enterprise Services. However, SAP SOA tools can be used to manage and catalog such localized and non enterprise services as well. The SAP SR provides rich taxonomy capabilities to catalog and manage all such services.

Summary

Web services should be the central integration strategy for all integration initiatives to provide a standards based integration to applications. However, not all web services are meant to be used at an enterprise level across the organization.

The SOA Governance body should approve all enterprise service candidates based on the above mentioned criteria, and use the SAP SOA toolset for cataloging and managing all service in the enterprise.

You need to consider how your system integrates with other systems in your organization. Clouds are good but one-off systems that use cloud but do not integrate with existing systems will diminish the payoffs of using a cloud.

You need to consider how your system and its data get integrated with existing data. This may need to be done both ways. Without access to existing data or data created and maintained by your cloud based system, you will again limit the potential of your system under design.

You may also need to integrate your application with your enterprise’s existing authentication and authorization solution.

Even if integration is not an immediate concern, an architectural choice can make it impossible to integrate even in future. You will need to worry about different networks and firewalls and if the protocols you will use to integrate these systems will work across these networks and firewalls.

Development Considerations

Once you start building the system, you will need to take into account how many engineers and support people are comfortable with the technologies you are using (in this case – something cloud based). You may need to take training and documentation time into account in your plans.

If you need to move large amounts of data between your internal computers and the cloud, you will need to either have sufficiently fast network connections or take the associated delays into account. This delay could play a major role in how you trouble-shoot issues, provide other kinds of support, take backup or just make the data collected by your application available to the users within your internal network.

Offshore development is a reality and if your application is either developed and/or maintained by a team working offshore, network speeds,  latency and round trip times will all come to play a major role. This means your choice of development and support tools and environments should work well over a bad network. VNC and Xserver-clients do not behave well, text terminals work better, light-weight HTML based clients work best. You also need to worry about moving large amounts of data in and out of the cloud from your development and support centers.

If you have specific regulatory compliance requirements, you need to be sure that the platform complies with the standard. For example, Amazon EC2 instances are not PCI (payment card industry) compliant so you should not plan on storing credit card information on Amazon EC2 instances. There could be regulations  regarding sending customer data out of country and when you store data in a cloud,  you (or even the provider) may have no control on where (which node) that data  could be at any given time.

I am not a big fan of using definitions as a starting point.  In technology many useful and successful concepts have escaped definition. People still argue about what exactly is an object in an object-oriented system.  There is no scarcity of definitions for software architecture. The internet cannot be pinned down to a simple definition. A definition, however, it is still useful in certain contexts.

While people might argue about the exact definition of cloud computing (which is probably a futile effort at this time), a working definition can be used to evolve a shared meaning. This is especially useful in software architecture where clear communication among stakeholders is extremely important. Software architects interact closely with business, users, developers, testers, compliance groups and others and before you move your architecture to the cloud, you should state exactly what cloud computing means  in the context of the system you are designing.

Definitions will differ based on whether you are a consumer of the cloud based  services or a provider and with more and more of private and hybrid clouds out there,  organizations and enterprises are fast becoming internal providers of cloud based  computing services which is blurring the line between providers and consumers of a  cloud “style” service.  Today, from a consumer’s point of view, a cloud based compute service:

• is implemented as a pool of servers though the consumer may not have direct access to the underlying pool.
• is characterized by very high degree of automation typically providing both  programmatic access (for integration) and graphical or command line user interfaces.
• provides an abstraction of compute services along with constraints that enable  the provider’s offering.
• is priced on a pay-as-you-go basis. Consumers only pay for what they use.

When you take these along with (effective) infinite scalability and pay-per-use utility model into consideration and design a system for the cloud, there are several issues that you should take  into account. I will discuss a few architectural considerations that must be taken into account while designing a cloud based system. The usual design principles and best practices for building a cloud based are not very different from what you probably already use as part of your enterprise architecture standards and/or other best practices for building large scale distributed systems; but some of these may be unique.

Going forward, SAP customers are tending to standardize on the SAP platform for running their core business operations. However, for development of custom applications, customers do not have a clear approach towards selecting SAP NetWeaver or the other mainstream platforms for development.

UI Technology Choice Criteria
Most enterprise vendors such as SAP, Oracle, Microsoft and IBM offer a complete technology stack with comparable features such as Portal, content management, knowledge management, enterprise search, data warehousing, middleware integration, mobility, application server and solution management tools.

A custom application is usually not isolated. In our opinion, the decision to select a development platform for custom applications should be determined based on the following criteria:

• Developer Skills: If your development team is most comfortable with SAP ABAP (rather than Java), then it would enable you to develop custom applications faster, accurate and perfectly aligned with business expectations. If your development team does not have SAP skills but Microsoft dotNET or Java, then it may be rational to develop custom applications on the Microsoft platform with integration to SAP.
• IT comfort to support the development platform: If your SAP IT team is not comfortable with supporting the SAP J2EE application server (for maintenance, patching, upgrade, transports, security), then it may make sense to stick with the SAP ABAP development platform.
• Depth of SAP Integration and SAP Business Processes: An SAP platform based development would accelerate development if there is significant SAP integration and business process interaction involved. A custom application is usually not isolated. If the majority of business data is resident in SAP, then an SAP platform based development would be faster and more accurate to end user requirements.
• Comfort of end users: This is an important criteria in determining the technology for end user applications. For example, if the end user has a strong liking for Adobe Flex based reports or dashboards, then, getting a similar look-and-feel on other platforms may be a challenge.

It should be noted that SAP NetWeaver has strong User Interface and data visualization capabilities. With the introduction of Adobe Flex integration, we can show “cool” tables, charts and graphics within an SAP application.

.NET based UI scenarios
If the development team is very comfortable using .NET for all application development, then .NET can be the natural choice for custom applications that have integration to SAP.

SAP WebDynpro based scenarios
SAP NetWeaver is a compelling choice for core business processes that fetch majority of their business data from SAP systems. SAP NetWeaver provides WebDynpro for User Interface development, SAP BPM for a guided business process and SAP BRM for business rules. Most of the development is through model based development,  thereby reducing manual coding to the “framework exit points or hooks”. Since SAP NetWeaver has a native integration to all SAP Business Suite systems, this reduces integration effort, increases accuracy of data mapping, and a much higher end user satisfaction with the business process.

Recommendation
As shown above, the use of a development platform is dependant on several criteria, and every customer has unique drivers to opt for a development platform.

In our experience, customers having both SAP and a nonSAP platform (such as Microsoft .NET) have benefited by developing SAP centric business processes on SAP NetWeaver. The SAP and Microsoft platforms co-exist and there is a conscious strategy and decision criteria to determine the use of SAP or Microsoft for a custom application.

 For our customers, this means that Nagarro continues to be a trusted resource for the newest, most advanced tools and technologies available.  We will keep you posted on our work with Google App Engine as it evolves.  If you have any experiences with Google App Engine to share, we would love to hear your comments as well.

Nagarro shares a love of mathematics with customer SIAM (the Society for Industrial and Applied Mathematics).  This year, Nagarro’s custom applications will be automating the judging process for the Moody’s Mega Math Challenge, an Internet-based math contest co-hosted with SIAM. Participating teams of students are given 14 hours to solve an open-ended, realistic, applied math-modeling problem focused on real-world issues.

The content spotlights applied mathematics as a powerful problem-solving tool, a viable and exciting profession, and a vital contributor to advances in an increasingly technical society. Scholarship prizes total $100,000.

If you know a high school math whiz from Maine through Washington, DC who might be interested in participating, please feel free to share information on the contest. Registration closes on February 26, 2010.

Who knows, the winners could be your future employees (or our future customers)…

Often, there are issues related to:
- Having a full time SAP consultant to address a functional area where there are few support requests or development requests
- Having a SAP consultant to manage and juggle many different functional areas. This leads to SLA issues when many issues pile up simultaneously
- Unable to address complex issues due to high costs involved in getting a Senior SAP consultant to the customer location
- Unable to implement advanced SAP modules (such as SAP APO) due to lack of inhouse talent for current and future support needs for these advanced modules

For all the above reasons, the customer organization continues to stick with the current implementation and the older SAP releases, and unable to innovate or implement advanced SAP products.

Remote consulting is catching up with many customers to address all the above issues.

Remote consulting allows you the following benefits:
- Access to advanced SAP skills and SAP talent. It is a fact that many Senior SAP consultants are unable to travel as much as they would like to! Hence, many Senior SAP consultants are constrained to serve their local geographic areas only.
- Cost reduction and control of SAP support budget. Remote consulting is always significantly cheaper than calling the Senior SAP consultant to physically arrive at our offices.
- Access to SAP functional services on Fractional basis. Many SAP service providers offer SAP services at less than full time basis. This is valuable for companies to get support for SAP services that have less support requests.

Today, many customer organizations are becoming receptive to the idea of SAP services delivered remotely. For many large global organizations, they are already geographically distributed over multiple locations and timezones and collaborating over the phone, Webex and VPN. For most organizations, remote consulting fits seamlessly with their current SAP team. Considering the advantages of access to sparse SAP talent and huge cost advantages, SAP remote consulting is the smart way to get access to SAP talent.

In this blog, we would present guidelines and best practices for usage of SAP PI in an Enterprise SOA environment.

Usage of SAP ESR and SAP SR for managing enterprise services:
- SAP ESR and SAP SR should be used for modeling and publishing enterprise services in the landscape.
- SAP ESR and SAP SR are provided by SAP PI and SAP CE infrastructures. Hence, the customer could install either SAP PI or SAP CE to get access to the SAP ESR and SAP SR instances.
- SAP ESR should be used to model and manage all SOA assets. We can use it to model SAP and nonSAP services. SAP ESR models can be used to directly implement the service implementation in SAP. With SAP ESR model, it is not possible to directly implement a .NET or J2EE service. However, the .NET or Java service signature can be made to conform to the Service definition described in SAP ESR.
- Enterprise Services Repository used to define re-useable services rather than classical interfaces
- SAP SR can be used to manage the publication of SAP and nonSAP services.

Usage Scenarios of SAP PI:
- All web services and enterprise services that are being exposed to the customer should go through SAP PI. This is to ensure scalability, security and availability, since going through a managed middleware ensures the quality of service (QoS) for the enterprise service. For example, we cannot expose an application web service directly to a customer in a B2B scenario since the security and load are unpredictable
- SAP PI is used to create abstraction between heterogeneous sender and receiver systems based on SOA standards to provide unified access to legacy systems.
- SAP PI 7.1 is for mainly leveraging functionalities for service enablement, and service and process orchestration
- Integration between non-SAP system and non-SAP system.
- Use SAP PI for both A2A, B2B integrations including EDI
- Use SAP PI for service enabling legacy applications that lack a framework for exposing web services.
- For SAP PI 7.1, many SOA enabling standards or WS standards are supported as part of this release making it the core technology enabler of Enterprise SOA. (e.g. WS Reliable Messaging, WS Security, SAML, WS Policy, WS PolicyAttachment  etc.)
- For a predominantly SAP landscape, SAP PI should be the strategic integration platform and the enterprise wide service bus

When not to use SAP PI in enterprise SOA:
- Recent release of the SAP platform (SAP ECC6) has native capability to expose enterprise services. The service should be modeled in SAP ESR and implemented in SAP ECC6. This service is readily available for consumption across the enterprise without the intervention of SAP PI or any other middleware.
- SAP PI usage is not recommended for synchronous communication since it places a significant load on the infrastructure services for servicing a synchronous request. Also, SAP PI cannot guarantee the QoS and response time in processing a sync request.
- For User Interface scenarios (such as a WebDynpro UI), the UI can directly consume enterprise services. SAP PI should not be used in UI driven scenarios if the backend is exposed as enterprise services.
- If a nonSAP backend such as .NET or J2EE or any other platform is exposing business services in a UI scenario, SAP PI is not needed for intermediation. The UI such as SAP Portal based User Interface through various SAP NetWeaver technologies (SAP CAF, SAP BPM, WebDynpro etc.) can directly consume nonSAP services.

In conclusion, SAP PI should be the strategic integration platform in a SAP centric landscape. However, SAP PI should be used as an intermediary if it adds value between the sender and the receiver entities.

Why is this so? Perhaps the problem is that when you are early in the SDLC, there are so many other challenges and moving parts that you have little time to worry about usability. You worry that bringing the “naïve” users in for design discussions will just derail the project or send it off on a tangent. On the other hand, if you wait till you are through with version 1, you have been compromised as well – it requires great courage to admit at this point that usability is poor and that major elements of the application have to be re-designed.

These are formidable challenges. Yet we at Nagarro recently had a very positive series of usability-related discussions with a major client, which may be useful to recount in this context.

The client is one of the world’s leading travel-related companies. We are building a business-critical application for this company that will be used by a handful of users. The IT project management on the client side had invested a lot of intellectual muscle into the functional and algorithmic design of the application. We too put in a lot of effort to make mockups of the entire application UI and these were approved by the client – but by the IT project management, not by the users. The users did see the application from time to time, but in short bursts and they definitely did not have enough time to play with it and give their feedback.

Then after we had a successful development release and were reviewing progress with senior folks on both sides, we all collectively realized that the usability of the application left quite a bit to be desired. It was a depressing moment.

It wasn’t just a matter of aligning data elements or fiddling with the color palette.

It wasn’t even a matter of trying to streamline workflows to reduce the number of clicks required for each task.

It was the fact that the application’s UI looked like it had been designed by highly analytical engineers and scientists, which it in fact had. Would the users – who were neither engineers nor scientists – find it easy to use? Would they buy into it? Would they find all the features that they would need?

Luckily, the client team comprised very intelligent and wise folks. Rather than blame the team members on either side, the client’s senior VP-level executive said, “We should think of this as continuous improvement. You all did a great job, but we now see it can make it even better. Let’s see this as a positive opportunity and move forward.”

So, no blame game, no recriminations, no requests for “free rework”. The gentleman basically had the wisdom to see that when you are building something highly innovative, you may have to iterate to get the design just right.

Still, on our part, we offered the client a free usability workshop with our best consultant, a person skilled at combining “right brain” creative thinking with the “left brain” analytical thinking required for software design. The users were the star participants in the workshop, which started from first principles – what exactly is it that the users want to achieve? Our engineering team got the chance to put itself into the shoes of the users and try to come up with metaphors and overarching design principles that would work for them and, hopefully, delight them.

The workshop ran for two days and turned our previous thinking on its head. Yet everyone was thrilled with the insights and we agreed we’d run such workshops for each new project. The client agreed to fund a few person-months of effort to upgrade the user interface. Perhaps the cost of the overall development rose by 10%. But as a result, the chances of the application being very successful and useful in the hands of the users probably doubled or tripled.

And that’s always the most important metric!

So far so good. With increasing globalization of the customer base, we are witnessing several movements that are making the ISVs to sit up and take action to preserve their market share. First of all, the customers themselves are getting acquired by larger companies, and “acquiring” globally recognized names. The decision making policies are now getting centralized, and all decisions need to adhere to “global standards” such as compliance of new software products, interoperability, multilingual multicurrency support etc. Additionally, the ISVs are seeing increasing competition from the larger “less specialized” players who are coming up with offerings to compete head-on with the ISVs. Finally, the customers are now less excited about buying “best of breed” applications, and leaning towards solutions from a single product vendor which complies to their global IT standards. For ISVs, deep functionality is not enough to break into larger customers who have a certain gate-criteria for qualifying all software solutions.

SAP is the predominant ERP globally; it is run by a large number of global corporations across Auto, High Tech, CPG, Retail, Chemical, Pharmaceutical, Oil&Gas etc. These global corporations have acquired global footprint across Americas, Europe and Asia with distributed IT management teams. For consistent management, most corporations mandate SAP compliance of all software solutions.

For an ISV looking to break-into the SAP customer base (which consists of a significant number of Fortune 1000 companies), it is almost mandatory to demonstrate SAP compliance. SAP compliance would be one of the earliest concerns to be brought up by the IT during the product evaluation and due diligence phase.

Then why wait ?

SAP compliance can be achieved through the following two approaches:
- Migration of the product’s important features to the SAP platform: Though this may seem like a more difficult approach, it is not. We have helped customers to rapidly build important features of the product using the SAP NetWeaver platform in a short span of a few weeks. The initial product release is adequate to support the ISV Marketing team to start socializing with the SAP customers.
- Integration of the ISV product with SAP: This is a common approach to integrate the ISV product with the SAP business suite. We typically use SAP PI (SAP XI) middleware platform to provide data conversion between the ISV product and SAP proprietary data formats.

Finally, nothing beats SAP Certification! An SAP certified product or SAP certified integration provides assurance to the customer that the deliverable meets SAP’s quality and data interoperability standards.

How do we do it ? ISVs are understandingly not SAP experts. The best way is to partner with an SAP Services provider who has previous experience in working with ISVs and has completed several such projects. The ISV should not need to learn SAP or get into the nitty-gritty of SAP Certification, all these activities are the responsibility of the SAP services provider.

What would be the effort ? As with any IT project, we need to set the boundaries and scope of work to reach the effort and costs. As a rough estimate, an ISV can achieve their initial goals through a 3 month project with a 3 person team; With capable SAP expertise available offshore, the offshore costs should hopefully justify an SAP migration project to make it into your engineering and marketing budgets for the next quarter.

A Bit of Background
The application database is SQL Server 2005, single instance (non replicated, non clustered). The database has patient data (Electronic Medical Records) and the end clients (clinics and hospitals) access this data via web services. The database had around 3 million records, which they expected to scale to around 5 million. The software product vendor was receiving performance related complaints from end customers, and had solicited our help in resolving these issues.

Analysis
The first step was to establish the nature of the performance problems. For this we worked with the business and development teams of ISV to determine the nature of the performance issues. Users had reported performance problems in the following situations:

• Longer than expected response times for transactions or queries.
• Insufficient transaction throughput to complete the required workload.
• Decrease in transaction throughput.

We needed to

• Determine if the performance degrades with increased load or with increase in data in the system.
• Determine the volume of historical data and if they cause long running queries.
• Examine the hardware configuration (RAID, multiple disk controllers) of disk subsystems.
• Take regular measurements of resource utilization and database activity and determine whether there is a disproportionate utilization of CPU, memory, disks or network. System Monitor (aka Performance Monitor) helps to identify potential hardware bottlenecks w.r.t. a few key counters (see http://www.sql-server-performance.com/articles/per/performance_audit_part2_p1.aspx) and potentially obvious performance problems.

The next step was to establish performance objectives. For this we developed a plan for measuring the system performance. The most common metrics used for measuring database application performance are:

• transactions per second (TPS)
• response time

We then needed to define clear objectives for the above metrics. Selecting a tool (we evaluated some of the many available; refer: http://www.testingfaqs.org/t-load.html) that can automate measuring the TPS and/or response times by generating peak / off–peak / average load conditions.

We would regularly monitor the application performance on these metrics to get a feedback on whether the optimization changes are reaping any improvements.

The next post shall cover a discussion of the various solution approaches that could be adopted to improve the database problems once the problems (or at least some pattern that may be imminent in the problem) have been identified. I shall describe the various areas and perspectives of looking for a solution to the database performance problems.

An SAP NetWeaver project would typically consist of identifying new business processes to be implemented, new SAP BW reporting, new data integrations etc. The SAP Netweaver is a platform that provides implementation of new processes and data visualization applications where the SAP ECC is at the backend. Hence, the need for SAP ECC core configurations is minimal.

As with any development project, the initial phase of the project needs close understanding of business requirements. This requires the Business process consultant to personally interact with the business and identify the business requirements through workshops, white boarding and discussions over printouts/charts/sheets etc. This process typically takes 10% of the total project effort for capturing the high level requirements. This phase of the projet has to be conducted onsite to fully interact with the business and understand their true requirements and aspirations.

Thereafter, the next level of details can be arrived at by the remote development team located at the offshore location. This includes creation of mock User Interface and mock process walkthrough to confirm the requirements. This is a partial development activity and can be efficiently conducted offsite. All artifacts can be discussed with the customer team remotely, with the partial assistance of the onsite consultant. This activity may take about 20% of the total project effort.

Thereafter, the design and development phase of the project can be totally conducted offshore. This phase takes 40% of the total project effort and needs access to deep SAP Netweaver expertise. We do not need close co-ordinatation with the business for this phase. Hence, this phase is ideally suited for offshore execution because it needs access to deep SAP Netweaver expertise and can lead to significant cost reductions for the customer.

The object is unit tested, and then provided to the customer for integration testing and acceptance testing. Thereafter, the approved objects can be deployed to production. All these activities take the remaining 30% of the project effort.

As we can observe, the initial phase of capturing high level business requirements needs an onsite presence to determine scope and negotiate with the Business to reach a technically feasable and cost-effective solution. Almost all the remaining activities can be efficiently conducted offsite to leverage the team knowledge and SAP Netweaver expertise available in the offshore team.

In our direct experience, we are observing customers to benefit through SAP NetWeaver offshoring by getting access to Netweaver experts located at offshore locations, and also get cost savings due to lower cost offshore pricing. This lives up to our motto of doing the “right activities at the right location”.

We recently caught up with Chris Roe, CTO of Qtrax, in New York to find out more about his experience as a Nagarro customer. Qtrax and Nagarro have been working around the clock for months preparing for the launch of Qtrax 1.0, the industry’s first free and legal peer-to-peer music download service.  Qtrax had a tall order – linking a massive music file system with ad-serving features while enabling millions of site visitors to rapidly download an unlimited number of high-quality music files.

Oh, and did we mention that they needed it done in just four months? Click on the video to hear how Nagarro rose to the challenge.

The problem of making such general rules about programming practices is that you tend to miss out the benefits by following these rules too literally in every context. Think it on the lines – Reflection or any feature for that matter, if included must have a valid reason for its presence. We should be able to see and know how and when to use it and when specifically to avoid it.

Reflection is slow; but the question is how slow is slow? One should worry about performance but this worry should not keep you away from making best use of a technology. If you look around in .NET world, you will realize that .NET framework itself uses a lot of reflection. Ability to store meta-data about your code and to be able to use it at runtime opens the door to a whole lot of possibilities limited only by your imagination.

If you rethink it it’s just one another way of storing and working with information; difference being that this information is stored in code itself. So instead of retrieving information from some XML file or database, you are pulling it in from the code itself. Therefore your concerns towards writing optimized code while reading data from XML (e.g., load once read many times) or from database (e.g., making fewer trips to database), etc. will not be much different when using reflection. You might want to fetch type info once and hold it for future calls as well instead of fetching it every time you need it.

As in case of database, we know where to use joins, on which columns to create indexes and which where clauses to use to get optimized execution time; in case of reflection too there is a well defined grammar to make optimized reflective calls. For example, using BindingFlags enumeration correctly, not trying to use IgnoreCase flag, and limiting your searches to public or static or instance members as required, will definitely give you best possible results. Joel Pobar’s post talks at length about how reflection works internally and is a good reference to understand how to avoid performance pitfalls.

Don’t just assume that reflection will hit the performance, but use it, test it, tweak it for performance optimization and enjoy the flexibility it provides. If you think your problem statement has a solution in reflection; don’t hesitate in going ahead just because someone says that it will be slow. If this turns out to be slow; there are ways to optimize it too. For instance Jon Skeet’s post talks about using Delegate.CreateDelegate to turn a MethodInfo into a strongly-typed delegate. This can improve performance massively in scenarios where you need to set the same properties multiple times. Essentially this means that a lot of the type checking is done once when you created the delegate, rather than on every invocation.

Let’s explore a scenario where reflection is no harm to use and probably the only way you can achieve the end result with.

You want dynamic extensibility to your desktop application; your revenue model includes selling various groups of features separately. You do not want to build multiple deployment packages and you want to be able to provide even more features to your customers after they have started using your application. How would you do it?

You might do this by developing a plug-in model in your application.

You may define an IMyAppPlugin interface which can be implemented in each of your application’s plug-in assembly implementing a specific feature. At minimum you may have following definition:

public interface IMyAppPlugin
{
    Guid Identifier { get; }
    string Name { get; }
    string Copyright { get; }
    bool Start(IHostApp hostApp);
    void Stop();
    bool Run();
}

You will not have any reference of these assemblies in your project at design time. Its only at runtime when you will try to discover these plug-in assemblies (using some mechanism such as looking in specific folder or registry, an XML file or anything else) and load it using reflection.

You will write code to find and create instance of the class that implements this IMyAppPlugin interface. Once you have this object you can initialize it by calling its Start() method passing your application’s reference as required. You may also write code to add access points (such as menu items) for this dynamic functionality to appear in your application’s user interface when this plug-in is successfully loaded.

A call to Run() method anytime after this will do what is desired.

So essentially what we are doing here is that we have loaded the assembly using reflection and we are calling methods dynamically again using reflection. If we just go by numbers, yes dynamically invoking a method is slow. Referring to Rick Strahl’s post we can say that invoking a method dynamically is about 3.5 to 4 times slower than directly calling of a method. But again the question needs to be asked – how slow is slow? Here we are talking about few milliseconds extra. Unless you are making calls in a loop for a large number of times, does it really matter in the larger perspective of a typical application? Most likely – No!

Remember .Net Framework and especially ASP.Net uses a lot of reflection internally to provide dynamic execution of code and controls.

Talking again about the above scenario, if not reflection, do we have some other relevant approach of dynamically extending the application? You may talk about using COM interfaces to achieve it – but reflection or no reflection – the essence of every approach is to dynamically resolve the address of the method to be called, which is the only major difference wrt static calls where this address is readily available.

This is just one scenario where reflection proves to be the most effective and simplified approach to work with. There are a lot of other scenarios where reflective access to properties, fields and methods provides enormous flexibility that simply wouldn’t be there otherwise.

With Dynamic Language Runtime (DLR) in .NET 4.0 we are heading towards a cleaner reflection. Eventually this may replace reflection how we see it today, due to its simplified code and caching advantages. In-fact as mentioned in this Jim hugunin’s post, it appears that DLR too internally makes good use of reflection.

In short, do not shy away from using reflection, just because you hear warnings about it. Always verify if these warnings apply to your project or scenario. Often they will not. As they say, “trust but verify!

At end there was Q&A session where I ended up raising my hand and asking a question “with so many technology choices available even in .NET for same tasks, the choice picking correct ones becomes tougher, with lack of guidance and gave several examples”. The reply was “That’s a great question” followed by a long answer – explaining the reason why this has happened and a nod of creating more practices and patterns around this topic. Highly encouraged, I ended up asking many more – till I was shut down by moderator! I ended up getting his autograph and a picture together using my low end camera phone – but a moment to remember!

With "The Gu"

A number of companies have had positive experiences with server virtualization over the last couple of years, and are looking to take the next step – virtualizing their desktop systems. Cloud computing, especially storage is finding increased traction in the midsize organizations, as they look to improve business continuity and disaster recovery plans. Another trend seems to be the increased focus on building relationships with external service companies in this market segment, as IT executives seek to do more with less.

With our expertise in software product development, manufacturing and SAP consulting, we knew Nagarro was a great fit for QISoft. But the winning the business was only the beginning. The real success story here is that Nagarro is making an immediate, positive impact on QISoft’s long-term competitive advantage.

So, is cloud computing merely hype? While there truly is value in what is provided by Amazon (EC2 and other services), Google’s App Engine, Microsoft’s Azure and a few others, the providers (or at least their marketing teams), if you ask me, have not helped a whole lot.  It seems like every company that used to provide virtual private servers is now a cloud computing provider. So are companies that used to provide hosting solutions, and every company that used to be a destination for an outsourced data center. Soon, we will have clouds of every other conceivable kind! When a term can be applied to pretty much anything, the term starts losing its meaning. Cloud computing, which I would put in the first stage of Gartner’s Hype-Cycle (hype followed by disillusionment followed by realization), is the latest buzzword that marketing teams love to apply to anything they can. I asked representatives from several companies that claim to provide cloud solutions what did they really mean when they talked about a “cloud”. When pushed, pretty much all of them ended up saying – “its just a buzzword”.

Getting back to the panel discussion, there were several strands of discussion which I would have liked to address in detail but couldn’t because of time constraints. So I decided to use this blog to move the discussion online (and to also share it with a wider audience). In this and subsequent posts, I will try to summarize as well as share my view of this technology, starting with some of the points from the panel.

• Cloud computing is nothing new: Cloud computing, like most other technologies, is an evolution, not a revolution. If it does end up changing the way we develop, deploy and manage software (and other components), we might start calling it a revolution but at this time, it represents incremental but very useful changes in several existing technologies and how it all comes together. Most offerings use virtualization, deliver services over the Internet and expose HTTP based APIs for accessing and controlling the services. But what is truly important for me is not whether its new or old; but whether it solves my technical and business problems. There are several class of applications and organizations for which EC2, Azure or Google App Engine is the right choice.  Moreover, I can not think of  a single service in the late 90s or early 2000s where I could provision Linux boxes in a matter or minutes by just signing up on a website using a credit card.  If nothing else, the sheer ease of use and affordability (at least in the beginning) is an innovation worth taking note of (and make use of if, it makes sense). My take on it – it works great when you use it to solve the problem it was designed to solve.
• There is no such thing as infinite scalability: The use of term “Infinitely Scalable” by the cloud computing service providers, in my opinion, is misguided. A better term to use would be “Sufficiently Scalable” but it does not have the same ring to it. Amazon’s or Google’s infrastructure can effectively provide “sufficient” scalability for small application. If you are mostly running on a couple of Xeons with 12GB of Ram or so, a typical spike in demand might need you to scale up by 4 or 5 times which can easily be handled by the Amazons of this world. I suspect, however, that you can not run another Amazon on Amazon’s infrastructure (especially during Christmas time!). There is a slightly different but related discussion about differences of scale and statistical models which I will not get into.
• What about the security of my data? Its no less secure that using a hosting solution of the past. While people like to complaint about Hypervizor breaches and how most cloud computing providers (especially those who provide infrastructure as a service)  use virtualization, the reality is that Hypervisors are far more secure that the operating systems they run. The two main and mostly related reasons for this is that the people who build Hypervizors realize this (and they are a pretty smart bunch), and that the Hypervizors are kept as small as possible. While I am not an expert in this area, the last time I was talking to one of the experts in Hypervizor and virtual machine security, he took pains to point out that inspite of the fact that it is bad for his business, Hypervizor breaches are extremely rare and that most Hypervizors are quite secure. I would, however, like to point out that what happens in case of a security breach is still not clear. We simply do not know enough about this stuff. And if you have contractual or legal requirements that prohibit you from moving your data into the cloud, you will simply have to wait until the lawyers catch up with the technology and the providers address these concerns directly through their SLAs.
• The clouds are not interoperable. Because every-body’s offering of “cloud” services is different, the questions of interoperability while being a very important question, does not make sense at this time. The really compelling offerings today are from Amazon (EC2 being the big one, but there is also SQS and S3 and a few others), Google (app engine) and Microsoft’s Azure.  EC2 gives you virtual Linux servers (and now windows too) and options of launching pre-configured servers. If all you are truly using is the Linux platform, nothing stops you from moving over to someone else who will also provide you Linux boxes (either virtualized or real). While you may have to make a lot of changes to migrate your Java application over to Google App Engine, taking your application out of the App Engine and deploying it somewhere else should not be a problem (assuming you have appropriately abstracted out the areas that use Google specific APIs). I have heard talks of Azure supporting Java (and a few other platforms) in the future and it will be interesting to see how portable Java applications will be between Google App Engine and Azure.
• What is the real cost of using these services? This is a tough one and merits at least a white paper of its own. I do not have a clear answer but I say this –  two main advantages of the cloud, the ability to scale up when you need to, and pay only for what you use go a long way in making cloud computing solutions cost effective. If you know your usage and it does not vary much, you do not really need to use a cloud computing provider.  There is work needed for moving an existing application either completely or partially into the cloud and you should not, as a matter of basic engineering principle, use a cloud simply because its the in-thing. It will not make sense and it will not be cost effective. You may, however, have other reasons for using Amazon or Google (for example, cloud computing payment models let you convert your capital expenses into operational expenses).

In my next post on cloud computing, I will list some of the prevalent definitions of cloud computing and try to factor out the characteristics that make most sense to me (especially for providers that we are most interested in) and try to come up with a working definition of the term that is most relevant to us as software architects/designers/engineers.

]]> http://www.nagarro.com/blog/cloud-computing-demystified-part-i/feed/ 4 http://www.nagarro.com/blog/a-better-way-to-test-email-sending-functionality-is-to-use-your-own-smtp-server/ http://www.nagarro.com/blog/a-better-way-to-test-email-sending-functionality-is-to-use-your-own-smtp-server/#comments Mon, 29 Jun 2009 07:09:14 +0000 Vaibhav Gadodia http://www.nagarro.com/blog/a-better-way-to-test-email-sending-functionality-is-to-use-your-own-smtp-server/ A large number of applications today have some kind of a notification mechanism built-in. More often than not, an email is used to send these notifications. Typically, while developing these applications, developers use their own inbox to send notifications to. Or they setup some dummy account to send test emails to; or sometimes they might send test mails to real users (it’s been known to happen).

There are many limitations to this approach:

• one has to deal with dummy messages arriving in the inbox, when all that you want to test is your email sending code.
• automating the testing of the email sending code will result in a large number of emails being generated and delivered to your inbox.
• code that is supposed to deliver large amounts of email will generate a lot of load on the Internet bandwidth (and yes that can be a problem).
• if you are testing on a copy of a production database, you might inadvertently end up sending test emails to real users.

There are other issues present as well. One solution to solving these problems is to use your own SMTP server to handle all these emails in a manner that it receives an email and doesn’t sent it forward. So, it is like a dummy SMTP server. But setting up an SMTP server is usually a hassle.

There are plenty of options out there which make your job much easier than you think it is. In this post, I am going to list some solutions which can be used (though there are many more). The basic premise behind this approach is to run a local SMTP server (either on the developer box, or the test server, or both, or somewhere on the LAN). This local SMTP server address is what is configured as the server address in your application which is being tested. Any mail sent to such a server is kept with the server and not forwarded, thus preventing all the above listed issues.

Following are some of the more popular out-of-the-box programs that can be used (click on the names to go to their websites):

• Papercut - this is a simple windows application which will run in your System Tray and simulate an SMTP server; it receives and stores any messages it receives. It allows you to view the messages. It sits in your system tray and alerts you whenever a new message arrives.
• Neptune – this is another application which will do a similar task. It is however a little more advanced. It has an admin API interface which allows you to run custom queries to analyze the messages that it processes. It doesn’t have a UI through which you can view the received message though – but sometimes all you need to check is if the messages are being delivered. Very useful in a bulk messaging scenario.
• smtp4dev – this is another option similar to Papercut, which again sits in the system tray and will let you know whenever any message arrives and will let you look at the message details.

There are many more out there which can be used to do similar things with some features here and there (for example, Neptune doesn’t generate all SMTP responses). Here’s a post by Phil Haack from long ago which is also interesting: A testing mail server for unit testing email functionality.

So, when you feel the need to test your email sending code the next time, it might make sense to have one of the tools (or something similar) available.

Get a look behind the scenes at Pacific Southwest Container, a midsize manufacturer of sustainable custom packaging. Nagarro’s custom applications enable PSC to optimize use of printing materials and equipment, reducing PSC’s costs and waste. End result? Cost savings are passed along to customers, while PSC improves its environmental footprint. Please view this short video and let us know what you think!

Per Wikipedia  

With ActiveRecord, a database table or view is wrapped into a class, thus an object instance is tied to a single row in the table. After creation of an object, a new row is added to the table upon save. Any object loaded gets its information from the database; when an object is updated, the corresponding row in the table is also updated. The wrapper class implements accessor methods or properties for each column in the table or view.

This pattern is often implemented by ORM and Object Persistence Tools like Hibernate (Java) and ActiveRecord (ruby). The implementation framework handles generating the necessary SQL appropriate for the database being used and executing them over a database connection.

Recently, in a couple of projects I used ActiveRecord Migrations to manage database setup and migrations.
Below, I shall present a walk though the steps that I followed to get a set of ActiveRecord migrations working on my Ubuntu-9.0.4 desktop.

1. Check if you have ruby installed:

     ruby -v

You’d see something like the following: ruby 1.8.7 (2008-08-11 patchlevel 72) [i486-linux]

If not, then install ruby by executing the following from a terminal window:

     sudo aptitude install ruby

Also, install rake, – ruby’s build program with capabilities similar to make using:

    sudo aptitude install rake

2. Install rubygems:

    sudo aptitude install rubygems1.9

3. Install the following ruby-gems

• ActiveRecord: sudo gem install activerecord
• Database adapter: I used the MySQL database; so I needed to install the mysql gem  sudo gem install mysql
  
  The above command results in the following error:
  
    extconf.rb:8:in `require’: no such file to load — mkmf (LoadError)

Fix: Being a Debian or Ubuntu user you’ll find that the Ruby standard distribution is split into lots of little packages in order to comply with Debian packaging Guidelines. Installing ruby only gives you the ruby binary and a subset of the libraries for Ruby. You’ll need to add more packages if you want to utilize more of Ruby’s standard library.

Hence, we need to install the ruby-dev package prior to installing the mysql gem.

  sudo aptitude install ruby-dev

4. Create directories to hold the migration definitions:

  mkdir ~/workspace/db_setup
  mkdir ~/workspace/db_setup/migrations

That’s all on the installation side. Now it’s time to write the migrations.

With ActiveRecord, each change to the database is termed as a migration. Migrations derive from ActiveRecord::Migration and enable a developer to generate a database structure using a series of Ruby script files (each of which is an individual migration) to define database operations. It’s a database-agnostic representation of the database. It uses generic data types for columns, like :binary, :string, :text, :boolean, etc. to define the kind of data to be stored in a column. A sample migration would look like the following:

File: migrations/001_create_institutions.rb
class CreateInstitutions < ActiveRecord::Migration
def self.up
create_table :institutions, {:id => true} do |inst|
inst.column :name, :string, :limit => 100, :null => false
inst.column :url, :string, :null => false
end

execute(’ALTER TABLE institutions ADD UNIQUE (name) ‘);

end

def self.down
drop_table : institutions
end
end

• The file-naming convention is important here:
  • ActiveRecord uses the numeric prefix to determine which migrations it needs to apply. To do this, it creates a table called as SCHEMA_MIGRATIONS in the database in which the VERSION column holds the number from the last migration added. When the migrations are run without a target version specified, it will only execute the migrations that have number greater than that stored in the database, lowest numbered first.
  • The class name should correspond to a camel case representation of the migration’s file name.
  • Typically your project would have a collection of these migrations, in files with prefixes as 001_xx, 002_xx…. and so on, each defining a change desired on the database.
• The two methods up and down should be functional complements of each other. The up method is executed when upgrading the database while down method is executed while reverting to an earlier version.
• You can do many things in a migration – create and delete tables, add and remove columns, set column constraints (such as nullability and column length) and also run code to initialize the data in a table.
• As a convenience, you also have the execute method available (shown as the last statement in the up method) to run SQL / commands directly. This has many implications: it lets you execute commands that use specific database dialects, like adding database constraints. The following script allows you to execute SQL commands from an external file, say create_db.sql:
  
    sqlCode = File.open(File.dirname(__FILE__) + “/create_db.sql”).read
    sqlCode.split(”;\n”).each do |stmt|
     execute stmt if stmt =~ /\S/
    end

ActiveRecord can read the database configuration from a YAML config file called as database.yml (place it under ~/workspace/db_setup) that looks like:

# The # character signifies a comment.
#
# Define db config per environment here.
# yml files are formatted in a very strict way; you cannot insert spaces and tabs randomly.
# Typically they use two (2) spaces to indent options.
#
# The space between the ‘:’ and the configuration values is required.
dev:

adapter: mysql
database: $DB_NAME
username: $USER
password: $PASSWORD
host: $HOST

A set of ActiveRecord migrations can be executed as
1. A part of a Rails application by invoking

  rake db:migrate RAILS_ENV=xx VERSION=yy

2. A standalone Ruby application: If it is not a part of a Rails application, there is no default Rakefile to use. So we’d need to create one (under ~/workspace/db_setup) with the following content:

require ‘rubygems’
require ‘active_record’
require ‘yaml’
task :default => :migrate

# Enable diagnostic logging to help debugging.
# Note that ActiveRecord colorizes output for viewing in a terminal.
# But when viewed as a normal file, it isn’t readable. So we disable it.
ActiveRecord::Base.colorize_logging = false
logFile = File.open(’database.log’, ‘w’) # set it to ‘STDERR’ to enable logging to Std. ERROR console
ActiveRecord::Base.logger = Logger.new(logFile)
end

To execute the migrations, run the following command from the directory containing the RakeFile:

  rake RAILS_ENV=xx VERSION=yy

Both the above mentioned ways lend themselves to be easily invoked from a multitude of build and CI tools, like ANT, CruiseControl, python scripts etc.

Thus we, saw how to configure ActiveRecord as a database agnostic mechanism of managing database setup and incremental migrations. It is relatively simple to use and highly extensible.

Although, some may argue that ActiveRecord leads to a high degree of coupling between application code and database structure; in many cases the issues due to this coupling is much easier to manage than to adopt an alternate complex solution.

Recently, we had the opportunity to give our opinion on the latest avatar of the Ado.Net Entity Framework to Jeffrey Schwartz from Application Development Trends magazine. Jeff was kind enough to publish our opinions in an article on the Visual Studio Magazine (thank you Jeff). The article is titled “Will Entity Framework 2 appease early adopters of LINQ to SQL?“. We, as can be deduced, were early adopters of LINQ (to SQL and others).

If you would like to see that question answered, I urge you to go and read that article. In this article, I wanted to discuss a little bit more about how we look at Entity Framework v2.

When Microsoft first announced the Ado.Net Entity Framework, we were unsure about what to do with it. We had already started training our developers on LINQ to SQL, and some of our projects were already using the technology. We found it (L2S) to be a very good way to implement the ORM layer of an application (and that worked very well in most cases). Additionally, we weren’t sure how mature the framework was. Sure, it looked perfect for some of our larger projects, but somehow it wasn’t fitting the bill (and we were not alone). And compared to LINQ to SQL, it felt quite lackluster in features.

The new version of the Entity Framework looks much better. It almost feels as if Microsoft went out of their way to ensure that Ado.Net EFv2 can stand on its own when compared to LINQ to SQL. However, by this time, we are pretty invested in LINQ to SQL, so there is no question of moving any of our existing projects over. But, we are very keen to try out the EFv2 in newer projects and internal applications. Specifically, we will looking at integrating it the next version of our Nagarro .Net Application Framework (which is a base framework that we use for most of our projects).

“Are we there yet?” – I would say, almost. The latest version of the Entity Framework looks a much more complete product as compared to the earlier version. Some of the features from an implementation-as-a-tool-in-a-software-development-services-shop perspective are really great:

• POCO support - the support for persistence ignorant objects make a great deal of difference because we can continue to leverage the features of the Entity Framework while using your own custom data layer logic alongside it. This is beneficial because this means we can at least consider using the EF in existing applications where the data model and the layers are already defined.
• First class code generation – this will come as a relief to most developers. The code generation engine, T4, is much more versatile that what we used to have earlier. This means that each application development team can customize the code generation a little bit according to their specific needs.

And of course there are hosts of improvements in general and there are tons of articles covering those.

From Nagarro’s perspective, I think we are almost there.

After initial analysis, it was clear that we needed to incorporate AJAX in multiple areas across the application. It was further decided to use one of the existing AJAX frameworks rather than writing AJAX “by hand”. After evaluating several AJAX frameworks, we selected GWT. Other AJAX frameworks like DOJO and YUI library are also good but they did not solve the above mentioned problems arising from a fat architecture and a complicated technology stack.

One of the primary reasons for selecting GWT was its unique approach to developing the complete application: both the client side and the server side components using Java. GUI development with GWT APIs is similar to Swing development and it is refreshing to forget JSP, JavaScript, and Struts and write everything using Java. I am very happy with experience so far and we have vastly improved presentation layer while retaining most of the service layer code.

With GWT, the AJAX front-end is written in Java that gets compiled into highly optimized Java Script. Developers do not need to write Java Script while end users are provided with dynamic and standards-compliant AJAX experience. JavaScript gets downloaded by the client browser at runtime where most of the client side processing is performed thus releasing the server side from handling presentation tasks.

I am impressed by the developer friendly features of GWT. Using Java in the presentation layer facilitates the use of design patterns and creation of reusable widget libraries. GWT’s approach to history management, internationalization, and service layer interaction is easy to understand and implement. I felt spoilt by the choice of available tools, default support for multiple browsers and ability to debug AJAX in hosted mode. I also noticed ancillary benefits like elimination of training on multiple technologies, reduced dependence on technology experts, and the overall speed of development.

I have heard arguments about Flex being a better RIA choice than GWT. I think this may be because some people may not be aware of other GWT widget libraries (like GWT-EXT) available today. Although GWT comes with a very good collection of useful and cool widgets but you can always use third party widget libraries which provide additional AJAX components like a special Tree or Grid component. We used GWT-EXT, a very rich widget library to achieve the desired functionality.

I found a few features missing – for example, there is no inbuilt support to load client side libraries on demand. Actually GWT loads all client-side libraries in client’s browser when the end-user accesses any module of the web application for the first time. So I think GWT may be a misfit for large enterprise applications. Also I faced problems due to lack of emulators for some of the Java classes like Calendar and our application POJOs contain Calendar objects. The good thing is that there are workarounds available to solve these problems.

I strongly recommend GWT if you are developing a mid size web application that needs enhanced usability and a desktop like look and feel.

While at the show, I had the opportunity to participate in a panel on social networking. Our discussion was an important reminder that the idea of “business to business” is giving way to “business to community.” In our constant endeavor to make our clients more competitive, we have been working closely with them to understand how social media impacts their organizations. I am constantly amazed by the possibilities of building custom applications using the Web 2.0 APIs to leverage this new medium in departments ranging from sales, marketing, customer support, product development, design innovation, training, and education.

Social Networking Panel at the Midsize Enterprise Summit

And last week, the NASSCOM Emerge blog examined Nagarro’s intellectually vigorous culture in a post about the NASSCOM Emerging 15 (I like It Here, Do You?).Nagarro stands out from the crowd as “a company where nerds love to work” as NASSCOM discusses its picks for the best small and medium sized IT-BPO companies to work for.

NASSCOM characterizes the top 15 emerging companies as “sought after by career seekers for their close knit, non-hierarchical work environment, culture of innovation and commitment to employee empowerment.” If this is the definition of a nerd-magnet, sign us up.

In today’s dynamic business environment, IT departments need to remain agile so they can align IT strategy with shifting business,market, competitive and economic conditions.  Outsourcing is one way to adapt resources to suit an ever-changing landscape but can also introduce risks if the right partner is not engaged or the partnership is mismanaged.

Companies rely on SAP deployments to run their core business operations.  As a result, outsourcing decisions must contemplate security controls and allow the IT department to maintain operational control.

In an effort to help you identify and manage risks, Nagarro’s new white paper addresses some common outsourcing concerns and objectives, and provides practical information that can help you more effectively address your current and future business requirements.

A quick  review of event processing in Windows Application tells us that there are 3 different types:

1. Quick Processing (less than a second) – Action performed is executed almost instantaneously without any significant wait period. In this case, you would generally not display any indication to the user about the background task execution. For instance, Open a dialog on the Click of a button.
2. Short Processing (1 second – 5 seconds) – User action consumed a few seconds in execution. In such cases, recommended practice is to show a wait cursor. For instance, Pick records from database to list them in the application screen requested by the user.
3. Long Processing (more than 5 seconds) - Actions which involve a significantly longer processing time, in such cases it is recommended to show progress bar indication to users.

In order to display a wait cursor (category 2 above) the following is the most common block of code seen in most applications:

public object GetData(int id)
{
    Cursor.Current = Cursors.WaitCursor;
    try
    {
        // Processing code here    
    }
    finally
    {
        Cursor.Current = Cursors.Default;
    }
}

The code piece above will have to be duplicated at all the places where we want to use wait cursor during processing, this would be a pretty tedious task. Now, let’s have a look at another approach to implement the same feature:

using System;
using System.Windows.Forms;

public class WaitCursor : IDisposable
{
    private Cursor cursor;

    public WaitCursor()
    {
        cursor = Cursor.Current;
        Cursor.Current = Cursors.WaitCursor;
    }

    public void Dispose()
    {
        Cursor.Current = cursor;
    }
}

The WaitCursor class above implements IDisposable interface, hence you may use this is the using block. Now see how simple it is to add a wait cursor to your application:

public object GetData(int id)
{
    using (new WaitCursor())
    {
        // Processing code here
    }
}

There are some very obvious benefits by doing it this way:

• It simplifies the code at several places into a single line statement.
• More importantly, we just ensured that a bug where some developer forgot to restore the cursor to normal can never happen (as long as we have trained our team to simply call the above code).
• Finally, it is always a best practice to encapsulate repeated code in one place.

One should always be on the lookout for such small coding patterns and implement them.

The first and most obvious change will be that programmers will start using many more threads and use current lock based approaches to synchronize access to shared data. Now detecting bugs in multithreaded programs is extremely difficult and as we write more concurrent programs, it will become increasingly important to have tools which can check our programs for correctness. Two such tools are CHESS from Microsoft research and jPredictor from UIUC. Both are still research projects but are definitely worth taking a look at and playing around with.

The transactional memory research community advocates getting rid of lock based approach for managing access to shared data. The idea comes from the database world where transactions have been around for a long time. Hardware transactional memory, hybrid transactional memory and software transactional memory, the three broad categories in which implementations are often split, are all under active research but there already are quite a few implementations out there for you to play around with. One of the core ideas behind using transactional memory for accessing shared data is to allow users to specify “what” needs to be synchronized and then let the machine take care of the “how” part (compare this to how you use locks today). This is supposed to greatly simplify the concurrent programming model but it is too early to say if it really does make the programming model simpler.

While  I cannot cover or summarize the work being done on transactional memory without making it into a small project, I can provide a few starting points that you may find interesting. Microsoft is doing very interesting work in this area and while a lot of it uses Haskel, there is a C-sharp API for software transactional memory that you should look at. If you are a Java programmer, you should take a look at DSTM2. And irrespective of what technology you use for building systems, I will recommend reading publications on transactional memory by Maurice Herlihy, Simon Peyton-Jones, Victor Luchangco and Mark Moir (there are others, I just picked up the four I see often in the literature).

Finally, we should not ignore the world of message passing. OpenMP and Erlang have been around for a long time and are designed for concurrent programming. OpenMP is an API you can use from C/C++ and FORTRAN to write parallel programs to take advantage of shared-memory architectures.  Erlang is a programming language in which you write processes that communicate exclusively using messages and if you write your Erlang programs correctly, you get the ability to scale to hundreds of processors for free.

Recently, I evaluated SharedCache, which according to their website is a high performance distributed and replication cache system built for .Net applications running in server farms. The system is made available to .Net applications through a Windows service.

 Some of the key features are listed of SharedCache are listed below:

• Works for both web and non-web solutions.
• Is extremely fast for fetch.
• It supports multiple topologies: distributed partitioned caching, replicated caching, and single instance caching. You can choose which you need based on your requirements (for example, you may want to choose replicated caching over single caching to ensure that you don’t have any single points of failure)
• In case of Caching, one of the most important aspect is purging. With SharedCache, you have a choice to make amongst the several that are supported:
  • Cache Item Priority
  • Least Recent Used Item (LRU)
  • Least Frequently Used Item (LFU)
  • Time based
  • Biggest Latency First (BLF)
  • Lowest Latency First (LLF)
  • Hybrid – Creating an index based on various item attributes such as: time in cache, Amount of requests, Size, Cache Life Time, etc.
• Configurable maximum cache size and the load factor when the system starts to purge cached items.

You can install the service by downloading it from CodePlex. And several samples and discussions are available at the author’s blog. The best thing about this service is that there is plenty of documentation to guide you.

Just try it out. Its pretty cool.

Today we announced that Nagarro has signed on more than two dozen new customers in the US and Europe in the past three quarters. And we continue to sign new deals almost every week. Like our customers, we at Nagarro are looking at the downturn as an opportunity to keep our focus sharp and our resources first-rate. Obviously we are glad to be maintaining a steady presence despite the ups and downs of the market, but more importantly, today’s news sends a strong signal that at the high end of the technology market, innovation and competition are alive and well.

One method to log exceptions is to log them in the cloud. And there exists a useful service which makes your job easier.

A service in the cloud represents a service that is hosted on the Internet, available to you through an API, so that you have no idea how it is physically implemented. This makes things simple for you since you don’t have to worry about writing implementation code for the exception logging and reporting framework. You only need to call it.

The service that I am discussing is called HopToad. The service offers a simple to use API which allow you to log exceptions from your application. The exceptions are stored in the cloud and are available to you from a central web-based console. All the logged exceptions are presented in easy to understand and navigable way. For example, you can see which errors are occurring most often, or which one is the latest error that came up and from where. You can look at error details and stack traces.

The service developed to be used from within RubyOnRails applications. However, they have an open API, and can be accessed through any technology stack. K. Robertson has made a .Net wrapper available for logging to HopToad. It’s called HopSharp. It is fairly straight forward to use (just drop the DLL in your application and you are good to go). I can imagine that it would be straight forward to write a Log4Net extension for this service as well.

When to use this?

So, when would you use something like this. While the final decision to use this would vary from project to project and client to client, I can think of the following scenarios:

• The application is a work in progress and you are expected to release a number of updates. Typically, in such scenarios, a lot of testing happens while the application is in deployed state. A logging framework such as this will allow you to monitor the application as it is being used, and priorities on bugs can be assigned based on frequency and severity.
• You have an application deployed at multiple locations and you would like to see a consolidated view of the exceptions being reported.
• As a way to save time and effort. This is a ready made framework where you don’t need to impart any effort for implementing an exception logging and notification system.

I am pretty sure that there are other such services out there.

Yet I just can’t stay away, and neither can you if you are still reading this.  So let me jump right in with my take…

Ruby Doesn’t Scale.  It can’t take you to the same places Java can.

I don’t mean performance-wise.  And I don’t necessarily believe it can’t scale complexity-wise.  I just think it won’t scale with the size of a team.  I can’t see an application equivalent to 2 million lines of Java code with 80 developers written in Ruby.  Here are some reasons why:

• the language and tools don’t facilitate writing APIs
• it’s too hard to protect code from the side effects of other unrelated subsystems
• the language and tools do not support for components or version management (see my posts on OSGi)
• one bad Ruby developer can do a lot more damage than one bad Java developer
• there are a lot of bad Ruby developers (I am one!)
• the tests don’t scale–no Ruby app can survive inadequate tests, and the while the amount of tests grows linearly with the size of the application, the cost of maintaining them grows exponentially.  This may be true of any tests but…
• you need more tests with Ruby–a Java app can survive without a complete unit test suite.  In fact, some large Java app can survive without tests at all.  Call me crazy but I’ve seen it.  What did we do before JUnit, after all?  I don’t think I’d want to work on one of those apps, but the point is, test code for Ruby is as essential as an interpreter to make a decent sized Ruby app work.
• there’s no meta-programming in Java; that’s kind of a bummer, but it also means no massive indecipherable frameworks collapsing under the weight of the author’s cleverness, frameworks which are so unfathomable they are as brittle as COBOL.

There, I said it.  Now that I did, I’ll make a couple of other observations about why I love Ruby anyway:

• A two million line Java program probably translates to 100K lines of Ruby code
• 80 Java developers are probably as productive as 20 equivalently skilled Ruby developers
• With only a couple of developers, I think there’s no limit to how big the application can be.  In fact, the language is much better suited for a very small team than Java because you can be so much more efficient if you know what you’re doing.
• The tools for Ruby are way behind Java, and there’s no reason to think they won’t catch up.  When they do, it’s entirely possible the gulf between the scalability of the languages will close substantially.  I certainly hope they will.  As far as the tools for writing Ruby go, I feel like I stepped back into 2002 when I went from Java/Eclipse to Ruby/{NetBeans/RadRails/TextMate}.  rake ~ make: when was the last time I typed in make when writing Java apps?
• The tools may be behind, but the available libraries, plugins and gems are already very rich.
• Ruby helped me remember why I love to write programs

The last point I threw in there apropos of nothing, other than to point out that while I do miss some things about writing Java apps, I’m happy to be writing in Ruby now, and hope to keep working on it for the indefinite future, or until our app gets over 100,000 lines of Ruby which should be… never.

P.S. Here’s my favorite entry in the Ruby vs. Java debate, brought to you courtesty of RailsEnvy.