Nagarro is proud to be listed alongside some of the industry’s most highly regarded outsourcing firms. We fared well in evaluations of our HR practices, management team strength and corporate social responsibility programs. But ultimately, we took the greatest pride in being able to provide objective third-party validation and glowing customer references. As we add this award to Nagarro’s accolades, we are once again grateful to all the customers, partners and employees who have helped Nagarro continue to achieve success.

As part of a company-wide upgrade to the SAP platform, our SAP Consulting team is using the SAP Netweaver ™ platform, SAP IS-Banking and numerous other SAP technologies to help catapult Patrimonio Hipotecaria to the next level of success. Our team will create a 12-fold increase in overall system capacity for Patrimonio Hipotecaria –while improving the consumer experience as well as streamlining internal efficiencies.

You can read more about this new customer engagement in our recent press release and new customer case study.

Since a very compelling business reason for using cloud computing is the ability to scale on demand and when needed, using the best practices and guidelines for designing large scale  distributed systems is a good starting point. While academia has a preference for   distributed transactions, ACID properties, and all kinds of consensus protocols, you are better off with loosely coupled systems where nodes will eventually reach a consistent state (as opposed to waiting for all nodes in a very large system to  reach a consistent state before proceeding). Considering the state of existing systems, loosely coupled self healing self regulating systems should be preferred over system with exotic distributed transactional properties and consensus protocols.

Be prepared to lose computational appliances. As you already know, stateful systems are bad for scalability. Avoid storing the state in server sessions. If you are designing a client-server system from scratch, do not store client sessions on the server. For typical web applications, this means that your HTTP calls should be stateless. GWT and other rich AJAX or Flex clients let you do this relatively easily.

You should also try to avoid solutions that try to replicate state across multiple nodes. This almost always leads to overly complicated solutions. You are better off being prepared for  loss of data and designing with the assumption  that client application state, if at all available on the server side, may not  be synchronized across all nodes. UDP, which is often used for discovery, and negotiating other information may not even be available on many cloud platforms.

These are important design decisions that you must make right up front.  Retrofitting these into existing solution might be impossible.

Software engineering

In addition to strictly technical design and runtime issues, you also need to take into account other software engineering issues which affect you development.

As already discussed, you need to have a pool of engineers who understand the technologies you will be using. You may find it hard to find many engineers who have had experience with large scale, self healing, self managing systems. While one of the objectives of most cloud platforms is to take these complexities away from the developers, you will need engineers who understand these concerns.

You also need to find out if the platform you have selected provides good tools for development, deployment and debugging. You should ask the same questions you ask when selecting any technology.

• Find out if the platform integrates with the IDE you are planning to use (or if the platform comes with an IDE that will serve your development needs).  I would recommend staying away from custom IDEs. Amazon, Google and Microsoft support industry standard IDEs for their respective platforms.
• You need to design for testability right from day one and your design should be testable on the platform.
• If your organization has an enterprise architecture, you need to be sure that your design and tools meet the standards and approaches laid down by it. Different network topologies, moving data in and out of the cloud and integrating systems running in the cloud with the systems inside the enterprise will come with its own set of challenges. While most enterprises have already started looking into cloud as a viable means for extending their computational needs, they may not be ready for a complete adoption.
• Security is an important aspect that you may not have full visibility into when you are running on someone else’s platform. Work with your security experts, quality assurance and compliance team to make sure everyone understands the security risks. A perceived security risk and lack of visibility could be a major problem while adopting any new technology (anyone remember trying to use Java RMI across enterprise firewalls?) and clouds are no exception. In fact the shared nature of the services makes security a bigger concern.
• Finally, you need to design for monitoring. It is best if the platform supports automated monitoring. Otherwise you may need to design for it right from day one. Even when the platform provides some tools, you may still have to monitors aspects not supported by the platform tools.

This discussing should provide you with a good starting point for creating your own best practices and guidelines. These are important design decisions that you should try to get right  up front.  Retrofitting these into existing solution might be impossible.

What is SOA

Service Oriented Architecture (SOA) is a software design strategy in which there is a service provider and many service consumers. SOA is similar to the distributed computing principles of loosely coupled architecture. SOA differs from the traditional client server architecture in terms of distributing the functionality over separate business objects exposed through service interfaces, loose coupling through independent business objects, durable interface contracts and promoting reuse of functionality through a loosely coupled architecture.

What is a Web Service

A web service is an interface implementing the industry standards of SOAP and WSDL (and usually HTTP) to expose functionality to a web service consumer. It is a technical implementation to expose functionality through a standards based protocol.

The use of web services has enabled a tremendous leap in enterprise integration. Till a few years back, most applications provided proprietary or rudimentary methods of integration, such as SAP provided SAP RFC, BAPI, IDOC based integrations. Such integrations warranted the use of middleware adapters to provide integration with SAP.

The use of web services now allows a direct integration between applications with the SAP business suite.

Hence, we recommend the usage of web services as a central integration strategy for all integration initiatives.

With the proliferation of web services, it is a challenge to catalog and manage the usage of services in the enterprise. Hence, SAP has provided a robust set of SOA toolsets to manage the usage of services in the enterprise.

What is an SAP Enterprise Service

An SAP Enterprise service is technically a web service, and it has the following attributes:

• Based on web services standards of WSDL and SOAP
• Based on SAP global data types
• The service has been modeled within SAP ESR using business objects, process components and the SAP enterprise model
• Published in the SAP SR (service registry)
• Guarantee on the availability and functional correctness

What if my Web Service does not qualify to be an Enterprise Service

There are scenarios where the web service interface exposed by an enterprise application is not globally relevant, and is really a technical interface to a localized business functionality. The usage of such a web service may be limited to the department in the organization and its particular IT applications having its own semantics.

Such web services are not Enterprise Services. However, SAP SOA tools can be used to manage and catalog such localized and non enterprise services as well. The SAP SR provides rich taxonomy capabilities to catalog and manage all such services.

Summary

Web services should be the central integration strategy for all integration initiatives to provide a standards based integration to applications. However, not all web services are meant to be used at an enterprise level across the organization.

The SOA Governance body should approve all enterprise service candidates based on the above mentioned criteria, and use the SAP SOA toolset for cataloging and managing all service in the enterprise.

You need to consider how your system integrates with other systems in your organization. Clouds are good but one-off systems that use cloud but do not integrate with existing systems will diminish the payoffs of using a cloud.

You need to consider how your system and its data get integrated with existing data. This may need to be done both ways. Without access to existing data or data created and maintained by your cloud based system, you will again limit the potential of your system under design.

You may also need to integrate your application with your enterprise’s existing authentication and authorization solution.

Even if integration is not an immediate concern, an architectural choice can make it impossible to integrate even in future. You will need to worry about different networks and firewalls and if the protocols you will use to integrate these systems will work across these networks and firewalls.

Development Considerations

Once you start building the system, you will need to take into account how many engineers and support people are comfortable with the technologies you are using (in this case – something cloud based). You may need to take training and documentation time into account in your plans.

If you need to move large amounts of data between your internal computers and the cloud, you will need to either have sufficiently fast network connections or take the associated delays into account. This delay could play a major role in how you trouble-shoot issues, provide other kinds of support, take backup or just make the data collected by your application available to the users within your internal network.

Offshore development is a reality and if your application is either developed and/or maintained by a team working offshore, network speeds,  latency and round trip times will all come to play a major role. This means your choice of development and support tools and environments should work well over a bad network. VNC and Xserver-clients do not behave well, text terminals work better, light-weight HTML based clients work best. You also need to worry about moving large amounts of data in and out of the cloud from your development and support centers.

If you have specific regulatory compliance requirements, you need to be sure that the platform complies with the standard. For example, Amazon EC2 instances are not PCI (payment card industry) compliant so you should not plan on storing credit card information on Amazon EC2 instances. There could be regulations  regarding sending customer data out of country and when you store data in a cloud,  you (or even the provider) may have no control on where (which node) that data  could be at any given time.

I am not a big fan of using definitions as a starting point.  In technology many useful and successful concepts have escaped definition. People still argue about what exactly is an object in an object-oriented system.  There is no scarcity of definitions for software architecture. The internet cannot be pinned down to a simple definition. A definition, however, it is still useful in certain contexts.

While people might argue about the exact definition of cloud computing (which is probably a futile effort at this time), a working definition can be used to evolve a shared meaning. This is especially useful in software architecture where clear communication among stakeholders is extremely important. Software architects interact closely with business, users, developers, testers, compliance groups and others and before you move your architecture to the cloud, you should state exactly what cloud computing means  in the context of the system you are designing.

Definitions will differ based on whether you are a consumer of the cloud based  services or a provider and with more and more of private and hybrid clouds out there,  organizations and enterprises are fast becoming internal providers of cloud based  computing services which is blurring the line between providers and consumers of a  cloud “style” service.  Today, from a consumer’s point of view, a cloud based compute service:

• is implemented as a pool of servers though the consumer may not have direct access to the underlying pool.
• is characterized by very high degree of automation typically providing both  programmatic access (for integration) and graphical or command line user interfaces.
• provides an abstraction of compute services along with constraints that enable  the provider’s offering.
• is priced on a pay-as-you-go basis. Consumers only pay for what they use.

When you take these along with (effective) infinite scalability and pay-per-use utility model into consideration and design a system for the cloud, there are several issues that you should take  into account. I will discuss a few architectural considerations that must be taken into account while designing a cloud based system. The usual design principles and best practices for building a cloud based are not very different from what you probably already use as part of your enterprise architecture standards and/or other best practices for building large scale distributed systems; but some of these may be unique.

Going forward, SAP customers are tending to standardize on the SAP platform for running their core business operations. However, for development of custom applications, customers do not have a clear approach towards selecting SAP NetWeaver or the other mainstream platforms for development.

UI Technology Choice Criteria
Most enterprise vendors such as SAP, Oracle, Microsoft and IBM offer a complete technology stack with comparable features such as Portal, content management, knowledge management, enterprise search, data warehousing, middleware integration, mobility, application server and solution management tools.

A custom application is usually not isolated. In our opinion, the decision to select a development platform for custom applications should be determined based on the following criteria:

• Developer Skills: If your development team is most comfortable with SAP ABAP (rather than Java), then it would enable you to develop custom applications faster, accurate and perfectly aligned with business expectations. If your development team does not have SAP skills but Microsoft dotNET or Java, then it may be rational to develop custom applications on the Microsoft platform with integration to SAP.
• IT comfort to support the development platform: If your SAP IT team is not comfortable with supporting the SAP J2EE application server (for maintenance, patching, upgrade, transports, security), then it may make sense to stick with the SAP ABAP development platform.
• Depth of SAP Integration and SAP Business Processes: An SAP platform based development would accelerate development if there is significant SAP integration and business process interaction involved. A custom application is usually not isolated. If the majority of business data is resident in SAP, then an SAP platform based development would be faster and more accurate to end user requirements.
• Comfort of end users: This is an important criteria in determining the technology for end user applications. For example, if the end user has a strong liking for Adobe Flex based reports or dashboards, then, getting a similar look-and-feel on other platforms may be a challenge.

It should be noted that SAP NetWeaver has strong User Interface and data visualization capabilities. With the introduction of Adobe Flex integration, we can show “cool” tables, charts and graphics within an SAP application.

.NET based UI scenarios
If the development team is very comfortable using .NET for all application development, then .NET can be the natural choice for custom applications that have integration to SAP.

SAP WebDynpro based scenarios
SAP NetWeaver is a compelling choice for core business processes that fetch majority of their business data from SAP systems. SAP NetWeaver provides WebDynpro for User Interface development, SAP BPM for a guided business process and SAP BRM for business rules. Most of the development is through model based development,  thereby reducing manual coding to the “framework exit points or hooks”. Since SAP NetWeaver has a native integration to all SAP Business Suite systems, this reduces integration effort, increases accuracy of data mapping, and a much higher end user satisfaction with the business process.

Recommendation
As shown above, the use of a development platform is dependant on several criteria, and every customer has unique drivers to opt for a development platform.

In our experience, customers having both SAP and a nonSAP platform (such as Microsoft .NET) have benefited by developing SAP centric business processes on SAP NetWeaver. The SAP and Microsoft platforms co-exist and there is a conscious strategy and decision criteria to determine the use of SAP or Microsoft for a custom application.

 For our customers, this means that Nagarro continues to be a trusted resource for the newest, most advanced tools and technologies available.  We will keep you posted on our work with Google App Engine as it evolves.  If you have any experiences with Google App Engine to share, we would love to hear your comments as well.

Nagarro shares a love of mathematics with customer SIAM (the Society for Industrial and Applied Mathematics).  This year, Nagarro’s custom applications will be automating the judging process for the Moody’s Mega Math Challenge, an Internet-based math contest co-hosted with SIAM. Participating teams of students are given 14 hours to solve an open-ended, realistic, applied math-modeling problem focused on real-world issues.

The content spotlights applied mathematics as a powerful problem-solving tool, a viable and exciting profession, and a vital contributor to advances in an increasingly technical society. Scholarship prizes total $100,000.

If you know a high school math whiz from Maine through Washington, DC who might be interested in participating, please feel free to share information on the contest. Registration closes on February 26, 2010.

Who knows, the winners could be your future employees (or our future customers)…

Often, there are issues related to:
- Having a full time SAP consultant to address a functional area where there are few support requests or development requests
- Having a SAP consultant to manage and juggle many different functional areas. This leads to SLA issues when many issues pile up simultaneously
- Unable to address complex issues due to high costs involved in getting a Senior SAP consultant to the customer location
- Unable to implement advanced SAP modules (such as SAP APO) due to lack of inhouse talent for current and future support needs for these advanced modules

For all the above reasons, the customer organization continues to stick with the current implementation and the older SAP releases, and unable to innovate or implement advanced SAP products.

Remote consulting is catching up with many customers to address all the above issues.

Remote consulting allows you the following benefits:
- Access to advanced SAP skills and SAP talent. It is a fact that many Senior SAP consultants are unable to travel as much as they would like to! Hence, many Senior SAP consultants are constrained to serve their local geographic areas only.
- Cost reduction and control of SAP support budget. Remote consulting is always significantly cheaper than calling the Senior SAP consultant to physically arrive at our offices.
- Access to SAP functional services on Fractional basis. Many SAP service providers offer SAP services at less than full time basis. This is valuable for companies to get support for SAP services that have less support requests.

Today, many customer organizations are becoming receptive to the idea of SAP services delivered remotely. For many large global organizations, they are already geographically distributed over multiple locations and timezones and collaborating over the phone, Webex and VPN. For most organizations, remote consulting fits seamlessly with their current SAP team. Considering the advantages of access to sparse SAP talent and huge cost advantages, SAP remote consulting is the smart way to get access to SAP talent.

In this blog, we would present guidelines and best practices for usage of SAP PI in an Enterprise SOA environment.

Usage of SAP ESR and SAP SR for managing enterprise services:
- SAP ESR and SAP SR should be used for modeling and publishing enterprise services in the landscape.
- SAP ESR and SAP SR are provided by SAP PI and SAP CE infrastructures. Hence, the customer could install either SAP PI or SAP CE to get access to the SAP ESR and SAP SR instances.
- SAP ESR should be used to model and manage all SOA assets. We can use it to model SAP and nonSAP services. SAP ESR models can be used to directly implement the service implementation in SAP. With SAP ESR model, it is not possible to directly implement a .NET or J2EE service. However, the .NET or Java service signature can be made to conform to the Service definition described in SAP ESR.
- Enterprise Services Repository used to define re-useable services rather than classical interfaces
- SAP SR can be used to manage the publication of SAP and nonSAP services.

Usage Scenarios of SAP PI:
- All web services and enterprise services that are being exposed to the customer should go through SAP PI. This is to ensure scalability, security and availability, since going through a managed middleware ensures the quality of service (QoS) for the enterprise service. For example, we cannot expose an application web service directly to a customer in a B2B scenario since the security and load are unpredictable
- SAP PI is used to create abstraction between heterogeneous sender and receiver systems based on SOA standards to provide unified access to legacy systems.
- SAP PI 7.1 is for mainly leveraging functionalities for service enablement, and service and process orchestration
- Integration between non-SAP system and non-SAP system.
- Use SAP PI for both A2A, B2B integrations including EDI
- Use SAP PI for service enabling legacy applications that lack a framework for exposing web services.
- For SAP PI 7.1, many SOA enabling standards or WS standards are supported as part of this release making it the core technology enabler of Enterprise SOA. (e.g. WS Reliable Messaging, WS Security, SAML, WS Policy, WS PolicyAttachment  etc.)
- For a predominantly SAP landscape, SAP PI should be the strategic integration platform and the enterprise wide service bus

When not to use SAP PI in enterprise SOA:
- Recent release of the SAP platform (SAP ECC6) has native capability to expose enterprise services. The service should be modeled in SAP ESR and implemented in SAP ECC6. This service is readily available for consumption across the enterprise without the intervention of SAP PI or any other middleware.
- SAP PI usage is not recommended for synchronous communication since it places a significant load on the infrastructure services for servicing a synchronous request. Also, SAP PI cannot guarantee the QoS and response time in processing a sync request.
- For User Interface scenarios (such as a WebDynpro UI), the UI can directly consume enterprise services. SAP PI should not be used in UI driven scenarios if the backend is exposed as enterprise services.
- If a nonSAP backend such as .NET or J2EE or any other platform is exposing business services in a UI scenario, SAP PI is not needed for intermediation. The UI such as SAP Portal based User Interface through various SAP NetWeaver technologies (SAP CAF, SAP BPM, WebDynpro etc.) can directly consume nonSAP services.

In conclusion, SAP PI should be the strategic integration platform in a SAP centric landscape. However, SAP PI should be used as an intermediary if it adds value between the sender and the receiver entities.

Why is this so? Perhaps the problem is that when you are early in the SDLC, there are so many other challenges and moving parts that you have little time to worry about usability. You worry that bringing the “naïve” users in for design discussions will just derail the project or send it off on a tangent. On the other hand, if you wait till you are through with version 1, you have been compromised as well – it requires great courage to admit at this point that usability is poor and that major elements of the application have to be re-designed.

These are formidable challenges. Yet we at Nagarro recently had a very positive series of usability-related discussions with a major client, which may be useful to recount in this context.

The client is one of the world’s leading travel-related companies. We are building a business-critical application for this company that will be used by a handful of users. The IT project management on the client side had invested a lot of intellectual muscle into the functional and algorithmic design of the application. We too put in a lot of effort to make mockups of the entire application UI and these were approved by the client – but by the IT project management, not by the users. The users did see the application from time to time, but in short bursts and they definitely did not have enough time to play with it and give their feedback.

Then after we had a successful development release and were reviewing progress with senior folks on both sides, we all collectively realized that the usability of the application left quite a bit to be desired. It was a depressing moment.

It wasn’t just a matter of aligning data elements or fiddling with the color palette.

It wasn’t even a matter of trying to streamline workflows to reduce the number of clicks required for each task.

It was the fact that the application’s UI looked like it had been designed by highly analytical engineers and scientists, which it in fact had. Would the users – who were neither engineers nor scientists – find it easy to use? Would they buy into it? Would they find all the features that they would need?

Luckily, the client team comprised very intelligent and wise folks. Rather than blame the team members on either side, the client’s senior VP-level executive said, “We should think of this as continuous improvement. You all did a great job, but we now see it can make it even better. Let’s see this as a positive opportunity and move forward.”

So, no blame game, no recriminations, no requests for “free rework”. The gentleman basically had the wisdom to see that when you are building something highly innovative, you may have to iterate to get the design just right.

Still, on our part, we offered the client a free usability workshop with our best consultant, a person skilled at combining “right brain” creative thinking with the “left brain” analytical thinking required for software design. The users were the star participants in the workshop, which started from first principles – what exactly is it that the users want to achieve? Our engineering team got the chance to put itself into the shoes of the users and try to come up with metaphors and overarching design principles that would work for them and, hopefully, delight them.

The workshop ran for two days and turned our previous thinking on its head. Yet everyone was thrilled with the insights and we agreed we’d run such workshops for each new project. The client agreed to fund a few person-months of effort to upgrade the user interface. Perhaps the cost of the overall development rose by 10%. But as a result, the chances of the application being very successful and useful in the hands of the users probably doubled or tripled.

And that’s always the most important metric!

So far so good. With increasing globalization of the customer base, we are witnessing several movements that are making the ISVs to sit up and take action to preserve their market share. First of all, the customers themselves are getting acquired by larger companies, and “acquiring” globally recognized names. The decision making policies are now getting centralized, and all decisions need to adhere to “global standards” such as compliance of new software products, interoperability, multilingual multicurrency support etc. Additionally, the ISVs are seeing increasing competition from the larger “less specialized” players who are coming up with offerings to compete head-on with the ISVs. Finally, the customers are now less excited about buying “best of breed” applications, and leaning towards solutions from a single product vendor which complies to their global IT standards. For ISVs, deep functionality is not enough to break into larger customers who have a certain gate-criteria for qualifying all software solutions.

SAP is the predominant ERP globally; it is run by a large number of global corporations across Auto, High Tech, CPG, Retail, Chemical, Pharmaceutical, Oil&Gas etc. These global corporations have acquired global footprint across Americas, Europe and Asia with distributed IT management teams. For consistent management, most corporations mandate SAP compliance of all software solutions.

For an ISV looking to break-into the SAP customer base (which consists of a significant number of Fortune 1000 companies), it is almost mandatory to demonstrate SAP compliance. SAP compliance would be one of the earliest concerns to be brought up by the IT during the product evaluation and due diligence phase.

Then why wait ?

SAP compliance can be achieved through the following two approaches:
- Migration of the product’s important features to the SAP platform: Though this may seem like a more difficult approach, it is not. We have helped customers to rapidly build important features of the product using the SAP NetWeaver platform in a short span of a few weeks. The initial product release is adequate to support the ISV Marketing team to start socializing with the SAP customers.
- Integration of the ISV product with SAP: This is a common approach to integrate the ISV product with the SAP business suite. We typically use SAP PI (SAP XI) middleware platform to provide data conversion between the ISV product and SAP proprietary data formats.

Finally, nothing beats SAP Certification! An SAP certified product or SAP certified integration provides assurance to the customer that the deliverable meets SAP’s quality and data interoperability standards.

How do we do it ? ISVs are understandingly not SAP experts. The best way is to partner with an SAP Services provider who has previous experience in working with ISVs and has completed several such projects. The ISV should not need to learn SAP or get into the nitty-gritty of SAP Certification, all these activities are the responsibility of the SAP services provider.

What would be the effort ? As with any IT project, we need to set the boundaries and scope of work to reach the effort and costs. As a rough estimate, an ISV can achieve their initial goals through a 3 month project with a 3 person team; With capable SAP expertise available offshore, the offshore costs should hopefully justify an SAP migration project to make it into your engineering and marketing budgets for the next quarter.

A Bit of Background
The application database is SQL Server 2005, single instance (non replicated, non clustered). The database has patient data (Electronic Medical Records) and the end clients (clinics and hospitals) access this data via web services. The database had around 3 million records, which they expected to scale to around 5 million. The software product vendor was receiving performance related complaints from end customers, and had solicited our help in resolving these issues.

Analysis
The first step was to establish the nature of the performance problems. For this we worked with the business and development teams of ISV to determine the nature of the performance issues. Users had reported performance problems in the following situations:

• Longer than expected response times for transactions or queries.
• Insufficient transaction throughput to complete the required workload.
• Decrease in transaction throughput.

We needed to

• Determine if the performance degrades with increased load or with increase in data in the system.
• Determine the volume of historical data and if they cause long running queries.
• Examine the hardware configuration (RAID, multiple disk controllers) of disk subsystems.
• Take regular measurements of resource utilization and database activity and determine whether there is a disproportionate utilization of CPU, memory, disks or network. System Monitor (aka Performance Monitor) helps to identify potential hardware bottlenecks w.r.t. a few key counters (see http://www.sql-server-performance.com/articles/per/performance_audit_part2_p1.aspx) and potentially obvious performance problems.

The next step was to establish performance objectives. For this we developed a plan for measuring the system performance. The most common metrics used for measuring database application performance are:

• transactions per second (TPS)
• response time

We then needed to define clear objectives for the above metrics. Selecting a tool (we evaluated some of the many available; refer: http://www.testingfaqs.org/t-load.html) that can automate measuring the TPS and/or response times by generating peak / off–peak / average load conditions.

We would regularly monitor the application performance on these metrics to get a feedback on whether the optimization changes are reaping any improvements.

The next post shall cover a discussion of the various solution approaches that could be adopted to improve the database problems once the problems (or at least some pattern that may be imminent in the problem) have been identified. I shall describe the various areas and perspectives of looking for a solution to the database performance problems.

An SAP NetWeaver project would typically consist of identifying new business processes to be implemented, new SAP BW reporting, new data integrations etc. The SAP Netweaver is a platform that provides implementation of new processes and data visualization applications where the SAP ECC is at the backend. Hence, the need for SAP ECC core configurations is minimal.

As with any development project, the initial phase of the project needs close understanding of business requirements. This requires the Business process consultant to personally interact with the business and identify the business requirements through workshops, white boarding and discussions over printouts/charts/sheets etc. This process typically takes 10% of the total project effort for capturing the high level requirements. This phase of the projet has to be conducted onsite to fully interact with the business and understand their true requirements and aspirations.

Thereafter, the next level of details can be arrived at by the remote development team located at the offshore location. This includes creation of mock User Interface and mock process walkthrough to confirm the requirements. This is a partial development activity and can be efficiently conducted offsite. All artifacts can be discussed with the customer team remotely, with the partial assistance of the onsite consultant. This activity may take about 20% of the total project effort.

Thereafter, the design and development phase of the project can be totally conducted offshore. This phase takes 40% of the total project effort and needs access to deep SAP Netweaver expertise. We do not need close co-ordinatation with the business for this phase. Hence, this phase is ideally suited for offshore execution because it needs access to deep SAP Netweaver expertise and can lead to significant cost reductions for the customer.

The object is unit tested, and then provided to the customer for integration testing and acceptance testing. Thereafter, the approved objects can be deployed to production. All these activities take the remaining 30% of the project effort.

As we can observe, the initial phase of capturing high level business requirements needs an onsite presence to determine scope and negotiate with the Business to reach a technically feasable and cost-effective solution. Almost all the remaining activities can be efficiently conducted offsite to leverage the team knowledge and SAP Netweaver expertise available in the offshore team.

In our direct experience, we are observing customers to benefit through SAP NetWeaver offshoring by getting access to Netweaver experts located at offshore locations, and also get cost savings due to lower cost offshore pricing. This lives up to our motto of doing the “right activities at the right location”.

We recently caught up with Chris Roe, CTO of Qtrax, in New York to find out more about his experience as a Nagarro customer. Qtrax and Nagarro have been working around the clock for months preparing for the launch of Qtrax 1.0, the industry’s first free and legal peer-to-peer music download service.  Qtrax had a tall order – linking a massive music file system with ad-serving features while enabling millions of site visitors to rapidly download an unlimited number of high-quality music files.

Oh, and did we mention that they needed it done in just four months? Click on the video to hear how Nagarro rose to the challenge.

The problem of making such general rules about programming practices is that you tend to miss out the benefits by following these rules too literally in every context. Think it on the lines – Reflection or any feature for that matter, if included must have a valid reason for its presence. We should be able to see and know how and when to use it and when specifically to avoid it.

Reflection is slow; but the question is how slow is slow? One should worry about performance but this worry should not keep you away from making best use of a technology. If you look around in .NET world, you will realize that .NET framework itself uses a lot of reflection. Ability to store meta-data about your code and to be able to use it at runtime opens the door to a whole lot of possibilities limited only by your imagination.

If you rethink it it’s just one another way of storing and working with information; difference being that this information is stored in code itself. So instead of retrieving information from some XML file or database, you are pulling it in from the code itself. Therefore your concerns towards writing optimized code while reading data from XML (e.g., load once read many times) or from database (e.g., making fewer trips to database), etc. will not be much different when using reflection. You might want to fetch type info once and hold it for future calls as well instead of fetching it every time you need it.

As in case of database, we know where to use joins, on which columns to create indexes and which where clauses to use to get optimized execution time; in case of reflection too there is a well defined grammar to make optimized reflective calls. For example, using BindingFlags enumeration correctly, not trying to use IgnoreCase flag, and limiting your searches to public or static or instance members as required, will definitely give you best possible results. Joel Pobar’s post talks at length about how reflection works internally and is a good reference to understand how to avoid performance pitfalls.

Don’t just assume that reflection will hit the performance, but use it, test it, tweak it for performance optimization and enjoy the flexibility it provides. If you think your problem statement has a solution in reflection; don’t hesitate in going ahead just because someone says that it will be slow. If this turns out to be slow; there are ways to optimize it too. For instance Jon Skeet’s post talks about using Delegate.CreateDelegate to turn a MethodInfo into a strongly-typed delegate. This can improve performance massively in scenarios where you need to set the same properties multiple times. Essentially this means that a lot of the type checking is done once when you created the delegate, rather than on every invocation.

Let’s explore a scenario where reflection is no harm to use and probably the only way you can achieve the end result with.

You want dynamic extensibility to your desktop application; your revenue model includes selling various groups of features separately. You do not want to build multiple deployment packages and you want to be able to provide even more features to your customers after they have started using your application. How would you do it?

You might do this by developing a plug-in model in your application.

You may define an IMyAppPlugin interface which can be implemented in each of your application’s plug-in assembly implementing a specific feature. At minimum you may have following definition:

public interface IMyAppPlugin
{
    Guid Identifier { get; }
    string Name { get; }
    string Copyright { get; }
    bool Start(IHostApp hostApp);
    void Stop();
    bool Run();
}

You will not have any reference of these assemblies in your project at design time. Its only at runtime when you will try to discover these plug-in assemblies (using some mechanism such as looking in specific folder or registry, an XML file or anything else) and load it using reflection.

You will write code to find and create instance of the class that implements this IMyAppPlugin interface. Once you have this object you can initialize it by calling its Start() method passing your application’s reference as required. You may also write code to add access points (such as menu items) for this dynamic functionality to appear in your application’s user interface when this plug-in is successfully loaded.

A call to Run() method anytime after this will do what is desired.

So essentially what we are doing here is that we have loaded the assembly using reflection and we are calling methods dynamically again using reflection. If we just go by numbers, yes dynamically invoking a method is slow. Referring to Rick Strahl’s post we can say that invoking a method dynamically is about 3.5 to 4 times slower than directly calling of a method. But again the question needs to be asked – how slow is slow? Here we are talking about few milliseconds extra. Unless you are making calls in a loop for a large number of times, does it really matter in the larger perspective of a typical application? Most likely – No!

Remember .Net Framework and especially ASP.Net uses a lot of reflection internally to provide dynamic execution of code and controls.

Talking again about the above scenario, if not reflection, do we have some other relevant approach of dynamically extending the application? You may talk about using COM interfaces to achieve it – but reflection or no reflection – the essence of every approach is to dynamically resolve the address of the method to be called, which is the only major difference wrt static calls where this address is readily available.

This is just one scenario where reflection proves to be the most effective and simplified approach to work with. There are a lot of other scenarios where reflective access to properties, fields and methods provides enormous flexibility that simply wouldn’t be there otherwise.

With Dynamic Language Runtime (DLR) in .NET 4.0 we are heading towards a cleaner reflection. Eventually this may replace reflection how we see it today, due to its simplified code and caching advantages. In-fact as mentioned in this Jim hugunin’s post, it appears that DLR too internally makes good use of reflection.

In short, do not shy away from using reflection, just because you hear warnings about it. Always verify if these warnings apply to your project or scenario. Often they will not. As they say, “trust but verify!

At end there was Q&A session where I ended up raising my hand and asking a question “with so many technology choices available even in .NET for same tasks, the choice picking correct ones becomes tougher, with lack of guidance and gave several examples”. The reply was “That’s a great question” followed by a long answer – explaining the reason why this has happened and a nod of creating more practices and patterns around this topic. Highly encouraged, I ended up asking many more – till I was shut down by moderator! I ended up getting his autograph and a picture together using my low end camera phone – but a moment to remember!

With "The Gu"

A number of companies have had positive experiences with server virtualization over the last couple of years, and are looking to take the next step – virtualizing their desktop systems. Cloud computing, especially storage is finding increased traction in the midsize organizations, as they look to improve business continuity and disaster recovery plans. Another trend seems to be the increased focus on building relationships with external service companies in this market segment, as IT executives seek to do more with less.

With our expertise in software product development, manufacturing and SAP consulting, we knew Nagarro was a great fit for QISoft. But the winning the business was only the beginning. The real success story here is that Nagarro is making an immediate, positive impact on QISoft’s long-term competitive advantage.

So, is cloud computing merely hype? While there truly is value in what is provided by Amazon (EC2 and other services), Google’s App Engine, Microsoft’s Azure and a few others, the providers (or at least their marketing teams), if you ask me, have not helped a whole lot.  It seems like every company that used to provide virtual private servers is now a cloud computing provider. So are companies that used to provide hosting solutions, and every company that used to be a destination for an outsourced data center. Soon, we will have clouds of every other conceivable kind! When a term can be applied to pretty much anything, the term starts losing its meaning. Cloud computing, which I would put in the first stage of Gartner’s Hype-Cycle (hype followed by disillusionment followed by realization), is the latest buzzword that marketing teams love to apply to anything they can. I asked representatives from several companies that claim to provide cloud solutions what did they really mean when they talked about a “cloud”. When pushed, pretty much all of them ended up saying – “its just a buzzword”.

Getting back to the panel discussion, there were several strands of discussion which I would have liked to address in detail but couldn’t because of time constraints. So I decided to use this blog to move the discussion online (and to also share it with a wider audience). In this and subsequent posts, I will try to summarize as well as share my view of this technology, starting with some of the points from the panel.

• Cloud computing is nothing new: Cloud computing, like most other technologies, is an evolution, not a revolution. If it does end up changing the way we develop, deploy and manage software (and other components), we might start calling it a revolution but at this time, it represents incremental but very useful changes in several existing technologies and how it all comes together. Most offerings use virtualization, deliver services over the Internet and expose HTTP based APIs for accessing and controlling the services. But what is truly important for me is not whether its new or old; but whether it solves my technical and business problems. There are several class of applications and organizations for which EC2, Azure or Google App Engine is the right choice.  Moreover, I can not think of  a single service in the late 90s or early 2000s where I could provision Linux boxes in a matter or minutes by just signing up on a website using a credit card.  If nothing else, the sheer ease of use and affordability (at least in the beginning) is an innovation worth taking note of (and make use of if, it makes sense). My take on it – it works great when you use it to solve the problem it was designed to solve.
• There is no such thing as infinite scalability: The use of term “Infinitely Scalable” by the cloud computing service providers, in my opinion, is misguided. A better term to use would be “Sufficiently Scalable” but it does not have the same ring to it. Amazon’s or Google’s infrastructure can effectively provide “sufficient” scalability for small application. If you are mostly running on a couple of Xeons with 12GB of Ram or so, a typical spike in demand might need you to scale up by 4 or 5 times which can easily be handled by the Amazons of this world. I suspect, however, that you can not run another Amazon on Amazon’s infrastructure (especially during Christmas time!). There is a slightly different but related discussion about differences of scale and statistical models which I will not get into.
• What about the security of my data? Its no less secure that using a hosting solution of the past. While people like to complaint about Hypervizor breaches and how most cloud computing providers (especially those who provide infrastructure as a service)  use virtualization, the reality is that Hypervisors are far more secure that the operating systems they run. The two main and mostly related reasons for this is that the people who build Hypervizors realize this (and they are a pretty smart bunch), and that the Hypervizors are kept as small as possible. While I am not an expert in this area, the last time I was talking to one of the experts in Hypervizor and virtual machine security, he took pains to point out that inspite of the fact that it is bad for his business, Hypervizor breaches are extremely rare and that most Hypervizors are quite secure. I would, however, like to point out that what happens in case of a security breach is still not clear. We simply do not know enough about this stuff. And if you have contractual or legal requirements that prohibit you from moving your data into the cloud, you will simply have to wait until the lawyers catch up with the technology and the providers address these concerns directly through their SLAs.
• The clouds are not interoperable. Because every-body’s offering of “cloud” services is different, the questions of interoperability while being a very important question, does not make sense at this time. The really compelling offerings today are from Amazon (EC2 being the big one, but there is also SQS and S3 and a few others), Google (app engine) and Microsoft’s Azure.  EC2 gives you virtual Linux servers (and now windows too) and options of launching pre-configured servers. If all you are truly using is the Linux platform, nothing stops you from moving over to someone else who will also provide you Linux boxes (either virtualized or real). While you may have to make a lot of changes to migrate your Java application over to Google App Engine, taking your application out of the App Engine and deploying it somewhere else should not be a problem (assuming you have appropriately abstracted out the areas that use Google specific APIs). I have heard talks of Azure supporting Java (and a few other platforms) in the future and it will be interesting to see how portable Java applications will be between Google App Engine and Azure.
• What is the real cost of using these services? This is a tough one and merits at least a white paper of its own. I do not have a clear answer but I say this –  two main advantages of the cloud, the ability to scale up when you need to, and pay only for what you use go a long way in making cloud computing solutions cost effective. If you know your usage and it does not vary much, you do not really need to use a cloud computing provider.  There is work needed for moving an existing application either completely or partially into the cloud and you should not, as a matter of basic engineering principle, use a cloud simply because its the in-thing. It will not make sense and it will not be cost effective. You may, however, have other reasons for using Amazon or Google (for example, cloud computing payment models let you convert your capital expenses into operational expenses).

In my next post on cloud computing, I will list some of the prevalent definitions of cloud computing and try to factor out the characteristics that make most sense to me (especially for providers that we are most interested in) and try to come up with a working definition of the term that is most relevant to us as software architects/designers/engineers.

]]> http://www.nagarro.com/blog/cloud-computing-demystified-part-i/feed/ 3 http://www.nagarro.com/blog/a-better-way-to-test-email-sending-functionality-is-to-use-your-own-smtp-server/ http://www.nagarro.com/blog/a-better-way-to-test-email-sending-functionality-is-to-use-your-own-smtp-server/#comments Mon, 29 Jun 2009 07:09:14 +0000 Vaibhav Gadodia http://www.nagarro.com/blog/a-better-way-to-test-email-sending-functionality-is-to-use-your-own-smtp-server/ A large number of applications today have some kind of a notification mechanism built-in. More often than not, an email is used to send these notifications. Typically, while developing these applications, developers use their own inbox to send notifications to. Or they setup some dummy account to send test emails to; or sometimes they might send test mails to real users (it’s been known to happen).

There are many limitations to this approach:

• one has to deal with dummy messages arriving in the inbox, when all that you want to test is your email sending code.
• automating the testing of the email sending code will result in a large number of emails being generated and delivered to your inbox.
• code that is supposed to deliver large amounts of email will generate a lot of load on the Internet bandwidth (and yes that can be a problem).
• if you are testing on a copy of a production database, you might inadvertently end up sending test emails to real users.

There are other issues present as well. One solution to solving these problems is to use your own SMTP server to handle all these emails in a manner that it receives an email and doesn’t sent it forward. So, it is like a dummy SMTP server. But setting up an SMTP server is usually a hassle.

There are plenty of options out there which make your job much easier than you think it is. In this post, I am going to list some solutions which can be used (though there are many more). The basic premise behind this approach is to run a local SMTP server (either on the developer box, or the test server, or both, or somewhere on the LAN). This local SMTP server address is what is configured as the server address in your application which is being tested. Any mail sent to such a server is kept with the server and not forwarded, thus preventing all the above listed issues.

Following are some of the more popular out-of-the-box programs that can be used (click on the names to go to their websites):

• Papercut - this is a simple windows application which will run in your System Tray and simulate an SMTP server; it receives and stores any messages it receives. It allows you to view the messages. It sits in your system tray and alerts you whenever a new message arrives.
• Neptune – this is another application which will do a similar task. It is however a little more advanced. It has an admin API interface which allows you to run custom queries to analyze the messages that it processes. It doesn’t have a UI through which you can view the received message though – but sometimes all you need to check is if the messages are being delivered. Very useful in a bulk messaging scenario.
• smtp4dev – this is another option similar to Papercut, which again sits in the system tray and will let you know whenever any message arrives and will let you look at the message details.

There are many more out there which can be used to do similar things with some features here and there (for example, Neptune doesn’t generate all SMTP responses). Here’s a post by Phil Haack from long ago which is also interesting: A testing mail server for unit testing email functionality.

So, when you feel the need to test your email sending code the next time, it might make sense to have one of the tools (or something similar) available.

Get a look behind the scenes at Pacific Southwest Container, a midsize manufacturer of sustainable custom packaging. Nagarro’s custom applications enable PSC to optimize use of printing materials and equipment, reducing PSC’s costs and waste. End result? Cost savings are passed along to customers, while PSC improves its environmental footprint. Please view this short video and let us know what you think!

Per Wikipedia  

With ActiveRecord, a database table or view is wrapped into a class, thus an object instance is tied to a single row in the table. After creation of an object, a new row is added to the table upon save. Any object loaded gets its information from the database; when an object is updated, the corresponding row in the table is also updated. The wrapper class implements accessor methods or properties for each column in the table or view.

This pattern is often implemented by ORM and Object Persistence Tools like Hibernate (Java) and ActiveRecord (ruby). The implementation framework handles generating the necessary SQL appropriate for the database being used and executing them over a database connection.

Recently, in a couple of projects I used ActiveRecord Migrations to manage database setup and migrations.
Below, I shall present a walk though the steps that I followed to get a set of ActiveRecord migrations working on my Ubuntu-9.0.4 desktop.

1. Check if you have ruby installed:

     ruby -v

You’d see something like the following: ruby 1.8.7 (2008-08-11 patchlevel 72) [i486-linux]

If not, then install ruby by executing the following from a terminal window:

     sudo aptitude install ruby

Also, install rake, – ruby’s build program with capabilities similar to make using:

    sudo aptitude install rake

2. Install rubygems:

    sudo aptitude install rubygems1.9

3. Install the following ruby-gems

• ActiveRecord: sudo gem install activerecord
• Database adapter: I used the MySQL database; so I needed to install the mysql gem  sudo gem install mysql
  
  The above command results in the following error:
  
    extconf.rb:8:in `require’: no such file to load — mkmf (LoadError)

Fix: Being a Debian or Ubuntu user you’ll find that the Ruby standard distribution is split into lots of little packages in order to comply with Debian packaging Guidelines. Installing ruby only gives you the ruby binary and a subset of the libraries for Ruby. You’ll need to add more packages if you want to utilize more of Ruby’s standard library.

Hence, we need to install the ruby-dev package prior to installing the mysql gem.

  sudo aptitude install ruby-dev

4. Create directories to hold the migration definitions:

  mkdir ~/workspace/db_setup
  mkdir ~/workspace/db_setup/migrations

That’s all on the installation side. Now it’s time to write the migrations.

With ActiveRecord, each change to the database is termed as a migration. Migrations derive from ActiveRecord::Migration and enable a developer to generate a database structure using a series of Ruby script files (each of which is an individual migration) to define database operations. It’s a database-agnostic representation of the database. It uses generic data types for columns, like :binary, :string, :text, :boolean, etc. to define the kind of data to be stored in a column. A sample migration would look like the following:

File: migrations/001_create_institutions.rb
class CreateInstitutions < ActiveRecord::Migration
def self.up
create_table :institutions, {:id => true} do |inst|
inst.column :name, :string, :limit => 100, :null => false
inst.column :url, :string, :null => false
end

execute(’ALTER TABLE institutions ADD UNIQUE (name) ‘);

end

def self.down
drop_table : institutions
end
end

• The file-naming convention is important here:
  • ActiveRecord uses the numeric prefix to determine which migrations it needs to apply. To do this, it creates a table called as SCHEMA_MIGRATIONS in the database in which the VERSION column holds the number from the last migration added. When the migrations are run without a target version specified, it will only execute the migrations that have number greater than that stored in the database, lowest numbered first.
  • The class name should correspond to a camel case representation of the migration’s file name.
  • Typically your project would have a collection of these migrations, in files with prefixes as 001_xx, 002_xx…. and so on, each defining a change desired on the database.
• The two methods up and down should be functional complements of each other. The up method is executed when upgrading the database while down method is executed while reverting to an earlier version.
• You can do many things in a migration – create and delete tables, add and remove columns, set column constraints (such as nullability and column length) and also run code to initialize the data in a table.
• As a convenience, you also have the execute method available (shown as the last statement in the up method) to run SQL / commands directly. This has many implications: it lets you execute commands that use specific database dialects, like adding database constraints. The following script allows you to execute SQL commands from an external file, say create_db.sql:
  
    sqlCode = File.open(File.dirname(__FILE__) + “/create_db.sql”).read
    sqlCode.split(”;\n”).each do |stmt|
     execute stmt if stmt =~ /\S/
    end

ActiveRecord can read the database configuration from a YAML config file called as database.yml (place it under ~/workspace/db_setup) that looks like:

# The # character signifies a comment.
#
# Define db config per environment here.
# yml files are formatted in a very strict way; you cannot insert spaces and tabs randomly.
# Typically they use two (2) spaces to indent options.
#
# The space between the ‘:’ and the configuration values is required.
dev:

adapter: mysql
database: $DB_NAME
username: $USER
password: $PASSWORD
host: $HOST

A set of ActiveRecord migrations can be executed as
1. A part of a Rails application by invoking

  rake db:migrate RAILS_ENV=xx VERSION=yy

2. A standalone Ruby application: If it is not a part of a Rails application, there is no default Rakefile to use. So we’d need to create one (under ~/workspace/db_setup) with the following content:

require ‘rubygems’
require ‘active_record’
require ‘yaml’
task :default => :migrate

# Enable diagnostic logging to help debugging.
# Note that ActiveRecord colorizes output for viewing in a terminal.
# But when viewed as a normal file, it isn’t readable. So we disable it.
ActiveRecord::Base.colorize_logging = false
logFile = File.open(’database.log’, ‘w’) # set it to ‘STDERR’ to enable logging to Std. ERROR console
ActiveRecord::Base.logger = Logger.new(logFile)
end

To execute the migrations, run the following command from the directory containing the RakeFile:

  rake RAILS_ENV=xx VERSION=yy

Both the above mentioned ways lend themselves to be easily invoked from a multitude of build and CI tools, like ANT, CruiseControl, python scripts etc.

Thus we, saw how to configure ActiveRecord as a database agnostic mechanism of managing database setup and incremental migrations. It is relatively simple to use and highly extensible.

Although, some may argue that ActiveRecord leads to a high degree of coupling between application code and database structure; in many cases the issues due to this coupling is much easier to manage than to adopt an alternate complex solution.

Recently, we had the opportunity to give our opinion on the latest avatar of the Ado.Net Entity Framework to Jeffrey Schwartz from Application Development Trends magazine. Jeff was kind enough to publish our opinions in an article on the Visual Studio Magazine (thank you Jeff). The article is titled “Will Entity Framework 2 appease early adopters of LINQ to SQL?“. We, as can be deduced, were early adopters of LINQ (to SQL and others).

If you would like to see that question answered, I urge you to go and read that article. In this article, I wanted to discuss a little bit more about how we look at Entity Framework v2.

When Microsoft first announced the Ado.Net Entity Framework, we were unsure about what to do with it. We had already started training our developers on LINQ to SQL, and some of our projects were already using the technology. We found it (L2S) to be a very good way to implement the ORM layer of an application (and that worked very well in most cases). Additionally, we weren’t sure how mature the framework was. Sure, it looked perfect for some of our larger projects, but somehow it wasn’t fitting the bill (and we were not alone). And compared to LINQ to SQL, it felt quite lackluster in features.

The new version of the Entity Framework looks much better. It almost feels as if Microsoft went out of their way to ensure that Ado.Net EFv2 can stand on its own when compared to LINQ to SQL. However, by this time, we are pretty invested in LINQ to SQL, so there is no question of moving any of our existing projects over. But, we are very keen to try out the EFv2 in newer projects and internal applications. Specifically, we will looking at integrating it the next version of our Nagarro .Net Application Framework (which is a base framework that we use for most of our projects).

“Are we there yet?” – I would say, almost. The latest version of the Entity Framework looks a much more complete product as compared to the earlier version. Some of the features from an implementation-as-a-tool-in-a-software-development-services-shop perspective are really great:

• POCO support - the support for persistence ignorant objects make a great deal of difference because we can continue to leverage the features of the Entity Framework while using your own custom data layer logic alongside it. This is beneficial because this means we can at least consider using the EF in existing applications where the data model and the layers are already defined.
• First class code generation – this will come as a relief to most developers. The code generation engine, T4, is much more versatile that what we used to have earlier. This means that each application development team can customize the code generation a little bit according to their specific needs.

And of course there are hosts of improvements in general and there are tons of articles covering those.

From Nagarro’s perspective, I think we are almost there.

After initial analysis, it was clear that we needed to incorporate AJAX in multiple areas across the application. It was further decided to use one of the existing AJAX frameworks rather than writing AJAX “by hand”. After evaluating several AJAX frameworks, we selected GWT. Other AJAX frameworks like DOJO and YUI library are also good but they did not solve the above mentioned problems arising from a fat architecture and a complicated technology stack.

One of the primary reasons for selecting GWT was its unique approach to developing the complete application: both the client side and the server side components using Java. GUI development with GWT APIs is similar to Swing development and it is refreshing to forget JSP, JavaScript, and Struts and write everything using Java. I am very happy with experience so far and we have vastly improved presentation layer while retaining most of the service layer code.

With GWT, the AJAX front-end is written in Java that gets compiled into highly optimized Java Script. Developers do not need to write Java Script while end users are provided with dynamic and standards-compliant AJAX experience. JavaScript gets downloaded by the client browser at runtime where most of the client side processing is performed thus releasing the server side from handling presentation tasks.

I am impressed by the developer friendly features of GWT. Using Java in the presentation layer facilitates the use of design patterns and creation of reusable widget libraries. GWT’s approach to history management, internationalization, and service layer interaction is easy to understand and implement. I felt spoilt by the choice of available tools, default support for multiple browsers and ability to debug AJAX in hosted mode. I also noticed ancillary benefits like elimination of training on multiple technologies, reduced dependence on technology experts, and the overall speed of development.

I have heard arguments about Flex being a better RIA choice than GWT. I think this may be because some people may not be aware of other GWT widget libraries (like GWT-EXT) available today. Although GWT comes with a very good collection of useful and cool widgets but you can always use third party widget libraries which provide additional AJAX components like a special Tree or Grid component. We used GWT-EXT, a very rich widget library to achieve the desired functionality.

I found a few features missing – for example, there is no inbuilt support to load client side libraries on demand. Actually GWT loads all client-side libraries in client’s browser when the end-user accesses any module of the web application for the first time. So I think GWT may be a misfit for large enterprise applications. Also I faced problems due to lack of emulators for some of the Java classes like Calendar and our application POJOs contain Calendar objects. The good thing is that there are workarounds available to solve these problems.

I strongly recommend GWT if you are developing a mid size web application that needs enhanced usability and a desktop like look and feel.

While at the show, I had the opportunity to participate in a panel on social networking. Our discussion was an important reminder that the idea of “business to business” is giving way to “business to community.” In our constant endeavor to make our clients more competitive, we have been working closely with them to understand how social media impacts their organizations. I am constantly amazed by the possibilities of building custom applications using the Web 2.0 APIs to leverage this new medium in departments ranging from sales, marketing, customer support, product development, design innovation, training, and education.

Social Networking Panel at the Midsize Enterprise Summit

And last week, the NASSCOM Emerge blog examined Nagarro’s intellectually vigorous culture in a post about the NASSCOM Emerging 15 (I like It Here, Do You?).Nagarro stands out from the crowd as “a company where nerds love to work” as NASSCOM discusses its picks for the best small and medium sized IT-BPO companies to work for.

NASSCOM characterizes the top 15 emerging companies as “sought after by career seekers for their close knit, non-hierarchical work environment, culture of innovation and commitment to employee empowerment.” If this is the definition of a nerd-magnet, sign us up.

In today’s dynamic business environment, IT departments need to remain agile so they can align IT strategy with shifting business,market, competitive and economic conditions.  Outsourcing is one way to adapt resources to suit an ever-changing landscape but can also introduce risks if the right partner is not engaged or the partnership is mismanaged.

Companies rely on SAP deployments to run their core business operations.  As a result, outsourcing decisions must contemplate security controls and allow the IT department to maintain operational control.

In an effort to help you identify and manage risks, Nagarro’s new white paper addresses some common outsourcing concerns and objectives, and provides practical information that can help you more effectively address your current and future business requirements.

A quick  review of event processing in Windows Application tells us that there are 3 different types:

1. Quick Processing (less than a second) – Action performed is executed almost instantaneously without any significant wait period. In this case, you would generally not display any indication to the user about the background task execution. For instance, Open a dialog on the Click of a button.
2. Short Processing (1 second – 5 seconds) – User action consumed a few seconds in execution. In such cases, recommended practice is to show a wait cursor. For instance, Pick records from database to list them in the application screen requested by the user.
3. Long Processing (more than 5 seconds) - Actions which involve a significantly longer processing time, in such cases it is recommended to show progress bar indication to users.

In order to display a wait cursor (category 2 above) the following is the most common block of code seen in most applications:

public object GetData(int id)
{
    Cursor.Current = Cursors.WaitCursor;
    try
    {
        // Processing code here    
    }
    finally
    {
        Cursor.Current = Cursors.Default;
    }
}

The code piece above will have to be duplicated at all the places where we want to use wait cursor during processing, this would be a pretty tedious task. Now, let’s have a look at another approach to implement the same feature:

using System;
using System.Windows.Forms;

public class WaitCursor : IDisposable
{
    private Cursor cursor;

    public WaitCursor()
    {
        cursor = Cursor.Current;
        Cursor.Current = Cursors.WaitCursor;
    }

    public void Dispose()
    {
        Cursor.Current = cursor;
    }
}

The WaitCursor class above implements IDisposable interface, hence you may use this is the using block. Now see how simple it is to add a wait cursor to your application:

public object GetData(int id)
{
    using (new WaitCursor())
    {
        // Processing code here
    }
}

There are some very obvious benefits by doing it this way:

• It simplifies the code at several places into a single line statement.
• More importantly, we just ensured that a bug where some developer forgot to restore the cursor to normal can never happen (as long as we have trained our team to simply call the above code).
• Finally, it is always a best practice to encapsulate repeated code in one place.

One should always be on the lookout for such small coding patterns and implement them.

The first and most obvious change will be that programmers will start using many more threads and use current lock based approaches to synchronize access to shared data. Now detecting bugs in multithreaded programs is extremely difficult and as we write more concurrent programs, it will become increasingly important to have tools which can check our programs for correctness. Two such tools are CHESS from Microsoft research and jPredictor from UIUC. Both are still research projects but are definitely worth taking a look at and playing around with.

The transactional memory research community advocates getting rid of lock based approach for managing access to shared data. The idea comes from the database world where transactions have been around for a long time. Hardware transactional memory, hybrid transactional memory and software transactional memory, the three broad categories in which implementations are often split, are all under active research but there already are quite a few implementations out there for you to play around with. One of the core ideas behind using transactional memory for accessing shared data is to allow users to specify “what” needs to be synchronized and then let the machine take care of the “how” part (compare this to how you use locks today). This is supposed to greatly simplify the concurrent programming model but it is too early to say if it really does make the programming model simpler.

While  I cannot cover or summarize the work being done on transactional memory without making it into a small project, I can provide a few starting points that you may find interesting. Microsoft is doing very interesting work in this area and while a lot of it uses Haskel, there is a C-sharp API for software transactional memory that you should look at. If you are a Java programmer, you should take a look at DSTM2. And irrespective of what technology you use for building systems, I will recommend reading publications on transactional memory by Maurice Herlihy, Simon Peyton-Jones, Victor Luchangco and Mark Moir (there are others, I just picked up the four I see often in the literature).

Finally, we should not ignore the world of message passing. OpenMP and Erlang have been around for a long time and are designed for concurrent programming. OpenMP is an API you can use from C/C++ and FORTRAN to write parallel programs to take advantage of shared-memory architectures.  Erlang is a programming language in which you write processes that communicate exclusively using messages and if you write your Erlang programs correctly, you get the ability to scale to hundreds of processors for free.

Recently, I evaluated SharedCache, which according to their website is a high performance distributed and replication cache system built for .Net applications running in server farms. The system is made available to .Net applications through a Windows service.

 Some of the key features are listed of SharedCache are listed below:

• Works for both web and non-web solutions.
• Is extremely fast for fetch.
• It supports multiple topologies: distributed partitioned caching, replicated caching, and single instance caching. You can choose which you need based on your requirements (for example, you may want to choose replicated caching over single caching to ensure that you don’t have any single points of failure)
• In case of Caching, one of the most important aspect is purging. With SharedCache, you have a choice to make amongst the several that are supported:
  • Cache Item Priority
  • Least Recent Used Item (LRU)
  • Least Frequently Used Item (LFU)
  • Time based
  • Biggest Latency First (BLF)
  • Lowest Latency First (LLF)
  • Hybrid – Creating an index based on various item attributes such as: time in cache, Amount of requests, Size, Cache Life Time, etc.
• Configurable maximum cache size and the load factor when the system starts to purge cached items.

You can install the service by downloading it from CodePlex. And several samples and discussions are available at the author’s blog. The best thing about this service is that there is plenty of documentation to guide you.

Just try it out. Its pretty cool.

Today we announced that Nagarro has signed on more than two dozen new customers in the US and Europe in the past three quarters. And we continue to sign new deals almost every week. Like our customers, we at Nagarro are looking at the downturn as an opportunity to keep our focus sharp and our resources first-rate. Obviously we are glad to be maintaining a steady presence despite the ups and downs of the market, but more importantly, today’s news sends a strong signal that at the high end of the technology market, innovation and competition are alive and well.

One method to log exceptions is to log them in the cloud. And there exists a useful service which makes your job easier.

A service in the cloud represents a service that is hosted on the Internet, available to you through an API, so that you have no idea how it is physically implemented. This makes things simple for you since you don’t have to worry about writing implementation code for the exception logging and reporting framework. You only need to call it.

The service that I am discussing is called HopToad. The service offers a simple to use API which allow you to log exceptions from your application. The exceptions are stored in the cloud and are available to you from a central web-based console. All the logged exceptions are presented in easy to understand and navigable way. For example, you can see which errors are occurring most often, or which one is the latest error that came up and from where. You can look at error details and stack traces.

The service developed to be used from within RubyOnRails applications. However, they have an open API, and can be accessed through any technology stack. K. Robertson has made a .Net wrapper available for logging to HopToad. It’s called HopSharp. It is fairly straight forward to use (just drop the DLL in your application and you are good to go). I can imagine that it would be straight forward to write a Log4Net extension for this service as well.

When to use this?

So, when would you use something like this. While the final decision to use this would vary from project to project and client to client, I can think of the following scenarios:

• The application is a work in progress and you are expected to release a number of updates. Typically, in such scenarios, a lot of testing happens while the application is in deployed state. A logging framework such as this will allow you to monitor the application as it is being used, and priorities on bugs can be assigned based on frequency and severity.
• You have an application deployed at multiple locations and you would like to see a consolidated view of the exceptions being reported.
• As a way to save time and effort. This is a ready made framework where you don’t need to impart any effort for implementing an exception logging and notification system.

I am pretty sure that there are other such services out there.

Yet I just can’t stay away, and neither can you if you are still reading this.  So let me jump right in with my take…

Ruby Doesn’t Scale.  It can’t take you to the same places Java can.

I don’t mean performance-wise.  And I don’t necessarily believe it can’t scale complexity-wise.  I just think it won’t scale with the size of a team.  I can’t see an application equivalent to 2 million lines of Java code with 80 developers written in Ruby.  Here are some reasons why:

• the language and tools don’t facilitate writing APIs
• it’s too hard to protect code from the side effects of other unrelated subsystems
• the language and tools do not support for components or version management (see my posts on OSGi)
• one bad Ruby developer can do a lot more damage than one bad Java developer
• there are a lot of bad Ruby developers (I am one!)
• the tests don’t scale–no Ruby app can survive inadequate tests, and the while the amount of tests grows linearly with the size of the application, the cost of maintaining them grows exponentially.  This may be true of any tests but…
• you need more tests with Ruby–a Java app can survive without a complete unit test suite.  In fact, some large Java app can survive without tests at all.  Call me crazy but I’ve seen it.  What did we do before JUnit, after all?  I don’t think I’d want to work on one of those apps, but the point is, test code for Ruby is as essential as an interpreter to make a decent sized Ruby app work.
• there’s no meta-programming in Java; that’s kind of a bummer, but it also means no massive indecipherable frameworks collapsing under the weight of the author’s cleverness, frameworks which are so unfathomable they are as brittle as COBOL.

There, I said it.  Now that I did, I’ll make a couple of other observations about why I love Ruby anyway:

• A two million line Java program probably translates to 100K lines of Ruby code
• 80 Java developers are probably as productive as 20 equivalently skilled Ruby developers
• With only a couple of developers, I think there’s no limit to how big the application can be.  In fact, the language is much better suited for a very small team than Java because you can be so much more efficient if you know what you’re doing.
• The tools for Ruby are way behind Java, and there’s no reason to think they won’t catch up.  When they do, it’s entirely possible the gulf between the scalability of the languages will close substantially.  I certainly hope they will.  As far as the tools for writing Ruby go, I feel like I stepped back into 2002 when I went from Java/Eclipse to Ruby/{NetBeans/RadRails/TextMate}.  rake ~ make: when was the last time I typed in make when writing Java apps?
• The tools may be behind, but the available libraries, plugins and gems are already very rich.
• Ruby helped me remember why I love to write programs

The last point I threw in there apropos of nothing, other than to point out that while I do miss some things about writing Java apps, I’m happy to be writing in Ruby now, and hope to keep working on it for the indefinite future, or until our app gets over 100,000 lines of Ruby which should be… never.

P.S. Here’s my favorite entry in the Ruby vs. Java debate, brought to you courtesty of RailsEnvy.

I feel privileged to have worked with Ward in several different contexts, and to have had a front row seat as he developed the first Wiki site and his Framework for Integration Testing (FIT), as well as a few other fascinating smaller projects along the way.  Even though I haven’t worked closely with him over any extended period, in the long run I think I’ve developed a sense of what he’s about as an innovator, and with the publishing of this interview have some reflections on what I’ve learned from him.

Ward to me is a classic idea guy.  He’s always thinking, always saying “what if?”  In many of the conversations we’ve had, I’m embarrassed to say I can hardly keep up.  If I can follow half of what he’s talking about I’m fortunate since that half is usually enough to cause me to go running back to my desk and start working on some new angle in my own work.

Not all of his ideas see the light of day, I’m pretty sure.  But when he hits on a good one, he hits it out of the park–nobody could deny that.  The Wiki wasn’t his first great idea and it definitely wasn’t his last.

So that’s one thing I’ve learned.  Having great ideas isn’t like making great movies, where if you’re Peter Weir you do nothing for five years and then suddenly come out with some brilliant new film, then go quiet again for another five years.  You have to constantly be working on it.  You have to come up with a lot of ideas, be constantly thinking about where to go next, and keep pushing them, trying them out, until you find the one that captures the imagination and the mindset of those around you, and really takes hold.  Put another way, it’s not about the batting average–it’s about the number of at bats.

Joel Spoelsky likes to make fun of smart people, people that say things like,  “spreadsheets are really just a special case of a programming language.”  Rightly so.  Those kinds of smart people are useless.  That’s not Ward.  It’s not enough to just be an idea guy–to always be drawing connections between seemingly unrelated things, or playing with the meaning of things to try to derive new value.  If you want to make your mark the way Ward has, your ideas need to have true value, to make people’s lives easier somehow.  That’s the second thing I learned from Ward.  Great ideas need to be a path to a better way.  The Wiki, XP, and Fit–they all fit the bill in that regard.

Also, Ward’s ideas have a common theme, in my mind.  They are simple but with incredible leverage. He doesn’t overthink anything.  He just starts off with a simple way of looking at something differently.  He challenges conventional thinking, not very overtly, but in a kind of subtle, subversive way, innocently asking, “what if?”

What if customers wrote tests?  (They can’t!)

What if programs told you how they behaved, instead of creating lots tests to ensure they behave how we expect?  (That won’t work!)

What if web surfers could edit the pages they view on the web?  (Chaos!)

Ward knows the most successful systems always start out as simple concepts.  It would be easy to get carried away, to get too ambitious and over-engineer, but he always showed restraint.  The original Wiki, still going strong, is not much changed from the original implementation (although I doubt it is still served over a modem).  Instead, Ward left it to others to explore new platforms, languages, and intentions.  And thus it blossomed.

Which brings me to my third observation on his success.  It reminds me of something my father always used to say: “There is no limit to the amount of good you can do if you don’t care who gets the credit.”  Ward never tried to own the ideas or control them.  He puts everything into the public domain because he feels that’s where ideas can flourish and do the most good.  I believe if he tried to control his ideas with trademarks or patents (some have offered to “help” him by registering Wiki as a trademark), it would have stifled their growth, and maybe we would never have MediaWiki or Wikipedia.

But the best lesson I learned is something I observed when Ward was with other people.  He brings out the best in those around him.  I think he has that effect because he’s such an enjoyable person to be around.  People gravitate towards him.  Humble, unassuming, with not the slightest bit of arrogance, he disarms the cynics and brings people together, getting them to leave egos at the door and get down to the business of talking about really interesting things.  And the best example of that for me is the interview he did with me, which I thoroughly enjoyed.  View it along with the other lightning interviews and I’m sure you’ll see what I mean.

We started out with a large, over-frameworked set of applications built with a creaky buildsystem cobbled together from perl scripts and ant build files. We had all the code crammed into two projects, one for core libraries and the other for application code. The applications were all part of a stack that had a lot of shared libraries and resources and the only thing that kept parts separated were the build scripts which extracted selected packages from the projects and built jars out of them. The jars would then be combined into the classpath for the corresponding application. So there was some attention to keeping components decoupled but nothing that actually enforced dependency management other than the build files. You couldn’t see any real structure emerge by browsing the two projects but after you ran the build you could browse the dozen or so different jar files in the targets directory to identify the common components, the application modules, client libraries for server communication, and the shared open source libraries.

Don’t bother reading that last paragraph again. It’s not worth it. It’s enough to know that things were a mess. We were a long way from a system based on a component architecture with every functional component of the system assigned to a standalone project with it’s dependencies and exports plainly elaborated in the IDE. And even if we did manage to completely reorganize everything, how could it possibly build? Our buildsystem would be useless, and at that moment there were about 25,000 lines of it to contend with.I spent about three weeks doing nothing but analyzing the situation. I didn’t change anything but played around with Equinox. I hacked our jarfiles to make them look like bundles. I wrote some PDE builders to see how they worked. I considered all possible options for breaking apart our two mammoth projects and putting everything back together again in. I read documentation, wikis, blogs and watched a number of EclipseCon presentations. I was already halfway through my alloted time and I had not made one change. Finally I decided to take the plunge. I sent out e-mails to the dev team and told them for the next few weeks (which included the Christmas Holiday) I would be moving code around, breaking the build often, and changing the way they did everything. Then I went to work on a sort of recursive process to break apart the projects and re-constitute them as OSGi bundles.

Here’s how it went:

It’s not hard to make any Java application run in an OSGi runtime. All you have to do is zip everything up into a single file–class files, jar files, resources–and add the appropriate manifest headers. You need to write a few lines of java that implements a bundle Activator by invoking your main() routine. Then you start up your OSGi runtime (Equinox) and have the configuration reference your zip file, which is now a bundle. No application code needs to be changed.

So that was my starting point. I created four different projects for the different applications that made up our product. These projects are OSGi Bundles. I went back to my build output folder from the old ant build and grabbed the jar files needed for each of the applications and copied them into my new bundles. Then I created an Activator class for each one that just called the main() method for the application. I was stunned when it actually worked. It only took a little while but of course I wasn’t actually compiling code. It was just a re-packaging of the binaries.

To get to the point of having all the source in OSGi bundles I was going to have to break down the big bundles into smaller ones and move the source over one at a time. This turned out to be an iterative process that looked something like this:

1. Pick one of the bundle projects I had just created.
2. Go through the jar files inside of it one at a time.For each one, I would create a new bundle corresponding only to that jar file. Let me use utilities.jar as an example. The bundle I created was called com.acme.utilities.
   1. I would take all the classes that appeared in utilities.jar, find the java source files in the old projects and move them into my new com.acme.utilities bundle source folder.
   2. Once I had that working, I would go back through all the other bundles I had already created and find any others that were using the utilities.jar. In those bundles I would delete thejar file and add the dependency on com.acme.utilities.
3. Now my new bundle project had no more jar libraries in it. It also had no source code. I had broken it up into a bunch of bundles like com.acme.utilities so now I could just delete it.

The trick was to work up the food chain starting from the bottom. Pick out the jar files which did not depend on anything else that wasn’t already a bundle. I started with 3rd party libraries and simply converted those jar files into bundles. It only takes a minute to do this in Eclipse or with the excellent BND utility from Peter Kriens. You add the 3rd party bundles to what’s known in Eclipse as the target platform for your application.

After this phase, I had gone from two huge projects to a dozen or so smaller ones, each more clearly identified with it’s function. Instead of one giant “libs” project I had com.acme.utilites.core, com.acme.utilities.swing, and com.acme.utilities.html, etc.

Along the way I discovered problems in our application, like circular dependencies.  These weren’t necessarily bugs but were isolated cases where someone had specifically broken a rule like accessing some view code from the model.  The IDE was catching problems like this.

One lesson I learned was to let go of our classloaders.  We were using classloaders in a number of places to try to scope visibility of jar files or look up classes dynamically by name.  OSGi now managed all of the issues we were trying to solve with classloaders.  Invariably when I encountered places in the code that were passing classloaders as arguments I could pretty much just remove them completely.  OSGi gives you the proper class visibility no matter where you are.When it was all finished I could look at our code and see real structure.  Bundles represented groups of related classes that often corresponded to some layer in the application, like persistence, ui, domain code, messaging, utilities, etc.  Not only could I see the dependencies but I could make strong assertions about them because they were actually being enforced.  I could find exactly who had visibility into classes down to the package level.  The public module APIs were clearly delineated from the implementation classes.  Not only was there loose coupling but there was also good cohesion.

Other unanticipated benefits to developers were immediately apparent:

• It was much easier to understand how things were being used and easier to change them.
• We have an environment that supports different Java VMs in different tiers, but we had to write code to the least common denominator.  Now we could move the major parts of the server to Java5 and utilize things like generics without worrying about breaking other tiers because we had separated out the bundles that were not going into the Java 1.3 applications.
• We didn’t have a 20 minute ant build anymore on the developer desktop.  It was part of the headless build only.  Because the same build artifacts were used to launch apps in the IDE and drive the headless build, they were pretty much in sync with each other so developers didn’t need to do a headless build.  Just edit code, launch application.
• We could create extension points for add-ons, rather than implement complicated hooks with classloaders and convoluted configuration files.  This is part of Equinox, not OSGi per se.  It’s the same kind of benefit you would get with Spring.
• Eclipse was much more responsive.  Apparently having lots of small projects instead of one giant one is more efficient for many tasks.  It didn’t improve Eclipse build times but it did help with other things like auto-complete and refactoring.

There were many, many other nice benefits yielded from this exercise, most of them unanticipated.

Would anyone yield these kinds of benefits from converting to OSGi?  I don’t think so.  We saw such a dramatic improvement because our existing infrastructure was so dilapidated.  Our code was old and the buildsystem brittle.  It’s a common scenario for many Java developers–dealing with the legacy of a system that became massively bloated over time.  We all look at systems like that and wistfully imagine rewriting layers with Hibernate, Spring, Struts, JSF–whatever.  Many times the benefit doesn’t justify the cost, or the rewrite turns into a classic Second System.  From the outset this didn’t seem much different.  Early on I worried this was going to be a very ambitious undertaking, and I had doubts about whether it would spin out of control.

It didn’t turn out like that at all.  In fact, this was my first experience giving a facelift to such an old application with relative ease.  I didn’t have to rewrite any code (other than fixing a few bugs).  I just moved classes around.  I did rewrite the buildsystem, but since the new build artifacts were a tiny fraction of the size of the old buildsystem, this effort paid for itself quickly.   It only took a long time because I was working alone and was being extremely cautious, keeping the system stable continuously throughout the process, rather than just break everything for two weeks and put it all back together again all at once.

Engineers understand the value of this kind of improvement immediately.  You can almost get Product Managers on board with the “SOA in a JVM” story.  But Sales and Marketing won’t bother to stop typing on their Blackberries once you start talking about OSGi.  For me this was mostly about extending the life of a very useful piece of software.  The legacy of older applications isn’t just the code, but all the energy, innovation and hard work that goes into them, regardless of how you feel about the end result.  We owe it to the original authors to make as much as we can of their work.  We’d want nothing less for our legacy.

If my picture wasn’t very clear, or you still don’t fully grasp the motivation for this, I’ll describe a specific example that many Java developers should relate to. But before I do, it might be a good idea to make a more detailed presentation of OSGi and Equinox, the implementation used by Eclipse. Rather than take up space here though, I think I’ll just include some references to articles which I think give the best overview:

This presentation by Jeff McAffer covers some OSGi basics as well as some of the really interesting areas being explored, such as dynamic provisioning of bundles (think of Java Webstart) and API tooling.

• 50 minute presentation by some Equinox implementers introducing OSGi and Equinox. Sample Code available for download.
• Equinox Documentation Page, pretty much all you need.
• Peter Kriens’ OSGi blog (thanks Abhijat!). This covers OSGi in more detail as well as a lot of the politics going on around competing component models.

On a recent project I was working on we ran into a sort of crisis. We had reached a point where development was no longer scaling. Incremental changes to the architecture required a tedious littany of changes to configuration files, build files, test scripts, library paths, IDE configurations, and installer scripts. These changes all had to be carefully coordinated and missed steps often didn’t show up until further downstream; something as simple as adding a new class could result in a ClassNotFoundError in testing if one of the many dependent configurations had not been updated with the new class. So much effort and diligence was required for even small changes that we could not keep up with the kinds of improvements necessary to remain competitive.

We decided to migrate our application runtime to the OSGi Equinox runtime. As it turned out our motivation had nothing to do with the issues above but with other business issues. What we discovered was that after making the investment in structuring our application into bundles, we were able to breathe new life into it. We could again make big changes to the “plumbing” with much less effort and could make stronger assertions about the different parts of the application–what could be safely refactored and what had fragile dependencies.

The improvements included but were not limited to:

• Going from 25,000 lines of ant code required for a full build down to about 200 lines of boilerplate configuration, plus about 200 lines of custom callbacks,
• We eliminated about seventy class file catalogs used to ensure extra classes were not inadvertently shipped in the wrong jar file or duplicated unnecessarily,
• Reducing the size of application distros by eliminating unused dependencies
• Surfacing previously unknown bugs based on dangling references to missing classes and libraries,
• Eliminating a large body of code devoted to managing extensions with segregated class spaces using custom class loaders.
• Going from managing four different runtime configurations for each application–the IDE classpaths, the IDE launchers, the runtime script classpaths, the build script classpaths–down to a single feature descriptor listing the OSGi bundles comprised by each application.

And probably the biggest improvement was in the introduction of the extension mechanism provided in Equinox to extend applications with a sort of dependency injection. This allowed us a new lease on life as a platform for custom configuration and specialized implementations instead of a product trying to be all things to all customers.

The investment required wasn’t trivial but it turned out to be much easier and simpler than I expected. I worked on the conversion myself over a period of weeks. We were able to pull the trigger on the new configuration after about two weeks (implemented over a slow holiday period) and have it fully baked with about a four week effort spread over time. Much of this effort consisted on eliminating the old build system, verifying that everything we did before was being done the same way using the new build system. Developers didn’t really have much downtime but it was very disruptive for them. I had to send out emails every few days or few weeks with instructions on how to do things differently. Most notes were joyous announcements that “you no longer have to” do some tedious routine task that you had to do before. For instance, developers no longer had to kick off an ant build before testing an app. They could launch it directly. They no longer had to update build scripts. We no longer had to wait for the CI server to inform us that a build broke because a dependency was missing. Now Eclipse would show us the missing dependencies. In fact, it did this before but because the configuration of the IDE was separate from the build it wouldn’t really mean anything.

To this day team members still come up to me occasionally to thank me for introducing OSGi, often after being reminded what things were like by having to go back to an old release build. I wish I could take credit but the truth is I never anticipated most of these benefits until I started the conversion.

Now clearly most of the benefit derived had to do with the painfully wrongheaded buildsystem we already had in place by the previous generation of developers. In their defense, it was developed when the only real build tool available was an early version of ant, before they were even using a modern refactoring IDE. And there are a lot of good choices for modern build tools which solve some of these problems (Maven and Buckminster are two that I would look at). But OSGi isn’t really a buildsystem or a configuration management system. It’s simply a runtime that enables solutions to these problems with a much more coherent, consistent and simplified approach.

Our buildsystem, our unit test runner, the IDE, the extensible platform, the launchers, the branding, dependency management–all of these mechanisms are now based on the same set of build artifacts, the Equinox configuration files (bundle manifests, feature and product descriptors), files which are managed with a very nice GUI.

I was expecting to start this entry in early December but I’m learning it’s hard to get a few hours of quiet time at the computer now with the baby and two year old. I’m adapting slowly! I hope to continue the OSGi discussion with the actual procedure for converting applications to OSGi and how the PDE headless build works.

But this appeal isn’t limited to distributed enterprise applications. Wouldn’t software developers like to see a Service Oriented Architecture for the “services” located within a single JVM?

Java Enterprise Software is never written from scratch. Typically it involves writing components that utilize 3rd party libraries, extensions to application frameworks, communications with different tiers. Just look at the class path of any Java Enterprise Application and you’ll see a long list of jar files including JDBC drivers, web application frameworks, validation libraries, Jakarta Commons libraries, and various client libraries. Also, typically you’ll see a number of entries developed by other groups in house: proprietary frameworks, common core utilities, i18n libraries, validation frameworks–on and on.

The end result of the typical class path of a Java application is a giant set of classes from many different libraries all thrown together into a big soup where any class technically can access any other public class, directly or indirectly. Furthermore, nothing guarantees that all the necessary parts are present. Just because everything compiles into jar files doesn’t mean the classes referenced at runtime are actually going to be there, or for that matter be the correct version.

In these applications, dependencies are hard to manage. Tools such as JDepend can be used to identify fragile APIs, problematic dependencies and tight coupling, but nothing really prevents developers from using classes they shouldn’t be using. For instance, once someone uses an Oracle specific class in the Oracle JDBC driver they will be unable to use a different vendor’s driver. Or some code might inadvertently reference an implementation class that is not part of the public API and break on a subsequent upgrade of the library.

Now imagine that the different libraries and frameworks were actually services. They had a public API which is accessible to other components and private implementation classes which are not. The dependencies of any service would be explicit, listing the other services and versions which are required at runtime for them to operate properly. These constraints would be enforced at compilation time as well as at runtime; if a required service in a particular version range is not available in either instance, the system fails fast with an indication of what’s missing. If a service tries to access a public class in a service not listed as a dependency, a class not found error occurs (an error also caught at compilation time). These libraries would still materialize as jar files, but these jar files would be service modules which can be moved in or out of an application even while the application is running. There would be no separate step of composing the classes of a jar file with globs in an ant file. The definition of a service implies the contents of the jar files–no need to maintain an ant file, just a manifest file for the service identifying it’s version and dependencies.

Some of this may already sound familiar. Inversion of Control (IoC) frameworks such as PicoContainer and Spring provide some aspects of the separation of implementation and interface already. Application servers use class loader hierarchies to enforce separation of class visibility within a single JVM. And various application architectural patterns such as Abstract Factories and Dependency Injection provide a means for protecting implementation classes and avoiding fragile APIs.

While these frameworks are all helpful they still leave the programmer in charge of introducing and utilizing them. In most cases, the JVM still loads classes from the big vat of class soup indicated by the class path. What is needed is something more fundamental, something that would completely “invert” the control of the whole application, executing everything as services, not the chosen few components. Instead of the JVM invoking main() on your main class with a single class loader for all your classes, a small application runtime is invoked that looks at all your services, wires up all the dependencies between the services, blocks any that have unresolved dependencies, then invokes start() on each of the services.

Hold it right there, you are probably saying. It already sounds complicated. You might be picturing some big API, or massive documentation you have to go out and read. Sign up for the classes, read the articles, copy the example code, and a week later you’ve got HelloWorld done. Stick with me and you’ll find out it’s not like that. The runtime I’m speaking about is called OSGi and while it is complicated in what it does and how it works, it’s remarkably simple to utilize with the right tool. And more likely than not, you are using that tool already.

OSGi is a standard for a Java component model that has several open source implementations. The most common implementation is calld Equinox. Equinox is the runtime used by the Eclipse IDE and all Eclipse RCP based applications. It’s the lowest level on the Eclipse application runtime stack. It provides the SOA architecture on which Eclipse runs, as of version 3.0. If you’ve programmed RCP applications you’re already familiar with Equinox, but even if you’re just an Eclipse user, you might have noticed that Eclipse is composed of plug-ins which all live in the plugins directory in the Eclipse installation. These plugins are generally jar files or directories that represent the various components of the application. There can be hundreds of these. As an Eclipse user, you may appreciate that you can extend Eclipse with new features by simply adding new plugin jars to the plugins directory and restarting the IDE. If you’ve looked at how Eclipse starts, you’ll see it simply invokes a small jar file on the command line–no reference to any application libraries or classes. As a plugin developer, you are more keenly aware of how these plugins are built, their lifecycle, and the actual mechanics of extending the IDE.

What you may not be aware of is that the plumbing for all of this is not part of the IDE, or even part of RCP, but is actually part of the Equinox runtime bundled with Eclipse. The main class invoked when you start up Eclipse is actually an Equinox entry point, and none of the Equinox code knows anything about Eclipse, RCP, SWT or any of that. This is very important because it means that Equinox can be used as the runtime for any application, including Swing and Web applications.

Eclipse/RCP Architecture built on OSGi

What does all of this mean for you? You have the means and the opportunity to be building your applications with a service oriented architecture, much more easily than you might expect, and I’ve only scratched the surface of the benefits.

You may have noticed it’s been a pretty long gap between my last entry and this one. I’m currently on a leave of absence while managing a new dependency in my family, in the form of a baby boy born on November 14. I’ll pick up this thread in a week or two and show you how we converted our 600,000 line multi-tier swing based application into a service oriented architecture, replacing over 30,000 lines of ant code with about 200 lines, and vastly improving its maintainability and longevity in the process.

Reduce Cognitive Load. This guideline in UI design is about minimizing the amount of things a user has to know in order to be able to use a tool effectively. Likewise, a programmer shouldn’t have to subclass five different classes just to effect one small change to an application. For instance, I often find that in order to add a new behavior interactive application I may have to learn the logging framework, the undo/redo mechanism, the localization framework, and the validation framework–on top of whatever complicated hierarchies exist in the model and views. Things usually reach this state when developers anticipating future needs and striving for reusability and generality in their frameworks end up sacrificing simplicity and, ultimately, usability.

Be consistent. The brake pedal on one model of car shouldn’t serve as the accelerator on another. That would be dangerous. It can also be dangerous to mix thread safe classes with classes that aren’t thread safe, or to have methods accept null String values in some places and throw exceptions in others. If you adopt an idiom in your public interfaces, like converting exceptions thrown by delegate APIs to local exceptions, or using immutable objects as enumerated types instead of ints, then try to stick with it. Consistency fosters a perception of predictability and stability.

…But not too consistent. Jakob Nielsen, a well known usability expert, suggested that you can take certain principles too far, and the principle of consistency is one of those. He felt that Mac interfaces went to excess in trying to maintain consistency, to the point of defying common sense. A famous example: since the operation of removing things from a desktop consisted of dragging them to the trash can, the operation for ejecting a floppy disk was to drag it from the desktop into the trash can, an operation guaranteed to unnerve a user the first time they try it. In software, programmers often step into the realm of over-engineering in an effort to enforce consistency. An action in a user interface may implement an Undoable interface, but that doesn’t mean all actions should be undoable. One might take pains to separate UI code from Domain objects, but if it means introducing intermediary classes and interfaces just to keep domain code from importing javax.swing packages then it may be better to diverge from consistency.

Avoid modality. In many applications users are confronted with modal dialogs which force them to take action before they can proceed in other parts of the application. While a modal dialog is showing, all other parts of the interface are disabled. On the surface this may seem helpful to the user but oftentimes it’s just a convenience for the programmer who doesn’t have the inclination to design the data model so that multiple activities can be conducted concurrently (i.e., changing the properties of an Employee in a dialog while simultaneously deleting that employee in the parent window).

The same thing applies in program design. Sometimes when one object collaborates with another it makes assumptions about the state of that object. Its operation is only guaranteed to be correct under those assumptions. For instance in some data access object there may be operations which result in updating a database object. But these operations might assume that a transaction has been started and fail in the worst possible way if that’s not the case. Users of this class must be aware these methods can only be called in a transaction and keep track of the state of the transaction. An alternative design might encapsulate individual operations in a full transaction, or offer more support keeping track of the transaction state.

Avoid this kind of modality in your classes. You can’t drop all preconditions but the fewer opportunities for failure, the more robust the application.

Utilize Affordances. Affordances are properties of an object that suggest some action can be taken. A door knob “affords” opening a door. A rectangle on a screen may not afford action, but if it is raised and rounded, resembling a button, it affords being pushed. For programmers, affordances might be properties of an interface which make it’s usage more clear. In a class, abstract methods afford overriding. If a class requires several instance variables be initialized when the instance is created, then the programmer should afford that action by adding parameters to the constructor to populate those variables, rather than rely on the caller to invoke setters. Good abstractions and well factored class hierarchies can afford subclass extensions, where flat class structures with duplicate code and concrete classes may not.

Adopt Standards. What more is there to say? Standards in User Interface design result in uniformity across user interfaces, giving the user a sense of familiarity when using an application for the first time. In programming, coding standards provide exactly the same benefit.

Provide Feedback. Feedback in user interfaces usually means giving some indication that a user action has been detected and interpreted. If the action was accepted then give an indication to the user of success. If the action results in an error, make that error known to the user directly. This could probably be interpreted several ways in the context of programming, but the best one would probably be the principal of “Fail Fast.” This means detecting an erroneous use of a class or method early and communicating the actual offense, preferably in a runtime exception, as opposed to letting it go and waiting until the effects of the error cause secondary exceptions which may not reveal the actual API violation. An example of this would be not checking for null values in method parameters but passing the null values on to subsequent calls until a NullPointerException appears in a lower level method. I don’t think you want to check every pre-condition to an exception since it makes methods very noisy, but a lot of times there are conditions which are more likely to be violated. Play the odds and enforce those when you come across them.

You can really go on and on with this idea. The principles of usability apply to any system that interacts with a user, and that includes a programming model. The goal of a great programmer should be the same as the great product designers: empower and delight your users. Programmers should never forget that ultimately their users are not CPUs but other programmers.

The idea of applying Human Factors to API design is not far off the idea of designing programs as works of literature. In both cases your main focus is your audience, not the computer. The common principle in both is readability, but the idea of API usability takes it a little further because your audience isn’t just trying to understand what it’s looking at but actually use and extend the work.

I believe these principles of usability apply not just to API design, but to programs in general. Ken Arnold gives a couple of examples of UI principles applied to API design, such as the use of Progressive Disclosure. That’s the principle that says you shouldn’t overwhelm a user with more detail than they need for a common task, but rather disclose the additional detail when the need arises as they move further along in some interaction. Give the user what they need to know when they need it and only when they need it. That way they won’t be distracted or confused by superfluous information.

For an API, this might mean segregating the methods of a complicated class into the methods most commonly used by client applications from the less frequently used methods needed in special situations or by advanced users. For an application designer it might mean using interfaces and abstract classes to hide the distracting implementation details of a class.

Think about other common principles for user interface design, such those described in the Apple Human Interface Guidelines. How can these be applied to program design? I’ll list some more examples in the next installment of this column.

It may help to view your program less as a novel read in a single sitting and more like a TV show that is tuned in once a week by busy people with short attention spans. The purpose is still to engage and inform, but you need to keep in mind your audience could be jumping in to the story just about anywhere. Like a viewer flipping through channels, happening upon an interesting show, there’s a good chance that the first time a colleague comes across your code is clicking through the frames in a stack trace. They will be debugging an application trying to understand some unexpected sequence of events and will stumble upon one of your classes. Or they might drop in to use some small part of your API. Maybe they jump into your code to extend some capabilities, add a new feature, or make small changes as part of a global refactoring. They just want to know enough to get the task done and keep going. They might watch an episode but then move on.

There are a lot of practical implications to this approach but for this article I’ll just focus on one: give the user the information they need in the context of just “dropping in.” Think about the questions they might have looking at your code in a debugger: What is this class for? What elements are in the collection? Where did this member variable get set? Don’t make them work to find out something you could have made apparent with very little effort to begin with. Your goal is to give them enough information about each element so they won’t have to get sidetracked tracking down other references to the element just to see how it’s used . For example:

• Don’t make it a precondition for them to read a big design document on your intranet just to be able to understand what your GizmoClass is. Put the information as close to the artifact as possible. In the code, or at least in the same directory as the source.
• Document all your classes and interfaces. This may seem like a no-brainer, but it’s not about spending more time writing comments. Instead, keep it as simple as possible. Avoid verbosity and forms. Just a sentence or two to explain what the class is (especially helpful if you couldn’t come up with a really good name). Maybe say what it’s used for. That’s usually about as much as someone needs to know. Refrain from a huge exposition on the overall design or the justification for the class. Save discussion of the class members for the member comments themselves. And if you can’t think of something meaningful to say, don’t say anything at all.
• Don’t duplicate information in an Interface and a Class. If you have to pick one, document the interface. Focus on the interface for describing what something is or represents, and in the implementation, anything noteworthy about the implementation.
• When declaring variables of collection types, if you say nothing else in the comment you should at least state the actual type of the elements of the collection.
• Method javadocs are critical for the casual audience, but not because they will show up in the HTML documentation. Most IDEs will provide javadoc comments in a tooltip or property sheet, saving the user the step of clicking through to the declaration, or opening up the javadocs in a new window.
• Make variable names as descriptive as possible. Not just your member or local variables. People tuning in to your code the first time will have a much easier time decrypting your for loops if instead of i or index you use names like row, month, or customerNumber.
• Consider “Hungarian Notation” for variable names. Many developers find this convention ugly and onerous, but you can’t deny the value of immediately recognizing a variables scope without having to jump to its declaration. There’s a wide spectrum of adherence to this principle; from the old C++ practice of limiting the convention to member variables with a ‘_’ prefix on one hand, to an array of prefixes identifying the scope of every variable.

The point of this is not to enumerate a list of documentation standards for your programs, but to get you to think like an author and empathize with your audience. Use your own experience learning and debugging other people’s code to determine what your audience needs from your documentation. Give them whatever they need to keep them from changing the channel before your program finishes.

What is the purpose of a computer program?

Your colleagues and you all write software.  You generate software prose on a daily basis, crafting these artifacts called programs, line by line.  You utilize a common language that you all agree upon, a language which bears scant resemblance to a spoken language.  So what is it that you are creating?  What is the purpose of these lines of code?

You might say a program elaborates how a computer is to execute an algorithm.  It describes what the system should do and how to do it.  As programmers, we are the narrators and the computer is our audience.  That’s basically what we were taught in our very first computer class.

Donald Knuth, Professor of the Art of Computer Programming at Stanford University, challenged this idea over twenty years ago when he first proposed the idea of “literate programming.”  Knuth felt that computer programs were first and foremost works of literature, to be read and appreciated by humans more than computers.  A programmer should take the same sort of care and attention in writing a program that an essayist takes in writing prose.  He devised a special language called “WEB” which was basically a programming language that resulted in two artifacts: one compiled into the machine language of the computer and the other compiled into a document for humans.  A program listing in WEB not only read like an article, it actually looked like one because it was based partly on Tex, the typesetting language Knuth developed in the 70’s for publishing technical papers.

This would have been a much more challenging idea in the days before open source and tools that readily made source available and searchable, like Archie and the World Wide Web.  It wasn’t as easy to find examples of source code and even book authors were reluctant to disclose real applications in all the gory detail.  Most publications relied heavily on pseudo-code when trying to present samples for human consumption.  Before object oriented programming and the subsequent wave of class libraries and frameworks, programs had to do much more.  Real programs were just too ugly and verbose to put on display.

Open source has heralded in a new era of visibility for software artifacts.  The authors of Linux and Apache know that it’s not enough for their APIs to work as expected.  It’s not just how the systems behave that falls under scrutiny, but the actual source code itself.  The source of the Java Class Libraries makes itself known to most java programmers on a daily basis when they click on a stack frame in the debugger.  Successful open source contributors are very self conscious about how the source code looks.  The computer may not judge their triply nested loops but other programmers certainly will.  Their audience is first and foremost other software professionals.

So our program is a work of literature, then what is its purpose?  The purpose of our program is not to describe for the computer what we want it to do, but to describe for the programmer what the computer will do.  If that sounds subtle, then think about this the next time you are editing code in your IDE.  Is your “audience” really the computer?  Or is it the person in the office next to you?

Most conscientious programmers like to think of their code as maintainable–well documented and readable–but even the best of us may be tempted by obscure or clever shortcuts when we think nobody is looking.  Focus your work on a human audience.  If you’ve ever done pair programming, you know that having someone looking over your shoulder makes you take a bit more care with what you type in.  So imagine that someone is always looking over your shoulder, or that every line is going to be subjected to a team code review, scrutinized by your peers.  Write the code not for yourself, not for the computer, but for them.  Take pride in how your program communicates its cleverness, not just in being clever.  Choose names that not only mean something but could actually be read out loud in something that sounds like English.  Comments shouldn’t be an afterthought–your disconnected scribbled notes in the sidebar.  They should be interweaved in the source, flowing with it and filling in the prose to help the reader move effortlessly through the narration of what the computer is doing.  And remember: the best writers are avid readers.

Those are just a few examples of how programs should play to an audience of humans and not a computer.  I’ll be covering more in upcoming installments of this column.

But you’ll seldom hear him talk about himself. He is modest to a fault, which is one of his endearing qualities.

Bill Kayser

There are other interesting sides to Bill. In 1996, he took a six month sabbatical to sail as a deck hand on the tall ship HMS Rose, scrubbing decks and rigging up sails. He plays soccer fanatically and has badly scarred knees to show for it. Plus he’s often up to something fun – one day we played a prank on his sister, calling her with exaggerated Indian accents, telling her that we were calling from Dell Service Center in Bangalore, and that she should be careful while using her computer because it “might catch fire”.

Bill lives in Portland, Oregon with his British wife Crista, two-year old son Felix, and Boris the cat.

Anyway, Bill has agreed to write a series of articles exclusively for Nagarro that talk about the philosophy of programming and architecture. The first article of the series, titled “The Literacy of Programs” will be out tomorrow. Over time, we might put new articles up on a Wiki or a blog (internal or external?) and then you can discuss the articles directly with Bill. If you have any ideas of how we should structure this Wiki/blog, or you can volunteer to set it up, please do write to me. Plus if you want to write a note to Bill at any time, you can do so at bill@kayser.org.

We are excited by this initiative and look forward to seeing how this will work out.