Basics of Agile Approach in Integration projects

Traditional software development philosophy is increasingly finding it difficult to meet the business expectations and challenges of the large and critical programs. With the increasing complexity in the enterprise and technology, the ‘get it right the first time’ idea of waterfall model bears a high risk for the stake holders to take. Agile development reduces the risk of ‘surprises’ and ‘failures’ through “iterative” and “incremental” development philosophy. The inspect-and-adapt” approach of agile development is an attractive alternative for stakeholders in terms of time and money.

Many tools and methodologies are available to implement and support the agile approach for development of software applications. Many articles and books have been written to cover the implementation of agile development methodology for software application development. However the usage of the agile method for large scale enterprise integration project is not commonly discussed.

This is due to the fact that integration projects tend to be very complex, involving many end points, often different application systems and technologies. Integration projects demand a level of certainty in the design phase, where the input and output to the system is well-defined. Large scale integration projects also tend to involve many developers working on different areas of integration pieces. These integration pieces must then be put together to an “integrated test” to ensure compatibility and inter-operability between one component and the others.

To avoid the risk of lately identified defects, surprises or failures in large scale integration projects as previously mentioned, there is a growing need to adopt an agile approach within the development cycle. This is slightly contradictive with most integration projects’ characteristic which demand design (especially for the input and output of the system) to be decided up front; often referred as a contract driven design.

In an agile paradigm, the product/deliverables is broken down into smaller units of work, known as a sprint. Each sprint team must present a shippable increment of work at the end of sprint time cycle, which usually span across two or three weeks development cycle.

In an agile paradigm, every aspect of development: requirements, design, etc. is continually revisited throughout the project lifecycle. When a team stops and re-evaluates the direction of a project at the end of the sprint cycle, there is a chance that the design or development is steered to another direction.

There can be more than one sprint team exist within a large enterprise integration project. Each team addresses different integration needs with different application systems. However each team may use some common sharable libraries, which may be developed collaboratively across all sprints.

To ensure interoperability of the developed components or the deliverables of each sprint cycle, an integrated testing must be conducted at the end of each sprint. This is a step forward from the unit testing, which is normally executed within the sprint cycle. A dedicated functional test team or system test team typically conducts this integrated testing. The test team works within its own time frame, its own environment – which sometimes can be more than one environment – and must use ready-to-test code base. Different deliverables from different sprint teams must be integrated and released to the test environment rapidly and frequently to meet the aggressive testing cycles of the integrated testing.

The key challenge of this test driven sprints is fixing all functional defects within the sprint cycle. This needs rapid deployment of the fixes to one or multiple test environments as the defects are reported in the integrated testing cycle. This is referred in this article as continuous integration.

Although continuous integration practice is practiced widely in software application development world, it is not a common practice in the enterprise application integration world. The implementation is also slightly different; where in the common application software development, the test tend to be isolated with specific goals involving just one or two stable end-points, enterprise application integration projects involve many different end-points that sometimes can be unstable and inconsistent with the environment being tested. Additionally some of the leading application integration platforms do not even have the tools to support this type of rapid deployment.