Enterprise Integration using Web Services

From last couple of days, while working on an integration activity I was going through some of the Enterprise Integration Patters (EIP) and was also having couple of round of discussion with my clients. During this period of time I got the idea of writing some notes on this topic so that if anyone is already working on this can get some help from the post (read "can help me by giving me back some new ideas").

EI has numerous patterns and we will be looking most importantly the webservice based models in this post.

There are two principal architectures for Web service interfaces

A> Synchronous Web services
B> Asynchronous Web services

These two architectures are distinguished by their way of request/response handling pattern. With synchronous services, clients invoke a request on a service and then suspend activity to wait for a response. With asynchronous services, clients initiate a request to a service and then resume their processing without waiting for a response. The service handles the client request and returns a response when available later. Upon availability of the response, the client retrieves the response for further processing.

However a Web service may combine synchronous and asynchronous architecture depending upon the types of work, the service performs and the available technologies.

A> Synchronous Web Services

Synchronous services are characterized by the client invoking a service and then waiting for a response to the request. Since the client suspends its own processing after making request, this approach is best when the service can process the request in a small amount of time. Synchronous services are also best when applications require a more immediate response to a request. Web services that rely on synchronous communication are usually RPC-oriented. Generally, consider using an RPC-oriented approach for synchronous Web services.

A credit card service, used in an e-commerce application, is a good example of a synchronous service. Typically, a client (the e-commerce application) invokes the credit card service with the credit card details and then waits for the approval or denial of the credit card transaction. The client cannot continue its processing until the transaction completes, and obtaining credit approval is a prerequisite to completing the transaction.

A stock quote Web service is another example of a synchronous service. A client invokes the quote service with a particular stock symbol and waits for the stock price response.

Pic: synchronous webservice model

Synchronous Web service mostly leverages the JAX-RPC servlet endpoint. Client makes the request to the servlet endpoint and then the servlet delegates the request to the service's appropriate business logic, which may reside in the service's Web tier or EJB tier. The service's business logic processes the request (It may access the business data through a persistence mechanism, if required) and When it completes its processing, formulates a response which is then returned to the client through the JAX-RPC servlet endpoint.

Pic: Synchronous webservice with JAX-RPC

A synchronous architecture like this also has the facility to expose a Web service interface to an existing J2EE application that may already have a browser/wireless/rich client interface.

B> Asynchronous Web Services

With asynchronous services, the client generates the request but doesn't wait for the response. Often, with these services, the client does not want to wait for the response because it may take a significant amount of time for the service to process the request and the nenxt operation is not directly dependent on the response (unlike the credit card response for the e-commerce application during check out).

Generally, an asynchronous class of services is used for document-oriented approaches.

A travel desk service is a good example of a document-oriented service that might get benefitted by using asynchronous communication, where the client sends a document to the travel service (may be for requesting arrangements for a particular trip or reimbursement for a trip). Based on the document's content and/or the workflow defined within the business logic of the service, it starts processing. Since the travel service might perform time-consuming steps in its normal workflow, the client cannot afford to pause and wait for these steps to complete.

So far so good. The synchronous calls are easy enough, since we do not need to think about a way out of delayed response tracking. Now in case of Asynchronous calls the service may make the result to the client's request, available in one of two ways:

1> the client that invoked the service periodically checks the status of the request using the ID that was provided at the time the request was submitted. (This is also known as polling.)
2> If the client itself is a Web service peer, the service calls back the client's service with the result.

While designing the architecture of asynchronous webservice based integration, we can think of multiple design patterns. The above architecture shows one of the recommended approaches to achieve asynchronous communication. In this architecture, the client sends a request to the JAX-RPC servlet endpoint on the web container. The servlet endpoint delegates the client's request to the appropriate business logic of the service. It does so by sending the request as a JMS message to a designated queue or topic. The JMS layer (along with the message-driven beans) makes asynchronous communication possible.

The above architecture shows the scenario in case the client itself is a Web service peer. The client application (which could be a direct client or another web container) generates a request and sends it via JMS to the Processing Center (Service A). "Service A" has a Web service invoker that sends the request to the destination application's Web service endpoint (Service B). The endpoint receives the request and acknowledges the receipt by sending an ID back to the service invoker. Later, when the supplier application fulfills the request, its Web service invoker sends the result to "service A" endpoint.

As I mentioned earlier, there could be numerous approaches to achieve this pattern and the above mentioned are only two of them. In case you are interested to add anything, please feel free to write note on the post.

SOA concept and role of an ESB in it

This posts is influenced by my one of my recent discussion about Service Oriented IT trends with couple of my friends. Since SOA being a recent buzz word and with couple of very good implementation of it present in current market from some of the software giants, it has become absolute necessity to understand the concept first.

Service-Oriented Architecture

Service-Oriented Architecture (SOA) has emerged as the leading IT agenda for infrastructure reformation, to optimize service delivery and ensure efficient business process management. Part of the paradigm shift of SOA are fundamental changes in the way IT infrastructure is designed—moving away from an application infrastructure to a converged service infrastructure. Service-Oriented Architecture enables discrete functions contained in enterprise applications to be organized as layers of interoperable, standards-based shared “services” that can be combined and reused in composite applications and processes.

In addition, this architectural approach also allows the incorporation of services offered by external service providers into the enterprise IT architecture. As a result, enterprises are able to unlock key business information in disparate silos, in a cost-effective manner. By organizing enterprise IT around services instead of around applications, SOA helps companies achieve faster time-to-service and respond more flexibly to fast-paced changes in business requirements.

In recent years, many enterprises have evolved from exploring pilot projects using ad-hoc adoption of SOA and expanded to a defined repeatable approach for optimized enterprise-wide SOA deployments. All layers of an IT SOA architecture have become service-enabled and comprise of presentation services, business processes, business services, data services, and shared services.

Fig: SOA Conceptual Architecture

Service Mediation Challenges

A major challenge for SOA initiatives is attributed to the inherently heterogeneous multi-vendor IT landscape in many enterprises, and the resultant individual silos of business information. Rather than incur the cost and complexity of replacing disparate components of legacy infrastructure, enterprises often choose to extend existing business applications as services for use in other business processes and applications.

The influx of Web service interfaces to functionality within existing packaged applications, often introduces services that do not adhere to established service and compliance guidelines. This is especially true if the services are published from core enterprise systems such as CRMs, Data Warehouses, and ERPs.

In the absence of robust and comprehensive service infrastructure solutions, developers have used a variety of “middleware” technologies to support program-to-program communication, such as object request brokers (ORBs), message-oriented middleware (MOM), remote procedure calls (RPC). More recently, IT infrastructure developers hard-coded complex integration logic as point-to-point connections to web services, in order to integrate disparate applications and processes. This inevitably resulted in complex service sprawls within enterprise IT environments. The following figure illustrates a typical static service integration scenario.

Fig: Service Sprawl Challenge

The following are other service related challenges attributed to heterogeneous IT architectures:

• Tightly-coupled business services integration due to complex and rigid hard-wired connections
  


• Difficulty managing deployed services due to disparate protocols and applications involved
  


• High total cost of ownership for the enterprise
  


• Impaired ability to reuse services
  


• Inherent replication of transport, transformation, security, and routing details
  


• Exponential redevelopment and redeployment efforts when service end-point interfaces change
  


• Inevitable service disruption that significantly impact service consumers

Enterprise architects and web service modelers with goals to streamline IT infrastructure now require enterprise service capabilities that address the following IT needs:

• Simplify access and updates to data residing in different sources
  


• Reuse services developed across the enterprise and effectively manage their lifecycle
  


• Provide dynamic configuration of complex integration logic and message routing behavior
  


• Enable run-time configuration capabilities into the service infrastructure
  


• Ensure consistent use of the enterprise services
  


• Ensure enterprise services are secure and comply with IT policies
  


• Monitor and audit service usage and manage system outages
  

Composite Applications and Service Layering

In an SOA initiative, composition is an integral part of achieving business flexibility through the ability to leverage existing assets in higher-order functions.Within a mature SOA environment, complete business applications are composed using existing services to quickly meet the business needs. Flexibility in the service provisioning process, is achieved by avoiding coding logic in service implementations.

Many organizations develop services at very granular levels and the proliferation of many small specific services are difficult to compose into broader logical services. Layering of Services is as a way of breaking out of the limitations of monolithic applications and shortening development, release and test cycles. By defining a layered approach to service definition and construction, the service infrastructure team can achieve the right mix of granular and course-grained services required to meet their current and future business demands. Service Layers typically comprise of the following services:

• Physical Services: that may represent functions that retrieve data in its raw form
  


• Canonical Services: that may define a standard view of information for the organization, leveraging industry-standard formats and supporting a very wide data footprint
  


• Logical Services: that provide a more client-specific granular view of information, generated at compile time using highly-optimized queries
  


• Application Services: that are consumed directly by applications in a line-of-business dependent fashion and may be exposed through presentation services

Service Bus Component of SOA

The core of SOA success depends on an Enterprise Service Bus (ESB) that supports dynamic synergy and alignment of business process interactions, continual evolution of existing services and rapid addition of new ones. To realize the benefits of SOA, it is imperative that IT organizations include a robust and intelligent service intermediary that provides a layer of abstraction to mask the complexities of service integration in heterogeneous IT environments, typical in today’s enterprises. While an intermediary layer of abstraction previously implied a platform for customizing enterprise applications, today it implies toolkits for service customization and scalable infrastructures that support loosely coupled service interactions with a focus on service mediation.

Fig: Enterprise Service Bus

ESBs have been instrumental in the evolution of integrated middleware infrastructure technology by combining features from previous technologies with new services, such as message validation, transformation, content-based routing, security and load balancing. ESBs use industry standards for most of the services they provide, thus facilitating cross-platform interoperability and becoming the logical choice for companies looking to implement SOA.

An ESB provides an efficient way to build and deploy enterprise SOA. ESB is a concept that has gained the attention of architects and developers, as it provides an effective approach to solving common SOA hurdles associated with service orchestration, application data synchronization, and business activity monitoring. In its most basic form, an ESB offers the following key features:

• Web services: support for SOAP, WSDL and UDDI, as well as emerging standards such as WS-Reliable Messaging and WS-Security
  


• Messaging: asynchronous store-and-forward delivery with multiple qualities of service
  


• Data transformation: XML to XML
  


• Content-based routing: publish and subscribe routing across multiple types of sources and destinations
  


• Platform-neutral: connect to any technology in the enterprise, e.g. Java, .Net, mainframes, and databases

Fig: ESB Architecture

A robust SOA suite offers:

• Adapters, to enable connectivity into packaged and custom enterprise applications, as well as leading technologies.
  


• Distributed query engine, for easily enabling the creation of data services out of heterogeneous data sources
  


• Service orchestration engine, for both long-running (stateful) and short-running (stateless) processes
  


• Application development tools, to enable the rapid creation of user-facing applications
  


• Presentation services, to enable the creation of personalized portals that aggregate services from multiple sources

Using ESBs offers greater flexibility for enterprises to connect heterogeneous resources, by eliminating the need for brittle high-maintenance point-to-point connections. Adding an ESB intermediary between service consumers and service providers, shields them from the implementation details of underlying service end-point interfaces, reducing or eliminating the redevelopment and redeployment impacts at the service-consumer level.

Best in class enterprises have achieved SOA success by harnessing high-speed enterprise-ready ESB intermediaries that strategically integrate service mediation capabilities and business process management functionality. Recognizing the significance of operational service management as a critical SOA success factor, they have implemented solutions that provide enterprise-class service scalability, reliability, customization and security. By adopting such solutions built specifically for management and governance of an SOA service lifecycle, these enterprises have obtained the following business benefits:

• Minimized costs by accelerating SOA deployment initiatives
  


• Ensured customer satisfaction by assurance of continuous service availability
  


• Insulated service consumers to changes in service infrastructure by virtualizing service end points
  


• Maximized ROI by leveraging shared services infrastructure and using consistent modeling methodologies
  


• Reduced integration burden by simplifying service interactions
  


• Improved effectiveness of SOA initiatives through accurate run-time governance of shared services
  


• Justification of SOA spending by inventory and tracking of run-time services
  


• Accurate cost benefit decisions by measuring the benefit or cost avoidance obtained through SOA
  

Fig: Enterprise Integration for SOA

Hope this will help to understand the concept and drive of the SOA initiative. My many thanks to one of the leading SOA implementation vendor document to put up this post together.