A real (business) problem for SOA and BPM

Core financial business processes like new account opening rely on SOA and BPM to be able to provide real improvement.

Point software 'solutions' pick off components of the problem, but the real need is for a complete end to end process. Although this can be implemented without employing the rigorous frameworks offered by BPM and SOA, the result may be a solution that is hard to change and expensive to maintain, much like the legacy systems of last century.

How does this relate to New Account Opening?

In this post I'd like to explain some of the concepts of SOA and BPM and how they relate to a real process like new account opening, to help readers picture how these technical worlds look when applied to a real business problem, and how to approach using the SOA and BPM frameworks in combination.

Through this discussion I'm going to use new account opening for annuities as an example, or at least the early presales stage of the process. Here is a high-level flowchart of the way the process typically runs (click to enlarge):

In more detail, this is what is actually happening:

1. Prospective customer chooses to by an annuity and with the help of a Producer (basically a sales agent) fills out an account application
2. The Producer uses this application information to create a Master Account for the customer, more as a holding place for additional information than anything else
3. A range of consent, profile, authentication and anti-money laundering information is collected and stored as electronic records (preferably supported by e-signature, but unlikely in the immediate future)
4. At this point the Producer continues on with the customer facing sales process, helping the customer select the appropriate product.
5. Behind the scenes the Broker / Dealer is responsible for using the information collected about the customer in (3) to ensure the suitability of the customer to the investment products being offered by the Producer in (4), based on product attributes coming from the Insurance Carrier
   
6. The Broker / Dealer is also responsible for ensuring that the Producer is licensed and authorized to offer the combined securities and insurance products to the customer (under the particular federal and state regulations)
7. Based on an appropriate product being selected by the customer, certain disclosures may be required, prior to the whole process moving to the next stage of validation, review and account setup

Complex processes make it all worthwhile

The new account opening process for annuities is complex due to the many interactions between the business partners (Producer, Broker / Dealer, Carrier) and customer, spanning potentially several weeks. Each of these constituents in the overall process has their own processes that must be followed, while accessing, creating and updating information in a range of business applications.

It is this combination of business process, information sharing and interaction with business applications that make this a decent illustration of the application of BPM and SOA.

What comes first? SOA or BPM?

Technology practitioners argue about whether SOA or BPM should be addressed first when architecting an improved business process. This is one of many examples of a top-down or bottom-up (business first or technology first) argument which seem to iterate indefinitely. The following sections provide examples that illustrate what I believe to be best practices.

SOA is not just integration

At one level SOA is about providing integrations with business applications - but doing it in a way that allows reuse across multiple parts of the business while minimizing the impact that a change to a system may have on all of the interconnected pieces. True SOA can be viewed as a way of exposing the valuable functions that business applications and legacy systems do as a set of meaningful services. These services are required by typically more than one task, user or other system, making the effort to expose them worthwhile.

You have a process, now see what systems you have

In new account opening for annuities, the Broker / Dealer can really benefit from SOA and BPM, since much of the process falls in their camp.

The original process flowchart for the Broker / Dealer doesn't explicity show all of the systems touched in this stage of the process, instead identifying the key tasks that have to be performed, either by a system or human operation. In a picture, it might look like this

The indivual systems boxes in pink don't directly relate to the required business tasks to be performed. This gives us is a decent reason to put in place a layer of abstraction, relating the business process depicted in the original flow chart to the backend systems. This enables the business analyst to avoid having to put a huge amount of system specific stuff into the business process diagram, retaining its value to the business, and making it more manageable long-term.

Abstract with Services

Since something has to go between the process tasks and the available (or maybe new) systems, we need to approach this from either top-down (business process defining what the technology should do) or bottom-up (available technology defining how the business process should run).

I would suggest that allowing the business process to drive the technology is the best approach to start with, recognizing that in the real world the legacy systems may just be too stubborn to allow this to work. Starting from top down then , we can look at integration as a set of services required to complete the tasks, rather than the technology approach that says 'here is a legacy system, here are all its functions, let's expose them as web-services'.

In this example, this is how some of the key services based on the business tasks could look:

To reiterate, for me the top-down approach is the first step in SOA: identify what I need to do, not everything I can do already. If it happens later on that I find out that its going to cost an arm and a leg to perform some of these tasks in their ideal form, then I can revert to a more pragmatic technology-based approach.

Services hide system complexity

Mapping the required services back to the available systems already in place we see the picture progress:


This helps to illustrate three points:

1. A service can touch multiple backend systems to achieve what it claims to do
2. Each capability offered by a service may perform multiple lower level functions against a single system
3. The complexity of the backend systems presented back to the business process is greatly reduced, enabling a business analyst to identify services that exist already, allowing him to reuse them

The value of SOA alongside BPM

The final point reinforces the idea of SOA. Only if the services are identifiable as valuable business functions will a business analyst be able to see what exists already and be able to reuse them. And it is reuse of services that makes SOA valuable beyond just being a technical abstraction layer between process and systems.

Using our sample process and services lets imagine the scenario where a current customer goes back to her sales agent (the Producer) and requests to buy an additional annuity. The process here is different - this is not a New Account Opening process, but the suitability components still have to be tested to ensure the products offered meet the known customer profile. In this scenario we see that this Add Product to Customer Portfolio process can make use of the same Suitability services as used by the New Account Opening process.

Now it is likely that the IT/SOA team does not have to make changes to the service or the backend systems to provide suitability to the new Add Product process. The business analyst defining the Add Product process can easily plug in the suitability service to the process.

If the services had been defined bottom-up from a more granular technology level, this would require the business analyst to be able to identify and assemble multiple technical functions that are really foreign to the business process to be able to complete the real task.

Summary

Complex business processes need BPM with all of its related tools for definition, coordination and management of change. SOA provides a framework for architecting valuable, reusable services to support the business process interaction with backend systems.

If the IT/SOA team build services based on technology functions alone, they produce a series of low-level web-services that are probably meaningless to business processes without further assembly of the right parts in the right order. Only by looking at services from a business level, rather than a technology function level can the services match the requirements of multiple business process, enabling a business analyst to identify and reuse them.

Despite believing this 'business first' approach to be best-practice, I will hedge my bets by saying that there is always room for pragmatism. Recognizing those systems and services that will be reused is the first step in setting priorities for perfect business services. The second step is the ability of the systems to be adapted to provide this. I have seen legacy systems that just would not allow simple definition of discrete services. Here the technology won out over the ideals of BPM and SOA.

Recognizing how to adapt BPM to embrace difficult systems is something I will talk about in a future post.

Some References

The thought-leaders from the BPM and SOA worlds have been visibly discussing the relationship between the two 'frameworks' in the last few days (and probably behind the scenes for much longer). Bruce Silver on BPMS Watch has provided good critical overview, and as he says Jesper Joergensen has written a good description of the relationship of BPM and SOA, effectively distilling some of the deep discussions in the SOA thread on Yahoo Groups. Steve Jones has also written a nice post illustrating why many customers probably mess up SOA, even under the expert guidance of an SI.

Updates:

Thanks to Rob Hill at Vignette for pointing out that I had incorrectly labelled the first flowchart, with Consumer instead of Carrier. This has been corrected

Also check out Steve Jones' detailed contribution to OASIS proposing a methodology for service architectures. This approaches services from the business direction, and correctly takes it a level higher than an individual business process as the starting point for enterprise SOA.

Technorati tags: Financial Services Technology New Account Opening SOA BPM reuse