The missing link

I don't know when it happened, but the evolution of building software applications for business, especially those based on business processes and BPM, has led to a missing link. Not a sub-human creature of limited intelligence, although certainly a large gap in knowledge and capabilities inside the teams that define and develop solutions.

Leading a recent business process management project, I ran across this missing link several times. And it has led me to compare the profile of the software projects I was part of maybe ten years ago. Back in the good-ol'-days, software projects seemed to have a nice neat demarcation of roles: the business analyst worked with end-users and management to define the requirements that fed the current needs of the business and the perceived improvements; a software architect worked with the business analyst to translate these requirements into a software architecture of blocks, arrows, UML, and written specifications; finally the developers worked with the architect and (if they were senior and knew how to tie a tie) with the knowledgeable users to work out and eventually code software logic.

With BPM, the missing link has evolved through the pressure of vendors, industry analysts, and probably financial pressure in organizations wanting new software solutions cheaper. The roles have morphed as vendors and analysts tell companies that their business analysts can define and build workable processes without the help of their IT team (although see my post: Your 'Systems Group' is the reason you business teams (are) disintegrated if you want to see why I think that it wouldn't work even if your IT team was involved). The reality is that with the right tools, business analysts can build working processes, though they probably don't want to, because real (rather than just workable) business processes are hard, detailed, time consuming beasts.

So what is the profile of a team in the modern world? A senior business analyst has turned highly strategic (maybe because there is more caché in a hand-waving strategy role?), but still gets his or her hands dirty from time to time defining high level processes using a vague understanding of something BPMN-like. A more junior business analyst takes some of the concepts and tries to make a vague guess of how this maps to the details of the business. The great leveller of industry standards says that if you use BPMN you are sure to get a best practice result (I lie), and then fill in some spreadsheets for the other details you find out along the way that might just be useful. Now throw this information at a consulting firm's expert in your chosen BPM product, telling them that the analysis is complete and there is no room in the budget for further analysis, so just get building. Actually sounds very Dilbert-esqe.

What is happening? The software architect role is the missing link. He or she was carefully superseded by the enforced architecture of the BPM foundation the solution will be built on, forgeting that there was more to the role than blocks and arrows. The business analysts do not really know how to interact with software developers, since their contact has only been with senior strategy people they one day aspire to be, so the level of detail they are able (or willing) to collect is variable. But they try. The consultants are therefore forced to try and meet in the middle by wearing the hats of architect, process analyst, business rules analyst, process designer and software (at least customization and UI) developer. With top quality, highly experienced consultants, there is a good chance that they will fit the gap without a nervous breakdown. For less experienced consultants (or maybe worse, consultants highly experienced in traditional software lifecycles), the gap and constant flux in project requirements, definitions and uncertain rules can be highly jarring.

So, who wants to evolve to be stronger, better, faster, and take some of the traits of the extinct software architect? I don't know - and I'm open to offers from anyone who has seen this pattern in projects or believes that the architecture role is still alive and well. My guess is that the profile of projects I've been involved in has led to the customer not wanting to pay for the people I mentioned: process analysts and business rules analyst. Or maybe there still is a software architect role in BPM under a different name.

Either way, this trend of half-hearted, strategy rather than details-oriented business analysis, coupled with do-it-yourself process developers, seems to present a big risk to BPM projects. There are a limited number of BPM implementation experts out there, so if you want to implement a BPM solution, pick well!

I hope someone can help me with some solid experience, or tell me that consulting firms should stop giving way to clients and force them to pay for more time and resources. Or maybe someone will tell me to pick a different BPM tool that makes analysis, implementation and deployment so easy that the missing link was in fact a natural and necessary part of software evolution.

Comments welcome!

A post from the Improving It blog

Download the podcast of this blog post

Your 'Systems Group' is the reason you business teams (are) disintegrated

One of the big selling points of business process management (BPM) is the way it can provide order to the activities of disparate groups of workers so they can work better and more efficiently. To achieve this, you have to answer the big question 'how did groups that have to work together so closely end up so separated?'. In parts of a businesses that handle customer requests and new accounts (such as insurance underwriting or claims, banking account management, credit card disputes, telco and utilities customer service), it is typical to see separate systems forcing the business groups apart. When groups of users have to operate systems that don't work the same, don't share data and require swivel-chair integration, the way they work together as teams disintegrates.

It is the 'Systems Group' (or groups with other vague technical-ish names) that runs the line of business systems. And it is these owners that perpetuate the separation of business teams by failing to integrate their systems. So it seems obvious where to place the blame - the 'Systems Group'. But it is not their fault.

Repeated observation has shown me that the systems group are experts in what they do: they understand their systems better than anyone. They understand how the systems work, how to keep them working, and how to work around their faults. Because of this they are also often the designers of new 'applications' that fill a need of the business, in the form of Access databases or enormous Excel macros, that further separate and distinguish the roles of individual business users. And they are rarely experts in integration, largely due to the time and patience they must expend on nursing their current systems through another painful situation.

I have a recent example on a project I was leading - I had to teach various highly experienced technical people in the system group about XML, what it was, what it looked like, why it was different from EDI, and how a BPM system could use it to capture data from other systems (including that big Excel spreadsheet that is core to the business). XML is really not that new a concept or technology, but the guys had not had the opportunity to understand it.

With a limited knowledge of the available technologies, and even more limited time to burn, an organization's 'systems group' is not the team to apply to even basic integration tasks. If it was, the integration would have been done already.

So who should do targeted integration associated with BPM projects? That consulting team you have onsite already to build the new BPM solution may cost more than your internal team per hour, and may not be pure software developers, but it is likely that they understand integration mechanisms better than most. And if they represent a decent boutique firm, even if they can't build the integrations themselves they should have access to a team that can. Although it may cost a little more overall, the integration will get done, which is more than can be said for the situation many companies find themselves in currently.

The 'Systems Group' is the reason that your business teams (are) disintegrated. But it is not their fault. So help them improve, the same as you are trying to do with your business processes.

A post from the Improving It blog

Download the podcast of this blog post

Stop Persevering, Start Thinking

Consulting and software projects do funny things to incredibly smart people - they make them dumb and irrational. I've experienced it first hand (though I don't admit in public that I rate as incredibly smart).

So what's the deal? We've all used some variation of the phrase 'you can't see the wood for the trees'. That's what tough projects can do to people. They get so involved in the details of the project that they either don't notice that the project is starting to fall apart, or don't notice that it was never really together enough in the first place to fall apart. Then disaster strikes - the shit hits the fan - the customer starts ranting (more than before) and the incredibly smart people fall into the worst trap possible; they keep doing what they were doing before, just now at nights and weekends.

There are only three people or roles of people that I think can help prevent, or at least fix this when it happens:

1) The project manager or leader - the person who is falling into the trap and taking the project with them

2) The project manager's boss, mentor, or someone close to the project - preferably someone the leader trusts

3) The client - the guy who is being the biggest pain in the ass that there is right now

As a project manager, how do you prevent this? Well, I don't know that you can, but try following your own rules maybe and put in some checks or balances for your own personal behaviour, independent of the project.

• How many hours have you worked this week? Is it significantly greater than the average project you run?
• Have you started to find excuses why you can't do things differently in the project (the client will say no to everything you suggest, its just impossible, etc)?
• Have you made your team members miss personal events, weekends, etc?

As a project manager, when you really feel you can't take an hour off, do just that. In fact take two. One hour, complain and moan to yourself, go for a walk, do something that forces you to get away from the PC. The second hour, start brainstorming about how you can change the project; what is the stuff that is unnecessary, what can you do differently. This is brainstorming. Nothing is impossible. You can work out in the third hour (three hours, impossible!) what is going to be most effective at getting back on track with minimal impact to the value of the project.

As the person the leader trusts, have you seen this issue getting bigger? How can you help? I'm no counsellor in this stuff, and have always enrolled in courses of 'tough love'. Painful for everyone involved, unfortunately, though often gets the job done. Help the leader brainstorm, and visualize where the value in the project comes from, which is often not the individual details. Help them understand how they might sell some of the changes to the client.

Then there's the client. Why in the world would you want to put your project at risk, along with the potential risk to your business and credibility? So maybe its time to start compromising a little. Screwing the vendor or consultant out of every last bullet point rushed into the last draft of the contract or scope may just lead to a valuable, expert team becoming so overwhelmed that your aim of getting the perfect system may result in you getting nothing. Or worse still, a vendor that provides a system that looks great until its put into practice (cos those little features and design points you bullied them into were actually meaningless or of minimal value). Finally you could end up with a system that is functional, but nobody wants to go near it, in-house or external, because its tainted with bad-project vibe.

So, when you find that you just need to persevere a little more (beyond the 70 hours a week you already are), or need to shout at the vendor beyond the point that its fair sport, its possibly time to stop, put down your tools, and do the hardest thing of all. Start thinking about what you can change, and what you can ask the client or vendor to be changed, to get the project back on track.

A post from the Improving It blog

Download the podcast of this blog post

Custom software development is really better than complex configuration?

Another week in Mexico City, and nothing chaotic or remotely strange (by Mexico standards) happened. Life is heading back to its usual modus operandi. Offices have thermal imaging cameras at their entrances to catch suspected flu carriers, but traffic on the streets is back to its chaotic self and everyone is happy that this signifies a return to normality.

As for my current project with the large multinational insurance company, things are getting interesting. The team has entered an iterative 'construction and validation' series of development 'sprints'. The Case360 product is doing exactly what it should: providing a flexible platform for rapid development of process and document management solutions. Its nice to know that the messaging I put around it from a product management and marketing perspective while in the company was really true! I have to hope that Global 360 manage to keep some level of focus on the product in the future, since it really is a great technology.

Some interesting facets of using a product of this type, which allows production solutions to be configured without coding (beyond a little script here and there), are:

(a) how fast you can put skeleton solutions together that you refine over a series of iterations

(b) what a waste of time documenting and formally designing a solution can be

(c) customers start to think this is so easy they can do it themselves

The power of a software product that allows configuration of meaningful solutions, based on its templates and best-practices, also carries some risks. The major one is (c). A deep understanding of what the product offers, and how all the available pieces fit together in a usable and maintainable way is required by the consultants doing the work. Otherwise the solution becomes a series of disconnected and confusing components for the end user to navigate (how many of you remember Lotus Notes applications?). And some complex requirements, though possible to meet, require some serious thought, prototyping, rework, and occasionally coding, that can be hard for a customer to comprehend.

Its strange to me that customers are often willing to accept the opaque nature of custom software development if the whole solution is based on this. But if a solution is largely configuration and clever reuse of templates, anything complex is looked at with contempt, even though that the final solution is far more manageable than most custom software. There's no pleasing some people!

A post from the Improving It blog

Trust is key in software projects

Building relationships with customers based on trust is difficult. When you are providing your customer services, rather than tangible deliverables upfront, this becomes even harder. Trust is essential in software projects of any style, since it absolutely will affect the perception and success of any deliverable. I don't proclaim to be an expert in the software services business, though I have run several large software implementation projects, while being involved in numerous others from the sidelines.

Right now, I'm running a business process management (BPM) and document management implementation for a global insurance company in Mexico City, while being involved in a technical consultant sideline role for the same company in Chile. If I had been the customer I might have chosen to stagger the implementations to make best use of the reusable components and experience, but the reality of budgets and timelines does not always allow. Either way, we have two projects running simultaneous, for the same parent company in different countries, following very different implementation styles:

• Chile - traditional (waterfall) model: analysis, design, development, testing, rollout

• Mexico - a modified scrum: iterative analysis and development with iterative timeboxed deliverables, final testing, rollout

The challenge from either approach is proving to be building the trust of customer early enough in the project to set out on the right track for the style of project you want to run. So here is what I'm seeing.

Large insurance companies continue to be conservative in nature, meaning that a traditional software development cycle is something they are familiar with and comfortable with. The problem is that the trap becomes paralysis through analysis. The development of the solution may never get started, because it is impossible to get the customer to agree on the scoping and initial analysis. In this case, I can't tell whether the design will deliver what the customer really needs, as they have no experience with the best practices or limitations of the tools they are building on. Within a reasonably well defined timescale they are going to get a set of applications that work according to the design. Trust in this environment has come from extremely tight scoping and analysis sign-off, and will hopefully build into a more flexible trust soon.

Taking another approach, I've been working on my customer to move them to a more iterative project. This has been uncomfortable for them, as they feel that they want to know what they will get before agreeing to the scope of the project. There are items they are clinging onto strongly that I would bet a fresh fish taco on that will be dropped (or at least significantly diluted) by the end of the development as they understand what these requirements mean in reality. The challenge here has been building enough trust between client and consultants early in the project that they will get a usable deliverable in their timeline. Perfection has never been promised, and there continues the risk that the scope which forms the contract is open to interpretation. The deal here rests on a combination of personal trust and constant expectation setting, aligned with a flexibility within a restrictive scope.

Which project will work best is yet to be seen. Chile has the best chance of meeting the customers initial requirements most closely, but has the equal risk that these requirements will not provide the full value that could be envisioned, requiring a further deep iteration. Mexico has the best chance of being delivered to time and budget, though with many gaps that the customer will return to in a second phase. Similar result, we'll see which gets closest in expectations met, time and budget.

My belief right now is that the iterative approach is a good fit for the tool we are using, and a loose BPM methodology. I'll be letting you know whether this was a good bet or not!

A post from the Improving It blog

Business continuity: how will your company cope?

I'm in Mexico City working on what could be (could have been?) a fun and challenging 3 month project, implementing business process management (BPM) and document management in a global insurance company. Now the city is faced with being the source of a potential pandemic of swine flu. This is giving me first hand experience of how companies handle business continuity planning (BCP) in different countries. And I'm reminded once again that, business continuity is not about having a bunch of backup tapes.

My earlier experiences with business continuity planning were working for a vendor of Imaging and Workflow in the UK, in the government and insurance sectors. Customers of any size believed that if they were going to implement such technologies to improve their business processes they would probably become dependent on them. They also seemed to buy the concept that if you were going to improve your business processes, you should take the opportunity to improve the security of your information and the resilience of your operations.

The whole concept of business continuity was reinforced to me with a very powerful punch one day when I was helping a large systems integrator to design a system for the UK government, to support around 6000 concurrent users. Not only were we discussing the value of near real-time backups to offsite locations, but added to that the logistics around providing a system to test business continuity. The testing would involve not just taking a server and unplugging it to test failover to a secondary system, but would also test how the plans for moving a set of essential users to a temporary location would operate, both technically and logistically.

Then the first of many amazing calls came in. The day was September 11th. We all know what happened next. Less obviously, the business continuity procedures of hundreds, if not thousands of companies were tested. There are many examples of those that worked and those that didn't. Some companies managed successfully to move their entire financial or trading operations overseas and absorb the increase in load for the remote resources with seemingly minimal effort. Other companies just vanished along with their information, their infrastructure and, extremely sadly, the knowledge in the heads of some of their key workers. Since those days, companies have learned their lessons, and some have unfortunately forgotten them.

I moved to the US in late 2003 to work in the same imaging and workflow business with the same profile of organizations. I was shocked at how little attention US companies and government paid to providing more than a bunch of backup tapes for business continuity. In the event of an office flooding, being struck by lightening, or something more catastrophic, these organizations would not only have to rebuild their complex servers from the ground up, they have to restore terabytes of information from tape, and procure PCs and temporary office space for their workers. Weeks of time, and in the case of the commercial organizations, hundreds of customers would be lost. In my opinion, US companies seemed unwilling to invest to mitigate risks such as these. Since some of those organizations are / were around the Gulf coast, I hope they are still around to tell the tale.

Things have changed at least from a remote infrastructure standpoint. Companies have the advantage that most of their professional workers, even here in Mexico City, own a PC and a broadband Internet connection. The infrastructure of temporary office space may be unnecessary. And I have watched many companies making the most of that today with the current flu situation, as they try to reduce the number of groups of people in close proximity. In the pair of office tower I work in, Torre de Esmeralda, an estimated 40% of people were working from home. The question is, if the minor tremor of yesterday had been more than 5.6, how many would have had adequate, ready to roll offsite servers and information available for those users, or others in a completely different city?

Business continuity planning, the technology of resilient business process management and document management, and the logistics to make it worth anything, is being brought back to the attention of companies worldwide.

A post from the Improving It blog

Mexico City - up front and personal

What an interesting time to be in Mexico City. Watching the inhabitants of this enormous city's response to what could turn out to be an incredibly nasty situation has been an education in the culture of the Mexican people. Watching how the government and companies are addressing the situation as well has also been interesting.

I've only been in Mexico for 7 days, which means that I had been here about 3 days when the news about the first cases started to be recognized as meaningful. Listening to the discussions about the issue in the office where I am working (a multinational insurance company) has been enlightening.

Day one of the news (at least the first day I was aware of it), people were joking around about about having the flu every time someone sneezed.

Day two, the company was distributing blue face-masks to everyone in the office, the doors were opened to ensure air circulation, and presumably the air-conditioning (if there really is any, as it always feels tropical) was turned down. People continued to joke about the flu, though the habitual greeting kiss was avoided as recommended by the government. Maybe 75% of the office population had their face masks on, both inside and outside the building. Given the speed that the face masks appeared, I wonder if they were kept in storage by the company for this eventuality. Business continuity planning at its finest.

Day three, and people seemed to be getting tired of the face masks. Those who had been wearing them, kept them round their necks, and I'm not sure if this was as a symbolic protective chalice, as a safety net in case they spotted someone who was sickly looking in their vicinity, or because they had not quite reached the stage where they had the confidence to take it off completely however much it was annoying them. Schools, universities and government institutions are shut. Hospitals and health centers are open 24/7. Keeping people apart in this crowded city is a tough proposition.

Day four. I'm not in the office as its a Saturday. A large handful of people are walking around the city wearing face masks, but I'd say the majority are not. The civil defense troops have been out in some parts of the city distributing face masks, and the metro is also handing them out. Unfortunately, I don't see the taco stand cooks wearing them, and I bet no one in the restaurant kitchens is. Maybe tonight would be a good night to stay in and cook!

The people of Mexico City seem to have taken this worrying situation in their stride, showing their typical relaxed, courteous (unless they're driving) and humorous character. The government has rolled its health plans into action quickly, calmly and without fuss, which is probably a good thing, as a city of 23 million inhabitants panicking could lead to more of a disaster than the potential spread of this nasty virus. The good thing is that people still seem to be getting on with their lives (based on the level of traffic outside my window).

Fingers crossed that the number show a positive decrease in cases over the weekend.

A post from the Improving It blog

Twitter... And back again...

Back in January I announced I was not going to blog any more, instead diverting my creative juices to Twitter. Well, there is only so much that can be said in 140 characters, so I've decided to make a trial comeback. To try and refresh the blog a little there is no big makeover, just a change in name. For the astute, you'll see the name changed from "Improving New Account Opening" to "Improving It". I wanted to give myself some latitude in what I could write about, while not scrapping the great (and mostly relevant) content that is already in the blog.

So for my relaunch, welcome (back) to Improving It. In this blog I'd like to have a conversation with anyone that stops by. If you are interested in technology for solving complex business, social or economic problems, remember to click the RSS, Twitter or Email links in the side menu, and I'll be there ready and waiting in your feedreader, inbox or Twitter. Or just stop by this site.

I'm going to try very hard to make this a conversation, so please comment, trackback, tweet or whatever technology you use. If I see you link me, I'm more likely to read whatever you are writing about. If its good stuff, I'll probably return the favor. As such, its time for a clean slate, so the blogroll / links will be cleared out to make room for current exciting stuff.

Enough of the admin, I look forward to chatting with you all soon about technology, business, the economy, the environment and whatever else takes our fancy!

A post from the Improving It blog

Changing to Twitter

Microblogging is apparently THE way to communicate and share discussions. Twitter, really little more than a central site for publishing and tracking SMS-style messages, is the great enabler in this. So if you are as tired of reading huge long blog posts as it seems I am of writing them, feel free to follow me on Twitter: http://twitter.com/improvingit

The name change is there as I'd like to broaden my discussions. My work world for a while was finserv solutions and pure IT. In blogging, I realized there is a lot more that interests me to chat about, so Improving It (or IT) gives me a little more freedom to discuss the issues that affect all of us in business and life - from technology, IT and green issues, to the whole human race just getting-along. And I hope that Twitter makes the discussion a lot more interactive (its hard to write an uninteractive diatribe in 160 characters).

I may post here from time to time, though I'd really like it if you chose to follow me on Twitter and join the disussion. The RSS feed is linked at the bottom of this page: http://twitter.com/improvingit

Thanks for reading and I hope you can follow me on Twitter!

A post from the Improving New Account Opening blog

The history of workflow and BPM

Congratulations to everyone in the States that survived 'death by turkey-sandwich' over Thanksgiving, and hopefully managed to spare just a moment to remember the meaning of the long weekend. Unlike Independence Day (where I just get jokes about how the Americans threw the English out, and my jokily sarcastic replies fall flat), I can buy into Thanksgiving.

Serious stuff over, its back to work now. My Monday was spent turning over a new leaf and making sure I get my head out of the tasks I'm doing long enough to look around and see what else is going on out there. I find that anything involving really using software (I've been building some demos with the sales engineers using Global 360's case management software) can quickly pull me down into the weeds of 'doing it right' rather than 'getting it done'. Anyone that has worked around Sales knows that the deadlines rarely let you really do anything 100%, so this can be a challenge for me.

In looking around, I came across this nice morning coffee post by a guy I hadn't read before, Arshad Nazim. In it he writes about the brief history of BPM, and in doing so I think answers the question that many long-time software people have when they first see BPM: 'what's the difference between BPM and workflow?'. Not mine to answer, but I think that Arshad does a good job of doing so. [Note: it seems the credit should go elsewhere, since Arshad forgot to credit Sandy Kemsley who wrote the whole series series A Short History of BPM, which was a pleasure to re-read in its original format on her Column 2 blog. I've moved the link to Arshad's post to here, so that readers can review the additions Arshad made subsequently].

I'm looking forward to starting to write more regularly again, so keep an eye out for my next 'original' post.

A post from the Improving New Account Opening blog