Issue 39, June 2006
So, Did That Project Meet Your Expectations?
Investment firms invest millions of dollars upgrading systems that are calculated to provide ROI through enhanced functionality, reduced operational costs, and lower risk. Most firms create detailed business cases that quantify the benefits and costs of each technology project and identify the exact timeframe that will provide a "break-even" on the investment. In keeping with this practice, after obtaining funds for the project, you will carefully place your business case on the shelf and leave it, undisturbed, for perpetuity.
Most firms, large and small, require some form of financial justification for every significant investment in their infrastructures. It is considered a "best practice" to gain the support and commitment from all stakeholders - those who will benefit from the investment and those who must pay for it (not always the same groups).
Then why is it also true that a large percentage of firms fail to review projects to make sure that they have met both tangible and intangible objectives? Several reasons exist, of which the one most cited is lack of discipline.
Discipline involves behavioral change. All infrastructure organizations - both technology and operations - should develop a practice of benchmarking their capabilities on periodic schedules, e.g., annually, upon major reorganization or acquisition, etc., to know where they stand and to keep on track with their goals. Benchmarking exercises highlight areas of strengths and weaknesses, giving you the information you need to prioritize your activities. Any firm that outsources its capabilities (technology or process) inserts a benchmarking clause into the outsourcing contract to make sure that it is getting the return that it expects from its investment. Why would you treat your internal capabilities any differently? To be effective a good benchmarking program should not require a new layer of overhead.
You can obtain meaningful and actionable results by incorporating a few techniques into your current operations:
Whether you are planning next year's budget, or your next system upgrade, be sure to specify all objectives in a measurable manner. The adage "If you can't measure it, you can't manage it" holds true here. Headcount changes and transactions per day are relatively straightforward metrics. However, you can also establish metrics for "improved customer satisfaction", e.g., increase retention rate by 10% or increase customer account size by 20%, and other "soft" goals by requiring that stakeholders articulate the real value that these goals will deliver.
Take a snapshot of your current operation before you initiate any changes. Express baseline metrics in the same terms used in the budget or business case, e.g., operations FTEs, maintenance dollars/month, so you can measure improvement at the end of the project.
Conduct a Post Mortem
At the end of the project or year, re-run the baseline activity to determine how far you have progressed. In time you will want to perform this task more frequently to allow for mid-course corrections.
Compare Results With Multiple Sources
Results frequently differ from expectations. Some of the reasons for these differences are obvious - lack of staffing or funding to effect the changes required - where others are more difficult to discern. Obtain performance data from your peers to see how effective their processes are. You may find that others are obtaining similar results; this is extremely useful information to have when you are trying to explain any shortfalls to management.
Experiment with these guidelines by applying them to your next small project. As you gain confidence in the approach, expand it to other, more complex initiatives and, eventually, to your entire organization. Over time you will notice that your team will pay closer attention to organizational and/or project targets, and start questioning whether what they are doing is helping to achieve those goals.
Base Lining Best Practices
Base lining is all about metrics. Metrics should be:
State the metrics in quantifiable terms, and include the units you are measuring. Transactions per day is clear; fewer staff is not.
Identify three to five specific goals that you must achieve with this initiative. These goals should relate to, or be restatements of, the objectives you specified in the business justification that you made to secure approval for the project.
You must be able to collect the data required to calculate the metric. And you need to be able to capture this data at the level of detail required to properly assess the operation.
Are you going to be able to obtain the same data once you have completed the project? If you materially change a process during the project you may find that you cannot make a fair "before and after" comparison.
You need to be able to act on the information. Just because some measurement deteriorates over time does not mean that you can affect the change.