Recently, I shared my meeting with a CIO from a Global 50 company. When I discussed with him the importance of IT performance management, this CIO said that no one talks to him about IT performance management. He went onto say that having a performance management system would ensure he gets the business transformation and ROI out of the IT toolset purchases that he makes. “What I need is best practice KPIs to measure along the journey,” he said. Given this, I will turn my attention in this blog to quality management and what measures matter for this activity.
What objectives do customers have for purchasing quality management software?
Once again, I’ve asked some of my savvy customers what their reasons are for purchasing a quality management tool. Here is what they told me:
1) They’re looking to decrease the man hours required for the business and IT to develop scripts for testing (functional, regression, etc)
2) They want to decrease the man hours used for defect tracking and reporting. This includes traceability of quality from requirements, development management, and performance validation.
3) They want the ability to ensure there are no performance issues with sociability testing for enterprise applications. One of my contacts told me, “Right now, an army of people perform testing (functional, integrated, regression) from IT and the business. And the business is always complaining that they do not have resources for the additional product testing.”
4) They’re looking out for the bottom line. Another person said, “These are the justifications to purchase a tool that frees up resources since the tools have a better ROI than bringing in high priced contractors at $200 per hour for testing.”
These are great objectives. And a balanced scorecard can demonstrate that IT organizations are achieving these benefits from a quality management tool. Clearly, such a scorecard should be based upon the business goals set for such a purchase. For purposes of this blog, we will use the four objectives above to guide us.
What would go into a Quality Scorecard?
So if we follow the four-quadrant model for Norton Kaplan, what key performance indicators (KPIs) should be in the scorecard for a quality management purchase?
First Quadrant—IT Value
The first quadrant in the scorecard focuses on IT value. Given the above objectives have a strong pecuniary bent to them, I recommend a series of KPIs here. These include the percent of projects at budget risk and the percent of change in project costs. Now, some of you may object to project measures here, but quality testing is a direct expense to the cost of a project. And since modern quality management happens as features are in fact developed, it is an appropriate measure of cost dimension of quality. The number of escaped defects is on the value side of the ledger. Now this is a strange name but it is about the number of defects that are discovered post release. This is the ultimate measure of value for quality because quality simply should capture defects prior to release—i.e. this number should be very small. The final one for value is % of Actual vs. Planned Tests Executed. This is a way of measuring whether I do what I say that I am going to do in terms of delivering value.
Moving to the customer quadrant, I want to see support for goal number 2. Given this, KPIs that make sense here look at the quality in responding to customer needs. Here I would look at two areas. The first includes the % of Tested Requirements and the Average Cycle Duration. The first KPI tells me the likelihood of a defect escaping into production and the second KPI tells me whether I am reducing the cycle time which is also relates to goal number three above. The second area is Defect Number of Escaped Defects and Average Time to Resolve Production. For customers, this should be “where the rubber meets the road”—I want limited escaped defects and when a production issues happen, the ability to quickly resolve it.
Third Quadrant—Operational Excellence
For the operational excellence quadrant, many KPIs make sense to track change. Specifically, we want to measure the quality of processes and the improvement made to them. These include things such as: % of Reopened Defects, % of Tested Requirements, % of Critical Defects, % of Requirements Traced to Tests, Average Cycle Duration, % of Automated Tests, Average Time to Review Requirement, Detected vs. Closed Defects Ratio, % of Rejected Defects, and % of Completed Tests. There are a lot of KPIs above. Together they measure the effectiveness and efficiency of quality and for that matter, project management processes. Clearly, increasing coverage and automation are critical to reducing cycle time. And reopens and coverage are critical to the quality of process.
Fourth Quadrant—Future Orientation
In future orientation, we want KPIs that among other things measure quality employee satisfaction from the business process change for future orientation. Clearly, if the training is not taking place, employee frustration and sentiment could go up and down respectively. These together show if an innovative, future oriented IT organization is being created.
With this said, we would like to hear back from you. What measures would you put on a quality management success scorecard? Where do you think we are missing something from the above list?
Solution page: IT Performance Management