Posted by Markus:
“You cannot manage what you cannot measure”
Everybody has heard this phrase before and it boils down to the need to develop metrics that describe how an organization or a process performs.
Those metrics are used in a variety of ways
- They often are being developed to describe subjective circumstances in a more factual way
- They serve as factual data to show the progress that one is making (historic comparison of metrics)
- They are being used to be able to benchmark against other organizations and create a ranking or compare performance
- They determine our bonuses 
Outside the EA world, metrics have been around for many years.
In the ERP world, KPI’s like Inventory Turns, DSO (Days Sales Outstanding), Return on Investment, Discounted Cash Flow, Meantime between Failure or Asset Utilization are being used to describe the efficiency and proficiency of organizations when it comes to business execution. The organizational performance measured through those KPI’s also reflects the maturity of organizations with regards to business processes, organizational structures and the alignment of objectives and execution.
We all know that Dell for example was the poster child of world-class Supply Chain Management capabilities. They not just defined the business model of “mass customization” but also were able to execute this business model through all levels of the supply chain which resulted in previously unheard inventory turnovers and minimal product lead times while keeping supplies at a minimum. Apple’s ability to not just design revolutionary products but at the same time, reduce time to market and product innovation cycle times dramatically is another example. Those are not just innovative companies, they demonstrated through their flawless execution that they were able to operate on a very high maturity level.
So how is maturity in the Enterprise Architecture world defined and how can we measure KPI’s in the EA world so that we can effectively manage and benchmark the performance of the Enterprise Architecture discipline?
In a Gartner Blog, Robert Handler describes a recent Gartner Poll in which
– 17.8% of the respondents said that “Demonstrable Financial Impact” is the most critical EA practice
– 2.2 % of the respondents said that “EA measurement and Reporting” is the most critical EA practice
Help me understand this!
How can you measure “demonstrable” financial impact when you pay almost no attention to EA Measurement and reporting?
This reminds me of the comment a business exec made who said he is still waiting to be invited to a project meeting in which the IT project AFTER completion is being reviewed with regards to the achieved (and promised) benefits and ROI. Interestingly enough in the Gartner Poll mentioned above, 51% of the participants responded with “Solid Business Vision” which of course is extremely easy to measure
How do large “EA mature” organizations measure their EA performance? What experiences have folks made with regards to defining “subjective” measurable Enterprise Architecture performance indicators?
Or does it all boil down to Enterprise Architecture being an art?
Thanks for your comments
Marcus, the question you pose is an interesting one. I believe the maturity of the EA Program is best measured in a larger context, and also in terms of total benefit to a company. The firm I am with – The BTM Corporation – measures the maturity of what we call Strategic Enterprise Architecture, or SEA, with three other functional areas – Governance & Organization (G&A), Managing Technology Investments (MTI) and Strategy & Planning (S&P) against a maturity scale of our framework. Each of these functional areas has a set of defined critical capabilities which we deem essential for mature business performance. The capabilities for the SEA are:
• Application Portfolio Management
• Asset Rationalization
• Business Architecture
• Enterprise Architecture Standards
• Technology Architecture
Each capability is evaluated within four dimensions for each interval on the maturity scale – integrated processes, organization, information and technology.
We have found through extensive research that companies which demonstrate operations at a high level of maturity across all dimensions for all capabilities, have been able to converge business and technology management, and outperforms competition within their industry groups significantly. SEA plays an important role in this, but needs to be measured along with other critical functional areas, and when that takes place executive management recognizes its value in the total business context.
Thanks for your insight into this.
Let’s use one of the capabilities you describe as an example:
Asset Rationalization
First of all one interesting discussion point is what is viewed as an asset in the EA world.
That in itself can easily morph into a philosophical discussion but let’s not go there.
Let’s assume we all have an understanding of what we mean when we use the term asset.
What would be good examples of measuring the maturity (and ultimately the enterprise capability) when it comes to asset rationalization.
How do we measure the degree of rationalization?
Number of retired applications?
Number of retired interfaces?
Number of decommisioned servers?
To use the value concept, examples for “objective” KPIs that could come into mind are:
- Current Ratio of IT asset value to Mio USD in Company Revenue
- Decrease in overall IT asset value over the last 12 months
- Ratio of IT asset value (CAPEX) to IT Asset Expenses (OPEX)
- Current Asset Value (weighted by life time value of asset) compared to Asset Investments
What I am really trying to get to are some standard metrics that could be used as benchmarks to demonstrate value of an EA initiative and the sum of those EA initiatives describes the maturity of an EA organization.
Executive Management understands the value of a supply chain initiative when inventory is reduced by y % and therefore free cash flow increases by x %.
The sum of all supply chain initiatives and the value to the bottom line (savings, cash flow, etc) represent an maturity indicator for a supply chain organization.
I think EA tends to describe benefits as strategic without pinning down the true bottom line impact.
That makes the development of true benchmarks difficult.
Markus
When we established standards for measuring the maturity of SEA (Strategic Enterprise Architecture)we did not go for “hard” benchmarks, and used criteria for the four dimensions (process organization, information and technology) the capability Asset Rationalization’s maturity was measured against. At the highest level of the maturity scale for the BTM Framework these are:
- Process; refine processes to link asset disposition roadmaps upstream to business and architecture roadmaps, and downstream to investment portfolios.
- Organization; make sure that the Office of Architecture & Standards (OAS) is properly staffed with business and technology personnel to develop, execute and manage the asset disposition roadmaps and investment programs together.
- Information; develop and document performance metrics to measure the impact of the disposition investment programs on the business.
- Technology; Extend the user base of the asset portfolios, disposition roadmaps and investment programs to include all relevant support and decision bodies and their stakeholders, and insure that the information is current and includes performance and benchmarking information.
In this manner, each company is free to define its own benchmarks and metrics, however, the maturity is defined by the way the capability is managed.
I see your point.
For me the focus of benchmarking was always externally focused.
Being able to compare the performance / maturity of an organization with peers in the industry is critical to understand risks to and potential for the competitive differentiation of an organization.
But to be able to benchmark against other organizations, clear defined metrics and KPI’s need to be established.
Example from your post:
How do you define “properly staffed”?
Without a clear approach how to define and measure metrics we will be comparing apples to oranges and there is too much room for interpretation.
Thanks for your comments
Markus
I see your point.
For me the focus of benchmarking was always externally focused.
Being able to compare the performance / maturity of an organization with peers in the industry is critical to understand risks to and potential for the competitive differentiation of an organization.
But to be able to benchmark against other organizations, clear defined metrics and KPI’s need to be established.
Example from your post:
How do you define “properly staffed”?
Without a clear approach how to define and measure metrics we will be comparing apples to oranges and there is too much room for interpretation.
Thanks for your comments
Markus