IT Service
Metrics 101
What to Measure, and How, When, and
Why to Measure It
Written by Mark
Katsouros.
November 13, 2012
As IT service providers, we all
know how important it is to develop good metrics. Without meaningful measurements, how are we
to gauge our level of performance, and thus how we improve, at delivering the
services most needed by our customers?
And, yet, it seems that so few of us are doing this correctly, if at
all. In order to deliver the most value
to our institution, we must know which
services will do that in the first place (something that is always shifting), how
efficiently and effectively we are delivering those services in order to be
competitive with alternatives, and how adequately we are managing/monitoring
services in order to maximize our provisioning and support capacity, and
respond to growth in demand.
Know the Vernacular: Metrics vs. Measurements
These two terms are not
interchangeable, and a lot of people get them confused with each other, but it
is important that we are all speaking the same language.
This is really straightforward,
actually.
Metrics are the things you
want to measure—the variables.
Measurements are the actual data—the values.
As a simple example, height is a metric, and
6’2” is a measurement.
That example
metric, and the measurements taken over time, can be used, for instance, to
determine how fast a child is growing (or to identify some other
trend).
Trends are the real value in all of this, and are discussed further
below, in more depth.
The Four Critical Categories of
Metrics
While some might argue that there are more, I have found
that most every meaningful metric falls into one of four high-level
categories: (1) Capacity, (2)
Performance, (3) Relevancy, and (4) Satisfaction. The first two types are typically gleaned via
operational measurements—instrumentation, monitoring, and logging. The latter two types are best gleaned via
customer feedback—subscription rates and satisfaction surveys. They are all truly critical, and are somewhat
interwoven, but let us look at each one independently:
- Capacity
generally refers to quantity—how much
of a finite resource is being consumed.
In the case of IT service metrics, this resource is a service (or an
identifiable, finite piece of a service, such as infrastructure). How much bandwidth is being consumed? What is our CPU utilization? How much disk space or memory is being
utilized? (And, of course, how fast are
these numbers growing? Again, trends are
discussed further below.)
- Performance,
in the context of IT services, is mostly about uptime and reliability, but it
is also often used to measure how “healthy” (fast, error-free, etc.) a service
is. How many nines of uptime have we
achieved? How quickly did the data arrive? How many packets were lost? What is the quality of the service? Are
we meeting the performance expectations identified in our service
level agreements?
- Relevancy
gauges how dependent your customers are (or their work is) on a particular
service. This metric is most critical in
terms of distinguishing levels of importance,
and thus prioritizing services—which ones you should continue to provide and,
perhaps more importantly, which ones you should not (or which ones you should
change to better meet your customers’ needs).
- Satisfaction should be pretty
self-explanatory. This is how satisfied
your customers are with a particular service—how they perceive your delivery of
it, or their level of gratification. Without this metric, you are only guessing at
how well your service delivery is received by customers. Of course, this metric is only applicable to
the services that customers deem most relevant to them. (No need to ask, obviously, how satisfied
someone is with a service having little importance to his/her job.)
The Importance of Continuous
Measurements towards Trending
Metrics are the means to an end, or, rather, the means to a trend.
While each of these categories is critical, it is the slope of the lines
plotted across time from which you can learn the most about your services. Is the service moving towards more importance
or less? (Is it time to dedicate more
resources to the service, or plan its end?)
At what rate are the service’s resources being consumed? Are we getting better or worse at delivering
it?
Taking a measurement in time, for example, of a satisfaction
score might yield an average score of seven on a scale of one to 10 (with one
being poor and 10 being excellent).
While seven might seem like a decent average score on its own, it could
be a sign of serious trouble if the past several average scores have been nines
and 10s, or a welcomed sign of great improvement if the past several average scores
have been extremely low. Which kind of
attention (or celebration) to give that average score of seven is totally based
on the context gleaned by the trend.
Knowing when to end services
(one of the most significant challenges in IT organizations), how to manage
just-in-time (optimal) resource growth, how much to focus (resources) on improving
service provisioning and/or customer service skills in general, and
understanding the overall fitness of your service management (and service
portfolio management) processes all comes down to metric-based trends.
A Valuable Tool: The Service Survey
As mentioned previously, service subscription rates and customer
surveys provide the best metrics for identifying trends to gauge service
relevancy. And, essentially, the only
surefire way to know what customers think about your organization’s ability to
deliver services is to ask them. A simple,
consistent, regularly-scheduled survey, that asks two critical questions for
each service, is all you need. Those two
questions are:
Ask these two questions for
each service that can be ordered from your service catalog. Additionally, depending on how centralized
your IT organization is, you may want to ask some questions to segregate the
results into appropriate demographics.
For instance, if you are the central IT service provider for your
institution, you will likely want to specifically differentiate between the
feedback of your institution’s other IT service providers (who might serve as
liaisons between you and their
department’s end-users) and that of
true
end-users. You might have totally
different staff serving these two very different constituencies, or a very
different provisioning process for each of them, so looking at this data
separately likely makes a lot of sense.
“What’s the Frequency, Kenneth?”
Dan Rather references aside, the frequency by which one takes
measurements is critical to maximizing the meaning of the trends produced. If measurements fluctuate dramatically (i.e.,
have a high standard deviation), they need to be recorded more often to
minimize the likelihood of drawing the wrong trending conclusions. More importantly, you want to ensure that you
can appropriately react to trend changes in a timely manner. So, the frequency of your measurements needs
to account for the time you might require to sufficiently take action (e.g.,
increase capacity, add redundancy to deal with an availability issue, improve
customer service training, or even kill a service).
In terms of the above relevancy
and satisfaction survey, one should balance the need for frequent measures with
that of not asking too much of one’s customers (by over-surveying them). One solution is to divide your customer base
into 12 groups and survey one group per month, thereby surveying every
individual only once per year, but still gathering monthly points on the graph
towards more rapid trending. Of course,
this assumes that your customer base is substantially large—large enough so
that one twelfth of it would still provide reasonably large sample sizes, and
thus un-skewed results.
A Fifth Metric Category
In addition to the four
critical categories of metrics above, some count costs among the set of formal
metrics that help expose important IT service trends.
IT cost metrics can include financial measures
such as labor, software licensing costs, hardware acquisition and depreciation,
and data center facilities charges, all mostly from general ledger systems, and
combines these with operational data from ticketing, monitoring, asset
management, time tracking, and project portfolio management systems to provide
a single, integrated view of IT costs by service, department, GL line item, and
project. Costs should further be broken down between capital (one-time) and
operational (ongoing) cost elements.
In
addition to tracking all of these costs, combining them with usage and
operational performance metrics can provide a measure of value or
Return on Investment.
Costs, budgets, performance metrics, and
changes to data points are tracked over time to identify trends and the impact
of changes to underlying cost drivers in order to help managers address the key
drivers in escalating IT costs and thus improve planning.
Storing Measurements and the Grand
Scheme of Things
Besides collecting, and gleaning meaning out of, metrics,
one of the challenges many IT organizations seem to face is being able to
summon and share those metrics when needed/requested. This is a critical part of utilizing metrics
successfully, as IT organizations are often large and complex, with completely
different employees or groups supporting different services.
It is important to recognize the hierarchical relationship
among metrics and two other fundamental service resource layers, (1) the
service catalog/portfolio and (2) Service Level Agreements or, often,
commitments / Operational Level Agreements (SLAs/OLAs). To be clear, the service catalog is a
public-facing menu of orderable
services and the service portfolio is a more-internal superset of the service
catalog that contains services (internal, retired, etc), service attributes,
and detailed data, not typically present in the catalog. SLAs are the performance and response
commitments made to customers of the service, and OLAs are agreements between
IT service providers that address roles, responsibilities, and response
commitments. Metrics should be leveraged
to support SLAs/OLAs, and, of course, also support other service measurements
more directly. The aforementioned hierarchical
relationship is a one-to-many-to-many structure:
In addition to storing detailed
metric data in the service portfolio, a higher-level view of metrics and metric
trends are often made available in the service catalog. In essence, they demonstrate that you are
meeting the service levels to which you have committed, and measuring other
important aspects of the service.
Perhaps equally important, this provides a single authoritative source
of metrics for both customers and service owners.
A Few Words of Caution
There are two risks, even dangers, of which you must be
cognizant:
First, developing metrics, towards enabling better, data-driven
decisions around a service, needs to be approached in a top-down fashion—with
service level agreements/commitments (and perhaps other service goals) driving
what you collect. One of the major rookie
sins of defining and gathering metrics is to start with those that are simply
easy to gather, in which case you can certainly end up with voluminous amounts
of data, very little of which may actually be useful. Instead, start with your service
goals—SLAs/OLAs, provisioning efficiency, customer satisfaction, and so on, and
then identify the metric data that will help you gauge how well you are meeting
those goals.
Second, developing the processes and structures for
gathering, storing, analyzing, and sharing metric data requires a fairly
monumental effort. The very last thing
you want to do with respect to metrics is to not respect the metrics, i.e., go through all of this effort, only
to end up not utilizing the results.
More than just a waste of time, this will create a huge morale suck for
those employees involved in the effort, as well as those service subscribers/customers
who would likely benefit from it. Act on
the metrics and metric trends you expose.
Limited resources demand that
our actions have a purpose, and that our purpose results in action.
Conclusion
Identifying appropriate metrics, taking reasonably-frequent measurements,
and monitoring for trends, particularly changes
in trends, is the only truly-reliable way to not only improve services towards
meeting customer demands and expectations (and ensuring that you are meeting
service level agreements/commitments), but it is the only way to know which
services deserve your focus and resources, and which ones deserve to be placed
on the chopping block (to free up focus and resources for more important
things).
Capacity and performance metrics are typically gleaned via
operational measurements—instrumentation, monitoring, and logging. Relevancy and satisfaction metrics are best
gleaned via customer feedback—subscription rates and satisfaction surveys. Cost metrics, both capital and operational
expenses, are obviously best gleaned from financial systems—general ledger,
payroll, order/trouble tracking, and service and project portfolios.
And do not gather a bunch of metric data just because you
can. Gather it in support of service
level agreements/commitments and other specific service goals. Then deliberately and visibly act on them! In other words, start with your high-level
service goals and then identify the metrics that can help gauge how well those
goals are being met. Again, action
without purpose is as detrimental to good service management (and employee
morale) as purpose without action.
Finally, remember that those trends and trend changes tell
the real stories, are what help us adequately grow (or shrink) services, and
ultimately maximize our effectiveness and efficiency in delivering the important
services on which most every business process of the modern enterprise so
depends.
Mark Katsouros is the Director of Network Planning &
Integration at the Pennsylvania State University. The opinions expressed
in this article are his and are not necessarily shared by the University, but
they just might be, as PSU is a pretty awesome institution.
