Problems With KPI's

 Thoughtful.

Two problems with your KPIs

Nicholas Zeisler -November 11, 202201 views  in CustomerThink

Anybody who’s read much of what I have to say about Customer insights and Voice of the Customer (VoC) knows I’m quite critical of legacy KPIs like CSAT and NPS.  Part of that, naturally, is my inherent wiseass contrarian nature, sure.  That said, if I didn’t really believe that there’s a better way to do things, I’d not have invented a new top-level CX metric of my own (the Brand Alignment Score).

Sure, it’s easy to beat up on “the way we’ve always done it” and call it a day, congratulating oneself on being so much better than the rest.  But it’s also important to understand the limitations not only of our own cleverness, but the principles underpinning our own arguments.  For example, it’s important to understand (and admit) that even the metric I made up doesn’t deliver a panacea.

So let’s talk about how bad metrics are bad in two categories:

Universal—these are properties of every top-level metric, and it doesn’t matter which one you use, there’ll always be problems

Specific—this is why this particular metric is bad, and ideally this other one is good.

There are some things that, no matter how great a new metric you develop or how clever you are (or think you are, or your LinkedIn circle of connections says you are), there’s not any easy way—least of all by simply switching to a new type of question or measuring scale—to overcome the shortcomings that exist for all VoC metrics.  This is the category one needs to be aware of before parading out a great new KPI…it’s what should keep you from thinking you’ve solved the world’s problems.

For example, every metric can be gamed or lead to unintended behaviors based on how it’s applied.  Even operational measures (think:  Average Handle Time, First Contact Resolution, etc.) often drive people to do something that negatively impacts the Customers’ experiences.  It’s like squeezing a balloon…that pressure and air goes somewhere and causes it to bulge out in a different place.  If you emphasize scores uber alles, you’ll get folks doing whatever it takes…to drive that score, not necessarily to drive good CX.  This is why you get service professionals often asking you to give them a 9 or 10 on the survey that’s on the way.  There’s no perfect metric that’ll keep that from happening if the attitude of the organization is centered simply on hitting that target.  Every KPI will get you to that bad place.

Similarly, CSAT, NPS, Customer Effort Score, and yes even Brand Alignment Score are all too vague.  I’ve written before about Score Data and Amplification Data and if nothing else, that concept emphasizes that no top-level KPI will ever give you any information about what to do about your CX…i.e., how to improve it.  It’s a good sign that an organization is simply paying lip-service to “doing CX” if they don’t look any further than their top-level KPIs.  If the purpose of “doing CX” isn’t centered around making your CX better, it won’t matter which metric you use because the key to acting on what you learn entails learning more deeply why and how you’re falling short, not simply that you have (then of course, there’s the huge matter of doing something about it).  So regardless of what KPI you’re using, if that’s all you’re using, you’re missing the biggest part of the picture.  ... ' 

Remote Access Cautions

 Good more balanced view of the recent SCADA threat in infrastructure.  Remote access is a classic potential security weakness.    Have myself been involved in the need to monitor and control industrial systems remotely   The failure was not including several elements in the remote connection.    Fix that now, early warnings are good.  Good points made here: 

Increased uptime? Check. Better access to outside expertise? Check. Improved first-time-fix rate? Check. in Tripwire

These are just some of the benefits of industrial remote access. Yet many customers are reluctant to embrace remote access. Not only that, but incidents such as the breach at the Oldsmar water utility might increase organizations’ reluctance to use remote access.

Using Oldsmar as an Example

The benefits of remote access should not be in dispute. So rather than making remote access the scapegoat, let’s consider the incident at Oldsmar water utility briefly.

It has been established that the nefarious actor was able to access the SCADA system via TeamViewer. The details of how they were able to gain access via TeamViewer is still unknown.

So, based on this information, TeamViewer is the villain, correct?

The answer is not binary. TeamViewer serves a legitimate purpose if used correctly. In this instance, to understand if TeamViewer was the right tool, let’s consider the application more closely.

As a water authority, the Oldsmar plant’s main KPI is to keep the plant operational 24/7 because we all want safe and clean drinking water when we start the tap! This means minimal downtime, timely notifications of any alarms and the ability to diagnose faults promptly. Remote access is an essential tool to achieve this objective. The remote user does not need access to the utility’s IT network to keep the plant operational. And, this is the key – IT and OT’s remote access needs are different .... " 

Maintaining the Measures

Like the general thought, measures are important, but what is driving and changing the measures?  Another example of maintaining the model in use.  Changes will happen.

Modern IT KPIs emphasize cloud, DevOps and user experience

When it comes to KPIs, IT ops teams have typically prioritized process-centric metrics, but recent technical and cultural shifts have started to change that.

By Alyssa Fallon, Assistant Site Editor

Key performance indicators drive IT operations teams to work more efficiently -- but what drives KPIs?

While specific metrics will always vary between organizations, the IT KPIs that enterprises track are evolving as a whole. Technical initiatives, such as cloud and DevOps adoption, as well as organizational changes that emphasize IT-business alignment, have IT shops eyeing a new, or at least more expansive, set of key performance indicators.

Technical drivers
The rise of cloud and DevOps has transformed IT in many ways -- including KPIs.

"DevOps has a unique set of KPIs in terms of how applications are developed, how they are provisioned, how they are maintained, how they operate, how frequently they may fail or need to change, and how quickly they can be fixed," said Carl Lehmann, principal analyst at 451 Research. These unique DevOps KPIs include metrics such as time to provision, time to upgrade and time to value.... " 

Make KPI's Great Again says Sloan Study

Via Michael Schrage ....

In Forbes.com  
Make KPIs Great Again, Study Urges   Joe McKendrick , CONTRIBUTOR

For every technology effort worth its salt, the output is tied to some form of key performance indicators (KPIs) that measure its effect on the business. Now, a new MIT Sloan Management Review-Google study calls these vaunted metrics into question. Despite this being the information age,  filled with data-driven organizations, nearly 30% of business leaders don’t even bother to use KPIs to drive change in their organizations.

"Our research indicates that KPIs are mismanaged and undervalued,” according to Michael Schrage, a research fellow at the MIT Sloan School’s Center for Digital Business and a coauthor of the report. The survey of 3,200 executives finds only 26% of senior managers strongly agree that their KPIs "are aligned with their organization’s strategic objectives." That's consult speak, by the way, for "only 26% have a clue as to what the heck the KPIs are supposed to be telling them." ... " 

KPI in User Experience

Always good to think about useful measures.

What is the most important KPI in User Experience?
by Magnus Revang  in The Gartner Blog

I have many conversations about User Experience with many different organizations. A recurring theme is one of “what do we measure?” and “how much do we invest?”. The key to both these questions lie with the single most important metric in User Experience:    .... " 

Linking Echo to Business Data Queries

Does get back to when is a voice interaction with your data most useful?  An added human channel to data.    Another means to deliver business intelligence beyond the dashboard.

Sisense-Amazon Echo integration lets you ask Alexa questions about your business data  by Ron Miller (@ron_miller)

Sisense, a company which helps customers link multiple data sources and summarize them in a single dashboard view, has been working on ways to understand the data outside of the context of a computer.

Today, the company announced an initiative called Business Intelligence Virtually Everywhere. That could involve asking questions to a voice-driven system like the Amazon Echo or linking important Key Performance Indicators (KPIs) to an intelligent lightbulb that changes colors to represent the current state of the KPI — for example, green for meeting the metric, orange for being in danger of falling short, or red for missing it.   ... " 

Too Much Data? First Use Right Measures

In era of hunger for bigger data, we can still get it wrong.  Its more important to have the right metrics.  Via Think with Google:

Weekly Thought-Starter
Too much data can make for muddled metrics. To drive business growth, focus measurement on your brand’s true KPIs, such as sales and lifetime value.  ....   

Marketers are capitalizing on this by delivering digital marketing strategies to meet consumers in these moments of need. But with mobile taking center-stage for many brands, a new set of challenges has arisen around how to quantify digital's value and capture growth opportunities. We chatted with Adam Lavelle, Chief Growth Officer at Merkle (a leading performance marketing agency) to discuss how they help clients not only measure the impact of digital campaigns, but also drive real business growth.    ... ." 

Performance Metrics

Bernard Marr concisely writes on the 'little data' of performance metrics.   This links well with my constant refrain, the data, big or otherwise, however cleverly analyzed, is only good as far as it relates to business process.  And the key measure of process is performance.  See my previous writings on KPI at the link below.

Building Better Loyalty Programs

Have noted that loyalty programs in general are looking very stale these days.  Nothing new or very engaging.

In K@W: 
Consumers today often belong to a host of retailer ‘loyalty programs,’ earning redeemable points when they spend. In many cases, there is no extra reward for building up a high balance of points in your account. So why do so many people stockpile their points?

This is the question explored by Wharton doctoral student Valeria Stourm and marketing professors Eric T. Bradlow and Peter S. Fader in their paper, “Stockpiling Points in Linear Loyalty Programs,” recently published in the Journal of Marketing Research. In this interview with Knowledge@Wharton, Fader talks about what the trio learned, what it means for the businesses that use loyalty programs and how this peculiar hoarding behavior reflects a phenomenon at the heart of consumer psychology. ... " 

Microsoft PowerBI Preview

Walter Riker points me to the Microsoft PowerBI site.   Preview free trial.  Which adds data visualization and dash boards to many MS and other data sources.    "Power BI works for your role ... Reduce risk and increase rewards - unlock insights to drive more informed and confident decisions. ... ".  Also now has an IOS App:  Microsoft Power BI   Which I have installed.  Will be looking more closely.

More on Power BI from article in the MSDN blog announcement:

" ... This morning at Microsoft’s Convergence Conference, we announced new products and services that will enable organizations to create a data culture in all parts of their organizations. Adopting a data culture starts with making information easily accessible to all - our Power BI offering is helping to accelerate that adoption by changing the way people access, use, manage and share data.

First unveiled in January, the new and improved Power BI is a cloud-based (software-as-a-service) business analytics service for non-technical business users. With just a browser or a Power BI mobile app, customers can visualize and analyze their data with easy-to-use, accessible tools and innovative technology  ...... 

With Power BI our customers can keep a pulse on their business by bringing all their data together and monitor it through a single pane of glass. Live Power BI dashboards and reports show visualizations and KPIs from data that reside both on-premises and in the cloud, providing a consolidated view across the business regardless of where the data lives."  ... '

Power BI Blog.

Unified Decision Processes Driven with Data

At the Webinar today from Innovation Excellence by Julie Anixter on Big Data and New Product development,  Kobi Gershoni and Tom Davenport highlighted the work done over the years by Procter & Gamble in driving analytics towards operational big data, and by using their Business Sphere and Data Cockpit models.   These are ecosystems that provide a means towards a unified decision model.   Linking analytics ever more closely to the decision process.   Some excellent examples provided.   Slides and recording here.  Glad to have been a part of that analytics emphasis there over a number of years.  The alumni of that effort continue to drive this ahead.

(Updated:  See this startup, which is doing fundamental work about getting NPD data entered into the system and thus an excellent first step for a unified system )

Social Media Influencing Companies

This question came up recently.  Do we have rough numbers about how much social media influence companies?  Of course it depends upon how you define influence and on what topic.  SmartKPIs takes a cut at this.

On Total Quality Mangagement

Another topic that has seen less talk lately: Total quality Management.  Here in the Smart KPIs Blog.  "... Total Quality Management – a means for improvement ... 
Customers’ increasing demand for superior quality products and services has made the market a very harsh environment. Organizations are looking for new ways to improve themselves in order to survive on the market and maintain their competitiveness. Therefore, continuous development has become a priority. ... " 

Measurement as a Path to Success

A favorite topic of mine.  Improving measurement can lead to better results, even if you don't apply advanced analytics to the measurements.  Because measurement leads to control.

 " ... Measuring Your Way to Success in Procurement and Supply Management
“Measurement is the first step that leads to control and eventually to improvement.” – H. James Harrington, CEO of the Harrington Institute.

Across the years, companies have traditionally and exclusively measured their success in terms of financial achievements. In the rapidly emerging business world it appeared the need for a much broader range of measures, in order to keep companies and organizations on track into achieving their goals. This is how the fulminating interest in dashboards, scorecards and KPIs can be explained, and Procurement and Supply Management is no different. .... "

How P&G Presents Data to Decision Makers

In the HBR:  This effort started long ago as a method to deliver data and models to executives, and has evolved much more broadly for many employees.  And has finally resulted  in what they call the Business  Sphere.  Note also the emphasis in using data visualization to simplify pattern recognition for users.  Later versions included the ability to include simple analytic methods like regression, clustering and forecasting.    Tom Davenport writes:

" ...if you work in a large organization and want it to make better use of data visualization, I'd argue that commonality is more important than creativity. If you can establish a common visual language for data, you can radically upgrade the use of the data to drive decision-making and action.  The best case I can cite for this argument is Procter & Gamble, which has institutionalized data visualization as a primary tool of management. Working with visual analytics software vendor Tibco Spotfire, P&G has put visual displays of key information on desktops — over 50,000 P&G employees now have access to a "Decision Cockpit" ... " 

The heat map shown is an interesting choice to emphasize.  See the article link for a full readable version. In general, heat maps distort comparisons between data, so they are readily scan-able, but are not a good means for interacting with complex data.  Other graphics shown in the article are more typical dashboard visualization styles.

P&G Decision Cockpit Detailed

P&G's CIO Filippo Passerini provides some detail about P&G's long in development Decision Cockpit, the brains behind the Business Sphere information delivery approach mentioned here previously.   " ... Sometimes you have to put the cart before the horse, according to Filippo Passerini, Procter & Gamble's Business Services Group President and CIO. That's how the consumer products giant came up with a decision-support environment now used by more than 60,000 employees worldwide to see what's happening in the business, why it's happening, and how to respond to changing market conditions. ...   At the IW500 Conference, P&G's Filippo Passerini demos analytic dashboards used to forecast performance and drive the right decisions....   " 

Choosing Relevant KPI's

A slide show in Information Management that has some useful information.  Though this contains useful information, it is also an example of how the style of information delivery can get in the way of the content.  I guess making this a slide show with lots of pictures allows for more advertising impressions.  Another invasion of the infographic.  For me it makes me what to exit quickly.

The Design of Procter and Gamble

I have been reading The Design of Business: Why Design Thinking is the Next Competitive Advantage by Roger L. Martin of the Rotman School of Management in Toronto. The book does an excellent job of introducing you to design concepts, and their applications in business.

I will review the introductory parts of the book in another post, but I want to point out that Chapter 4 of the book, pp. 75- 103, called Transforming the Organization, are devoted to describing P&G's embracing of the design concept in the 2000s, and the remarkable results that came about. I was hired into P&G in the late 70s, very much an analytically and engineering oriented problem solver. I saw this transition occur and although I was skeptical at first, saw a number of very useful results.

A number of new design processes were created to make the best use of mysteries, heuristics and algorithms within the knowledge funnel (well defined in the book) to further the abilities of the company. Top down organizations were set up under the leadership of Claudia Kotchka, to help people understand design.

A method of solving 'wicked problems' was instituted to remove some of the friction in moving innovation forward. Connect & Develop was created to effectively bring in external innovation. I would have included more about how P&G's innovation centers were used to establish laboratories to test the ideas. They also engaged Global Business Services to use IT methods to build 'decision cockpits' to help groups work together.

As the book points out: the idea of 'design is NOT about making things pretty', but how to spark innovation and make it work. It is about continuously demonstrating and letting people experience the value of design in business.

Anyone interested in P&G's approach, and how this is credited by many with saving P&G in the early 2000s should read this book. It's business writing of the best kind, describing both the theory and why it works with great examples. Experience at several other companies is also covered. Its also a nice slice of corporate history at P&G.
-

Executive Dashboard Designs

Many companies are in the process of developing executive dashboards, cockpits, consoles, portals or whatever you want to call them. To what degree have these been designed to be most effective? Designer Edward Tufte has addressed these issues in a number of 'ask ET' pieces at his web site. Worth a look, these discussions provide a number of designs and discussions by designers.

" ...Can you share any guiding principals or "best practices" in the presentation of Key Performance Indicators to the senior executives of a corporation? Discussion.

Executive Dashboard
I'm developing an executive dashboard, and I haven't been satisfied with the business graphics that are widely available (e.g. gauges, dials, stoplights). I decided to make a "Zen" version of a KPI status indicator, using as little color as possible, and incorporating E.T's innovative "Spark Line" metaphor for display of trends. The graphic below shows the proposed KPI display across the top of a browser screen with a descriptive example in the middle. Any feedback would be wonderful!
Discussion..."