The Metrics Mess

I recently saw the perfect illustration of how we can get ourselves completely tangled up in unproductive activity by measuring the wrong thing. In this case, it was someone on Twitter who thought they had hit the jackpot because they had hundreds of followers. Further, this person was offering advice on how to increase the number of followers his readers had. This struck me as misguided at best. To be honest, there are folks I follow whom I'm sure don't realize I exist. Equally, there are folks who follow me, but I'm largely oblivious to them because our paths don't cross very often. So the numbers alone don't tell the whole story and may, in fact, tell a misleading story.

The real issue isn't size of following as much as it is scope of impact. How many of these folks are really paying attention to you? How many do you actually affect? Unless you know this, you don't have a good understanding of your interaction with Twitter. Admittedly, there are Twitter stars whom everyone likes to follow. And, assuming we follow because of their established reputations, we're more likely to pay attention to what those Twitter stars say. For the rest of us in the Twitter mob, however, the number of our followers is a poor (and possibly inaccurate) proxy for our impact.

Coming back to law firm knowledge management, take a moment to consider whether your efforts to measure the wrong thing are leading you into unproductive activity. Don't focus on bulk -- focus on impact. For example, counting how many times a particular document is opened via your portal or document management system may be interesting but not helpful. What you really want to know is how many times was it opened and actually used? And, how often was it exactly the thing the user was searching for? In the latter two cases, you learn much more about the quality of your content and the quality of your search engine.

Consider the following: a document was opened 10 times and used each time, but then opened 20 times and discarded because it was not on point. For someone looking at bulk alone, they'd say, the document was opened 30 times, declare victory and go home. However, someone measuring impact would say it was used 10 times not 3o, and then would ask why. When you ask that question you create the possibility of learning and insight. That's when you know you're on the path to using metrics intelligently.



[permission to use granted under a creative commons license]