I planned to write one post that shows my own approach to Social Media Scorecarding but found there’s too much to discuss – so this first of two posts talks about what is wrong with two of three Agency Approaches to Measuring Social Media I will discuss below.

First, I read Agency Approaches to Measuring Social Media by Nathan Gilliatt recently and his post comparing Conversation Impact , Fluent 2009 and Digital Footprint Index, lead me to think deeply about all three approaches and come up with my own scorecard, based on the Digital Footprint Index with I will publish in my next post.

I think the Oglivy’s Conversation Impact and Razorfish’s Fluent Social Media scorecarding are hard to maintain – maybe building indicators that are hard to compile and maintain are not worth the effort – I’ve thought long and hard about this.

I’ve settled on Digital Footprint Index as the best scorecarding approach, even though there are few practical examples – and it’s the only approach where a Scorecard can be created entirely from a Social Media Listening Platform such as Radian6 (or in my case, Radian6 and Alterian/Techrigy/SM2) alone.

Before I show you my scorecard for a small non-profit that is totally unconnected with any Public Relations Social Media metrics work I’m currently working on – I want to point out why I suggest avoiding Oglivy’s Conversation Impact and Razorfish’s Fluent Social Media scorecarding. As Nathan Gilliatt points out in his post …..

• Conversation Impact, Ogilvy proposes a framework with three sets of metrics that correlate to the traditional marketing funnel: Reach and positioning, based on a combination of web analytics, media analysis, and search visibility; Preference, based on media analysis and traditional research; and Action, based on measurable client objectives (such as sales conversions).
• Social Influence Measurement, Razorfish (with TNS Cymfony and Keller Fay Group)The SIM score, as introducing in the Fluent 2009 report, compares sentiment for a company to sentiment for its industry. The report also mentions share of voice and weighting for influence, although the formulas for the metric do not.

… both of these approaches rely, to some extent, on hybrid sources of information that can be unreliable and time consuming to compile and maintain. While it’s true that Radian6 and other monitoring platforms are beginning to incorporate hybrid sources of data (Radian6 has Compete, WebTrends and Salesforce data, to name a few – don’t know what Cymfony and Keller Fay Group incorporate since I’ve never worked on that platform) the current state of the field suggests that no platform has all the data you’ll need to do Conversation Impact or Fluent today, though it’s possible that Social Media listening platforms might someday have all the data needed in one place. That suggests, eventually, it might be possible to build either of these scorecards from one platform, but DFI is only one, I think, you could today.

I believe the Conversation Impact approach is flawed, Web Analytics data will vary from platform analyzing the same exact website, and many of the metrics needed are based on implementation, which will also depend on who sets up and configures the site analytics. When a firm wants to get ahold of a client’s analytics they often can’t get all the needed data, and even if they could, it’s often not in the right form for use with a Social Media Scorecard.

Search Visibility depends on keywords used and the method to query search engines – which is highly dependent on what package to run a ranking report is used and when it is run and on what engines it’s run against. The same ranking report run during a weekday and on the weekend might yield entirely different results because the size of search engine indexes changes hourly, and is often much smaller over the weekend, and even at night – esp with Google, which still holds the lion’s share of search traffic (not withstanding Bing coming into the picture).

To be fair, positioning data can now be obtained from Google Analytics, so, it’s possible that ranking data could make it into a Social Media Dashboard.

Conversion data that is used to measure a clients’ objectives is also pretty hard to get a hold of in a reliable way (though easier than good Web Analytics data, in my opinion) – and when you put the hybrid data together, with the social media listening platform you get a subjective “mess” that is both time consuming to compile, almost impossible to maintain past a campaign launch (read my post PANEL: The New Socialism – Marketing Industry Growth Engine? – OMMA Global)

Another problem with both Conversation Impact and Fluent is they’re better suited for Brand Measurement than Campaign Measurement – here’s one example – SIM score in the Fluent 2009 report is compares sentiment for a company to sentiment for its industry – that might not be so hard to do for a Ford Sedan against all Ford Cars and against the rest of the Car Industry – but what about measuring the SIM score of particular Razor Model, against all the other Razor models out there – researching that and then writing filters to pick up all that data is a nightmare – and maintaining it over time (if anyone cares to, is an even worse nightmare – given the current state of listening platforms – much of the data your picking up isn’t relevant and has to be filtered out). While one could come up with a Razor blade model, and it’s competitors, the problem gets almost unmanageable when your taking something more exotic, like a vacation package or service – and the metrics will be entirely subjective.

The only measurement platform Nathan Gilliatt mentioned his post that I suggest building on is the Digital Footprint Index. Here’s what I like about the DFI Index -

1. The Digital Footprint Index is based on the physical dimensions of Height, Width and Depth, and fits well with how we visualize and move in 3D space. According to Nathan Gilliatt:

• Digital Footprint Index, Zócalo Group (with DePaul University) evaluates a brand’s online presence in three dimensions: Height, which represents the total volume of brand mentions; Width, based on consumer engagement with online content; and Depth, based on message saturation and sentiment.

2. I looked at the DFI Index and the Scorecard can be done almost entirely from Radian6 with some help from Alterian/Techrigy/SM2 – this makes the scorecard much easier to customize, maintain and build on top of.

3. DePaul University contributed to the formulation of Digital Footprint Index and I trust a university to come up with more thought out solution to measurement.

I also thought about the hybrid approach of Oglivy’s Conversation Impact and Razorfish’s Fluent Social Media scorecarding as something that might be seen as a good thing – because if your using 10 sources of data, and one is off, you still have the other 9…ha! ha! On the other hand, if your looking at progress from month to month, or week to week, and some of your data is off, and you have 10 imputs to your formula – figuring out which one is off can become headache.

On the other hand, using Digital Footprint Index, Zócalo Group (with DePaul University) and just Radian6 with Alterian, you can still run into a problem with some of the data they’re gathering as it might not always be reliable, or correctly categorized – or your topic profile might not be sufficiently targeted enough to capture the right data – but at least – you can fix a problem in your topic profile and if Radian6 has a problem with some data they’ve gathered, you can take it up with them and they can try to correct an issue.

In my next post, to be published shortly, I’ll present my own interpretation of the Digital Footprint Index.