Thought about the Audience Engagement KPI the Altimeter/Web Analytics Demystified  Social Marketing Analytics Framework put forth and how it might be measured.  While Share of Voice is easy to compile and chart (although applying share of voice is more difficult) qualifying Audience Engagement is not as easy because shares and trackbacks (for a site that is not a blog) are harder to gather (and don’t always show up in Site Analytics).

Upon further reflection, I find the Audience Engagement KPI metric somewhat confusing in weather it is trying to measure the Engagement of a piece of web content (a blog post or tweet) or a website/blog.  I suppose it makes more sense to take Audience Engagement as a measurement of a particular piece of content – like a blog post – where there are comments and trackbacks and shares, and where the total page views can be divided into the sum of the above.

Another thing that’s confusing me is calling this KPI “Audience Engagement” … which implies, to me, that I’m measuring the engagement of an audience with my content (the whole website) .. if I wanted to measure the engagement of my audience with a particular piece of content I created … maybe the name of this KPI ought to be called something else – like “Item Audience Engagement” … something that keys into a separate score for every piece of content rather than an aggregate score for all my content – etc.

On the other hand, since the Share of Voice KPI was applied to a brand or website – shouldn’t all the KPI’s either apply to an entire site … or just a piece of content ?  What confused me is the SOV metric was for a site, but the Audience Engagement KPI was for a piece of content – and doesn’t easily work if you try to get the audience engagement of a site.

While calculating the Audience Engagement of a blog post might make sense,  aggregate site numbers make more sense, yet the pathway to achieve those numbers is missing from this KPI.

For one thing, collecting the number of comments, historically, is easy enough for a blog (you can look at the blog backend or Technorati) but getting the number of shares and trackbacks is more difficult.    Total Views would be the total number of pageviews of a website or blog where the comments/shares/trackbacks were directed to, taken at a domain level.

And TrackBacks were first created for blogs yet they are a built in to the Audience Engagement equation even if you site is not a blog.  While it’s possible to get the total number of Trackbacks for a WordPress blog with the addition of a few lines of code to your blog template it may not be possible to to get trackback counts in many types of website content.

Meanwhile Trackbacks are much easier to get on a post level than at the blog level.  While I seem to recall there are tools that can track aggregate trackbacks to a website (trended over time?), try as I might, I can’t find one. That tells me this KPI should be re-written, as in several instances it will not be implementable (or you’ll just have to ignore trackbacks all together – and that is fuzzy analytics where we should have none).

The number of shares as a metric is also confusing – a share can be measured for a post on Facebook much as a RE-Tweet is.  You can go into Facebook and see how many times a piece of content you can see is shared – but since your content isn’t always on Facebook. counting the number of shares may not be applicable in  many cases.   I suppose for Twitter, they could have substituted ReTweets for Shares.

The number of comments is applicable both to a post or blog and is obtainable in both cases from the backend of a blog.  Comments can appear in many times of Social Sites and depending how much we have to keep track of, can be counted manually or perhaps, with a automated tool.   For example the total number of comments my blog has 3,295 Comments (gotten out of WordPress) – but this option doesn’t let me count the number of comments my blog got over time (as a trend).

Assuming I can’t get a good number for Trackbacks or Shares to a website (I can’t) what does my Audience Engagement look like for my blog?

3295 /124,188  = .27%   This isn’t very helpful to me – I’d need this over time , charted for me by software, but that isn’t happening anytime soon.

To sum up, measuring the Audience Engagement KPI ought to be clearly written to the website level or the post (content) level, but not both.  Also, this KPI  represents a circumstance where it’s best not to publish what you can not support  in the first place.  I cannot show you a concrete examples for a website as too much that is needed by this KPI is missing and I’m not able to recreate it on the fly, no less.

Finally, while I wanted to use the Audience Engagement KPI on Havana Central website (not a particular piece of content) - as I did with Share of Voice.

Even if a KPI is a long equation – the elements of that equation should be easy to obtain or derive – there should be no ambiguity about the formula or how it is applied - this KPI is extremely vague (as to weather it is to be applied to  a piece of content (mention) or a website as a whole) – therefore, poorly written,  and by getting rid of trackbacks and shares (because they may be too hard to gather in some cases) we might get a formula that will still “kinda” works - but ultimately defeats the purpose of what we’re trying to measure – and way too much “fuzzy” math is in the world right now.

Encouraged by various comments on Twitter I’m  trying out the Altimeter/Web Analytics Demystified  Social Marketing Analytics Framework for myself.  Even though there are 12 KPI’s spread across 4 business objectives – I doubt most businesses will use all of them – plus  I can only test 9 (have no access to “Facilitate Support” platforms or the use cases around them).

This post will only look at Share of Voice - and it will be a long.   Also, there are several ways to frame SOV for a business and the choice of how it’s done will be arbitrary.   It is great to have a Framework to fit into – but when you try to apply it to real problem – a real business – the results may be surprising.

As my use case – I’m working with a local restaurant chain in New York City as one of my clients and part of my agreement with Havana Central is that I can use them for Case Studies – still, I will be selective on what information is sharable .

Also – I do my work on the fly – much as I paint (when I do paint) much of the preparation happens in my mind before starting – been thinking about writing this post all day- letting my subconscious work on it before starting.

Earlier today – made an observation about the 12 KPI’s Altimeter and WAD came up with do not appear to address direct Return on Investment and wonder if  retail business establishments have  much to dig into here.    Right now, I think these KPI’s are designed  more for  large corporations than brick and mortar businesses.  Still, I will work have to work with the KPI’s in front of me.

Fostering conversations is the first Business Objective – are we communicating with customers and encouraging them to spread the word about Havana Central?  The first KPI is Share of Voice and is fairly easy to calculate.  But here’s where I run into the first red flag.

Who are the competitors?

Havana Central is a local NYC Cuban cuisine restaurant chain – comparing Havana Central them to larger national chains doesn’t  work – it would be  unfair to compare  a local chain to  The Olive Garden, Pizza Uno, Ollie’s and Houlihans – even as these restaurants are technically competitors – the share of voice metric needs to be modified in context of a business that is only competing, at this time, in Manhattan (a 4th branch in WestChester is in the works).

But, as we have one restaurant in Times Square, another in Union Square and yet another on the upper West Side – coming up with a list of competitors is problematic and open to interpretation.

So the first KPI must be applied somewhat subjectively- to even apply – at all.  Would the share of voice then be Havana Central against all local NYC Latin restaurant chains or would it be Havana Central against all local Times Square, Union Square and West End eating establishments – this is too big a problem.   It might be tempting to go back to the client and ask them (who they think their competitors are - and what if it’s Olive Garden and Pizza Uno plus some local establishments? – since they don’t know how to do it – that’s why they come to us – we’re the specialists) we may get mislead.  On the other hand, if we decide to come up with a search that’s too broad, or too specific – we’ll get results that are widely different and neither helps the client.

As an Analyst I will do this for Share of Voice (since there are no real guidelines – this is truly the Wild West side!).

Even going after local restaurants around each location of Havana Central is a problem – as there are so many possibilities – literally a jungle of restaurants to choose from - so you can’t apply this metrics without considerable interpretation – meaning I will get entirely different results that another analyst faced with the same question.

On one hand – using “analytics” and KPI speak, suggests our results ought to be precise – but the reality is – they are anything but precise.  My earlier posts showed how different each Social Monitoring platform is in Volume, Classification, Geo-Location and Sentiment – plus the analyst will skew the data according to the way they understand the task at hand.  This goes directly against what most customers/stakeholders expect of these tools – and it’s an industry problem – where Social Media Monitoring is today.

I will look at the last year.

Share of Voice>

“Havana Central” AND restaurant

——————————————

“New York” AND restaurant AND (“Times Square” OR “Union Square” OR “West End”)

Even here – should the competition be just Cuban or Latin Restaurants or all restaurants – clearly – that will largely be up to the client (in this case Havana Central – but since they don’t know how to measure Social Media – we must be sure, as mentioned above – to point out what is the best approach), since I’m acting in behalf of the client – it’s my decision – and for all I know, Eric T. Petersen will shoot me down and say …. no … do it this way or do it that way – except – he or John Lovett can’t -  as no one knows how to apply this metric - it’s just a general guideline - without the analyst and good insights – it’s worthless, perhaps even a detractor.

Using Sysomos – this is what I got – Havana Central got 2% of the online chatter of restaurants in the area – if that is to be believed (Twitter has a lot to do with it – if we discount Twitter – we get .5% or 4 times less.

The Share of Voice of Havana Central appears to have increased this year while the overall chatter of local area restaurants hasn’t – suggesting that having a Social Media community manager is, in fact, making a difference.  Does it translate into more mojitos being sold? … We don’t know – and this KPI will not tell us.

What if I geo-located and just considered queries that came from Manhattan – Sysomos can do handle that while Radian6 can’t – nor can SM2/Techrigy and I don’t think Scout Labs attempts to geo-located down to a city – so even here - what Altimeter/WAD came up with leaves a lot to the imagination of how your going to implement but since it’s a Framework – not a final solution - maybe that’s ok .

The local results for New York look  too good to be believed - of all the local chatter that actually came from “here” that Sysomos Map could pull up (they don’t use IP Address but rather – just the information they can find in the content they pick up – and it’s about 85% accurate – I’ve observed, based on other data pulls I’ve done.

Turns out Sysomos Map is picking up very little data that’s based just on NY so maybe the percentages aren’t that accurate – since we don’t really have much data.

Another problem is we many tweets and blog posts might use the name of the restaurant in the tweet, etc – but since we can’t know the totality of what we’re competing with – we must accept a  fuzzy query – fuzzy data – and the insights that come out of it or give up trying to measure Social Media, at all.

Based on the above – it looks like Havana Central is doing something right.

Looking at the guidelines that are published on page 13 of the Social Marketing Analytics Framework increases or decreases in your or your competitors online content is thought to be the main lever for Share of Voice.

After looking at the information I could get my hands or derive myself – decided that content wasn’t really the main driver – there’s been little change in Havana Central’s website for at least 6 months and whatever increases that were driven via Social Media to the site came mostly from Twitter.   In other words, Share of Voice has almost nothing  to do with content (if we were talking organic search traffic – that would be different ) – it has to do with people who are mentioning the brand Havana Central.

So maybe, what we need to look for in Share of Voice is the “Voice” part – what is being said more or less then before – or compared to competitors.

With that in mind I thought about local chatter of those who said, in one way or another, “I’m at Havana Central” but didn’t find anything that really showed there was that much more happening now than before.

At the end of the day – Share of Voice as a KPI might not be particularly relevant unless we have some business objectives around fostering dialog and specific areas where that is going to happen – otherwise we don’t have anything to put the changes we’re measuring online against.

I promised to take a look at SAS Social Media Analytics - had missed the announcements earlier this month while I was in London – and when I spoke at the Sentiment Analysis Symposium the week before last – SAS was present in force but didn’t really talk much about their new Social Media Analytics platform (which surprised me – as why were they present if not to speak about it)?

Anyway – here’s a video of that SAS® Social Media Analytics does:

There are some nice things here – like breakdowns by source – that mimic Web Analytics tools – the very thing I was criticizing about most Social Media platforms two years ago (Techrigy -then, Radian6 then) is largely solved by SAS® Social Media Analytics.

If SAS® Social Media Analytics just pulls your raw data and makes it into something like the video shows – that would be nice – though hardly revolutionary.  In fact, most of what is being shown in this video could be accomplished, more or less, by any of the other platforms I have used – but I’d need to do a little extra work to create tables Excel – etc.

More information from the SAS site says that:

Data integration and storage

• Captures online conversations from popular social networking sites like Facebook, MySpace and Twitter.
• Gathers customer reviews from thousands of review sites like Epinions.com, CNET and TripAdvisor.
• Identifies and integrates influential blog postings.
• Sources conversations both externally and via internal CRM systems, including Salesforce.com.
• Continuously captures and retains more than two years of conversation history.

Ok, that’s nice … agreed

Data and text mining

• Tailors topic classification.
• Customizes business rules specific to a company.
• Applies sentiment to topics within a hierarchy specific to company goals.
• Continuously improves the accuracy of sentiment identification by deploying both a statistically derived and business rules-driven

I think this stuff becomes important if your a big company that has defined processes – then this platform will allow you to take that set of processes and bring into SAS Social Media Analytics.

Media analyst workbench

• Searches source documents having verbatim comments that are the basis of analysis.
• Interprets how sentiment was applied to each document.
• Identifies each concept and the sentiment associated to that concept.
• Adjusts sentiment manually at source document level.
• Provides robust query and reporting capabilities to further analyze document repository.

Media intelligence portal

• Provides dashboard-driven insights that quickly show how core business concepts are performing.
• Enables a quick view of year-over-year comparisons of sentiment against specific business objectives.
• Drills into daily data to understand if a significant event is having inordinate impact.
• Shows verbatim comments to get context behind positive or negative sentiment.
• Shows media source to understand where advocates and critics tend to congregate.
• Shows the influence of specific sources of commentary in the blog and microblog space.

Multilingual support

• Arabic
• Chinese (both Simplified and Traditional)
• Dutch
• English (US/UK)
• French (French/Canadian)
• German (New/Old)
• Italian
• Japanese
• Korean
• Polish
• Portuguese (Portugal/Brazil)

There’s a lot to like here – but it will also take a lot of customization to get the value out of this platform – though that might not be an issue for the target audience.

I’d like to get my hands on a working copy I can customize myself – then I can tell you more.