This post is the first in a series of comprehensive comparisons of 5 social media monitoring platforms I have access to which will be part of my course at Monitoring Social Media Bootcamp taking place in London on March 31st, 2010 (two months from today) - Radian6, Alterian/Techrigy, Sysomos, Brandwatch,  Biz360 Community Insights.  I also have access to Crimson Hexagon Buzz Monitor but am not including it in this analysis – I’ll be happy to include Crimson if asked.

I’ll use the John Battelle approach and develop my content and thoughts online, and let the community of readers comment – and your thoughts will add to my course.  In return, I’ll share with you what I know.

Sound fair?    Let’s start with the query – “Social Media Week New York City“  which is going to be taking place next week and will give me three things  I need to test and compare these platforms on

1. An Event
2. A Location (test Geo-Location accuracy)
3. Sentiment
4. Influencer Lists of Authoritative bloggers
5. A Summary of Social Media Week NYC after it’s over using 5 platforms.  Which ones will end up being most useful?

A couple of thoughts before I start  – and this series of posts will take time to complete.

1. How these  platforms perform today may be different in 6 months or a year from now – I am analyzing them as they are today.  In the future, the same tests should be run again.
2. There is no “control group” – we don’t know what is absolutely true here or not – there is no ideal monitoring solution – but there are monitoring solutions and results that are more useful than not – and I suspect, the value of these platforms is the usefulness and “cleanness of the data” .
3. I choose “Social Media Week New York” because there is a known attendee list (if I’m willing to data-mine for it) and I can judge geo-location better if I know who’s attending and tweeting / posting in NYC about Social Media Week NYC.

For today – I’m going to compare the tracking of Social Media Week – which hasn’t happened yet – but there have been a lot of posts about it up to this point.  Plus, the same exact query has been entered into all 5 monitoring platforms.

1.1.  Volume of Online Messaging about Social Media Week NYC – Jan 1-30th, 2010

Radian6 – shows 1364 items or posts

Alterian/Techrigy/SM2 – shows 1551 items or posts

Sysomos – 628 posts or items

BrandWatch - 147 posts or items – there may be more later – may still be processing

ddd

Biz360 shows 845 items or posts

Just from the comparisons above it’s clear that each platform is counting differently -   Without knowing the back end filtering each platform is doing it’s hard to know, based on this, what is the right number of items or posts is or should be.

Who knows with absolute certainty how much “content” was created on the web about Social Media Week NYC 2010 in the last month, so far?    How can I prove that any of these numbers is absolutely right or wrong?   I can’t.

The main thing all platforms catch is the a spike of data on 1/29th

However, I can look at the items each platform collects to evaluate how useful the information could be to me.

My next post will examine the quality of the data each platform picked up.  Please bear with me in my approach – I actually think going step by step, comparing each platform to each other for each “dimension” (similar to Web Analytics Dimensions) is the best way to get started.   As this series goes on I will use each platform for a deep dive – testing it out.

What I learned so far is each platform counts differently for the same exact query – therefore – we may want to consider how each platform is collecting data off the web.  I invite vendors and interested readers to come up with suggestions – perhaps those who program these platforms have a better idea than I do of why the numbers are different.

I think the part that might scare people about the difference in volume of posts or items is that lack of agreement on how to crawl or count social media content in the first place.  Suppose there could also be differences in how my query is processed by each platform – some, like Sysomos and Biz360 allow you to specify distance between words (exact, tight, loose) while others like Radian6, Techrigy and Brandwatch don’t appear to specify it – but may still process it.

All the more reason there should be some set of standard ways of processing information – more about that in a future post.