My first two posts in this series on Comparing Social Media Monitoring Platforms for coverage of Social Media Week NYC and Comparing Social Media Monitoring Platforms on content about Social Media Week NYC 10 examined volume and media types captured by each platform – with very different results in each case.

But what about Sentiment?

Radian6 – 81% Positive and 19% Negative Sentiment

I used “Widget Keywords” which is to say, I told Radian6 to use anything that had “social media week” in it for sentiment analysis.  I could have used Topic Profile keywords which would have broken down the results differently, similar to what Techrigy does with “Tone”.

Alterian/SM2/Techrigy – 91% Positive and 9% negative sentiment

Considering the number of posts each platform considered, the results for sentiment, on the face of it – between Radian6 and Techrigy are not as far off as I’d thought they would be.

Sysomos – had 76% Positive and 1% Negative with 23% neutral - again, not as far off as I thought they’d be.

But Brandwatch was really far off from the rest with only 1% positive and 1% negative – and the rest of the data was “neutral”.

It’s clear Brandwatch is much more strict with assigning sentiment – maybe that’s good – -not sure yet.

Biz360 had 16% Positive and 4% Negative with 81% Neutral  - quite a difference from the rest.

Of course, we need to look at what each platform considers to be negative vs. positive.  More to the point – would the same content be found in each platform to be negative – or might a negative post or comment in one platform be considered to be positive by another?

Whew ..that is a hot !!!  How far off can these platforms be ?  Remember – same exact query (Social Media Week New York City) same time period.

Radian6 – none of the 22 tweets that were flagged negative -  were actually negative  when I read then visually.

Look – Radian6, to their credit – added Sentiment Analysis by keywords just last month. I want to say it’s not the platform’s fault that people (me) don’t know how to configure or use it might be intended.

On the other hand, I put in “Social Media Week” as the keyword to do sentiment around – on a topic profile about Social Media Week NYC  – so …. you figure it out.    Let me put it another way – if you really care about the accuracy of your results around sentiment – best to look at them by hand and reclassify them to what you think they are – Radian6 may not be smart enough to tell sentiment.  There- I tried to be nice.

Alterian/SM2/Techrigy – total wash out - most of the results don’t make sense – only one or two actually have any connection with Social Media Week New York City – and they’re not negative – at least, not negative about Social Media Week -

I realize the Sentiment Analysis part of  Techrigy is going to be updated on February 1st, and it will be interesting to look at these results – then.

People I know at Techrigy have been very nice to me and bend backwards to please – and I really appreciate their support for me over the last two years -  on the other hand, as an analyst, a blogger whose opinion people respect in this area, for this query – the results Techrigy returned were less than useless – they actually are a detractor.  I’m glad there’s an update in a few days – Sentiment Analysis needs an update, badly.

BrandWatch – 2 negative results – both were negative

Yes, let’s call a “spade a spade” - these two tweets do have a negative tone - look for yourself.   Congratulations Giles Palmer – you  and your team won this challenge -  maybe the English got this one right – sentiment analysis -  at least in this case – the results make sense.

Sysomos - the sentiment results were generally good -  I did not find any negative blog posts according to Sysomos, so I looked at the positive and neutral posts and I found them to be reasonable in their classification of Social Media Week Content – many of the Mashable posts about Social Media Week were listed  and it’s fair to read that as “positive” since they support and give attention to Social Media Week.

Unlike Brandwatch, Sysomos didn’t find any negative tweets and classified them all as neutral - perhaps that is not far from the truth – I would rather have a result be classified as neutral than have it be flagged as positive or negative by the system when it’s clearly not, having looked at it visually.   Sysomos treated news stories in mainstream media as either neutral or positive – again, Sysomos assignment of sentiment was reasonable – so maybe it’s a tie between Brandwatch and Sysomos.

What about Biz360? – the negative sentiment selections were pretty poor and looked like the same things that Sysomos flagged as positive, Biz360 flagged as negative.

It’s really hard to imagine most of what was picked up by Biz360 could be negative – just look at the titles of the content it selected.  I do like that Biz360 qualifies the reach of a piece of content while the other platforms that are being looked at here do not offer that feature, today.

Taking this all in – I need to say that I seriously question a “marketer” who builds a Social Media Monitoring platform is going to come up with a satisfactory solution for many of the things that are really important to measure – and to accurately gauge.

Radian6 is getting better – but the problem is, as far as sentiment goes – there’s too much that has to be done on a configuration level to get sentiment right – I know they  will continue to improve – but they also come from a “marketing” background and the Flash Interface, that makes their product look better than the others, is also, at time, frustrating to work with.

Alterian/Techrigy/SM2 – well …. they know Sentiment isn’t their strong point – they have a good technology and have improved alot over the years – yet they too, are built from a marketing perspective – and that can be a problem, because marketers aren’t that good at technology – that’s the same issue as Biz360, in my opinion – but look – I could be wrong – hell, I hope I am.

Brandwatch – I don’t know what their origins are – but I have met and spoken to Giles Palmer – who founded Brandwatch several years ago – and they consider themselves to be the #1 platform in Europe – their sentiment analysis seems to be better than most, based on my experience with it so far – and from what Giles tells me – they do a lot of work to make it and keep it that way.

Sysomos, the more I work with it, the more I like – Sysomos was built by a programmer and a university- the University of Toronto – and their backend is able to splice and dice data very well – Sysomos is more like a programming think tank that grew into marketing – yes,  their interface could improve – but the sentiment analysis and noise suppression are excellent.

So, there you have it – Sentiment Analysis – buyer beware – for the time being, if your going to have large amounts of data you need to score with automated sentiment analysis – I think you’ll be best off with Sysomos or Brandwatch, all things being equal.

And look, I’m open to discussion and reexamining my results (I don’t know a lot of what goes on behind the interface)- clearly – a lot more deep dives into the data of all the platforms are needed – but that’s precisely what most Social Media Monitoring Platform make it hard to do – and yes, a couple of analysts types have looked at these platforms, including Forrester and Nathan Gilliatt who will be at Monitoring Social Media Bootcamp with me in London in two months and a sample of his ebook is encouraging (and the new book he’s working on is going to be even better).

Nathan’s approach, as far as I can tell – is to look at these platforms more as a “buyers” guide  – similar to what Phil Kemelor does with Web Analytics platforms for CMSWatch.

My approach is hands on – how well does these platforms work for what we use them for?

I don’t think anyone else in the field actually does that - and this post alone – shows that.

By the way, if you want to get far more than even what I can write here – hire me to help you – and please join me at Monitoring Social Media Bootcamp in London on March 31st. 2010.

Tickets are on sale now.

The first post in this series examined the volume of “buzz” around Social Media Week NYC that is taking place next week here in New York.   What is the buzz made up of?

Remember – I’m using exactly the same query in all the platforms for the same period of time – the only way to do this.

Radian6

Alterian/Techrigy/SM2

Note: comparing Radian6 and Techrigy we already see a problem - Radian6 counts 390 blogs while Techrigy counts 66 blogs – Radian6 gets 964 Tweets while Techrigy gets 450, but both have very little coverage of Facebook, which is still mostly a walled garden.

Sysomos has 307 (closer to Radian6)  blog posts and 165 News mentions (similar to Techrigy) while Tweets are a sampled and Sysomos says as much.

Brandwatch numbers are much lower – but we don’t know if that’s good or bad yet.  News had 30 items while 72 blog posts and 24 tweets were found.

Biz 360 had 677 tweets and 168 Blog posts

My take away is that most people will never get the opportunity to meaningfully compare monitoring platforms in the way I am doing … because they can not get access to the platforms and don’t have the curiosity to test them the way I have. But now you can see just how different they are in their abilities, today, to track the same query across social media on the web.

On another level – it probably doesn’t matter that much each platform is so far apart in results it collects, because people will tend to believe the version of the truth that suits them the best.   Still, without the knowledge of how different they really are – an informed decision can rarely or ever get made.

My next post will be about Sentiment Analysis of these 5 platforms  – whew – that is a hot topic!

This post is the first in a series of comprehensive comparisons of 5 social media monitoring platforms I have access to which will be part of my course at Monitoring Social Media Bootcamp taking place in London on March 31st, 2010 (two months from today) - Radian6, Alterian/Techrigy, Sysomos, Brandwatch,  Biz360 Community Insights.  I also have access to Crimson Hexagon Buzz Monitor but am not including it in this analysis – I’ll be happy to include Crimson if asked.

I’ll use the John Battelle approach and develop my content and thoughts online, and let the community of readers comment – and your thoughts will add to my course.  In return, I’ll share with you what I know.

Sound fair?    Let’s start with the query – “Social Media Week New York City“  which is going to be taking place next week and will give me three things  I need to test and compare these platforms on

1. An Event
2. A Location (test Geo-Location accuracy)
3. Sentiment
4. Influencer Lists of Authoritative bloggers
5. A Summary of Social Media Week NYC after it’s over using 5 platforms.  Which ones will end up being most useful?

A couple of thoughts before I start  – and this series of posts will take time to complete.

1. How these  platforms perform today may be different in 6 months or a year from now – I am analyzing them as they are today.  In the future, the same tests should be run again.
2. There is no “control group” – we don’t know what is absolutely true here or not – there is no ideal monitoring solution – but there are monitoring solutions and results that are more useful than not – and I suspect, the value of these platforms is the usefulness and “cleanness of the data” .
3. I choose “Social Media Week New York” because there is a known attendee list (if I’m willing to data-mine for it) and I can judge geo-location better if I know who’s attending and tweeting / posting in NYC about Social Media Week NYC.

For today – I’m going to compare the tracking of Social Media Week – which hasn’t happened yet – but there have been a lot of posts about it up to this point.  Plus, the same exact query has been entered into all 5 monitoring platforms.

1.1.  Volume of Online Messaging about Social Media Week NYC – Jan 1-30th, 2010

Radian6 – shows 1364 items or posts

Alterian/Techrigy/SM2 – shows 1551 items or posts

Sysomos – 628 posts or items

BrandWatch - 147 posts or items – there may be more later – may still be processing

ddd

Biz360 shows 845 items or posts

Just from the comparisons above it’s clear that each platform is counting differently -   Without knowing the back end filtering each platform is doing it’s hard to know, based on this, what is the right number of items or posts is or should be.

Who knows with absolute certainty how much “content” was created on the web about Social Media Week NYC 2010 in the last month, so far?    How can I prove that any of these numbers is absolutely right or wrong?   I can’t.

The main thing all platforms catch is the a spike of data on 1/29th

However, I can look at the items each platform collects to evaluate how useful the information could be to me.

My next post will examine the quality of the data each platform picked up.  Please bear with me in my approach – I actually think going step by step, comparing each platform to each other for each “dimension” (similar to Web Analytics Dimensions) is the best way to get started.   As this series goes on I will use each platform for a deep dive – testing it out.

What I learned so far is each platform counts differently for the same exact query – therefore – we may want to consider how each platform is collecting data off the web.  I invite vendors and interested readers to come up with suggestions – perhaps those who program these platforms have a better idea than I do of why the numbers are different.

I think the part that might scare people about the difference in volume of posts or items is that lack of agreement on how to crawl or count social media content in the first place.  Suppose there could also be differences in how my query is processed by each platform – some, like Sysomos and Biz360 allow you to specify distance between words (exact, tight, loose) while others like Radian6, Techrigy and Brandwatch don’t appear to specify it – but may still process it.

All the more reason there should be some set of standard ways of processing information – more about that in a future post.