Radian6 – 81% Positive and 19% Negative Sentiment
I used “Widget Keywords” which is to say, I told Radian6 to use anything that had “social media week” in it for sentiment analysis. I could have used Topic Profile keywords which would have broken down the results differently, similar to what Techrigy does with “Tone”.
Alterian/SM2/Techrigy – 91% Positive and 9% negative sentiment
Considering the number of posts each platform considered, the results for sentiment, on the face of it – between Radian6 and Techrigy are not as far off as I’d thought they would be.
Sysomos – had 76% Positive and 1% Negative with 23% neutral - again, not as far off as I thought they’d be.
But Brandwatch was really far off from the rest with only 1% positive and 1% negative – and the rest of the data was “neutral”.
It’s clear Brandwatch is much more strict with assigning sentiment – maybe that’s good – -not sure yet.
Biz360 had 16% Positive and 4% Negative with 81% Neutral - quite a difference from the rest.
Of course, we need to look at what each platform considers to be negative vs. positive. More to the point – would the same content be found in each platform to be negative – or might a negative post or comment in one platform be considered to be positive by another?
Whew ..that is a hot !!! How far off can these platforms be ? Remember – same exact query (Social Media Week New York City) same time period.
Radian6 – none of the 22 tweets that were flagged negative - were actually negative when I read then visually. Look – Radian6, to their credit – added Sentiment Analysis by keywords just last month. I want to say it’s not the platform’s fault that people (me) don’t know how to configure or use it might be intended.
On the other hand, I put in “Social Media Week” as the keyword to do sentiment around – on a topic profile about Social Media Week NYC – so …. you figure it out. Let me put it another way – if you really care about the accuracy of your results around sentiment – best to look at them by hand and reclassify them to what you think they are – Radian6 may not be smart enough to tell sentiment. There- I tried to be nice.
Alterian/SM2/Techrigy – total wash out - most of the results don’t make sense – only one or two actually have any connection with Social Media Week New York City – and they’re not negative – at least, not negative about Social Media Week -
I realize the Sentiment Analysis part of Techrigy is going to be updated on February 1st,and it will be interesting to look at these results – then.
People I know at Techrigy have been very nice to me and bend backwards to please – and I really appreciate their support for me over the last two years - on the other hand, as an analyst, a blogger whose opinion people respect in this area, for this query – the results Techrigy returned were less than useless – they actually are a detractor. I’m glad there’s an update in a few days – Sentiment Analysis needs an update, badly.
BrandWatch – 2 negative results – both were negative
Yes, let’s call a “spade a spade” -these two tweets do have a negative tone- look for yourself. Congratulations Giles Palmer – you and your team won this challenge - maybe the English got this one right – sentiment analysis - at least in this case – the results make sense.
Sysomos -the sentiment results were generally good - I did not find any negative blog posts according to Sysomos, so I looked at the positive and neutral posts and I found them to be reasonable in their classification of Social Media Week Content – many of the Mashable posts about Social Media Week were listed and it’s fair to read that as “positive” since they support and give attention to Social Media Week.
Unlike Brandwatch, Sysomos didn’t find any negative tweets and classified them all as neutral - perhaps that is not far from the truth – I would rather have a result be classified as neutral than have it be flagged as positive or negative by the system when it’s clearly not, having looked at it visually. Sysomos treated news stories in mainstream media as either neutral or positive – again, Sysomos assignment of sentiment was reasonable – so maybe it’s a tie between Brandwatch and Sysomos.
What about Biz360? – the negative sentiment selections were pretty poor and looked like the same things that Sysomos flagged as positive, Biz360 flagged as negative. It’s really hard to imagine most of what was picked up by Biz360 could be negative – just look at the titles of the content it selected. I do like that Biz360 qualifies the reach of a piece of content while the other platforms that are being looked at here do not offer that feature, today.
Taking this all in – I need to say that I seriously question a “marketer” who builds a Social Media Monitoring platform is going to come up with a satisfactory solution for many of the things that are really important to measure – and to accurately gauge.
Radian6 is getting better – but the problem is, as far as sentiment goes – there’s too much that has to be done on a configuration level to get sentiment right – I know they will continue to improve – but they also come from a “marketing” background and the Flash Interface, that makes their product look better than the others, is also, at time, frustrating to work with.
Alterian/Techrigy/SM2 – well …. they know Sentiment isn’t their strong point – they have a good technology and have improved alot over the years – yet they too, are built from a marketing perspective – and that can be a problem, because marketers aren’t that good at technology – that’s the same issue as Biz360, in my opinion – but look – I could be wrong – hell, I hope I am.
Brandwatch – I don’t know what their origins are – but I have met and spoken to Giles Palmer – who founded Brandwatch several years ago – and they consider themselves to be the #1 platform in Europe – their sentiment analysis seems to be better than most, based on my experience with it so far – and from what Giles tells me – they do a lot of work to make it and keep it that way.
Sysomos, the more I work with it, the more I like – Sysomos was built by a programmer and a university- the University of Toronto – and their backend is able to splice and dice data very well – Sysomos is more like a programming think tank that grew into marketing – yes, their interface could improve – but the sentiment analysis and noise suppression are excellent.
So, there you have it – Sentiment Analysis – buyer beware – for the time being, if your going to have large amounts of data you need to score with automated sentiment analysis – I think you’ll be best off with Sysomos or Brandwatch, all things being equal.
And look, I’m open to discussion and reexamining my results (I don’t know a lot of what goes on behind the interface)- clearly – a lot more deep dives into the data of all the platforms are needed – but that’s precisely what most Social Media Monitoring Platform make it hard to do – and yes, a couple of analysts types have looked at these platforms, including Forrester and Nathan Gilliatt who will be at Monitoring Social Media Bootcamp with me in London in two months and a sample of his ebook is encouraging (and the new book he’s working on is going to be even better).
Nathan’s approach, as far as I can tell – is to look at these platforms more as a “buyers” guide – similar to what Phil Kemelor does with Web Analytics platforms for CMSWatch.
My approach is hands on – how well does these platforms work for what we use them for?
I don’t think anyone else in the field actually does that - and this post alone – shows that.
Note: in the post following this one – I put out a theory that Google is the main driver of this increase – which makes sense to me since Google is the driver of so much Web 2.0 Traffic – see 400% Rise in Social Media Traffic due to Search Engines – Google for more information and the next piece of this puzzle.
Used Alterian/Techrigy/SM2 platform to monitor what was being said about each brand in Debruyn’s guide with additional queries for “social media monitoring” and “social media buzz”, etc.
I noticed a curious thing – since this summer the volume of conversations about Social Media (focusing on monitoring Social Media) increased 400% from what it was at the beginning of this year, and for all of 2008.
I questioned the accuracy of the data I collected from Alterian, as I this profile took up most of my 200,000 searches limit and wondered if it didn’t actually complete – and that’s what I was seeing.
Used Google Insights for Search and found, more or less, a similar pattern of “Social Media” searches increasing by 400% this year – and compared it to SEO, SEM searches, which haven’t changed.
That gets me to the reason I am writing this post – not so much to find out about the 400% increase, as interesting as it is, as to see which of the social media monitoring firms, which make a living monitoring the web, are generating buzz around their own brands, and highlights of what what was being said about them.
More later, as I complete this study and try to figure out what it all means.
Posted by Marshall Sponder on December 12, 2009 | Link It
URL Link
HTML Link
BBCode Link
Trackback
I never know quite where I’m going to end up when I write a post – I started with Seth Godin’s Traffic Truth Tricks and ended up ………
Well …. it’s nice Seth Godin is giving some value (while getting liberally re-tweeted … nothing against that) in his post about Discover the truth about a site’s online traffic – where he mentions …….
… You can see the referrals and traffic to an individual bitly twitter URL by copying the URL and adding + sign to it. For example, if you see something like this in a tweet:http://bit.ly/870Ry9just copy it and paste it with the plus and you’ll seehttp://bit.ly/870Ry9+.I think that’s pretty neat. You can also track topretweetson an hourly or daily basis.
I didn’t know that … I bet a lot of people did not know about it, either. If you provide value to the community – you ought to be re-tweeted(and Seth asks for your subscription right at the end of the post)…..
My observation is what “the community” wants and finds valuable is not often the same thing the individual who is posting, finds valuable – more about that in a minute. Many of my best posts, by my own estimation, hardly got re-tweeted, at all, yet a post I just made a comment about (with a list I didn’t come up with – but found value with – and noted it) got re-tweeted 253 (and counting) and viewed more than anything else I’ve ever written – at Social Media Today – this weekend, it was a syndicated copy of my post on 26 Tools for Social Media Monitoring that I spent about 10 minutes writing at 1AM in the morning – more as a quick note and what I read earlier in the day (or the day before). Notice that Social Media Today “tweaked” my title, I notice editors often do that – don’t know if it helped, or not. Other posts of mine, that I worked on and put my soul into, often spending a few hours on them, that I thought were more original and better – never got picked up, at all.
As far as the Traffic Truth, via Compete or Quantcast vs. Alexa – they’re all wrong compared to site analytics, as much as 40% under or over counting – but we’ve been down that road before – it’s not a new topic – most people know Compete is better for overall trending and competitive intelligence when you have site analytic data (we almost never do have it).
But, Seth Godin’s Traffic Trick doesn’t really go far enough – because it’s only about Bit.ly, and that’s not enough to cover the whole universe of where content is being created and shared.
1. BackType to the Social Media Today post shows several bitly URLs that were tweeted -
http://bit.ly/info/6DG3Mi which, when expanded … turn out to be 1001 clicks as of now (I checked the other bitly urls listed.
1,001
Total Clicks
All clicks on the aggregate bit.ly linkbit.ly/6kLto6
Have no idea what the traffic from TinyUrl, Short.to, Ff.im or j.mp was – and I don’t know, off hand, of anyone integrating that traffic – Seth’s traffic trick doesn’t really work, unless you want a quick Bit.ly fix.
Plus, let’s not forget my original post – at Webmetricsguru.com – shouldn’t my content be “summed up” … it is, more or less, the same thing … right? What’s the combined views of my content, including all syndication? Who’s able to answer that question? I should think Seth Godin would care about that much more than I do, since he’s syndicated all over the place – way more than I am. How come he doesn’t provide a quick trick on that one? He can’t – he can’t solve this problem – he would not even try – it’s not a quick fix.
My post on Webmetricsguru.com
WebMetricsGuru » 26 Tools for Social Media Monitoring
Tweets 22; Shares 2, Likes 1, Comments 0; Shares 0; Comments on Page 0; Searching…View All
Locations:
United States 11; Russian Federation 6; United Kingdom 3 View All
We’d have to add the 72 clicks from my post to the 1001 from Social Media Today and get something like 1073 clicks, plus whatever else was at the other shorted urls – which aren’t easy to collect (but maybe someone ought to try).
Now, Compete, Alexa or Quantcast don’t give stats on individual urls – and Compete is at-least a month behind – so they’re not helpful here.
On the other hand, Social Media Monitoring Tools, like Radian6, Techrigy, Sysomos – don’t pick up the slack and they ought to try – they’re all keyword based, but … not url based or topic based, but they ought to try to be.
For example, let’s take Radian6 – if I put a url in the source filter – will it take care of all the other url shortening and syndication? I’m trying it right now – Seth Godin might be into Traffic tricks – I’m not. What will Radian6 comes up with?
I put all the urls I could possible get into the source filter – suppose I could add Google Sidewiki urls on my site and Social Media Today (but SideWiki doesn’t work on the url level, yet), since Radian6 now tracks those, but I left those out, for now.
I created a topic profile that would be sure to pick up anyone talking about my post, in either place.
And I added “Sentiment Keywords”, a new feature Radian6 added that allows them to do Sentiment Analysis by letting a user tell Radian6 the phrases or entities they think are important – VERY NICE – Kudos to Radian6 for adding this … real stuff – that we can use – not Seth Godin’s Traffic Tricks … intended for a quick marketing pickup and some additional subscribers to his blog (who could end up being his clients or buying his books, stuff, down the line).
I also set my Influencer Settings to be thus (see below).
Last, but not least – I ensured that only keyword relevant content from the sites I am using in my source filter would be considered.
The results – didn’t work too well – didn’t aggregate the traffic but it did make the “Influencer List” and the “Word Clouds” more accurate – in a general way. I know Radian6 and the rest of the tools, wasn’t really designed to answer the questions I’m asking – to be fair – and neither is Seth Godin, to be fair – again – he’s a Marketer – don’t expect good analytic from him – he sells ideas – makes you feel good – and then he’s collects his money, and walks away – to the next gig or speaking engagement, or book.
Honestly, that’s OK, I don’t expect any more from Seth Godin, than that.
Getting to the Word Cloud – much better – not that I added Source Filters and Sentiment Keywords – “Social Tools”, “Social Media Tools” and “online reputation listening” are the basic focuses – it doesn’t do “meme analysis” yet (covered that in my slideshare presentation at Monitoring Social Media 09, below) – but it’s not bad, for what it is now.
See my presentation to find out how this word map could have been a lot better
Meanwhile, the Influencer List got a lot better – even though it’s not about my post – but just people who would be receptive to the topics in my post.
If I wanted to create an Influencer List with this basic data – I could – it take a few hours, but much of the noise, I think, is eliminated by limiting the sources of data and putting sentiment in around keywords I define ahead of time.
And I’ll argue the people on the bottom of the list are probably more likely to be receptive to me (if I wanted to contact them, than the people on the top of it).
However, to end this long post – no tool really can do more than get part of the answers – simple Traffic Tricks from Seth Godin don’t work well in this , nor to the Social Media Monitoring platforms, though I think, this is an area they ought to tackle next year, for 2010.
Nick started by showing me Sysomos Map which is an analysis tool used to produce reports and insights (see image 1) and contrasted it to Sysomos HeartBeat (see image 2), a real time search trend tool used to keep track of what is happening now.
I got an appreciation on how Sysomos is somewhat different than the other Social Media Monitoring platforms I’ve worked with so far -
87% accuracy in sentiment analysis of general text
91% accuracy in sentiment analysis of product reviews
85% accuracy in sentiment analysis of movies and celebrities
That’s not bad! What I mean to say, at 87% Sentiment Analysis accuracy you an almost do what most people wanted to do with Social Media Monitoring platforms, in the first place, score sentiment of a large amount of information in an automated way, with a certain degree of confidence (that’s not possible to do with Alterian/Techrigy/SM2, Radian6 or any of the free tools out there – though it may/may not be true of Crimson Hexagon).
Besides identifying influencers, Sysomos can map influencers - the map can be added to in realtime and altered while the pages automatically refresh.
Because information is collected and stored atomically, extremely rich reporting can be done, including geo-demographic profiling, isolating influencers by city and state level with a high degree of confidence – this seems to be entirely missing from the capabilities of anything else I’ve worked with. For example, I mentioned that Alterian and Radian6 are next to useless for geo-location queries, regardless of what the companies tell you and wrote about it in a post on finding local blogs and bloggers where I gave up on Alterian and Radian6 for hyper local searches as the platforms were not designed for it. In the future, this may change as more geo-local data is transmitted (ie: using mobile devices with geo location turned on – we’re seeing this appear first with Twitter – but it will NOT end there)
While Radian6 and Alterian technically do have the same capabilities – for example
- drill down by location, age, sex, (Sysomos can also break down traffic by industry)
I suspect, it’s not so much what Sysomos does, but the way in which it does it, where it’s strengths are.
The “BuzzGraph” looks similar to what some tools like Crimson Hexagon produce, though the style of the map is somewhat different – and the inclusion of Facebook and MySpace data (data that can be gotten from public crawling of profiles where they can be found on Google, for example, is powerful, and features people are asking for, more and more.
To sum up this post on Sysomos – I was impressed with how much thought went into the data architecture and reporting capabilities of Sysomos. It’s a powerful platform and an example of how fast the Social Media space is evolving.
I’m happy to say that Nick Koudas will be at Monitoring Social Media 09 in London and the Influence Scorecard gathering in New York during November – and I’m looking forward to that.
To read more about Monitoring Social Media 09 – check out these two posts
A leader in the web analytics and digital media recruitment marketplace, IQ Workforce provides access to some of the most sought-after full-time and contract talent in the corporate world. If you need help finding serious web analytics talent or want to take your career to the next level, call IQ Workforce!
![Reblog this post [with Zemanta]](http://img.zemanta.com/reblog_e.png?x-id=6c22f96e-4b56-4753-8615-8baa9a97bdff)
![Reblog this post [with Zemanta]](http://img.zemanta.com/reblog_e.png?x-id=9da75cc2-1b28-409d-970b-6759e796f090)
Tweets 
Shares 
Shares 
Comments on Page 
![Reblog this post [with Zemanta]](http://img.zemanta.com/reblog_e.png?x-id=d4d73019-9757-41a1-97cd-f77aa96d4015)
