I downloaded FreshNetworks Social media monitoring review 2010 earlier today (while reading it on my iPhone) and had some thoughts about the review.
One one hand, I would have liked to have had the time and resources to do this study myself.
On the other hand, there are several “holes” in the accuracy of the study – I saw several instances of mis-information that did not match up with my own experience working with the platforms listed below and lend me to put a question mark over the whole report.
Just want to note that in my writings there is a good amount of Art and Instinct - which allows me a certain freedom which I try not to abuse – but I try as hard as I can to stay true to the technical nature of the subjects I write about. I know as well as the next person the easiest way to lose credibility when dealing with anything to do with technology any kind of measurement – is to be inconsistent or inaccurate – one or two facts that are wrong or even misstated and blow the creditability of the rest of the report/study (which actually had a lot of good stuff in it).
The social media monitoring review covered 7 platforms
…. Alterian, Brandwatch, Biz360, Neilsen Buzzmetrics, Radian6, Scoutlabs and Sysomos performed when monitoring conversations about global coffee brand Starbucks. We compared over 19,000 online conversations and have written about the following topics:
Starting off positive, I’m glad Fresh Networks went about this social monitoring review in a structured way and applied the same criteria across platform. I did a similar set of posts in February about Social Media Week NYC –
Comparing Social Media Monitoring Platforms on Geo-Location about Social Media Week NYC #SMWNYC 10 – I covered Geo-Location in more depth than the report.
Comparing Social Media Monitoring Platforms on Sentiment Analysis about Social Media Week NYC 10 – Came up with proposed solutions to the Sentiment Analysis dilemma
Comparing Social Media Monitoring Platforms on content about Social Media Week NYC 10 – my post examines the same inconsistencies in data crawling and data consistency as in the report but goes deeper into figuring out why the problem is happening and proposing solutions.
2.1. Coverage (on page 6)
Sysomos database actually goes back to January 1, 2007 – I’ve accessed data more than 3 years old many times. Also, Radian6′s limitation of 30 days back only covers a newly created profile (which is the case here) – in all other cases you can continue collecting data indefinitely after that and chart it to your hearts’ content.
It’s also untrue that Sysomos does not cover UK, US and France, and the founder of Sysomos told me but a few weeks ago that 25% of Sysomos clients are in the UK – and if Sysomos didn’t cover the UK, I don’t see how that would be possible (see below – you can click on the image to see it full size – with UK, US and I’m sure you’d see France I could fit it in the image).
However, I did verify proximity searches (terms within a certain distance from each other) are missing from current versions of Radian6, Alterian and Scout Labs - I recall Marcel LeBrun telling me Proximity Search is present in Radian6 (see below):
…. On the proximity operations, we actually can use proximity queries against our index, but it is something that we have not enabled in the UI. We try to achieve powerful capabilities balanced with ease of use and this type of operation isn’t intuitively obvious to many users, but perhaps it is something we should look to enable as an “advanced user” feature. I’m wondering, is this something you would have liked to see at the topic profile configuration level or as a capability right in the widgets?
Ok, Fresh Networks can’t be expected to have known that – but I did – and I did it alone – with out being part of a larger consulting group.
Getting back to Radian6 – maybe they should have included nearness operators a long time ago – maybe they are under estimating the intelligence of their users. In my experience, many a time have I and people I worked around swore at Radian6 because of “noise” issues – issues that nearness operators could have helped eliminate. Maybe it was true 4 years ago the average user “then” might not want to deal with nearness operators – but is it true any longer. I think not. I bet most users would welcome and easily use proximity operators if they were made available in user interface.
And while they’re at it – Radian6 should add “cut and paste” functionality to queries on a macro level – right now inputting queries can be a laborious process, indeed – I’d rather do my queries ahead of time in a word processor or Notepad and simply paste it in. Again, the interface was limited based on an assumption that users could not handle the complexity, an assumption that is no longer true (if it ever was true).
Also, I have a hard time believing Attensity/Biz360 doesn’t cover location at all – though they have an estimated “reach” metric next to most listings – my suspicion is we’d uncover geo-location in Biz360 if we dug deep enough – but I just checked and could not find it, either.
Meanwhile, on page 6 it’s mentioned that Scout Labs doesn‟t allow you to extract Twitter conversations with sentiment, but that’s not true. When Scout Labs finds sentiment in a tweet – it will state that and allow you to export a report with that information.
- Bout 2 take my mom 2 da salon get her nails n hair done, then takin her 2 Havana central 4 a nice lil diner.
Sentiment:
havana central: neutral machinePublished: May 09, 2010
- Sunlightsquare Latin Combo – Havana Central http://sori.la/IBel2789 링크i님: 듣다보면 정간다 ㅎ
Sentiment:
Published: May 09, 2010
What the study says about Sentiment Analysis I largely agree with:
We suspect the main reason people have latched onto sentiment is because it gives the impression that the plethora of web conversations can be summarised in a single number. Businesses track numbers and sentiment is often the KPI of choice for social media. This is dangerous. Sentiment is more nuanced than a single number and using an automated tool to assess how people feel puts too much faith in the today‟s software.
In Section 2.3 I found the geo-location categories accurate (but incomplete); along with the URL, IP Address and Language of a document as clue of where it originated from should be added an additional measure – looking at the content of a document/page for references of it’s authors’ location – Sysomos MAP does a good job of this and roughly 1 out of every four mentions has geolocation down to the state/city level based largely on what’s in the document – and I found this analysis to be roughly 80% accurate.
As far as Duplication (section 2.4) I don’t think we can expect any platform to show consistent or uniform results before some industry standard definitions are set and that hasn’t happened yet.
Also, the study seemed to make Data Latency (section 2.5) more of an issue than it is or needs to be and I don’t think anyone I’ve talked with is too worried they are not getting the information fast enough.
In regards to the findings – I agree with the points in section 3.1 (Initial Set up test) but suspect the problem is not so much in the level of training (though the study is correct that inconsistent results come out of having people using the tools who don’t understand their strengths, limitations or even what the tool is designed to do, and what it’s not). I view this as more of a “cultural” or communications problem than a training issue.
Yet the more depth of analysis – in other words – the more the information you get out the platform that is granular and directly usable – the more likely the interface you’ll use to pull it out of will suck and be too hard to understand how to operate for the average user.
The report agreed with my own overall experience regarding Sentiment Analysis and ranked Sysomos MAP the leading platform of the 7 (I bet that makes uber expensive Nielsen, growl).
Issues Management I wrote about yesterday -
I guess it’s “universal” that all platforms agreed to look for “Guns”, but the rest was very spotty – but Alterian will be the first one to tell you, as Connie Bensen, Alterian Community Manager told me last year, accuracy of the platform has a lot to do with populating the data dictionary.
09/27/09 @ 7:31 pmHi Marshall,
The dictionary in SM2 is fully customizable so you can make the sentiment analysis as accurate as you’d like.Connie
Community Strategist
Techrigy / Alterian
@cbensen
The issue is not so much around any of these platforms – as long as they can be trained and there is enough time and money to pay people to train them (in this case, refine how the platform makes meaning out of language it encounters) the platforms will perform well enough. I think that’s what Connie is saying and what Fresh Network / Fresh Minds didn’t do.
I don’t blame them for that - no one wants to spend the time neccessary to train any of these platforms- they want it to work out of the box and it doesn’t work well out of the box if you care about what your monitoring and you need to know what it all means.
But none of the platforms in existence is there yet – unlike or Windows programs that have common functions and use cases – Social Media Monitoring do not share a common interface, and that the differences just begin there and go on and on.
But now we need to go back and look at the FreshNetworks Social media monitoring review 2010 report remembering one of the platforms were trained enough to measure what they are actually capable of doing if the platforms were properly configured – which they were not. Isn’t that what Connie Bensen said to me when I mentioned Alterian’s Sentiment Analysis sucked? … She threw it back at me and said if I trained Alterian it would provide me with all the accuracy I need.
And when you think about it – why do we expect these tools can work super well, out of the box, without advanced configuration?
Almost any large software project I ever worked on usually had a lot of configuration hours (ie: Back in the day, when I worked with HP Openview and HP OpertionsCenter – these tools required a few months of configuration to be useful at all).
Sure, I don’t want Microsoft Word, Excel or even my browser software to be so complicated I can’t operate it – but let’s face it – the more customized and accurate you need a platform to be, the more likely you’ll have to a good deal of spend time or money (or both) to get what you want. Why should that surprise anyone?
And finally, the reason why programs we typically use such as Microsoft Word, Excel, Firefox, Google Chrome, etc, appear simple and clear cut – is that they have been designed with several common use cases in mind.
In a Social Monitoring project, any project, your starting from scratch and trying to derive meaning from tools and platforms that aren’t too good at providing the meaning of anything.
So, I feel it’s unfair to say almost all the monitoring tools examined in this report sucked at Sentiment Analysis without taking responsibility and owning up that most users, including the people at Fresh Networks (and me) want something simpler that does work out of the box and that solution doesn’t exist yet (and we have to learn to live with it till it does exist).
Overall, I think FreshNetworks Social media monitoring review 2010 is an excellent study but with several lapses, most that I pointed out. In the past I’ve pointed out that it takes a while to learn what a platform is capable of and not – usually I’m tested by “fires” of life to have to figure out what these tools are capable of in a short period of time while many demands are on my plate, competing for attention at the same time.
I’m just not certain or seeing evidence of just how extensively those who tested these 7 platforms really understood them and their strengths and limitations – and I base that on the inconsistencies I pointed out earlier in this post.
![Reblog this post [with Zemanta]](http://img.zemanta.com/reblog_e.png?x-id=52bab2f8-ec55-47ce-a10b-2457148defd7)
