Sentiment Analysis best done by humans

Posted by Marshall Sponder on April 18, 2010 | Link It

I’m working on an reputation analysis of a international training organization who has expressed concerns about their online reputation.    After pulling the data from Sysomos MAP and comparing the sentiment score against human scoring – I’ve decided that if you care about Sentiment Accuracy – it’s best to have humans evaluate sentiment.

Also noted that human scoring of documents has an additional advantage that never seems to be spoken about  – the more “we” get involved in the output of our monitoring (i.e.:  by devising scoring metrics and applying them to the data at hand) the more “engaged” and satisfied, I feel, we will be with the monitoring programs we devise.

I’ve been seeing this more and more, and have taken that to mean that community managers and analysts that are fully “invested” in the data they collect (they touch it, in other words) the more satisfied they are with what they are doing.

And, as a result of doing all this sentiment “scoring” we have more confidence in the results than if we let a machine program do it.

Which reminds me – not only did I present at the Sentiment Analysis Symposium last week but I also presented on Sentiment Analysis in London last month – and did a deck for it.

How To Monitor Sentiment And Benefit From The Insight This Offers

In the case of my client – i found about 22% of the sentiment around them was negative – but that was after I looked at everything – and that’s actually a fairly large amount of negative sentiment.
By the way, I’m a fan of using Automated Sentiment Analysis – problem is – there’s no standards around this and the current implementations and technologies are still too immature to handle many of the tasks Sentiment Analysis is used for.
Reblog this post [with Zemanta]

Filed in Sentiment Analysis, Sentiment Analysis Symposium
Post a Response


Post comment as
twitter logo facebook logo
Sort: Newest | Oldest

Trackbacks

  1. Text Classification for Sentiment Analysis – Naive Bayes Classifier «streamhacker.com says:
    May 10, 2010 at 9:32 am

    [...] assumptions and very little code we're able to get almost 73% accuracy. This is somewhat near human accuracy, as apparently people agree on sentiment only around 80% of the time. Future articles in this [...]

  2. Text Classification for Sentiment Analysis – Naive Bayes Classifier «streamhacker.com says:
    May 10, 2010 at 9:32 am

    [...] assumptions and very little code we're able to get almost 73% accuracy. This is somewhat near human accuracy, as apparently people agree on sentiment only around 80% of the time. Future articles in this [...]

  3. Why analytics companies should stop focusing on “accuracy” in automated sentiment analysis | context analytics says:
    May 14, 2010 at 11:35 am

    [...] new tool can identify sentiment “better than most humans”. Just a few days later, I read a post this week claiming that ”sentiment analysis [is] best done by [...]

  4. WebMetricsGuru » Is there any point to doing Sentiment Analysis? says:
    May 15, 2010 at 10:10 am

    [...] new tool can identify sentiment “better than most humans”. Just a few days later, I read a post this week claiming that ”sentiment analysis [is] best done by [...]

  5. Why analytics companies should stop focusing on “accuracy” in automated sentiment analysis | Beyond says:
    August 27, 2010 at 12:56 pm

    [...] new tool can identify sentiment “better than most humans”. Just a few days later, I read a post this week claiming that ”sentiment analysis [is] best done by [...]

  6. El Mito de los Software de Inteligencia Competitiva: Automatización vs Inteligencia « Fin del Marketing.com says:
    September 17, 2010 at 3:14 pm

    [...] Hace unas semanas, Hugo Zunzarren (colega y experto en Inteligencia Competitiva) nos presentaba un post en el cual hacía mención a las falsas expectativas que generan en el mercado quienes comercializan este tipo de software, fundamentalmente cuando nos enfrentamos con información cualitativa, hoy en auge para el análisis de las Redes Sociales. A menos que exista evidencia explicita, un robot no es capaz de discernir si un comentario tiene connotación positiva, neutra o negativa (a pesar de que se está insistiendo con cierto éxito en algoritmos que se aproximen a los resultados ciertos), ya que este no solo depende de las palabras en sí mismas, sino del contexto y del tono del comentario. Hugo concluía en la escasa fiabilidad de los análisis automáticos, citando la afirmación categórica de Jason Falls y Marshall Sponders,  expertos en la medición cualitativa de los mensajes: el análisis lo hace mejor un humano. [...]





UPCOMING SPEAKING