I just had a post of mine at AllAnalytics.com appear this morning on Wealth Analytics -   I now have a regular column there and will have content appearing once or twice a month – you’ll need to go to AllAnalytics.com to read the post.

On my way to London tonight for a couple of book signings and public speaking events, so my posting activity might be a little lighter than normal.

Here’s the presentation for today’s Social Media Club presentation on my book, Social Media Analytics. Register here

https://www3.gotomeeting.com/register/893757958

Not certain it’s an open webinar (may be for Social Media Club members, but go ahead, try to dial into it at 2PM EST, if you have the time.

Over the last month I have been reading  – Pulse: The New Science of Harnessing Internet Buzz to Track Threats and Opportunities by Douglas W. Hubbard – Kindle Edition, which I greatly enjoyed reading and finished  Sunday night.   It’s hard for me to put into words all the things I picked up by reading Doug Hubbard’s latest book – I also have a Kindle copy of his last book – so I was familiar with him, but I noticed his new book last month, I felt I had to drop everything else I was reading just then, and focus on his book – a wise decision, as it turns out.

I also made so many annotations and notes that I maxed out Amazon’s limit for this book – here they are – Amazon Kindle – PULSE – Doug Hubbard- marshall sponder annotations-9-20-11 in case anyone wants to read all of them.

….
…..

Douglas Hubbard puts forward that, despite all the measurement and search platforms out there – the ” Internet itself is almost entirely underutilized as a measurement instrument of society” which is largely true because people (as I noted) are using the Internet, or Big Data, or “The Pulse” in a half baked way, instead of visualizing “big trends”, which is what Hubbard has done.

He also makes a point to distinguish this kind of research from what it’s not, which was the right thing to say – otherwise people would mix it up and attack what he’s doing by confusing it with something else.:

the Pulse Is Not (Necessarily or Entirely) Related To:

1. Diminishing privacy
2. Using the Internet for lead generation
3. The use of business tools commonly referred to as business intelligence and predictive analytics

4. Online versions of traditional survey methods

(Kindle Locations 310-315). Wiley. Kindle Edition.

I found Hubbard’s analogy to the 19th century’s search for “The Big Picture” and the use of Statistics, map making and rapid communications as an energy movement that pushed society to evolve faster and faster, thought provoking. As I read on I saw the author’s ideas dovetailed my own, and enriched them.   For example, listen to this:

“…In World War II, for example, the Allies used a method they called “content analysis”—the study of recorded communications—as a method of military intelligence.”

….. For the clerks, the job was a fairly mechanical procedure; often they were instructed just to categorize content by keywords. In some ways, the procedure was similar to modern “sentiment analysis.” When electronic, textual communications between individuals (emails, texts, blogs, etc.) came into wide use, the same methods could be applied in an automated way with the advantage of having much larger data sets analyzed much faster.”

I guess, what that says to me is what is abundantly clear when I deal with many of my colleagues – who look at data analytics, web analytics, search analytics and now Social Media Analytics as as a “glorified Data Clerk”.    It’s exactly why I wrote Social Media Analytics – to revolt against the perceptions that  real value (economic/political) lies with the Biz Dev/MarCom/Sales People, the executive function – and the Analysts are just a bunch of “clerks” – busy fixing up “data”.   I’d laugh, if it wasn’t so painfully true.

Getting on to another point in the book – Douglas Hubbard has an axe of his own to grind with critics who scoff at what he calls “Pulse Data” because it isn’t perfect yet and has noise in it:

“…Again, the existence of noise is not the same as the lack of signal.”

“….most of the data we need for the Pulse tends to be concentrated in a few major sources.”

It turns out, and Hubbard makes a good argument for it, that if we had the right 10 or 15 sources, we’d have about 90% of the data we need for analysis, and the rest (millions of sources) would only supply about 10%, and I’ll have more to say about that in a minute.

Using the “Pulse” data (which I’d personally call “Big Data”) for economic and political predictions and benchmarking was a really good chapter of the book, but not my favorite; I already accepted and experienced much of what was being discussed, intuitively working out my own solutions, over the years, though not as robustly as Doug Hubbard has, with a very scientific bent.  Also, Hubbard’s explanation of “graph theory” helped me better understand the tools I already use, but how, perhaps, they fundamentally were formed to function under.

Another interesting observation, which goes along with my own, is how Twitter reached a critical mass (a.k.a – The Pulse) in mid 2008; I have written about it in my book – that Social Media seemed to pick up and become more prominent as a term around late 2008, early 2009.

Perhaps, more than anything else in the Pulse: The New Science of Harnessing Internet Buzz to Track Threats and Opportunities - my ideas totally resounded with Douglas Hubbard in the differences between “exploratory research” vs. Decision Support research where we know exactly what we’re looking for and how to describe it, and program the responses to it.

Probably without realizing it, Douglas Hubbard got a lot of people upset, were they to read between the lines, as I did – and draw the conclusions I did -probably far beyond what he originally meant and in London next week, I’ll talk about it and some of the solutions to it.   Here goes a short “kernel” of my thoughts about this.

A lot of the Decision Support intelligence, as Hubbard points out, for many of the things that surround us such as automatic controls in Nuclear Plants, Disc Breaks, Trading Systems, Banking and Economic Systems, to name few, is largely pre-programmed.   It has to be that way because if something, for example, goes wrong in a Nuclear Plant, most of the actions need have to happen very quickly, much more quickly than a human can respond.   Much of this information is programmed because we understand triggers, thresholds, causes and effects, and can therefore, program responses, that while not perfect, work the majority of the time.

On the other hand, with Exploratory Research, especially using the Real Time Web (a.k.a – social media monitoring, social media analytics, social listening, at el)  little of any of this “scenario” programming and automation are put in place – EVEN THOUGH THEY COULD HAVE BEEN – and should have been.  Hubbard makes a strong argument on where the failure lies, though without the value judgements I am putting on it – and offers his own solutions.

“….just track data because the measurements simply seem interesting and important. The hope is that if the measurements were surprising in some way, further investigation and analysis would lead to some kind of action. This method can be very useful for purely academic research, but even then only for initial discovery (after which specific research goals and hypotheses are proposed). However, in a business setting, this method probably is acceptable only when the cost of waiting on a decision is negligible, the consequences of a poor decision are low, or no specific decision scenarios or actions can even be imagined. It is highly unlikely that these conditions—especially the condition that no actions can even be imagined—exist for most important business decisions.”

Hubbard, Douglas W. (2011-04-04). Pulse: The New Science of Harnessing Internet Buzz to Track Threats and Opportunities (Kindle Locations 2962-2967). Wiley. Kindle Edition.

What’s truly radical with Hubbard’s view, which is my own, is that most Social Analytics, as it is practiced to day, is a waste of time, and he tells us why.

“….decisions that depend on real-time data can be defined in advance.”

“….The difference between the exploratory approach and the programmed, decision-focused approach is not that decisions couldn’t be defined in advance in the case of exploratory measurement—it is just that they weren’t.

This is where most of the failure really lies – and it tends to fall on Self Serve platforms more than the full service, where most are “on their own” trying to make sense of data without doing the necessary, and considerable “housekeeping” to make the listening and analytics worth doing in the first place – I bet because hardly anyone values it, or would pay for that work, then complaining the analytics systems produce results that are not actionable – when in reality – they could not be actionable because they were not properly calibrated in the first place.

Hubbard’s solution is to preform a “Forensic Decision Analysis” to improve future decisions.  And then Hubbard drops the Bomb -

If managers find some set of measures interesting but it looks like they are going to treat the measurements as purely exploratory, I ask: Why do you care about these measurements?

What decisions would be made differently if this measurement was surprisingly high or surprisingly low? What is the “threshold” of this measurement? That is, where is the point where, if this value exceeded it or failed to exceed it, you would engage in an alternative action? For example, if the positive buzz on Twitter about a new product drops, how far would it have to drop before you took action?

….The value of a real-time estimate of public mood, economic conditions, or the popularity of a product is fully exploited only if we attempt to work out at least some decision scenarios in advance. If you wait for something out of the ordinary to occur with your dashboard metrics and then start to worry about what the measurement means and how you should react, you may as well not worry about using real-time data.You would be just as well off waiting for some regular monthly report based on large and deliberate surveys and hoping you aren’t missing too many opportunities in the meantime.

Hubbard, Douglas W. (2011-04-04). Pulse: The New Science of Harnessing Internet Buzz to Track Threats and Opportunities (Kindle Locations 3025-3028). Wiley. Kindle Edition.

In my mind, Douglas Hubbard just “blew away” 90% of  Marketing/Communications reason and rationale of spending money on Social Media Analytics.  After all – if few companies, and almost no agencies are willing to put analysts up front, taking the actual time to gather triggers and interesting metrics to capture - why bother doing any kind of Social Listening?

And that’s what I walked away from, after reading this book – Hubbard clearly tells the reader how to do this research, how to gather the data from managers to build the necessary triggers and data mashups to make Real Time listening actionable – but it’s hard to see how that will ever happen when the Data Analyst is basically seen as a “Clerk” and Data Lackey – and all your Biz Dev, Creatives, Communications types,  Sales people, are running business, including the Analytics.

… we can argue if this is really  true – I maintain it is, most of the time.

For me the book ends with this exchange – the rest (forecasting the future) didn’t excite me as much as the part I just went over:

here is the kind of conversation an analyst might have with a manager who wants to track a new set of data:

Manager:.I think it would be interesting to measure the positive-to-negative buzz (P/N) ratio from Twitter on each of our new products, each of our competitors, and also to forecast consumer spending.

Analyst:. That sounds very interesting. What will you do with this information?

Manager:. It would help us develop a feel for which products we should discontinue, which to promote more vigorously, or even if we need to develop something new.

Analyst:. Okay, but when would you do that? How bad would the positive-to-negative buzz ratio on a product need to be before you promote it differently or discontinue it?

Manager:. It depends on lots of factors.

Maybe if sales have fallen below a certain point but consumer spending is increasing, negative buzz about the specific product might be to blame.

Maybe we should fix something about the product or maybe it’s just a fundamental demographic shift or change in tastes, and we should discontinue the product.

Analyst:.Great! So let’s try to spell out those factors and model them now so we don’t waste critical time analyzing them after you decided you need to react. Let’s just imagine that the P/N ratio of product X dropped from 1.2 to 0.4 in the last year.

……..

……

Hubbard goes on to describe the outcome of this kind of conversation:

At some point the analyst should get to some actual calculations to determine the optimal actions under different conditions. Ideally, this kind of discussion would eventually lead to some well-defined actions with well-defined sets of triggers.

Hubbard, Douglas W. (2011-04-04). Pulse: The New Science of Harnessing Internet Buzz to Track Threats and Opportunities (Kindle Locations 3050-3051). Wiley. Kindle Edition.

Anyway, that’s a lot different than anything I’ve seen around me, for at least a couple of years.  Hubbard has essentially, demolished the entire reason for doing real time social analytics without “Forensic Decision Analysis” … but the reality is … that’s exactly what we should have been doing all along.

Thanks Douglas Hubbard for writing a fantastic book – albeit, not the easiest to read, but well worth it.  I highly recommend it to everyone and it very much complements my own book, so after you get done reading mine (if your planning or have read Social Media Analytics) pick up a copy of Pulse: The New Science of Harnessing Internet Buzz to Track Threats and Opportunities by Douglas W. Hubbard – Kindle Edition – it’s some of the best money you’ll ever spend.