I guess the HitWise thing got to Matt Cutts - as he ususally does not talk about metrics and today he just did.
"I’ve been reading the brouhaha about Hitwise’s press release about MySpace and Yahoo!, and I wanted to talk about metrics a bit.
"....At some point, one of the metrics services (which shall remain nameless) came to Google so that we could try to reconcile our data with their claims. I wasn’t in the meeting, so afterwards I caught an engineer and asked what happened; why did our numbers differ by so much? “They solicit people to install an application for them” was the answer. “But that’s a horrible methodology!” I said. “That would get you a ton more novice users; expert users wouldn’t see the value and probably wouldn’t install the application as much.” The other engineer agreed.
After mopping up the HitWise report he goes after Nielson for the Podcasting metric
"
Now that you’re appropriately jaded and cynical, let’s look at something out there right now. Here’s a recent post that appeared on Podcasting News:
Nielsen: Podcasts More Popular than Blogging
July 12, 2006Nielsen//NetRatings announced today that 6.6 percent of the U.S. adult online population, or 9.2 million Web users, have recently downloaded an audio podcast. 4.0 percent, or 5.6 million Web users, have recently downloaded a video podcast.
These figures put the podcasting population on a par with those who publish blogs, 4.8 percent …
The issue is the defination of the metric
"The thing to notice is that Podcasting News contrasted downloaders of podcasts with producers of blogs."
It would have been better if Nielsen//NetRatings could have measured the number of people who download podcasts with the number of people who read blogs (but don't download podcasts). Hmm...that would be hard to measure ...I mean you can get the number of people downloading Podcasts - but getting the number of people who read blogs? No wonder these metrics services go out of their way to make up measurements that only work because of the way they collect (and can collect) the data.
"Okay, if you think more people podcast than blog, raise your hand. Anyone? No? The thing to notice is that Podcasting News contrasted downloaders of podcasts with producers of blogs. The headline might have been technically correct; it would probably not be correct if the headline were “Podcasting More Popular than Blogging” (notice how I turned “podcasts” into a verb?). Yet that article was at the top of Techmeme, and your average reader could easily miss the distinction.
The story has a happy ending. I went back this morning to check if it was still on Techmeme, and Scoble and another podcasting site are calling people on it. In the instance above, the Nielsen numbers may have been completely accurate, but you still have to analyze how someone takes those numbers and think critically about what claims they make (or imply).
Since I make my living, partially, off of metrics - I know how they can be make up and spun - nuff said (for today).
Comment Preview