Avinash Kaushik wrote a very good post on the different types of testing you can do to pull metrics from a website.  

Here’s my take -  Avinash thinks 80% of the time we are wrong about what a customer wants / expects from our site experience.  I think that’s true and my experience with customers suggests they have their own vision of what a site is for and how it should operate that does not conform with actual visitor behavior.

A/B Testing is the easiest to put in place but outside the simple AdWords ad rotation - I’ve not done it and it’s really hard to get large corporate clients to do it if they’re not thinking it’s important; that’s my experience.   Also, in a dynamic environment - with pages changing on a weekly basis web teams are pretty overworked anyway - and might not be set up to easily create two or three versions of the same page and test.

Multivariate Testing might be something that some of my clients can do, but none of them actually do it. 

"For example for my blog I can create “modules” of the core page content, the top header, and each element of the right navigation (pages, categories, links, search etc). In a multivariate test I could move each piece around and see which one worked best."

Experience Testing is what many corporations are being to do with large websites and adaptive portals where navigation is changed based on who the customer is.

"With experience testing you don’t actually have to create three or four websites, but rather using your site platform you can easily create two or three persistent experiences on your websites and see which one your customers react to best. Since any analytics tools you use collect data for all three the analysis is the same you do currently."

The amount of programming work involved is very large; also standard Metrics software is really not built, at this time, to handle different versions of the same page that only a few customers might see.  Imagine having a page that’s somewhat different for every customer - how do you decide how much traffic a page got when it’s different "home page" for everyone (or almost everyone who say, logs in and identifies themselves).

I haven’t yet seen any of  the more expensive analytics platforms actually contend with what personalization will mean in terms of metrics tracking.