Putting A/B Testing in Its Place (15 Aug 2005)
For many sites, the ultimate goals are not measurable through user actions on the server. Goals like improving brand reputation or supporting the company's public relations efforts can't be measured by whether users click a specific button. Press coverage resulting from your online PR information might be measured by a clippings service, but it can't tell you whether the journalist visited the site before calling your CEO for a quote.
Similarly, while you can easily measure how many users sign up for your email newsletter, you can't assess the equally important issue of how they read your newsletter content without observing subscribers as they open the messages.
A second downside of A/B testing is that it only works for fully implemented designs. It's cheap to test a design once it's up and running, but we all know that implementation can take a long time. Before you can expose it to real customers on your live website, you must fully debug an experimental design. A/B testing is thus suitable for only a very small number of ideas.
Article URL: http://www.useit.com/alertbox/20050815.html
Read 370 more articles from Jakob Nielsen sorted by
Next Article: Simplicity is hard work