The Value of A/B Testing: Small Experiments That Lead to Big Results


AB-TestingThe use of randomized experiments to determine the most effective marketing or communications approach – known as A/B testing – is an extremely valuable tool for companies aiming to make the biggest impact on key stakeholders. However, according to Pete Koomen, president of website optimization software company Optimizely, the method is not implemented nearly enough. At the BRITE ’13 conference, Koomen shared personal experiences to demonstrate the very real value that A/B testing can contribute to developing results-driven communications programs. The most compelling of his examples included the 2008 Obama presidential campaign and the countless tests its analytics team ran onwww.barackobama.com website content and email subject lines. Koomen noted that even the slightest word phrasing could drive visitors to action (e.g., donating funds, signing up to volunteer). “It was an extremely powerful technique for [influencing] decisions,” he said. However, given the deep investment in time and resources needed for A/B testing, Koomen observed over time that companies tended to avoid using the technique – a fact that he and his business partner Dan Siroker quickly recognized as a major business opportunity.

The success of the 2008 campaign spoke for itself. Koomen estimated that methodical experimentation accounted for roughly $75 million more in donations to President Obama’s campaign and 4 million new website registrants. These results motivated Koomen and Siroker, both former Google product managers, to found Optimizely in 2009. They created a simple program that even small and medium-size businesses could utilize without having to depend on specialized in-house talent to run experiments. Organizations with limited resources could take advantage of marketing tactics that Amazon and other major blue-chip companies have been using for years to increase traffic and user conversion.

At BRITE ‘13, Koomen shared some best practices and lessons learned from running over 100,000 tests for clients and identifying the most effective approaches for achieving business objectives. For the Obama campaign, this entailed what Bloomberg BusinessWeek called “strange, incessant, and weirdly over familiar e-mails” due to the unusual, extremely casual tone Obama’s team usedjourney to office in 2008. The fundraising team found that the most successful subject heading “Hey” alone brought in millions of dollars in funding.

A few things that Koomen recommends businesses keep in mind as they take stock of their websites’ performance are:

  • Define quantifiable success metrics. One of the most important parts of testing. As exemplified by the Obama campaign, Koomen states that the campaign staffers did a good job of attracting people to the official website, but turning the site’s visitors into subscribers had proved more challenging and converting email signups to paying donators even more so.  By tweaking the website to optimize those two KPIs – subscribers and payers – the new website outperformed the old version by about 40%.
  • Explore before you refine. Koomen cautions against refining and optimizing in favor of exploring first to ensure you are aware of allpotential solutions before selecting one to improve.  Otherwise, there is a chance the best solution will be missed.
  • Less is more. Reducing optionality can have a major impact on a website’s effectiveness. Koomen cites a client which removed a series of links related to its product portfolio and company background from its shopping cart page and saw a 16% improvement in the dollars per visitor.

clip_image002

Continue reading more on brandleadership.wordpress.com



Categories: Digital Advertising, Measurement

Tags: ,

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Gigaom

Technology news, trends and analysis covering mobile, big data, cloud, science, energy and media

The healthy rawr chic

You must be willing and determined to achieve a state of mind, body, and soul beyond what this world imagines could be possible. —Mayra Mejia BA MS

Brand Leadership Blog

The leading forum worldwide for executives and researchers addressing the challenges of building and sustaining great brands

INSIDEDIGITAL

Intelligence for Brand Marketers

Digital Buzz Blog

INTELLIGENCE FOR BRAND MARKETERS

ciaobasta

Startups, entrepreneurs, ideas. Let's make it informal

Brian Solis

INTELLIGENCE FOR BRAND MARKETERS

HubSpot Marketing Blog

INTELLIGENCE FOR BRAND MARKETERS

ReadWrite

INTELLIGENCE FOR BRAND MARKETERS

Marketing Pilgrim - Internet News and Opinion

INTELLIGENCE FOR BRAND MARKETERS

advergirl

INTELLIGENCE FOR BRAND MARKETERS

Posts from the Econsultancy blog

INTELLIGENCE FOR BRAND MARKETERS

Get Elastic Ecommerce Blog

INTELLIGENCE FOR BRAND MARKETERS

%d bloggers like this: