A/B and Multivariate Testing Landscape

In our most recent whitepaper we investigated the current state of A/B and Multivariate testing, focusing on how the practice is currently used and the obstacles companies face in implementing this type of an online strategy.

Our analysis, taken from surveys given at the eMetrics Marking Optimization Summit, found that 52% of online marketing managers are currently engaged in A/B or Multivariate testing and that an even greater percentage plan to begin testing within the next year.

A high-level overview of the findings:

• A variety of testing platforms are considered before making a final decision
• Experiments vary in frequency, page type and page element
• Practitioners often use more than one platform to meet their needs
• Pretest hypothesis often fail
• Lack of Budget was cited as the main reason by those who currently do not do any testing
• Complexity of Tools and lack of best practices were two top challenges faced by those engaged in Testing
• A/B and Multivariate testing is “worth it”
• Behavioral targeting is a small but growing practice

Surprisingly lack of budget was one of the top reasons why companies were not involved in A/B or Multivariate Testing. But if you look at the chart below you will see that companies that do conduct A/B and Multivariate testing find it that it’s worth it. Majority of the respondents who engage in testing said that either it is worth it or can’t live without it. So if you are one of those who think because of lack of budget you can’t do A/B MVT, think again? Your Return will make it worthwhile to do testing. You will be able to move budget from other areas such as Ad spend to A/B testing. Do your ROI calculations. You will be surprised with the positive results you will get with developing a culture of continuous testing.

You can download the whitepaper from ZeroDash1 website.

——————————————————————————–

Looking to fill your Web Analytics or Online Marketing position? Try WebAnalytics Job Board

New Position

Web Data Analyst at Alzheimer’s Association (Chicago, IL)

———————————————————————————

Site Optimization, Behavioral Targeting and eMetrics

I attended eMetrics on May 4th and 5th.
On May 4th, I taught a full day class on Web Analytics for Site Optimization. This class was a part of the WAA base camp series.
This class showed the tricks, tools and techniques needed to improve the websites, increase engagement and improve conversion rates. Some of the topics that were discussed in this class were

• Building a measurable site
• Designing for navigation vs. conversions
• Improving online persuasion
• KPIs for ecommerce, content sites, and intranets
• Understanding visitor behavior
• A/B & multivariate testing
• Using advanced metrics such as frequency, recency and latency to increase customer engagement and conversions.

Next day I took the topic of optimization to next step and talked about Behavioral Targeting and how you can use it to increase your online marketing ROI. The topics I covered in my presentations

• What is Behavioral Targeting?
• How does it work?
• What Determines Behavior?
• What can you target?
• Why should we use it?
• Description and examples of Network and Site-Side Targeting
• 5 Step Process for you to successful Behavioral Targeting
• ROI Model – Is BT for you?
• Privacy Issue surrounding BT, results from a survey on privacy and a 5 step process to ensure you don’t run into privacy issues

It is clear that the interest in this topic is increasing. It was evident by the use of this term in several other presentations at the conference and the attendance in my session.

If you missed this presentation and are interested in learning about Behavioral Targeting email me. If you attended the session, I would love to hear your comments and questions.

I wish I could have stayed for the all 4 days of the conference but I had to come back to Seattle to take my US citizenship exam. I am happy to announce that I have I successfully completed the citizenship exam.

What Web Site Optimization tool are you using?

What Web Site Optimization tool are you using?

I conducted an informal poll on my blog to see what Web Site Optimization tools are the readers of my blog using on their site. I only had one question, I will be conducted a more formal survey in future. For now I wanted to share the results of this poll.

Here are the results:

The results are based on 65 respondents and respondents were allowed to choose more than one option.

As we see Google Optimizer by far is the leader used by 43% of the respondents. No big surprise there, it is free easy to implement and easy to use. “Other” , which referred to the tools other than those mentioned in the poll, was second with 21% of the respondent. Followed by Offermatica, In-house tool, Optimost and Touchclarity.

During past few days I have talked to many companies who are now starting to seriously look for a website optimization tool. So far either they had no A/B testing or used in-house ad-hoc A/B testing tool (hack).

What tool are you using?

Web Analytics is about taking actions

Earlier this year in my 2007 Predictions I wrote:

“Web Analytics won’t be standing alone – Marketers will want 360 view of the customers. Integration of various data sources and tools will be expected from web analytics and other supporting tool vendors. Omniture started the trend with Omniture Genesis and this will continue we will see more acquisitions and partnerships similar to Omnitures.

Web Analytics will be about taking actions – More and more marketers would like to take actions and not just report the findings. It just won’t be about what happened, it will be about taking action to drive sales, user satisfaction, lead generation etc. Incentives and bonuses will be tied to the online KPIs. Optimization and Behavioral Targeting will become a common term used by marketers.”

Proving me right once again, Omniture acquired Offermetica, a leading A/B testing and multivariate testing company.

According to the press release

“The acquisition is a key part of Omniture’s strategy to deliver the most comprehensive solution for optimizing online business. Offermatica leverages Omniture’s recent acquisition of Touch Clarity as the two are highly complementary and together address the spectrum from A/B testing and multivariate testing to behavioral targeting. Offermatica enables companies to define and test the structure and elements of their sites, and Omniture TouchClarity™ enables companies to deliver the optimal content to any individual at any time….This combination, with Omniture’s analytics as the underlying foundation, provides the industry’s first and only integrated site optimization suite.”

..A year ago it became very clear to us that our customers wanted to leverage their data through a range of testing and optimization tools. Once we acquired the leading behavioral targeting technology, our customers continued to validate our thinking that in addition to behavioral targeting, A/B testing and multivariate testing were distinct and critical components of their long-term online strategy. Combining these capabilities into a single, integrated offering, built on patented behavioral targeting and testing technology, answers the market need for a complete optimization solution,” said Josh James, CEO and co-founder of Omniture.

I will also take this opportunity to announce that all my predictions for 2007 have come true. I will be doing a follow-up post pretty soon.

If you like this post, you might want to subscribe to my blog feed.Click here to subscribe to Web Analysis, Behavioral Targeting and Online Advertising

What does Website Optimization mean?

What does word “Optimization” or “Website Optimization” mean to you? I asked this question a while ago on the yahoo web analytics group. The reason I asked this question was because I came across and article on Media post (Google Tries To Redefine Optimization ) where author was questioning the name of the tool that Google was calling “Website Optimizer”. According to the author “Website Optimization” means making sites naturally position well in Google and not a tool for doing multivariate or A/B testing.

I thought I had a clear idea of what an optimization or Website optimization mean but after reading this article I was not sure if everybody has the same definition of Website optimization. Just do a search on Website Optimization on Google and you will see what I mean, you will find that this query yields paid searches from Search Engine Optimization companies, A/B Testing companies and company which allows optimizing load time of your site.

According to Merriam Webster dictionary Optimization is an act, process, or methodology of making something (as a design, system, or decision) as fully perfect, functional, or effective as possible; specifically : the mathematical procedures (as finding the maximum of a function) involved in this

So in terms of website it translates into an act or process or methodology of making a website as fully perfect, functional or effective as possible. Website optimization is a continuous process of improving a site to achieve business goals while delivering a great user experience. Web Site Optimization include
1. Serving websites faster and uptime of the site
2. Search engine optimization (SEO)
3. Campaign optimization (media optimization, A/B testing, multivariate testing)
4. Landing page optimization (A/B testing, multivariate testing)
5. Conversion rate optimization (A/B testing, multivariate testing)
6. Website design
7. Usability

As you can see all the above aspects are managed by different people (or organizations). Depending on who you ask you will get a different definition of Website Optimization. Their role in the organization influences how they think about optimization which results in different definitions. All the people are correct in their definition however each definition alone is too narrow

The people responsible for different pieces of website optimization do not understand the complete picture and are locked in their own definition of website optimization. The writer of this article (Google Tries To Redefine Optimization) probably had SEO background and hence he had a different definition than a web analyst would have. I can see how then name Website Optimizer can be confusing to a lot of people since it covers only one aspect of website optimization.

As a web analyst you should be aware of (and involved in) all the aspects of website optimization and only then you can develop a continuous process to improve the site. By being involved does not mean you have to actually do it but you have to make sure the web analytics data is driving the decisions. It should also be your responsibility to educate all stakeholders about how best to optimize the site based on the data. You have to act as glue, connecting different aspects of optimization.

Sidebar: Here are the response I got from the yahoo web analytics group users

I intended to write this post a long time back but got caught up in other things and now finally I found sometime so there it is.

Google and Behavioral Targeting Part III – Google Buys Doubleclick

Google buys Doubleclick for record 3.1 Billion.

Remember 1999 – 2000 when Doubleclick tried Behavioral Targeting but had to shut down efforts due to privacy concerns. Since then things have changed. Many Behavioral Targeting networks have sprung up. Dave Morgan, founder of Tacoda provided several reasons why BT will work now.

Will Google revive BT capabilities of Doubleclick? As I wrote earlier in my blog posts Google is preparing for the largest BT network (see my previous articles) and this is one of the biggest step deep into that direction and further confirms what I wrote in my article.
See my old posts at
Google and Behavioral Targeting
Google and Behavioral Targeting Part II

Here are quotes from Sergey Bin, Google’s Co-Founder & President, Technology and Eric Schmidt, CEO of Google on Doubleclicks’s acquistion. Source: Yahoo
“It has been our vision to make Internet advertising better – less intrusive, more effective, and more useful. Together with DoubleClick, Google will make the Internet more efficient for end users, advertisers, and publishers,” said Sergey Brin.
“DoubleClick’s technology is widely adopted by leading advertisers, publishers and agencies, and the combination of the two companies will accelerate the adoption of Google’s innovative advances in display advertising,” said Eric Schmidt.
So far Google analytics, Adsense, Google checkout codes were only available on small – medium size websites. With this deal the big brand sites will have Google code on their sites as well (DoubleClick Publishers and Advertisers). Obviously by having a code on virtually every site on the internet Google will have such a wealth of information about individuals (see my previous article) that it will be foolish for them to not use that information to target individuals with ads that match their interest shown by online (soon they will tie in offline with something like GoogleTV, Google Times, Google Radio and so on) behavior.

Another effect of the widespread code of Google could be the death of companies like Alexa, Compete comscore etc. Google can provide the internet usage data that won’t be based on a sample of those who participate (voluntarily or by installing some kind of application such as a toolbar) but on the Google cookies which will be on almost every single computer connected to the internet. However, this business might not be so lucrative for them. Why would they want to let others know how people use internet when they can use it to make it Googlenet (formerly known as internet) or GWW – Google Wide Web (formerly know as World Wide Web).

You are about to enter the world of Targeting.

FOX INTERACTIVE MEDIA ACQUIRES STRATEGIC DATA CORPORA

Fox Interactive Media (FIM), a division of Rupert Murdoch’s News Corp., has acquired interactive advertising-technology company Strategic Data Corp (SDC).
SDC’s system automatically optimizes the selection of creative for each impression to maximize profitability by combining sophisticated statistical and predictive algorithms, demographic and geographic segmentation, and performance tracking.
SDC’s technology will enable FIM to deliver highly-targeted graphical performance-based advertising on billions of Web pages viewed each day across its growing network. Fox Interactive Media, which includes MySpace, IGN, Direct2Drive, AmericanIdol.com, AskMen.com and more, is one of the most visited networks on the Internet with more than 135 million worldwide unique visitors each month and is the number one most viewed network in the U.S. with over 40 billion pages viewed each month.
SDC has specialized in developing display ad serving and yield-optimization technology that significantly increases the revenue for its clients. SDC’s clients include large ad networks and publishers.
Source: http://www.strategicdatacorp.com/

FIM also has a deal with Google to serve text ads. According to FIM, Google deal will continue and SDC acquisition is only for graphical ads.

Having campaign optimization capabilities within the company and the massive amount of data on the advertising they they will be able to built very effective and powerful ad network. As I predicted earlier this year, Optimization and Behavioral Targeting will be become very common and this deal is a step in theat direction, soon I expect to hear Behavioral Targeting related announcements from FIM.

Omniture Acquires Touch Clarity and WebTrends Announces Partner Integration

Yesterday Omniture announced that it will acquire Touch Clarity, a Behavioral Targeting Company. Within few minutes a WebTrends announced partner integrations with leading digital marketing vendors, including email marketing firms ExactTarget, Responsys and Silverpop; on-site behavioral targeting firms Kefta and Touch Clarity; and consumer opinion and customer voice firms ForeSee Results and OpinionLab.

This promoted me to write this article because this is exactly what I predicted.

You can read the fill press release at
• Omniture Acquires Behavioral Targeting Company Touch Clarity
• WebTrends Announces Marketing Lab Partners to Deliver More Measurable, Automated Relationship Marketing

Anybody who read my predictions for 2007 (http://webanalysis.blogspot.com/2007/01/my-predictions-for-2007.html) might have already seen this coming.

Here were my 3 predictions for 2007, and this acquisition proves all 3 of them true. My new comments are in Bold

1. Web Analytics won’t be standing alone – Marketers will want 360 degree view of the customers. Integration of various data sources and tools will be expected from web analytics and other supporting tool vendors. Omniture started the trend with Omniture Genesis, and this will continue, we will see more acquisitions and partnerships similar to Omnitures.

- Both the above press releases confirm that this prediction has come true. Customers are demanding this stronger integration. I wrote a follow-up on this same predictions last week, you can read it here http://webanalysis.blogspot.com/2007/02/follow-up-on-my-web-analytics.html

2. Web Analytics will be about taking actions – More and more marketers would like to take actions and not just report the findings. It just won’t be about what happened, it will be about taking action to drive sales, user satisfaction, lead generation etc. Incentives and bonuses will be tied to the online KPIs. Optimization and Behavioral Targeting will become a common term used by marketers.

- Omniture acquisition of Touch Clarity, a Behavioral Targeting company confirms this prediction has come true too. WebTrends announcing partner integration with Kefta (Optimization and Behavioral Targeting) and Touch Clarity (Behavioral Targeting) confirms that marketers are demanding Optimization and Behavioral Targeting. It is not just about reporting, it is about taking actions. In near future I am expecting to see a lot of case studies about what actions customers took and how it affected their bottom line.

3. Behavioral Targeting – Only few main behavioral network players will be left and some of the existing ones with poor networks will either go out of business or be sold. See my previous article on why size of network matter. Behavioral Targeting won’t exist in isolation. Web Analytics tool will have to support behavioral targeting and visa versa. Also, on-site behavioral targeting will become very common.

- Part of this prediction has come true. I have yet to see any consolidation in behavioral network players. But as the above two press releases confirm “On-Site Behavioral Targeting” is becoming common.

Web Analytics is maturing. Gone are the days when marketers were just satisfied by learning like
“Traffic went up”, “Traffic went down” etc. It is about deeper understanding of the visitors behavior (Web Analytics), segmenting them (Web Analytics and Behavioral Targeting), taking actions based on this deeper understanding e.g. targeting relevant content, products etc. (Email, Behavioral Targeting, Optimization).

So I am 2.5 on my 3 predictions. I have already declared 1 more prediction come true (http://webanalysis.blogspot.com/2007/02/follow-up-on-my-web-analytics_09.html). So for the year I am 3.5/5 so far and the year has just started. I feel good about my predictions.

Questions, Comments???

Follow-up on my Web Analytics Predictions for 2007 Part I

In January I wrote my web analytics predictions for 2007. Two of my predictions were

Web Analytics won’t be standing alone – Marketers will want 360 view of the customers. Integration of various data sources and tools will be expected from web analytics and other supporting tool vendors. Omniture started the trend with Omniture Genesis and this will continue we will see more acquisitions and partnerships similar to Omnitures.
Web Analytics will be about taking actions – More and more marketers would like to take actions and not just report the findings. It just won’t be about what happened, it will be about taking action to drive sales, user satisfaction, lead generation etc. Incentives and bonuses will be tied to the online KPIs. Optimization and Behavioral Targeting will become a common term used by marketers.
Today I read an article Report: Web Analytics Market Pumped for Growth which hits on the same points that I predicted.
Here are two quotes from this article
“Many Web analytics companies started collecting data about Web site visits and providing reporting tools to analyze that data,” Megan Burns, Forrester Research senior analyst told CRM Buyer. “Now they’re moving to the next level of value, which is enabling people to act on that data and becoming a platform for managing interactive marketing activities.” This is exactly what I predicted “ Wen analytics won’t be standing alone and it will be all about taking actions on the data.
Here is another quote from Jason Palmer of WebTrends
“As an application category, Web analytics is evolving beyond data collection and reporting, claimed WebTrends’ Jason Palmer. “Traditionally, Web analytics has been about performance management, tracking Web site behavior and usage,” he said. “Four or five years ago, it began evolving into campaign reporting and most recently, campaign management and optimization.””
It is nice to see the validation by industry experts and leaders. The year has just begun I am sure by end of this year we will see similar quotes from a lot of industry leader and experts. I would like to declare that my prediction number 4 has already come true and number 3 is on its way to become true.

As always, I would love to hear your comments and feedback.